how do i access the clearnl.conf custom variables then?
or - how do i configure and access env variables that way?

the use case is simple:
i wanna fetch data from an sql table, inside a task.
so i want to execcute a query, and then do some operations on it, from within a task, to do that i have to connect to the db,
and i don't want the connection details to be logged

yeah

ok, thank you!

Hey joey, i configured ssh locally for my pc as well, and now it works.

are you guys planning in the future a feature which will allow me to specify the connection type, without relation to what i am running locally?

@<1523701070390366208:profile|CostlyOstrich36>
hey john, let us know if you need any more information

yes,
so basically I should create a services queue, and preferably let it contain its own workers

another question: if i save heavy artifcats, should my services worker ram be at least as high? (or is it enough for the default queue workers to have that)

ok, yeah, makes sense. thanks John!

i need to read and write, i do have access from genesis autoscaler when i set off all firewall rules. but this is not recommend by microsoft.

I need to add specific firewall rules for the genesis machines, to allow them to authorize to my azure blob storage

ok so, idk why it helped, but setting base_task_id
instead of base_task_name in the pipe.add_step
function, seems to overcome this

only sometimes, the pipeline runs using local machines

WebApp: 3.16.3-949 • Server: 3.16.1-974 • API: 2.24

?

plus, is there an option to configure the agent configuration? for example we are using:

force_git_root_python_path: true

can we do it there as well?

we use the clearml hosted server, so i don't know the version

the unsuccesslful:
Using cached meteostat-1.6.5-py3-none-any.whl (31 kB)
Requirement already satisfied: neuralprophet==0.5.3 in /usr/local/lib/python3.9/site-packages (from -r /tmp/cached-reqs58y_jg9f.txt (line 9)) (0.5.3)
Requirement already satisfied: numpy==1.23.5 in /usr/local/lib/python3.9/site-packages (from -r /tmp/cached-reqs58y_jg9f.txt (line 10)) (1.23.5)
Requirement already satisfied: pandas==1.5.3 in /usr/local/lib/python3.9/site-packages (from -r /tmp/cached-reqs58y_jg9f.txt (lin...

the successful, which is aborted for some reason (but at least the enviorment is setup correctly)
the end of it is :

• urllib3==1.26.15
• virtualenv==20.23.0
• wcwidth==0.2.6
• Werkzeug==2.3.2
• widgetsnbextension==4.0.7
• xgboost==1.7.5
• yarl==1.9.2
  Environment setup completed successfully
  Starting Task Execution:
  2023-04-29 21:41:02
  Process terminated by user

from which we run the task

it is installed as a pip package
but i am not using it in the code

just a sec

yeah

and then it works, doesn't try to install any other packages

only clearml probably

basically, only test.py need the packages, but for somereason pipeline_test installs them as well

thank you! i'll try it

so i think debian (and python 3.9)

the base image is python:3.9-slim

thanks for the help 🙂

(im running it on docker)

ok so i accidentally (probably with luck) noticed the max_connection: 2 in the azure.storage config.
canceled that, and so now everything works