Hi, could you please be more specific? It is hard to guess how to configure things without a specific example. Isn't there any clearml.conf example that does this? I thought it is a common case....
I'm creating a new autoscaler instance, where should configure all this? in the extra configuration section? to write a custom .conf file and put it there?

Hi thanks,
It did help.
My problem is now that the agent running on aws can't find my docker image (which is stored in AWS ECR private repo).
How can I give it access to ecr? I haven't seen any key-value in the conf sdk.aws section that talks about authenticating to ECR.

You can use the autoscaler init script to issue a docker login command

Check out the agent environment variables, you have one there the allows you to specify the python binary and bypass any installation - this flag should be passed into the docker image

See here - specifically CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL
You'll need to make sure this env var is set before execution starts (you can set that in the custom bash script for the autoscaler instances).
This is actually not a very common scenario since the agent is designed to recreate the environment on the fly, building a docker image containing everything is not required, takes time and should be done again every time you need to change something, so we recommend just specifying your requirements and letting the agent do the work 🙂