Okay so the only missing thing of the puzzle I think is that it would be nice if this propagates to the autoscaler as well; that then also allows hiding some of the credentials etc 😮

propagates meaning the autoscaler will use it to spin cloud instances?

Yeah, obviously it assumes everyone will be talking to the same server 🙂

Sorry, I misspoke, yes of course, the agents config file, not the queues

Than yes 😄

Holy crap this was a light-bulb moment, is this listed somewhere in the docs?
It solves so much of my issues xD

Well, that's actually not the current behaviour, and I'm pretty sure we had a good reason for that - I'll try to remember why 🙂

and I don't think it's in the docs - we'll add that

Very welcome update, please use some highlighting for it too, it's so important for a complete understanding of how the remote execution works

Exactly; the cloud instances (that are run with clearml-agent ) should have that clearml.conf + any changes specified in extra_clearml_configuration for the scaler

Well, let me check 🙂

Right, but that's as defined in the services agent, which is not immediately transparent

and I don't think it's in the docs - we'll add that

Actually SuccessfulKoala55 , there is something like that happening behind the scenes.
I have an AWS Autoscaler running on a services queue, so the autoscaler inherits the configuration used by the services agent, right?
Now, when my autoscaler launched new EC2 instances, they used the same fileserver as the one that was defined in the services agent too 🤔

Hi UnevenDolphin73 , as far as I can remember, the autoscaler does not use its own configuration as the default for the instances it spins up

Hi UnevenDolphin73 , queues don't have any notion of configuration. Agent running tasks remotely use their own configuration file to execute tasks.

As in here SuccessfulKoala55 - https://github.com/allegroai/clearml/blob/master/clearml/automation/cloud_driver.py#L169

Let me know if you do; would be nice to have control over that 😁

I guess it does not do so for all settings, but only those that come from Session()