Actually SuccessfulKoala55 , there is something like that happening behind the scenes.
I have an AWS Autoscaler running on a services queue, so the autoscaler inherits the configuration used by the services agent, right?
Now, when my autoscaler launched new EC2 instances, they used the same fileserver as the one that was defined in the services agent too 🤔
Okay so the only missing thing of the puzzle I think is that it would be nice if this propagates to the autoscaler as well; that then also allows hiding some of the credentials etc 😮
propagates meaning the autoscaler will use it to spin cloud instances?
Hi UnevenDolphin73 , as far as I can remember, the autoscaler does not use its own configuration as the default for the instances it spins up
As in here SuccessfulKoala55 - https://github.com/allegroai/clearml/blob/master/clearml/automation/cloud_driver.py#L169
Hi UnevenDolphin73 , queues don't have any notion of configuration. Agent running tasks remotely use their own configuration file to execute tasks.
Sorry, I misspoke, yes of course, the agents config file, not the queues
and I don't think it's in the docs - we'll add that
and I don't think it's in the docs - we'll add that
Very welcome update, please use some highlighting for it too, it's so important for a complete understanding of how the remote execution works
Exactly; the cloud instances (that are run with clearml-agent ) should have that clearml.conf + any changes specified in extra_clearml_configuration for the scaler
Well, that's actually not the current behaviour, and I'm pretty sure we had a good reason for that - I'll try to remember why 🙂
I guess it does not do so for all settings, but only those that come from Session()
Right, but that's as defined in the services agent, which is not immediately transparent
Yeah, obviously it assumes everyone will be talking to the same server 🙂
Let me know if you do; would be nice to have control over that 😁
Holy crap this was a light-bulb moment, is this listed somewhere in the docs?
It solves so much of my issues xD