However, If I edit directly the OmegaConf in the UI than the port changes correctly. I'd still prefer to override the Args so I can change entire sub-configuration e.g. ['dataset=cifar'] to ['dataset=imagenet'] instead of having to change all the parameters inside the OmegaConf

Hi AgitatedDove14 , what I meant is that if it is possible to associate ec2 instances of the autoscaler to a IAM role in order to grant permissions to applications running on that instances, which could be for example the access to a s3 buckets that can be accessed only with a certain IAM role permissions. I'm not completely sure that what I'm saying makes sense, but I refer to something similar as it's specified here https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role...

Actually I had the same issue even with that value set to False

Hi AgitatedDove14 , FriendlySquid61 ! I managed to grant permission to the AWS autoscaler to spin instances using the instance profile as suggested by FriendlySquid61 . The instances are created and terminated correclty, however the new instances don't executed the queued task and shutdown immediately. I noticed that the clearml credential at
self.web_server = Session.get_app_server_host()
self.api_server = Session.get_api_server_host()
` self.files_server = S...

Hi AgitatedDove14 , I noticed that in the Hydra parameters section it is not possible to add as parameters keys string with dots: .(dot) $(dollar) and space are not allowed in parameter key. However, it's very useful to add parameters with the dot to change something in a sub-configuration as, for example, training.max_epochs=10 . Do you think it's possible to allow this?

Also, if I want to modify another parameter, e.g. ui.height I have this problem:

Thank you!!

If it can help understand, this is what I'm doing

Hi Sapir, no that didn't solve the problem unfortunately. I ssh into the machine (after removing shutdown so that it doesn't terminate) and from the log I saw the error : "clearml_agent: ERROR: Connection Error: it seems *api_server* is misconfigured. Is this the ClearML API server http://apiserver:8008 ? So it is a credential problem

Yes it was set to nvidia/cuda:10.1-runtime-ubuntu18.04... ok I'll try again and see if that was the problem, thank you

Please let me know if my explanation is not really clear

After the agent finished installing the "requirements.txt" it will put back the entire "pip freeze" into the "installed packages", this means that later we will be able to fully reproduce the working environment, even if packages change (which will eventually happen as we cannot expect everyone to constantly freeze versions)

This would be perfect

Nice, I'll try also with the extra_bash_script, thank you!

Yes the workaround it's working 🙂

Yes it does 👍 Btw, at the moment I added import(s3fs) in my entry point and it's working, thank you!

Back to the feature request, if this is taken care of (both adding a missed package, and the S3 upload), do you still believe there is a room for this kind of feature?

Well, I can set import(s3fs) even if I don't really use it in my own code. One problem could be if this happen for a lot of packages, therefore I'd need to add this import to all my entry points of all my repos. While if I just download the right packages from the requirements.txt than I don't need to think about...

Make sure you have the S3 credentials in your agent's clearml.conf

Ok this could be a problem, as right now I'm using ec2-instances with a instance-profile (I use it in the autoscaler) so they have by the default the right s3 permissions. But I'll try it anyway

Hi AgitatedDove14 , thank you for your answer!
At the moment I can't configure both internal/external with the same dns. Before changing the server infrastructure, i'm trying a workaround where I upload the artifact with the internal file server path, and then I upload a string artifact which is the first artifact url where I replace the internal dns with the external dns, and use it to download the artifact from the UI.

Hi AgitatedDove14 , I'm interested in this feature to run the agent and force it to install packages from requirements.txt. Is it available?

AgitatedDove14 that seems like the best option. Once the aws autoscaler is inside a docker container I can deploy it inside a kube pod or a job. This, however, requires that I slightly modify the clearml helm chart with the aws-autoscaler deployment, right?

if in the "installed packages" I have all the packages installed from the requirements.txt than I guess I can clone it and use "installed packages"

I've just seen it is a know issue https://clearml.slack.com/archives/CTK20V944/p1611763839133700 . Has a new version been released meanwhile?

As an example, in Task.create() there is the possibility to install packages using a requirements.txt, and if not specified, it uses the requirements.txt of the repository. I'd like something like for Task.init() if possible

My problem right now is that Pytorch Lightning need the s3fs package to store model checkpoint into s3 buckets, but in my "installed packages" is not imported and I get an import error

Yes I think it's only related to the UI. Do you think It can be fixed somehow? It would be the easiest way to launch new experiments with a different configuration

I'm using ClearML 0.17.5

Hi AgitatedDove14
I implemented the pipeline manually as you suggested. I also used task.wait_for_status() after each task.enqueue() so I was able to implement a full pipeline in one script. It seems to be working correctly. Thank you!

I created this toy example so you don't need any external conf files. Btw if I first launch the task as python example.py port=80 than the task will print the message "Is this a webserver" correctly. If then in the UI I clone the same task, overrides the port with ['port=43'] , for example, and run the experiment, I will still get the message "Is this a webserver" so the port didn't change

Hi AgitatedDove14 , sorry for the late reply. Btw, I tried with the latest RC and the issue is still there. So if I clone an experiment, modify an overrides params eg ['training.max_epochs=10'] my experiment run the old configuration. Therefore it seems that it doesn't change the OmegaConf configuration.

Ok now I noticed that If I change the value of the port inside the Hydra parameters section ( not the overrides) It does actually change also in the experiment. The overrides doesn't seem to be working