Hi SuccessfulKoala55 , not really wrong, rather I don't understand it, the docker image with the args after it

Hi CostlyOstrich36 , I am not using Hydra, only OmegaConf, so you mean just calling OmegaConf.load should be enough?

Done in https://github.com/allegroai/clearml-server/issues/166

This works well when I run the agent in virtualenv mode (remove --docker )

I am using an old version of the aws autoscaler, so the instance has the following user data executed:
echo "{clearml_conf}" >>/root/clearml.conf ... python -m clearml_agent --config-file '/root/clearml.conf' daemon --detached --queue '{queue}' --docker --cpu-only

Ok, but that means this cleanup code should live somewhere else than inside the task itself right? Otherwise it won't be executed since the task will be killed

Ok, I could reproduce with Firefox and Chromium. Steps:
Add creds (either via the popup or in the settings) Go the /settings/webapp-configuration -> Creds should be there Hit F5 Creds are gone

that would work for pytorch and clearml yes, but what about my local package?

Alright, I will try now

Thanks, the message is not logged in GCloud instances logs when using startup scripts, this is why I did not see it. 👍

Ok thanks! And for this?

Would it be possible to support such use case? (have the clearml-agent setting-up a different python version when a task needs it?)

Thanks AgitatedDove14 ! I created a project with a default output destination to a s3 bucket but I don't have local access to this bucket (only agents have access to it for security reasons). Because of that, I cannot create a task in this project programmatically locally because it tries to access the bucket and fails. And there is no easy way to change the default output location (not in the web UI, not in the sdk)

I was rather wondering why clearml was taking space while I configured it to use the /data volume. But as you described AgitatedDove14 it looks like an edge case, so I don’t mind 🙂

Awesome, huge thanks to the team!

I am now trying with agent.extra_docker_arguments: ["--network='host'", ] instead of what I shared above

AgitatedDove14 I cannot confirm at 100%, the context is different (see previous messages) but it could be the same bug behind the scene...

Yes, I get 0 afterwards

AgitatedDove14 I eventually found a different way of achieving what I needed

ok, and if not the case, it will fall back to 3.8, right? Would it be possible to support such use case? (have the clearml-agent setting-up a different python version when a task needs it?)

I created a snapshot of both disks

same as the first one described

CostlyOstrich36 , this also happens with clearml-agent 1.1.1 on a aws instance…

Yea so I assume that training my models using docker will be slightly slower so I'd like to avoid it. For the rest using docker is convenient

 you mean “docker” was not installed and it did not throw an error ?

Yes docker was not installed in the machine

Yes you must make sure the docker can mount a persistent folder for you to work on.

Ok, it would be nice to have a --user-folder-mounted that do the linking automatically

The workaround I could find for now is to add the following to CONTAINER > SETUP SHELL SCRIPT:
mkdir -p ~/git/credential chmod 0700 ~/git/credential git config --global credential.helper 'cache --socket ~/git/credential/socket'

Ok interestingly using matplotlib.use('agg') it doesn't leak (idea from here )

Update: I successfully isolated one of the reason, mem leak in matplotib itself, I opened an issue on their repo here

Also enable_git_ask_pass is not dumped into the logs when an experiment start btw

Thanks! Will test now

Alright, thanks for the answer! Seems legit then 🙂