Good to know!
I think the current solutions are fine. I will try it first and probably will have some more questions/problems 🙂

Okay. It works now. I don't know what went wrong before. Probably a user error 😅

Perfect! That sounds like a good solution for me.

I will create a minimal example.

Just like in the git repo.

Hi @<1523701087100473344:profile|SuccessfulKoala55> Thank you very much.
Is there some way to verify the server uses the correct configuration files? (E.g. see it in the logs/web ui). I Just tried it does not work.
At least I can see the async_delete service complains about a missing secret, so I can start debugging there. I am using the same config as for my agents, but somehow for async_delete it does not work...

I just wanna avoid that ClearML leaves files lingering around. Btw: a better default behavior in my opinion would be to delete tasks only after files have been deleted. And only with the force option to delete the task anyways!

Let me try it another time. Maybe something else went wrong.

Or better some cache option. Otherweise the cron job is what I will use 🙂 Thanks again

Yea, but before in my original setup the config file was filled. I just added some lines to the config and now the error is back.

Wait, nvm. I just tried it again and now it worked.

For example in our case we do reinforcement learning and the we would call a script like this: python run_openai_gym.py some_ http://package.my _agent .

I have my development machine where I develop for multiple projects. I want to configure clearml differently based on the project. Similar to .vscode , .git or .idea at the project level.

Thank you for the quick reply. Maybe anyone knows whether there is an option to let docker delete images after container exit?

Mhhm, good hint! Unfortunetly I can see nowhere in logs when the server creates a delete request

Here is the minimal example: https://github.com/mctigger/test_clearml

So just tried again and still it does not work.
This is what is in .ssh on my clearml-agent
-rw------- 1 tim tim 1,5K Apr 8 14:28 authorized_keys -rw-rw-r-- 1 tim tim 208 Apr 29 11:15 config -rw------- 1 tim tim 432 Apr 8 14:53 id_ed25519 -rw-r--r-- 1 tim tim 119 Apr 8 14:53 id_ed25519.pub -rw------- 1 tim tim 432 Apr 29 11:16 id_gitlab -rw-r--r-- 1 tim tim 119 Apr 29 11:25 id_gitlab.pub -rw-rw-r-- 1 tim tim 3,1K Apr 29 11:33 known_hosts

Thank you very much 🙂

I am wondering cause when used in docker mode, the docker container may have a CUDA Version that is different from the host version. However, ClearML seems to use the host version instead of the docker container's version, which is a problem sometimes.

Thank you. Yes we need to wait for carla to spin up.

btw: Could you check whether agent.package_manager.system_site_packages is true or false in your config and in the summary that the agent gives before execution?
I start my agent in --foreground mode for debugging and it clearly show false , but in the summary that the agent gives before the task is executed, it shows true .

I am still not getting why it is a problem to just update the requirements at any time... 😕

Yea, it was finished after 20 hours. Since the artifact started uploading when the experiment finishes otherwise, there is no reporting for the the time where it uploaded. I will debug it and report what I find out

Yea, correct! No problem. Uploading such large artifacts as I am doing seems to be an absolute edge case 🙂

Yea, and the script ends with clearml.Task - INFO - Waiting to finish uploads

I see a python 3 fileserver.py running on a single thread with 100% load.

481.2130692792125 seconds
Done

Nvm, I took a look at conda history and there I see it

So my network seems to be fine. Downloading artifacts from the server to the agents is around 100 MB/s, while uploading from the agent to the server is slow.

Agent runs in docker mode. I ran the agent on the same machine as the server this time.