Is this working in the latest version? clearml-agent falls back to /usr/bin/python3.8 no matter how I configure clearml.conf Just want to make sure, so I can investigate what's wrong with my machine if it is working for you.

Thank you very much for the fast work!

One last question: Is it possible to set the pip_version task-dependent?

I think sometimes there can be dependencies that require a newer pip version or something like that. I am not sure though. Why can we even change the pip version in the clearml.conf?

Yea, but doesn't this feature make sense on a task level? If I remember correctly, some dependencies will sometimes require different pip versions. And dependencies are on task basis.

The one I posted on top 22.03-py3 😄

Thank you very much! 😃

Unfortunately, I do not know that. Must be before October 2021 at least. I know I asked here how to use the preinstalled version and AgitatedDove14 helped me to get it work. But I cannot find the old thread 😕

I am going to try it again and send you the relevant part of the logs in a minute. Maybe I am interpreting something wrong.

Thanks for answering. I don't quite get your explanation. You mean if I have 100 experiments and I start up another one (experiment "101"), then experiment "0" logs will get replaced?

Thank you very much!!!

AgitatedDove14 I have to problem that "debug samples" are not shown anymore after running many iterations. What's appropriate to use here: A colleague told me increasing task_log_buffer_capacity worked. Is this the right way? What is the difference to file_history_size ?

But would this not have to be a server parameter instead of a clearml.conf parameter then? Maybe someone from clearml can confirm MortifiedDove27 's explaination?

Thanks, that makes sense. Can you also explain what task_log_buffer_capacity does?

AgitatedDove14 Could you elaborate?

AgitatedDove14

MortifiedDove27 Sure did, but I do not understand it very well. Else I would not be asking here for an intuitive explanation 🙂 Maybe you can explain it to me?

You suggested this fix earlier, but I am not sure why it didnt work then.

Afaik, clearml-agent will use existing installed packages if they fit the requirements.txt. E.g. pytorch >= 1.7 will only install PyTorch if the environment does not already provide some version of PyTorch greater or equal to 1.7.

Thanks for answering, but I still do not get it. file_history_size decides how many past files are shown? So if file_history_size=100 and I have 1 image/iteration and ran 1000 iterations, I will see images for iteration 900-1000?

Yea, will do so in 30min

The agent and server have similar hardware also. So I would expect same read/write speed.

I am getting permission errors when I try to use the clearml-agent with docker containers. The .ssh is mounted, but the owner is my local user, so the docker containers root does not seem to have the correct permissions.

When you say it is an SDK parameter this means that I only have to specify it on the computer where I start the task from, right? So an clearml-agent would read this parameter from the task itself.

Hi TimelyMouse69
Thank you for answering, but I do not think these methods do allow me to modify anything the is set in clearml.conf. Rather they just do logging.

Maybe this opens up another question, which is more about how clearml-agent is supposed to be used. The "pure" way would be to make the docker image provide everything and clearml-agent should do not setup at all.
What I currently do instead is letting the docker image provide all system dependencies and let clearml-agent setup all the python dependencies. This allows me to reuse a docker image for more different experiments. However, then it would make sense to have as many configs as possib...

I am currently on the Open Source version, so no Vault. The environment variables are not meant to used on a per task basis right?

I mean if I do CLEARML_DOCKER_IMAGE=my_image clearml-task something something it will not work, right?

I don't know actually. But Pytorch documentation says it can make a difference: https://pytorch.org/docs/stable/distributions.html#torch.distributions.distribution.Distribution.set_default_validate_args

Is there a way to specify this on a per task basis? I am running clearml-agent in docker mode btw.

Thank you very much, didnt know about that 🙂