Ok, nice. Makes sense!

In the beginning my config file was not empty 😕

As in if it was not empty it would work?

clearml==0.17.4
` task dca2e3ded7fc4c28b342f912395ab9bc pulled from a238067927d04283842bc14cbdebdd86 by worker redacted-desktop:0
Running task 'dca2e3ded7fc4c28b342f912395ab9bc'
Storing stdout and stderr log to '/tmp/.clearml_agent_out.vjg4k7cj.txt', '/tmp/.clearml_agent_out.vjg4k7cj.txt'
Current configuration (clearml_agent v0.17.1, location: /tmp/.clearml_agent.us8pq3jj.cfg):

agent.worker_id = redacted-desktop:0
agent.worker_name = redacted-desktop
agent.force_git_ssh...

Okay, now I understand!

Is this working in the latest version? clearml-agent falls back to /usr/bin/python3.8 no matter how I configure clearml.conf Just want to make sure, so I can investigate what's wrong with my machine if it is working for you.

Thank you very much for the fast work!

One last question: Is it possible to set the pip_version task-dependent?

I think sometimes there can be dependencies that require a newer pip version or something like that. I am not sure though. Why can we even change the pip version in the clearml.conf?

Yea, but doesn't this feature make sense on a task level? If I remember correctly, some dependencies will sometimes require different pip versions. And dependencies are on task basis.

The one I posted on top 22.03-py3 😄

Thank you very much! 😃

Unfortunately, I do not know that. Must be before October 2021 at least. I know I asked here how to use the preinstalled version and AgitatedDove14 helped me to get it work. But I cannot find the old thread 😕

I am going to try it again and send you the relevant part of the logs in a minute. Maybe I am interpreting something wrong.

Or maybe a different question: What is not

Artifacts and Models. debug samples (or anything else the Logger class creates)

?

Also it is not possible to use multiple files server? E.g. log tasks on different S3 buckets without changing clearml.conf

Makes sense, but this means that we are not able to tell clearml-agent where to save on a per-task basis? I see the output_destination set correctly in clearml web interface, but as you say, clearml-agent always uses its api.fileserver ?

In my case I use the conda freeze option and do not even have CUDA installed on the agents.

Thanks for answering. I don't quite get your explanation. You mean if I have 100 experiments and I start up another one (experiment "101"), then experiment "0" logs will get replaced?

Thank you very much!!!

AgitatedDove14 I have to problem that "debug samples" are not shown anymore after running many iterations. What's appropriate to use here: A colleague told me increasing task_log_buffer_capacity worked. Is this the right way? What is the difference to file_history_size ?

But would this not have to be a server parameter instead of a clearml.conf parameter then? Maybe someone from clearml can confirm MortifiedDove27 's explaination?

Thanks, that makes sense. Can you also explain what task_log_buffer_capacity does?

AgitatedDove14 Could you elaborate?

AgitatedDove14

MortifiedDove27 Sure did, but I do not understand it very well. Else I would not be asking here for an intuitive explanation 🙂 Maybe you can explain it to me?

You suggested this fix earlier, but I am not sure why it didnt work then.

Afaik, clearml-agent will use existing installed packages if they fit the requirements.txt. E.g. pytorch >= 1.7 will only install PyTorch if the environment does not already provide some version of PyTorch greater or equal to 1.7.

Thanks for answering, but I still do not get it. file_history_size decides how many past files are shown? So if file_history_size=100 and I have 1 image/iteration and ran 1000 iterations, I will see images for iteration 900-1000?

Yea, will do so in 30min

The agent and server have similar hardware also. So I would expect same read/write speed.

I am getting permission errors when I try to use the clearml-agent with docker containers. The .ssh is mounted, but the owner is my local user, so the docker containers root does not seem to have the correct permissions.