I ll add creating an issue to my todo list

Okay, thanks for the info! I am currently not using k8s, but may be good to know for the future.

Same error 😕

Hey, thank you for answering.
I know this issue and I have it sometimes, but my current issue is a direct result of me trying to make SSL work. So I am not asking for help in solving my problem, but only for help how to debug. Finding out which step leads to the artifact not being deleted (e.g. the fileserver cannot be reached by from wherever the delete request is send)

For now I can tell you that with conda_freeze: true it fails, but with conda_freeze: false it works!

@<1523701205467926528:profile|AgitatedDove14> Thank you very much for your guidance. Setting these manually works for me!

Good idea. No, clearml-agent does not crash and works fine afterwards. Then it is probably some other problem with my machine. Thank you!

No reason in particular. How many people work at http://allegro.ai ?

Hey Martin, thank you for answering!
I see your point, however in my opinion this is really unexpected behavior. Sure, I can do some work to make it "safe", but shouldn't that be default. So throw an error without clearml.conf and expect CLEARML_USE_DEFAULT_SERVER=1 ` .

via Python?

Perfect, works! 🙂

Thanks for researching this issue. If you have time, you can create the issue since you are way more knowledgeable, but I can also open it if you do not have time 🙂

Sounds like a good hack, but not like a good solution 😄 But thank you anyways! 🙂

Could you elaborate on that:
"So the agent failed to actually restore it from the git (files that are not added are not considered part of the git diff, this is usually git behavior)."

I just started a task from this environment and it fails on the agent.

Yep, this install PyTorch CPU

However, because of the import carla it is added to the task requirements and clearml-agent tries to install it, although it is meant to be included at runtime.

The default behavior mimics Python’s assert statement: validation is on by default, but is disabled if Python is run in optimized mode (via python -O). Validation may be expensive, so you may want to disable it once a model is working.

In the beginning my config file was not empty 😕

Thank you very much for the quick answer. Still so confusing to me that so many things are configured client side 😄

I will debug this myself a little more.

Ok. I just wanted to make sure I have configured my agent properly. Just want to make sure I have to set it on all agents.

I am still trying to solve the add_requirements + importlib combo. If I use detect_with_freeze I can not use add_requirements and if I use automatic code analysis it will not find all packages because of importlib .
For now I come to the conclusion, that keeping a requirements.txt and making clearml parse the requirements from there should be the most robust solution. Unfortunately, there seems to be no way to do this with Task.init .

That sounds good! 🙂

Could be clean log after restart. Unfortunately, I restarted the server right away 😞 I gonna post if it happens again with the appropriate logs.

So actually deleting from client (e.g. an dataset with clearml-data) works.

By host you mean the machine on which the agent is running? How does clearml-agent find the cuda_version?

That I understand. But I think (old) pip versions will sometimes not resolve a package. Probably not the case the other way around.

Yes

Depends on how you start the task afaik. I think clearml-task uses requirements.txt by default, but otherwise clearml will parse your files dependencies or if you changed in clearml.conf it will use your conda/pip environment to generate the requirements.