I guess I’ll get used to it 😄

Nice, thanks!

might be worth documenting 😄

Will it freeze/crash/break/stop the ongoing experiments?

AgitatedDove14 I am actually considering rolling back to 1.1.0, so 1.3.0 is not really an option for now

yes what happens in the case of the installation with pip wheels files?

When installed with http://get.docker.com , it works

as it's also based on pytorch-ignite!

I am not sure to understand, what is the link with pytorch-ignite?

We're in the brainstorming phase of what are the best approaches to integrate, we might pick your brain later on

Awesome, I'd be happy to help!

AnxiousSeal95 Any update on this topic? I am very excited to see where this can go 🤩

Yes, not sure it is connected either actually - To make it work, I had to disable both venv caching and set use_system_packages to off, so that it reinstalls the full env. I remember that we discussed this problem already but I don't remember what was the outcome, I never was able to make it update the private dependencies based on the version. But this is most likely a problem from pip that is not clever enough to parse the tag as a semantic version and check whether the installed package ma...

torch==1.7.1 git+ .

Ho nice, thanks for pointing this out!

5,6 mins exactly

Thanks AgitatedDove14 ! I created a project with a default output destination to a s3 bucket but I don't have local access to this bucket (only agents have access to it for security reasons). Because of that, I cannot create a task in this project programmatically locally because it tries to access the bucket and fails. And there is no easy way to change the default output location (not in the web UI, not in the sdk)

yes exactly

/data/shared/miniconda3/bin/python /data/shared/miniconda3/bin/clearml-agent daemon --services-mode --detached --queue services --create-queue --docker ubuntu:18.04 --cpu-only

I would probably leave it to the ClearML team to answer you, I am not using the UI app and for me it worked just well with different regions. Maybe check permissions of the key/secrets?

but if you do that and the package is already installed it will not install using the git repo, this is an issue with pip

Exactly, that’s my problem: I want to remove it to make sure it is reinstalled (because the version can change)

I think that since the agent installs everything from scratch it should work for you. Wdyt?

With env caching enabled, it won’t reinstall this private dependency, right?

I have two controller tasks running in parallel in the trains-agent services queue

If I don’t start clearml-session , I can easily connect to the agent, so clearml-session is doing something that messes up the ssh config and prevent me from ssh into the agent afterwards

Should I try to disable dynamic mapping before doing the reindex operation?

Adding back clearml logging with matplotlib.use('agg') , uses more ram but not that suspicious

I must say I am confused

So I created a symlink in /opt/train/data -> /data

Or even better: would it be possible to have a support for HTML files as artifacts?

Hi AgitatedDove14 , that’s super exciting news! 🤩 🚀
Regarding the two outstanding points:
In my case, I’d maintain a client python package that takes care of the pre/post processing of each request, so that I only send the raw data to the inference service and I post process the raw output of the model returned by the inference service. But I understand why it might be desirable for the users to have these steps happening on the server. What is challenging in this context? Defining how t...

Ok, now I would like to copy from one machine to another via scp, so I copied the whole /opt/trains/data folder, but I got the following errors:

Could be, but not sure -> from 0.16.2 to 0.16.3

Thanks! I will investigate further, I am thinking that the AWS instance might have been stuck for an unknown reason (becoming unhealthy)

Thanks a lot, I will play with that!