SuccessfulKoala55 Since 2 hours I get 504 errors and I cannot ssh into the machine. AWS reports that instance health checks fail. Is it safe to restart the instance?

Will it freeze/crash/break/stop the ongoing experiments?

AgitatedDove14 I am actually considering rolling back to 1.1.0, so 1.3.0 is not really an option for now

yes what happens in the case of the installation with pip wheels files?

but not as much as the ELB reports

When installed with http://get.docker.com , it works

as it's also based on pytorch-ignite!

I am not sure to understand, what is the link with pytorch-ignite?

We're in the brainstorming phase of what are the best approaches to integrate, we might pick your brain later on

Awesome, I'd be happy to help!

AnxiousSeal95 Any update on this topic? I am very excited to see where this can go 🤩

Yes, not sure it is connected either actually - To make it work, I had to disable both venv caching and set use_system_packages to off, so that it reinstalls the full env. I remember that we discussed this problem already but I don't remember what was the outcome, I never was able to make it update the private dependencies based on the version. But this is most likely a problem from pip that is not clever enough to parse the tag as a semantic version and check whether the installed package ma...

torch==1.7.1 git+ .

Is it because I did not specify --gpu 0 that the agent, by default pulls one experiment per available GPU?

Ho nice, thanks for pointing this out!

5,6 mins exactly

Thanks AgitatedDove14 ! I created a project with a default output destination to a s3 bucket but I don't have local access to this bucket (only agents have access to it for security reasons). Because of that, I cannot create a task in this project programmatically locally because it tries to access the bucket and fails. And there is no easy way to change the default output location (not in the web UI, not in the sdk)

yes exactly

/data/shared/miniconda3/bin/python /data/shared/miniconda3/bin/clearml-agent daemon --services-mode --detached --queue services --create-queue --docker ubuntu:18.04 --cpu-only

I would probably leave it to the ClearML team to answer you, I am not using the UI app and for me it worked just well with different regions. Maybe check permissions of the key/secrets?

but if you do that and the package is already installed it will not install using the git repo, this is an issue with pip

Exactly, that’s my problem: I want to remove it to make sure it is reinstalled (because the version can change)

I think that since the agent installs everything from scratch it should work for you. Wdyt?

With env caching enabled, it won’t reinstall this private dependency, right?

I have two controller tasks running in parallel in the trains-agent services queue

If I don’t start clearml-session , I can easily connect to the agent, so clearml-session is doing something that messes up the ssh config and prevent me from ssh into the agent afterwards

Interesting! Something like that would be cool yes! I just realized that custom plugins in Mattermost are written in Go, could be a good hackday for me 😄 to learn go

Should I try to disable dynamic mapping before doing the reindex operation?

Adding back clearml logging with matplotlib.use('agg') , uses more ram but not that suspicious

I must say I am confused

So I created a symlink in /opt/train/data -> /data

Or even better: would it be possible to have a support for HTML files as artifacts?

AgitatedDove14 WOW, thanks a lot! I will dig into that 🚀

AgitatedDove14 Yes exactly, I tried the fix suggested in the github issue urllib3>=1.25.4 and the ImportError disappeared 🙂

And I am wondering if only the main process (rank=0) should attach the ClearMLLogger or if all the processes within the node should do that

I’ve set dynamic: “strict” in the template of the logs index and I was able to keep the same mapping after doing the reindex