so the task they execute must have clearml installed?

Alright, thanks for the answer! Seems legit then 🙂

Seems to works, I started a last one to confirm!

Yes, I will try 🙂

Another error that just popped up:

Indeed, I actually had the old configuration that was not JSON - I converted to json, now works 🙂

So it seems like it doesn't copy /root/clearml.conf and it doesn't pass the environment variables (CLEARML_API_HOST, CLEARML_API_ACCESS_KEY, CLEARML_API_SECRET_KEY)

Ok, now I would like to copy from one machine to another via scp, so I copied the whole /opt/trains/data folder, but I got the following errors:

thanks Jake, good to know!

Nevermind, nvidia-smi command fails in that instance, the problem lies somewhere else

I am running on bare metal, and cuda seems to be installed at /usr/lib/x86_64-linux-gnu/libcuda.so.460.39

I'll try with that; https://github.com/allegroai/clearml/compare/master...H4dr1en:add-aws-params

Usually one or two tags, indeed, task ids are not so convenient, but only because they are not displayed in the page, so I have to go back to another page to check the ID of each experiment. Maybe just showing the ID of each experiment in the SCALAR page would already be great, wdyt?

but most likely I need to update the perms of /data as well

There is no need to add creds on the machine, since the EC2 instance has an attached IAM profile that grants access to s3. Boto3 is able retrieve the files from the s3 bucket

then print(Task.get_project_object().default_output_destination) is still the old value

no it doesn't! 3. They select any point that is an improvement over time

Same, it also returns a ProxyDictPostWrite , which is not supported by OmegaConf.create

To be fully transparent, I did a manual reindexing of the whole ES DB one year ago after it run out of space, at that point I might have changed the mapping to strict, but I am not sure. Could you please confirm that the mapping is correct?

the deep learning AMI from nvidia (Ubuntu 18.04)

Yes, I would like to update all references to the old bucket unfortunately… I think I’ll simply delete the old s3 bucket, wait or his name to be available again and recreate it where on the other aws account and move the data there. This way I don’t have to mess with clearml data - I am afraid to do something wrong and loose data

Will it freeze/crash/break/stop the ongoing experiments?

Yes I did, I found the problem: docker-compose was using trains-server 0.15 because it didn't see the new version of trains-server. Hence I had trains-server 0.15 running with ES7.
-> I deleted all the containers and it successfully pulled trains-server 0.16. Now everything is running properly 🙂

I am already trying with latest of pip 😞

So it looks like the agent, from time to time thinks it is not running an experiment

if I want to resume a training on multi gpu, I will need to call this function on each process to send the weights to each gpu

Yes it would be very valuable to be able to tweak that param, currently it's quite annoying because it's set to 30 mins, so when a worker is killed by the autoscaler, I have to wait 30 mins before the autoscaler spins up a new machine because the autoscaler thinks there is already enough agents available, while in reality the agent is down

mmmmh I just restarted the experiment and it seems to work now. I am not sure why that happened. From this SO it could be related to size of the repo. Might be a good idea to clone with --depth 1 in the agents?
Or more generally, try to catch this error and retry a few times?

venv mode

And since I ran the task locally with python3.9, it used that version in the docker container