thanks for your help!

using Trains Agent 🙂

Answering myself: Yes, Task.set_base_docker RTFM!!!

The cloning is done in another task, which has the argv parameters I want the cloned task to inherit from

I can live with the current setup for now

I am running on bare metal, and cuda seems to be installed at /usr/lib/x86_64-linux-gnu/libcuda.so.460.39

Ok, this I cannot locate

Ok I have a very different problem now: I did the following to restart the ES cluster:
docker-compose down docker-compose up -dAnd now the cluster is empty. I think docker simply created a new volume instead of reusing the previous one, which was always the case so far.

Isn't it overkill to run a whole ubuntu 18.04 just to run a dead simple controller task?

But I can do:
` $ python

import torch
torch.cuda.is_available()
True
torch.backends.cudnn.version()
8005 `

Update: I successfully isolated one of the reason, mem leak in matplotib itself, I opened an issue on their repo here

Yes, actually thats what I am doing, because I have a task C depending on tasks A and B. Since a Task cannot have two parents, I retrieve one task id (task A) as the parent id and the other one (ID of task B) as a hyper-parameter, as you described 👍

Indeed, I actually had the old configuration that was not JSON - I converted to json, now works 🙂

yes, exactly: I run python my_script.py , the script executes, creates the task, calls task.remote_execute(exit_process=True) and returns to bash. Then, in the bash console, after some time, I see some messages being logged from clearml

This one doesn’t have _to_dict unfortunately

yes SparklingHedgehong28 🙂

This https://stackoverflow.com/questions/65109764/wildcard-search-issue-with-long-datatype-in-elasticsearch says long types can be converted to string to do the search

Stopping the server Editing the docker-compose.yml file, adding the logging section to all services Restarting the serverDocker-compose freed 10Go of logs

As you can see, more hard waiting (initial sleep), and then before each apt action, make sure there is no lock

I am using 0.17.5, it could be either a bug on ignite or indeed a delay on the send. I will try to build a simple reproducible example to understand to cause

but it is set here, right?

TimelyPenguin76 That sounds amazing! will there be a fallback mechanism as well? often p3.2xlarge are on shortage, would be nice to define one resources req as first choice (eg. p3.2xlarge) -> if not available -> use another resources req (eg. g4dn)

But I would need to reindex everything right? Is that a expensive operation?

I am using clearml_agent v1.0.0 and clearml 0.17.5 btw

I'll try with that; https://github.com/allegroai/clearml/compare/master...H4dr1en:add-aws-params

I can ssh into the agent and:
source /trains-agent-venv/bin/activate (trains_agent_venv) pip show pyjwt Version: 1.7.1

AgitatedDove14 Unfortunately no, I already had the problem before using the function, I added it hoping it would fix the issue but it didn’t

Yes, I get 0 afterwards

venv mode