but it is set here, right?

I get the same error when trying to run the task using clearml-agent services-mode with docker, so weird

interestingly, it works on one machine, but not on another one

Ok, now I get ERROR: No matching distribution found for conda==4.9.2 (from -r /tmp/cached-reqscaw2zzji.txt (line 13))

There is an example in the https://github.com/allegroai/clearml/blob/master/docs/datasets.md#workflow section of the linked I shared above

The file  /tmp/.clearml_agent_out.j7wo7ltp.txt   does not exist

yes

, causing it to unregister from the server (and thus not remain there).

Do you mean that the agent actively notifies the server that it is going down? or the server infers that the agent is down after a timeout?

I want in my CI tests to reproduce a run in an agent because the env changes and some things break in agents and not locally

ho wait, actually I am wrong

CostlyOstrich36 , actually this only happens for a single agent. The weird thing is that I have a machine with two gpus, and I spawn two agents, one per gpus. Both have the same version. For one, I can see all the logs, but not for the other

No, they have different names - I will try to update both agents to the latest versions

I am sorry to give infos that are not very precise, but it’s the best I can do - Is this bug happening only to me?

I think it comes from the web UI of the version 1.2.0 of clearml-server, because I didn’t change anything else

CostlyOstrich36 Were you able to reproduce it? That’s rather annoying 😅

So two possible cases for trains-agent-1: either:
It picks a new experiment -> show randomly one of the two experiments in the "workers" tab no new experiment in default queue to start -> show randomly no experiment or the one that it is running

Thanks a lot for the solution SuccessfulKoala55 ! I’ll try that if the solution “delete old bucket, wait for its name to be available, recreate it with the other aws account, transfer the data back” fails

SuccessfulKoala55 I was able to make it work with use_credentials_chain: true in the clearml.conf and the following patch: https://github.com/allegroai/clearml/pull/478

same as the first one described

So it looks like the agent, from time to time thinks it is not running an experiment

When an experiment on trains-agent-1 is finished, I see randomly no experiment/long experiment and when two experiments are running, I see randomly one of the two experiments

by mistake I have two agents started in one machine

the latest version, but I think its normal: I set the TRAINS_WORKER_ID = "trains-agent":$DYNAMIC_INSTANCE_ID, where DYNAMIC_INSTANCE_ID is the ID of the machine

yes SparklingHedgehong28 🙂

Hi @<1523701205467926528:profile|AgitatedDove14> @<1537605940121964544:profile|EnthusiasticShrimp49> , the issue above seemed to be the memory leak and it looks like there is no problem from clearml side.
I trained successfully without mem leak with num_workers=0 and I am now testing with num_workers=8.
Sorry for the false positive :man-bowing:

I think that somehow somewhere a reference to the figure is still living, so plt.close("all") and gc cannot free the figure and it ends up accumulating. I don't know where yet

Is it exactly agg or something different?

Early debugging signals show that auto_connect_frameworks={'matplotlib': False, 'joblib': False} seem to have a positive impact - it is running now, I will confirm in a bit

Nope, I’d like to wait and see how the different tools improve over this year before picking THE one 😄

I am happy if I can be of any help to fix that 😄