SuccessfulKoala55 They do have the right filepath, eg:
https://***.com:8081/my-project-name/experiment_name.b1fd9df5f4d7488f96d928e9a3ab7ad4/metrics/metric_name/predictions/sample_00000001.png

But that was too complicated, I found an easier approach

Ok AgitatedDove14 SuccessfulKoala55 I made some progress in my investigation:
I can exactly pinpoint the change that introduced the bug, it is the one changing the endpoint "events.get_task_log", min_version="2.9"
In the firefox console > Network, I can edit an events.get_task_log and change the URL from …/api/v2.9/events.get_task_log to …/api/v2.8/events.get_task_log (to use the endpoint "events.get_task_log", min_version="1.7" ) and then all the logs are ...

yes, in setup.py I have:
..., install_requires= [ "my-private-dep @ git+ ", ... ], ...

yes, something like that

Hi @<1523701205467926528:profile|AgitatedDove14> @<1537605940121964544:profile|EnthusiasticShrimp49> , the issue above seemed to be the memory leak and it looks like there is no problem from clearml side.
I trained successfully without mem leak with num_workers=0 and I am now testing with num_workers=8.
Sorry for the false positive :man-bowing:

ok, so there is no way to cache it and detect when the ref changes?

even if I move the Github workers internally where they could have access to the prod server, I am not sure I would like that, because it would pile up test data in the prod server that is not necessary

I can ssh into the agent and:
source /trains-agent-venv/bin/activate (trains_agent_venv) pip show pyjwt Version: 1.7.1

with the CLI, on a conda env located in /data

🚀 Thanks @<1523701205467926528:profile|AgitatedDove14> !

AgitatedDove14 I now tested with a real experiment, it works, but I saw two issues:
It first doesnt detect torch, downloads it but then says that it is already installed so it doesn't install it. One of the dependency of my repository is another repository (repo-2 in the logs). Both my repositories require numpy . When installing the first repository, it says Requirement already satisfied: numpy in /home/workeruser/.local/lib/python3.6/site-packages . Correct. But then it says `...

Hi @<1523701087100473344:profile|SuccessfulKoala55> I was able to find the issue, I was creating a queue and worker subprocess that were not properly cleaned up

For the moment this is what I would be inclined to believe

Sure 🙂 Opened https://github.com/allegroai/clearml/issues/568

Still investigating, task.data.last_iteration is correct (equal to engine.state["iteration"] ) when I resume the training

I am not using hydra, I am reading the conf with:
config_dict = read_yaml(conf_yaml_path) config = OmegaConf.create(task.connect_configuration(config_dict))

ha nice, where can I find the mapping template of the original clearml so that I can copy and adapt?

Also I can simply delete the /elastic_7 folder, I don’t use it anymore (I have a remote ES cluster). In that case, I guess I would have enough space?

Sorry, I refreshed the page and it’s gone 😅

On the cloned experiment, which by default is created in draft mode, you can change the commit to point either a specific commit or the latest commit of the branch

Well no luck - using matplotlib.use('agg') in my training codebase doesn't solve the mem leak

Could you please point me to the relevant component? I am not familiar with typescript unfortunately 😞

Yes, but I am not certain how: I just deleted the /data folder and restarted the server

You are right, thanks! I was trying to move /opt/trains/data to an external disk, mounted at /data

Sure, where can I find this file?

I wouldn't do it, this is less code to maintain from your side and honestly too much auto magic makes it difficult for the user to control the environment (ie. to understand what happens behind the scenes). I am not sure what switching back will solve, here the wheel should have been correct, it's just the architecture of the card that is incompatible

I did that recently - what are you trying to do exactly?

If the reporting is done on a subprocess, I can imagine that the task.set_initial_iteration(0) call is only effective in the main process, not in the subprocess used for reporting. Could it be the case?