I'll try with that; https://github.com/allegroai/clearml/compare/master...H4dr1en:add-aws-params

For some reason the configuration object gets updated at runtime to
resource_configurations = null queues = null extra_trains_conf = "" extra_vm_bash_script = ""

Still getting the same error, it is not taken into account 🤔

Adding back clearml logging with matplotlib.use('agg') , uses more ram but not that suspicious

Hi @<1523701205467926528:profile|AgitatedDove14> @<1537605940121964544:profile|EnthusiasticShrimp49> , the issue above seemed to be the memory leak and it looks like there is no problem from clearml side.
I trained successfully without mem leak with num_workers=0 and I am now testing with num_workers=8.
Sorry for the false positive :man-bowing:

With a large enough number of iterations in the for loop, you should see the memory grow over time

Hi SuccessfulKoala55 , super that’s what I was looking for

Hi CostlyOstrich36 , most of the time I want to compare two experiments in the DEBUG SAMPLE, so if I click on one sample to enlarge it I cannot see the others. Also once I closed the panel, the iteration number is not updated

Ok to be fair I get the same curve even when I remove clearml from the snippet, not sure why

Ok so what is the value that is set when it is run by the agent? agg ?

I also tried task.set_initial_iteration(-task.data.last_iteration) , hoping it would counteract the bug, didn’t work

AgitatedDove14 In theory yes there is no downside, in practice running an app inside docker inside a VM might introduce slowdowns. I guess it’s on me to check whether this slowdown is negligible or not

Ok no it only helps if as far as I don't log the figures. If I log the figures, I will still run into the same problem

clearml doesn't change the matplotlib backend under the hood, right? Just making sure 😄

is there a command / file for that?

Hi @<1523701087100473344:profile|SuccessfulKoala55> I was able to find the issue, I was creating a queue and worker subprocess that were not properly cleaned up

from the ClearML UI

Now I'm curious, what did you end up doing ?

in my repo I maintain a bash script to setup a separate python env. then in my task I spawn a subprocess and I don't pass the env variables, so that the subprocess properly picks up the separate python env

Nevermind, I just saw report_matplotlib_figure 🎉

I’d like to move to a setup where I don’t need these tricks

might be worth documenting 😄

that’s perfect, thanks!

If I manually call report_matplotlib_figure yes. If I don't (just create the figure), no mem leak

This is what I get with mprof on this snippet above (I killed the program after the bar reaches 100%, otherwise it hangs trying to upload all the figures)

The rest of the configuration is set with env variables

Downloading the artifacts is done only when actually calling get()/get_local_copy()

Yes, I rather meant: reproduce this behavior even for getting metadata on the artifacts 🙂

in my clearml.conf, I only have:
sdk.aws.s3.region = eu-central-1 sdk.aws.s3.use_credentials_chain = true agent.package_manager.pip_version = "==20.2.3"

Thanks for the help SuccessfulKoala55 , the problem was solved by updating the docker-compose file to the latest version in the repo: https://github.com/allegroai/clearml-server/blob/master/docker/docker-compose.yml
Make sure to do docker-compose down & docker-compose up -d afterwards, and not docker-compose restart

Ha nice, makes perfect sense thanks AgitatedDove14 !

Here is the console with some errors