yes

ok, now I actually remember why I used _update_requirements instead of add_requirements: The first overwrites all the other, the later only add to the already detected packages. Since my deps are listed in the dependencies of my setup.py, I don't want clearml to list the dependencies of the current environment

I had this problem before

/data/shared/miniconda3/bin/python /data/shared/miniconda3/bin/clearml-agent daemon --services-mode --detached --queue services --create-queue --docker ubuntu:18.04 --cpu-only

And I can verify that ~/trains.conf exists in the su home folder

from the ClearML UI

Alright then! 👍

Thanks, I will create an issue. I am fine with both ways :)

And I do that each time I want to create a subtask. This way I am sure to retrieve the task if it already exists

AnxiousSeal95 The main reason for me to not use clearml-serving triton is the lack of documentation tbh 😄 I am not sure how to make my pytorch model run there

Thanks!

Alright, how can I then mount a volume of the disk?

Yea thats what I thought, I do have trains server 0.15

Awesome, thanks!

Would you like me to open an issue for that or will you fix it?

5,6 mins exactly

That would be amazing!

AgitatedDove14 WOW, thanks a lot! I will dig into that 🚀

It broke the shift holding to select multiple experiments btw

I am also interested in the clearml-serving part 😄

meaning the RestAPI returns nothing, is that correct

Yes exactly, this is the response from the api server when I try to scroll down on the console to get more logs

AgitatedDove14 SuccessfulKoala55 I just saw that clearml-server 1.4.0 was released, congrats 🚀 🙌 Was this bug fixed with this new version?

Hi TimelyPenguin76 ,
trains-server: 0.16.1-320
trains: 0.15.1
trains-agent: 0.16

v0.17.5rc2

This is what I get with mprof on this snippet above (I killed the program after the bar reaches 100%, otherwise it hangs trying to upload all the figures)

Hi @<1523701205467926528:profile|AgitatedDove14> @<1537605940121964544:profile|EnthusiasticShrimp49> , the issue above seemed to be the memory leak and it looks like there is no problem from clearml side.
I trained successfully without mem leak with num_workers=0 and I am now testing with num_workers=8.
Sorry for the false positive :man-bowing:

Disclaimer: I didn't check this will reproduce the bug, but that's all the components that should reproduce it: a for loop creating figures and clearml logging them

mmmh it fails, but if I connect to the instance and execute ulimit -n , I do see
65535while the tasks I send to this agent fail with:
OSError: [Errno 24] Too many open files: '/root/.commons/images/aserfgh.png'and from the task itself, I run:
import subprocess print(subprocess.check_output("ulimit -n", shell=True))Which gives me in the logs of the task:
b'1024'So nnofiles is still 1024, the default value, but not when I ssh, damn. Maybe rebooting would work

Ok so what is the value that is set when it is run by the agent? agg ?