Thanks @<1523701087100473344:profile|SuccessfulKoala55> ! Are alive workers sending ping to notify the server that they are alive or does the server infers that they are alive based on the last communication?

Ok, after:

SmugDolphin23 Actually adding agent.python_binary didn't work, it was not read by the clearml agent (in the logs dumped by the agent, agent.python_binary = (no value)

And so in the UI, in workers&queues tab, I see randomly one of the two experiments for the worker that is running both experiments

When installed with http://get.docker.com , it works

So it seems like it doesn't copy /root/clearml.conf and it doesn't pass the environment variables (CLEARML_API_HOST, CLEARML_API_ACCESS_KEY, CLEARML_API_SECRET_KEY)

Nice! What is the default value?

in an ec2 machine

how would it interact with the clearml-server api service? would it be completely transparent?

Thanks!

Ho and also use the colors of the series. That would be a killer feature. Then I simply need to match the color of the series to the name to check the tags

It failed as well

(Even if I explicitely do my_task.close() )

Nevertheless there might still be some value in that, because it would allow to reduce the starting time by removing the initial setup of the agent + downloading of the data to the instance - but not as much as I described initially, if instances stopped are bound to the same capacity limitations as new instances launched

Yes AnxiousSeal95 , stopped instance meaning you don’t pay for it, but just its storage, as described https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Stop_Start.html . So AgitatedDove14 increasing the IDLE timeout would still make me pay for the instance while they are idle.

Do you get stopped instances instantely when you ask for them?

Well that’s a good question, that’s what I observed some time ago, but according to their https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/...

Interesting - I can reproduce easily

Hi AgitatedDove14 , so I ran 3 experiments:
One with my current implementation (using "fork") One using "forkserver" One using "forkserver" + the DataLoader optimizationI sent you the results via MP, here are the outcomes:
fork -> 101 mins, low RAM usage (5Go constant), almost no IO forkserver -> 123 mins, high RAM usage (16Go, fluctuations), high IO forkserver + DataLoader optimization: 105 mins, high RAM usage (from 28Go to 16Go), high IO
CPU/GPU curves are the same for the 3 experiments...

(docker was install with sudo snap install docker )

Could you please point me to the relevant component? I am not familiar with typescript unfortunately 😞

There is a pinned github thread on https://github.com/allegroai/clearml/issues/81 , seems to be the right place?

sure, will be happy to debug that 🙂

AgitatedDove14 I am actually considering rolling back to 1.1.0, so 1.3.0 is not really an option for now

So either I specify in the clearml-agent agent.python_binary: python3.8 as you suggested, or I enforce the task locally to run with python3.8 using task.data.script.binary

I don't know why it didn't detect it in first place

I assume you’re using a self-hosted server?

Yes

Ho nice, thanks for pointing this out!

So it is there already, but commented out, any reason why?

Thanks AgitatedDove14 ! I created a project with a default output destination to a s3 bucket but I don't have local access to this bucket (only agents have access to it for security reasons). Because of that, I cannot create a task in this project programmatically locally because it tries to access the bucket and fails. And there is no easy way to change the default output location (not in the web UI, not in the sdk)

yes, the only thing I changed is:
install_requires=[ ... "my-dep @ git+ ]to:
install_requires=[ ... "git+ "]

torch==1.7.1 git+ .