Sure 🙂 Opened https://github.com/allegroai/clearml/issues/568

Yes AnxiousSeal95 , stopped instance meaning you don’t pay for it, but just its storage, as described https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Stop_Start.html . So AgitatedDove14 increasing the IDLE timeout would still make me pay for the instance while they are idle.

Do you get stopped instances instantely when you ask for them?

Well that’s a good question, that’s what I observed some time ago, but according to their https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/...

Sure, where can I find this file?

Thanks for the help SuccessfulKoala55 , the problem was solved by updating the docker-compose file to the latest version in the repo: https://github.com/allegroai/clearml-server/blob/master/docker/docker-compose.yml
Make sure to do docker-compose down & docker-compose up -d afterwards, and not docker-compose restart

I also did run sudo apt install nvidia-cuda-toolkit

Relevant SO issue > https://stackoverflow.com/questions/31829587/docker-container-logs-taking-all-my-disk-space

yes -> but I still don't understand why the post_packages didn't work, could be worth investigating

Ho and also use the colors of the series. That would be a killer feature. Then I simply need to match the color of the series to the name to check the tags

hooo now I understand, thanks for clarifying AgitatedDove14 !

Thanks for your help 🙏

Very cool! Looking foward to the trains-agent venv caching 🤩

Yes exactly

now I can do nvcc --version and I get
Cuda compilation tools, release 10.1, V10.1.243

Hi SuccessfulKoala55 , there it is > https://github.com/allegroai/clearml-server/issues/100

Yes, not sure it is connected either actually - To make it work, I had to disable both venv caching and set use_system_packages to off, so that it reinstalls the full env. I remember that we discussed this problem already but I don't remember what was the outcome, I never was able to make it update the private dependencies based on the version. But this is most likely a problem from pip that is not clever enough to parse the tag as a semantic version and check whether the installed package ma...

The cloning is done in another task, which has the argv parameters I want the cloned task to inherit from

yes

@<1523701205467926528:profile|AgitatedDove14> I see other rc in pypi but no corresponding tags in the clearml-agent repo? are these releases legit?

What is latest rc of clearml-agent? 1.5.2rc0?

it would be nice if Task.connect_configuration could support custom yaml file readers for me

There is a pinned github thread on https://github.com/allegroai/clearml/issues/81 , seems to be the right place?

Hi SuccessfulKoala55 , yes indeed

trains-agent-1: runs an experiment for a long time (>12h). Picks a new experiment on top of the long one running trains-agent-2: runs only one experiment at a time, normal trains-agent-3: runs only one experiment at a time, normalIn total: 4 experiments running for 3 agents

Thanks!3. I don't know, I never used Highcharts 🙂

So either I specify in the clearml-agent agent.python_binary: python3.8 as you suggested, or I enforce the task locally to run with python3.8 using task.data.script.binary

Yes that’s correct - the weird thing is that the error shows the right detected region

Interesting idea! (I assume for reporting only, not configuration)

Yes for reporting only - Also to understand which version is used by the agent to define the torch wheel downloaded

regrading the cuda check with

nvcc

, I'm not saying this is a perfect solution, I just mentioned that this is how this is currently done.
I'm actually not sure if there is an easy way to get it from nvidia-smi interface, worth checking though ...

Ok, but when nvcc is not ava...

Here are the logs of the agent :)
` (base) user@worker:~$ tail -f /tmp/.clearml_agent_daemon_outjdups8t2.txt
sdk.development.worker.log_stdout = true
sdk.development.worker.report_global_mem_used = false

+----------------------------------+--------+-------+
| id | name | tags |
+----------------------------------+--------+-------+
| 54e4a62a402d5135612ba7b12cfe4e57 | docker | |
+----------------------------------+--------+-------+

Starting infinite tas...

You already fixed the problem with pyjwt in the newest version of clearml/clearml-agents, so all good 😄

The task is created using Task.clone() yes