Is there any logic on the server side that could change the iteration number?

Mmmh unfortunately not easily… I will try to debug deeper today, is there a way to resume a task from code to debug locally?
Something like replacing Task.init with Task.get_task so that Task.current_task is the same task as the output of Task.get_task

Note: Could be related to https://github.com/allegroai/clearml/issues/790 , not sure

Sorry both of you, my problem was actually lying somewhere else (both buckets are in the same region) - thanks for you time!

ok, so there is no way to cache it and detect when the ref changes?

even if I move the Github workers internally where they could have access to the prod server, I am not sure I would like that, because it would pile up test data in the prod server that is not necessary

Alright, experiment finished properly (all models uploaded). I will restart it to check again, but seems like the bug was introduced after that

There is a pinned github thread on https://github.com/allegroai/clearml/issues/81 , seems to be the right place?

Hi SuccessfulKoala55 , super that’s what I was looking for

CostlyOstrich36 good enough, I will fallback to sorting by updated, thanks!

The part where I'm lost is why would you need the path to the temp venv the agent creates/uses ?

let's say my task is calling a bash script, and that bash script is calling another python program, I want that last python program to be executed with the environment that was created by the agent for this specific task

select multiple lines still works, you need to shift + click on the checkbox

DeterminedCrab71 Please check this screen recording

I’ll definitely check that out! 🤩

I want to make sure that an agent did finish uploading its artifacts before marking itself as complete, so that the controller does not try to access these artifacts while they are not available

@<1523701205467926528:profile|AgitatedDove14> I see other rc in pypi but no corresponding tags in the clearml-agent repo? are these releases legit?

agent.cuda_version = 110 agent.cudnn_version = 0

ok yes, that makes sense 👍

This is no coincidence - Any data versioning tool you will find are somehow close to how git works (dvc, etc.) since they aim to solve a similar problem. In the end, datasets are just files.
Where clearml-data stands out imo is the straightfoward CLI combined with the Pythonic API that allows you to register/retrieve datasets very easily

I asked this question some time ago, I think this is just not implemented but it shouldn’t be difficult to add? I am also interested in such feature!

Awesome, thanks WackyRabbit7 , AgitatedDove14 !

I must say I am confused

AgitatedDove14 After investigation, another program on the machine consumed all the memory available, most likely making the OS killing the agent/task

I think my problem is that I am launching an experiment with python3.9 and I expect it to run in the agent with python3.8. The inconsistency is from my side, I should fix it and create the task with python3.8 with:
task.data.script.binary = "python3.8" task._update_script(convert_task.data.script)Or use python:3.9 when starting the agent

The task requires this service, so the task starts it on the machine - Then I want to make sure the service is closed by the task upon completion/failure/abortion

And I am wondering if only the main process (rank=0) should attach the ClearMLLogger or if all the processes within the node should do that

yes

I am not sure what you mean by unless the domain is different ? Personal Access Token are designed such that to allow cloning a private repo, the user has to give the PAT full access to repos, including public repos. So it should also work with all other git repos

It broke the shift holding to select multiple experiments btw

yes exactly

Yes that’s what I did initially, but eventually I decided that it’s too much complexity added for nothing really, I’d rather drop omegaconf and if one day clearml supports it out of the box take advantage of it