Because of a server error I can't download the log so I attached a screenshot. In the log I see only the following reports (without a summary table/plot).

also tried to connect from to dataset from CLI and recieved connection error:

Sending you to private CostlyOstrich36

yes

Hi, joining in into this thread. I just have the same error with my task, which is not an optimization one.
created virtual environment CPython3.7.5.final.0-64 in 232ms
creator CPython3Posix(dest=/home/ubuntu/.clearml/venvs-builds/3.7, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/ubuntu/.local/share/virtualenv)
` added seed packages: pip==22.0.4, setuptool...

I had a task which I have cloned and reset a bunch of times, when I created the test as a new one, the error didnt appear again

No it was also a fluke..

What will happen if I disable the cache? Is there a way to find out which experiment is hung and why? in order to avoid this?

https://app.slack.com/client/TT9ATQXJ5/activity-page/thread/CTK20V944-1650783838.033619

CostlyOstrich36 1.3.2

Does this relate to the error below? from reading the issue I didn't see anyone mentioning this error -
clearml_agent: ERROR: Failed cloning repository.
1) Make sure you pushed the requested commit:
(repository=' https://gitlab.com/data_science_team/____ ', branch='ilanit', commit_id='b5c___', tag='', docker_cmd='ubuntu:18.04', entry_point='training/____.py', working_dir='src')
2) Check if remote-worker has valid credentials [see worker configuration file]

Hi, the URL contains some details which I wouldn't like to share on this thread. Can I send it to one of you in private message?

CostlyOstrich36 The application problem was indeed solved 🙂 but the plots one didn't

Clearing my cookies solved the issue, Thanks 🙂

I will elaborate on the situation:
I have 2 agents - training and training_2. They are both listening to the services queue, but only 'training' pulls the tasks. At the beginning I had 2 tasks in the services queue. Then, agent 'training' pulled one and is currently executing it, but for some reason - it also pulled the 2nd task into it's queue, that is although this agent is not free and I have another agent which is: 'training_2'.

CumbersomeCormorant74 As you can see in the attached - there were 2 experiments at the same time, but only one agent pulled the task, even though the second agent was free and listening to the queue.

SuccessfulKoala55 I can't share the logs.

Can you check if there is a situation where there are tasks pending in that queue while one of the workers is idle?

In what queue ? In services there are no pending tasks because they were all pulled by 'training'.

just to maker sure, how do you start the agents? Are you using the

--services-mode

option?

I used clearml-agent daemon --detached --services-mode --queue services --docker ubuntu:18.04 --cpu-only

Just to clarify again - when I start the agents I run :
clearml-agent daemon --detached --queue training
and then: clearml-agent daemon --detached --services-mode --queue services --docker ubuntu:18.04 --cpu-only
This is why there are 'training' and 'training_2' queues.

We have been trying to resolve the issue. I will comment here again if any more problems arise. Thanks!

My questions are:

1. how can I avoid creating tens of new cache files?
2. do you happen to know why this lock is created and how it is connected to the above error (in the link - regarding "failing to clone.. ")

I really like the first idea, but I already see a problem if I make changes to the file, I will need to re-upload it every time?
I am currently using the repo cache, but unfortunately every time I run a new task with an existing cached repo, a new cache file is created.. very weird behaviour which I have already described in previous threads ( https://clearml.slack.com/archives/CTK20V944/p1651393535530439?thread_ts=1651063782.712379&cid=CTK20V944 )

I don't know if it has anything to do with it but I now see that the repo which is cloned and save in the cache is actually a different branch than the one which is pulled by the agent.

From the log file:
repository = https://gitlab.com/data_science_team/PROJECT_NAME branch = MYBRANCH
SOMENUMBER IP### DEBUG
warning: redirecting to https://gitlab.com/data_science_team/PROJECT_NAME.git/

CostlyOstrich36 Another clarification:
The master branch cache is stored at ". clearml/vcs-cache " - the code file doesn't exist there + the problem described above is occuring in this folder (multiple cache files of same repo).
While my branch is stored at " .clearml/venvs-builds/3.7/task_repository/"

yes of course I specified it with :
torch.save(state, f_path)
where f_path = os.path.join(tempfile.gettempdir(), 'trained_models') + '\ http://my_model.pt '

The problem is with the path I am trying to save the model to.Therefore my question How do I extract paths inside ClearML?

it is finalised

In the child dataset task I see the following :
ARTIFACTS - > STATE:
Dataset state
Files added/modified: 1 - total size 518.78 MB
Current dependency graph: {
"0385db....": [], ()
"94f4....": ["0385db..."]
}
child task is 94f4.. and parent task is "0385db..." but what does the () line means?

Another question: Is there a way to group together Dataset tasks (i.e redefine their parent) after the tasks have been finalized? In the same context: is there a way to change the dependency graph in the clearml dashboard after the task creation and finalization ?

Thanks!!