AgitatedDove14 Yes with the command you shared I can now ssh again manually to the agent, but I still clearml-agent will raise the same error

can it be that the merge op takes so much filesystem cache that the rest of the system becomes unresponsive?

Here is (left) the data disk (/opt/clearml) and right the OS disk

Super! I’ll give it a try and keep you updated here, thanks a lot for your efforts 🙏

SuccessfulKoala55 I am using ES 7.6.2

Ha nice, good one! Thanks!

I am not sure what you mean by unless the domain is different ? Personal Access Token are designed such that to allow cloning a private repo, the user has to give the PAT full access to repos, including public repos. So it should also work with all other git repos

Ho I wasn't aware of that new implementation, was it introduced silently? I don't remember reading it in the release notes! To answer your question: no, for gcp I used the old version, but for azure I will use this one, maybe send a PR if code is clean 👍

So I need to have this merging of small configuration files to build the bigger one

I did change the replica setting on the same index yes, I reverted it back from 1 to 0 afterwards

Thanks!

Ok thanks! And for this?

Would it be possible to support such use case? (have the clearml-agent setting-up a different python version when a task needs it?)

I just move one experiment in another project, after moving it I am taken to the new project where the layout is then reset

but it is set here, right?

v0.17.5rc2

(by console you mean in the dashboard right? or the terminal?)

Ok, by setting PyJWT==1.7.1 in the setup.py of the experiment pip did not enforced the update

CostlyOstrich36 Were you able to reproduce it? That’s rather annoying 😅

I think my problem is that I am launching an experiment with python3.9 and I expect it to run in the agent with python3.8. The inconsistency is from my side, I should fix it and create the task with python3.8 with:
task.data.script.binary = "python3.8" task._update_script(convert_task.data.script)Or use python:3.9 when starting the agent

yes SparklingHedgehong28 🙂

AgitatedDove14 Same problem with clearml==1.1.5rc2 😞 , I also tried with backend==gloo , still same problem

I need to investigate further

It broke the shift holding to select multiple experiments btw

I will let the team answer you on that one 🙂

mmmh there is no closing of the task happening at that point. Note that just before the task.upload_artifact, I call task.logger.report_table("Metric summary", "Metric summary", 0, df_scores) , if that matters

I am sorry to give infos that are not very precise, but it’s the best I can do - Is this bug happening only to me?

CostlyOstrich36 super thanks for confirming! I have then the follow-up question: are the artifacts duplicated (copied)? or just referenced?

I guess I can have a workaround by passing the pipeline controller task id to the last step, so that the last step can download all the artifacts from the controller task.

ha nice thanks 🙏

The jump in the loss when resuming at iteration 31 is probably another issue -> for now I can conclude that:
I need to set sdk.development.report_use_subprocess = false I need to call task.set_initial_iteration(0)