Hi VexedElephant56 , does it fail when it runs on the agent?
Hi @<1691983266761936896:profile|AstonishingOx62> , its not only the output destination field but where all artifacts/debug/datasets were saved. This is usually logged in Mongo so you would need to run some migration script to change all the urls to the new ip
I'm assuming your dictionary is made from non basic types (like object of a sort)
What do you have inside this dict?
Hi @<1546303293918023680:profile|MiniatureRobin9> , do you have some stand alone script that reproduces this behaviour for you? Are you both running the same pipeline? How are you starting the pipeline
Does the other PC have the package locally there somewhere?
What optimization method did you use? Can you add a log of the HPO app? It's in hidden projects (Go to settings and enable viewing hidden projects).
Also, what are the values that the cloned experiments get?
And finally 🙂
Can you try with values 0 and 1 without the decimal points at the end?
Hi @<1817731748759343104:profile|IrritableHippopotamus34> , I think it should be safe.
Can you provide a self contained contained snippet that reproduces this behavior?
After this passes - and you refresh the page even with the popup stuck in infinite loop - does the project get deleted? I think it might be a UI issue
Hi @<1594863230964994048:profile|DangerousBee35> , it sounds like some sort of network lag. I assume you are using app.clear.ml?
I'd check network latency from the instances starting in GCP to the server.
Hi @<1523701260895653888:profile|QuaintJellyfish58> , yes this is correct. You can also set your files_server in clearml.conf to point to S3 bucket as well and this way debug samples will be saved there as well
Can you copy paste the error you got?
Hi Chen, please go to settings -> configuration and mark 'show hidden projects'
What command did you run? What were you trying to do? What was the setup?
Hi @<1523701842515595264:profile|PleasantOwl46> , what version of ClearML are you running? Also, when you navigate to the webUI, do you see any console errors in developer tools (F12)? Are the API requests returning empty?
Hi @<1655744373268156416:profile|StickyShrimp60> , is it possible you're using different ClearML SDK versions?
You need to separate the Task object itself from the code that is running. If you're manually 'reviving' a task but then nothing happens and no code is running then the task will get aborted eventually. I'm not sure I understand entirely what you're doing but I have a feeling you're doing something 'hacky'.
Hi CheerfulGorilla72 ,
What are you logging? Can you provide a small snippet or a screenshot?
Hi @<1552101474769571840:profile|DepravedLion86> , do you have something that reproduces this behavior?
What happens if you use the settings I pasted?
Browser thinks it's the same backend because of the domain
Strange. Can you add your clearml.conf from the agent machine? Please make sure to obscure all secrets 🙂
If you want this to be applied to all jobs that run on that agent then yes. Otherwise you can set it up on the task level as well
You need the agents to run the various pipeline steps + controllers
Hi CurvedHedgehog15 ,
Can you please provide a short snippet to reproduce this?
In which section are we looking at currently in the comparison?
Hi AbruptWorm50 ,
After cloning the experiment you can actually edit the installed packages and specify which package version you want.
You can also do this via code using this method:
https://clear.ml/docs/latest/docs/references/sdk/task#taskadd_requirements
When the SDK runs it takes the default in ~/clearml.conf
@<1523701132025663488:profile|SlimyElephant79> , it looks like you are right. I think it might be a bug. Could you open a GitHub issue to follow up on this?
As a workaround programmatically you can set Task.init(output_uri=True) , this will make the experiment outputs all to be uploaded to whatever is defined as the files_server in clearml.conf .
Hi @<1739455977599537152:profile|PoisedSnake58> , in the log you have the location of the cloned repo printed out.
For CLEARML_AGENT_EXTRA_PYTHON_PATH you need to provide it with a path