Can you please add the full log of the execution?
In the task hyper parameters section you have a section called Hydra. In that section there should be a configuration called _allow_omegaconf_edit_
, what is it set to?
Then these should be by default killed by the ClearML server after a few hours. How long was it stuck?
Hi @<1523701295830011904:profile|CluelessFlamingo93> part of the server is a service that kills such tasks, I think this is what you're looking for - None
Hi CrabbyKoala94 ,
Are you running a self hosted server?
Not sure, I would ask the DevOps team to check
Can you try upgrading to the latest agent version? pip install -U clearml-agent
Same repo as the private repo?
I think you might find this video helpful
Also, can you verify that you still have the clearml-agent process running? top
/ htop
AbruptWorm50 , regarding the debug samples:
When were they registered?
Were you able to view them before?
Where are they registered?
DistinctShark58 , Hi 🙂
You can change all the default ports. This can be done through the docker compose for example.
Hi @<1584716355783888896:profile|CornyHedgehog13> , you can only see a list of files inside a dataset/version. I'm afraid you can't really pull individual files since everything is compressed and chunked. You can download individual chunks.
Regarding the second point - there is nothing out of the box but you can get a list of files in all datasets and then compare if some file exists in others.
Does that make sense?
AbruptWorm50 , can you try deleting your cookies/data on your browser to see if you manage to load the debug samples? I think this might be related: https://github.com/allegroai/clearml/issues/637
Can you add a snippet of how you're presenting/generating the matplotlibs?
AbruptWorm50 , it looks like the application issue was solved for us 🙂
Hi Chen, please go to settings -> configuration and mark 'show hidden projects'
StaleButterfly40 , I'm trying to get an estimate of what you have because if the content is too large the preview isn't shown....
Hi @<1750689997440159744:profile|ShinyPanda97> , I think you can simply move the model to a different project as part of the pipeline
Hi @<1635088270469632000:profile|LividReindeer58> , I'm afraid that isn't available in the open source version
RoughTiger69 Hi 🙂
Regarding your questions:
Moving certain tasks/datasets from server to server would require a data merge. This process basically requires merging the databases (mongodb, elasticsearch, files etc.). I think it's something that can be done in the paid version as a service but not in the free. I think if you'd like to 'promote' tasks to production you can either work on a special project for that OR upload the models to S3 and then re-run the experiment and point it to the...
StaleButterfly40 , do the inputs/outputs have previews or are they just file links?
Hi @<1654294834359308288:profile|DistressedCentipede23> , do you mean completely circumventing that basic function of the agent and simply pointing it to a specific script/env?
VexedCat68 , how is your ~/clearml.conf
configured? And is there any chance you were having network problems at the time you tried to run it?
Hi VexedCat68 , what is the error that you're receiving? Do you have a print of it by chance? Also what are you trying to do with 'File 2'?