VexedCat68 , I don't think such an example exists, but if you create one it would be great if you opened a PR for the open source 🙂
Hi @<1572032783335821312:profile|DelightfulBee62> , I think 1 TB should be enough. I would suggest maybe even having 2T just for the safe side
Hi @<1710827340621156352:profile|HungryFrog27> , to answer your questions:
- You can disable commit logging in your clearml.comf file - None
- Not sure what you mean. Things that are logged on the server are console logs, scalars/metrics, configurations and execution information (git, branch, commit id and uncommitted changes). These are all saved in the backend. Can you elaborate to w...
I think this is what you're looking for - the agent integration
None
Hi RoundMosquito25 , you can reset a failed task and this will bring it back to draft mode
I think this would be right up your alley 🙂
https://clear.ml/docs/latest/docs/references/sdk/model_outputmodel
And you can update weights later using:
https://clear.ml/docs/latest/docs/references/sdk/model_outputmodel#update_weights
Hi @<1523701717097517056:profile|ScantMoth28> , what version of ClearML are you using? Are you using a self hosted server or the community one?
And if you run the same code locally everything is reported correctly?
AbruptWorm50 , regarding the debug samples:
When were they registered?
Were you able to view them before?
Where are they registered?
AbruptWorm50 , can you try deleting your cookies/data on your browser to see if you manage to load the debug samples? I think this might be related: https://github.com/allegroai/clearml/issues/637
AbruptWorm50 , you can send me. Also can you please answer the following two questions?When were they registered? Were you able to view them before?
Also, you mention plots but in the screenshot you show debug samples. Can I assume you're talking about debug samples?
AbruptWorm50 , I also see the HPO app is missing, I'm told this is under investigation.
AbruptWorm50 , it looks like the application issue was solved for us 🙂
https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_mongo44_migration
Looks like what you might need 🙂
SkinnyPanda43 , I think so yes 🙂
Hi, SkinnyPanda43 , from what version did you upgrade to which version?
I see, I suspect you might need to do some migrations
GiganticTurtle0 , which ClearML version are you using? From what I can see in the documentation to add the new parameters, you'll have to task.connect() again to add the new args
Hi @<1523701066867150848:profile|JitteryCoyote63> , you mean a global "env" variable that can be passed along the pipeline?
I'm accessing both using SSH tunneling & the same domain
I guess we found the culprit 🙂
Please check what you get for events.debug_images
in network section of developer tools (F12) when trying to view the preview in the dataset
Hi @<1820993257639776256:profile|DeepOwl31> , are all workers on the same machine?
Yes, it can be controlled via the local clearml.conf
file
From the screenshots provided you ticked 'cpu' mode AND I think the machine that you're using n1-standard-1 is a cpu only machine, if I'm not mistaken.
Hi @<1787291173992271872:profile|BlandCormorant75> , how are you logging those plots? Can you provide a stand alone snippet that reproduces your behaviour?
Hi @<1582904448076746752:profile|TightGorilla98> , you would need to edit these addresses in mongodb/elastic for the new address. A migration script would do the job
Are you running in docker mode? You could maybe use another docker image that has python in it.
You should check the status of that container
Hi @<1736919317200506880:profile|NastyStarfish19> , the services queue is for running the pipeline controller itself. I guess you are self hosting the OS?