Try it as the first option after clearml-agent: clearml-agent --debug daemon --docker --foreground
Hi RoughTiger69 , how are you running the pipeline? Locally or on agents? How is the controller running?
Hi @<1557537273090674688:profile|ThankfulOx54> , HyperDatasets are part of the Scale & Enterprise licenses. You can see more here: None
Hi IrritableJellyfish76 , yes. It is available only for Scale & Enterprise versions
Hi @<1523707653782507520:profile|MelancholyElk85> , in a section right under the default S3 credentials in clearml.conf
you have a section to specify per bucket 🙂
SwankySeaurchin41 , I don't think pipelines were mentioned in the video. Are you looking for something specific?
ClearML should log all OmegaConf automatically according to this: https://clear.ml/docs/latest/docs/fundamentals/hyperparameters#hydra
Might as well take a look at this example as well 🙂
https://github.com/allegroai/clearml/blob/master/examples/frameworks/hydra/hydra_example.py
SwankySeaurchin41 , I encountered it somewhere before. This appears to be the code 🙂
https://github.com/thepycoder/urbansounds8k
Hi ConvolutedSealion94 , I think you're right. Maybe open a github issue for this to be added?
Hi @<1564060257435521024:profile|PerfectShrimp1> , currently not supported. Maybe open a GitHub feature request for better filterings
However it would be advisable to also add the following argument to your code : Task.init(..., output_uri=True)
SwankySeaurchin41 , what do you mean? Can you give a specific example?
ReassuredTiger98 , does it happen on any experiment with debug images or only on running experiments?
Hi @<1697056708469198848:profile|HollowPeacock63> , not sure I understand. What exactly are you trying to do?
CluelessElephant89 , I've added screenshots. Tell me if those help 🙂
ReassuredTiger98 , I played with it myself a little bit - It looks like this happens for me when an experiment is running and reporting images and changing metric does the trick - i.e reproduces it. Maybe open a github issue to follow this 🙂 ?
I think that's what's there. In the Scale & Enterprise version ClearML usually works together with customers to provide a glue layer for K8s or even SLURM
Interesting idea. Can you open a github issue for it for better tracking?
Hi ElegantCoyote26
I'm not sure what you mean, you create endpoints using clearml-serving
What exactly are you looking for?
I think this is what you're looking for - the agent integration
None
I think this would be right up your alley 🙂
https://clear.ml/docs/latest/docs/references/sdk/model_outputmodel
And you can update weights later using:
https://clear.ml/docs/latest/docs/references/sdk/model_outputmodel#update_weights
It's all configured by the helm chart, it is the glue layer between K8s & ClearML
BattyDove56 , that was my suspicion as well, that's why I wanted to see the logs 🙂
Hi BattyDove56 , it looks like your elasticsearch container is restarting. Is this the issue still? Can you check the container logs to see why it's restarting? I think this is what might be causing the issue with ClearML server not raising properly
BattyDove56 , the warning doesn't seem related. As I mentioned before, you need to check the elastic logs to see what's the issue. post them here so we can look together 🙂
The following command should give you something:docker logs --follow clearml-elastic
What errors are you getting?
Hmmm Regarding your issue you can use the following env vars to define your endpoint
https://clear.ml/docs/latest/docs/configs/env_vars/#server-connection
What is your usecase? Do you want to change the endpoint mid run?
Hi @<1569133683275730944:profile|CrabbyDove13> , the PyCharm plugin is for working with remote environments. I don't think you need is with VSCode since this capability is covered by clearml-session
You don't load the configuration during from clearml import Task
config is loaded during Task.init()
So you can make all your configuration additions up to the point Task.init()
is run