Hi @<1557537273090674688:profile|ThankfulOx54> , HyperDatasets are part of the Scale & Enterprise licenses. You can see more here: None
but have our own custom logging of metrics etc.
Are those custom metrics reported to the ClearML server or stored somewhere else?
Just wondering how tricky integrating a trigger would be for that
I guess it really depends on your current implementation currently
You can see my answer in channel
Hi @<1539417873305309184:profile|DangerousMole43> , in that case I think you can simply save the file path as a configuration in the first step and then in the next step you can simply access this file path from the previous step. Makes sense?
Hi @<1542316991337992192:profile|AverageMoth57> , yes you can 🙂
What if you remove the Point package? Does it run then?
Can you add the full log?
Try it as the first option after clearml-agent: clearml-agent --debug daemon --docker --foreground
You can create a queue through the UI. You can go into Workers & Queues tab -> Queues -> "New Queue"
You can also create new queues using the API as well
https://clear.ml/docs/latest/docs/references/api/queues#post-queuescreate
Hi @<1655744373268156416:profile|StickyShrimp60> , is it possible you're using different ClearML SDK versions?
Hi @<1655744373268156416:profile|StickyShrimp60> , I think it would be good to open a GitHub issue if there isn't one 🙂
@<1681111528419364864:profile|SmoothGoldfish52> , it will be saved to a cache folder. Take a look at what @<1576381444509405184:profile|ManiacalLizard2> wrote. I think tar files might work already. Give it a test
It's an interesting question!
I think that the instances are terminated when they are spun down therefor the behavior should be the same as if you terminated them yourself manually
On prem is also K8s? Question is if you run the code unrelated to ClearML on EKS, do you still get the same issue?
Did you see any other errors in the server logs? Is the artifact very large by chance?
Can you create a standalone script that reproduces this? @<1736194481398484992:profile|MoodySeaurchin62>
I'm not sure. But you can access the clearml.conf file through code
Now try logging in
Do you mean to kill the clearml-agent process after the task finishes running? What is the use case I'm curious
Hi ConvolutedSealion94 , you should use the InputModel/OutputModel modules for working with models:
https://clear.ml/docs/latest/docs/references/sdk/model_inputmodel
This makes getting models very easily directly by their IDs (Models have unique IDs). For example:
https://clear.ml/docs/latest/docs/references/sdk/model_inputmodel#get_local_copy
Or:
https://clear.ml/docs/latest/docs/references/sdk/model_inputmodel#get_weights_package
Hi @<1639799308809146368:profile|TritePigeon86> , where in the documentation did you see these parameters active_duration, job_started and job_ended ?
I'd suggest using the API directly, fetch a task and compare it's start to end time.
Hi TartBear70 ,
Did you run the experiment locally first? What versions of clearml/clearml-agent are you using?
Hi @<1570220858075516928:profile|SlipperySheep79> , I see no reason you shouldn't be able to. Are these the only actions you're doing? Does it happen to you with only decorators or also pipelines from tasks & functions?
If you want this to be applied to all jobs that run on that agent then yes. Otherwise you can set it up on the task level as well
Hi @<1577468638728818688:profile|DelightfulArcticwolf22> , I think this is what you're looking for - CLEARML_AGENT_SKIP_PIP_VENV_INSTALL
CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL
CLEARML_AGENT_FORCE_SYSTEM_SITE_PACKAGES
None
Hi @<1523702000586330112:profile|FierceHamster54> , do you mean you changes the policy during upload?
Does it go back to working if you revert the changes?