CumbersomeParrot30 , try setting the following environment variable to true:CLEARML_SKIP_PIP_VENV_INSTALL
WackyRabbit7 , isn't this what you need?
https://clear.ml/docs/latest/docs/references/sdk/automation_controller_pipelinecontroller#get_running_nodes
Hi @<1754676274460102656:profile|CurvedStarfish68> , it takes some time until metrics are updated. I suggest deleting a few more experiments and waiting.
Hi @<1671689469218000896:profile|PleasantWalrus77> , is this AWS S3 or something like minio?
Can you add a snippet of how you're presenting/generating the matplotlibs?
Hi WorriedRabbit94 , what do you see in the execution section of the experiment when you run it locally?
What version of clearml
are you using? Can you try in a clean python virtual env?
Hi @<1529271098653282304:profile|WorriedRabbit94> , I'll ask the guys to take a look at this and what is required for it.
Hi @<1668427950573228032:profile|TeenyShells80> , you would need to configure it in the clearml.conf
of the machine running the clearml-agent
Does this happen always?
Hi OddShrimp85 , you mean bash script? I don't think there is something built in to run a script afterwards but I'm sure you could incorporate it in your python script.
I'm curious, what is the use case?
AppetizingMouse58 , might have some input here 🙂
Hi @<1523701504827985920:profile|SubstantialElk6> , thanks for the heads up 🙂
I would suggest googling that error
Hi @<1523704674534821888:profile|SourLion48> , making sure I understand - You push a job into a queue that an autoscaler is listening to. A machine is spun up by the autoscaler and takes the job and it runs. Afterwards during the idle time, you push another job to the same queue, it is picked up by the machine that was spun up by the autoscaler and that one will fail?
Hi @<1523704674534821888:profile|SourLion48> , yes and yes. It's all part of the docs - None
I'm not sure I understand this config, is this an autoscaler for GCP or AWS?
And are they the same tasks?
I mean if you were to run the 'failing' task first, it would run, correct?
Hi @<1729309137944186880:profile|GrittyBee73> , models are unique objects in the system so each one of them has a unique ID. By default they will be named the same. However, you can add versioning on top in any way that you want. You can either add tags or even add metadata on top of them and then add custom columns according to this metadata so you can filter by versions.
What do you think?
You can do it in one API call as follows:
https://clear.ml/docs/latest/docs/references/api/tasks#post-tasksget_all
DepravedSheep68 , do you mean when registering your data?
Or when running something and uploading to a s3?
BrightMosquito10 simply re-run it with the new version 🙂
Hi SoreHorse95 ,
Does ClearML not automatically log all outputs?
Regarding logging maybe try the following setting in ~/clearml.conf
sdk.network.metrics.file_upload_threads: 16
Hi @<1679299603003871232:profile|DefeatedOstrich25> , you mean you're on the community server? Do you see any sample datasets in the Datasets section?
Hi EnormousCormorant39 ,
is there a way to enqueue the dataset
add
command on a worker
Can you please elaborate a bit on this? Do you want to create some sort of trigger action to add files to a dataset?