Reputation
Badges 1
2 × Eureka!Hi Jax! Thanks for the feedback, we really appreciate it 😄
MLFlow is able to support a multitude of models from dev to deployment. Is ClearML and ClearML-Serving going to support as much as well?
Do you mean by this that you want to be able to seamlessly deploy models that were tracked using ClearML experiment manager with ClearML serving?
I believe in such scenarios, a custom engine would be required. I would like to know, how difficult is it to create a custom engine wit...
Thank you so much, sorry for the inconvenience and thank you for your patience! I've pushed it internally and we're looking for a patch 🙂
I'll update you once I have more!
Thank you so much! In the meantime, I check once more and the closest I could get was using report_single_value()
. It forces you to report each an every row though, but the comparison looks a little better this way. No color coding yet, but maybe it can already help you a little 🙂
Hi CurvedHedgehog15 , so my previous reply does assume you have reported a scalar for each individual FAR level. Then you can add individual levels as shown in the gif. But like you siad, that might actually cause you to loose your overview in the scalars tab.
So I don't think there's an immediate way to do this in ClearML right now, but would you mind opening an issue on github for it? It might be interesting to add it to the tool?
Nice! Well found and thanks for posting the solution!
May I ask out of curiosity, why mount X11? Are you planning to use a GUI app on the k8s cluster?
Great to hear! Then it comes down to waiting for the next hugging release!
Ah I see 😄 I have submitted a ClearML patch to Huggingface transformers: None
It is merged, but not in a release yet. Would you mind checking if it works if you install transformers from github? (aka the latest master version)
Now worries! Just so I understand fully though: you were already using the patch with success from my branch. Now that it has been merged into transformers main branch you installed it from there and that's when you started having issues with not saving models? Then installing transformers 4.21.3 fixes it (which should have the old clearml integration even before the patch?)
Ah I see. So then I would guess it is due to the remote machine (the clearml agent) not being able to properly access your clearml server
RoundMosquito25 it is true that the TaskScheduler
requires a task_id
, but that does not mean you have to run the pipeline every time 🙂
When setting up, you indeed need to run the pipeline once, to get it into the system. But from that point on, you should be able to just use the task_scheduler on the pipeline ID. The scheduler should automatically clone the pipeline and enqueue it. It will basically use the 1 existing pipeline as a "template" for subsequent runs.
Great! Please let me know if it works when adding this permission, we'll update the docs in a jiffy!
It should, but please check first. This is some code I quickly made for myself. It did make tests for it, but it would be nice to hear from someone else that it worked (as evidenced by the error above 😅 )
That looks like a bug, would you mind copy pasting this into a github issue? 🙂 AgitatedDove14 is there something else this could be?
Nice find! I'll pass it through to the relevant devs, we'll fix that right up 🙂 Is there any feedback you have on the functionality specifically? aka, would you use alias give what you know now or would you e.g. name it differently?
This update was just to modernize the example itself 🙂
Hi VictoriousPenguin97 ! I think you should be able to change it in the docker-compose file here: https://github.com/allegroai/clearml-server/blob/master/docker/docker-compose.yml
You can map the internal 8008 port to another port on your local machine. But beware to provide the different port number to any client that tries to connect (using clearml-init
)
And it seems you are fully correct, log_dataset_statistics
is not even called in preprocessing.py
🤔 That seems like a bad oversight from our end. If you add it yourself, does it work then? Add it on line 140, just after dataset_task.flush
the line itself is: self.log_dataset_statistics(dataset_task)
also change line 84 to values=self.metadata['label'],
You found it, so if you want you can open a PR and fix it so you get credit for it 🙂 If not, I can do it ...
Can you walk us through how you set up your jupyter instance? If we can recreate your error, we'll be able to help much faster. What's the command you're using to set it up, on which OS are you running it and so on 🙂 Also, have you checked your jupyter server is running on port 8888? Chances are something else is using 8888, so jupyter might be running on some other port like 8889 instead, so ClearML is trying to get a kernel from a completely different service.
For the record, this is a minimal reproducible example:
Local folder structure:
` ├── remove_folder
│ ├── batch_0
│ │ ├── file_0_0.txt
│ │ ├── file_0_1.txt
│ │ ├── file_0_2.txt
│ │ ├── file_0_3.txt
│ │ ├── file_0_4.txt
│ │ ├── file_0_5.txt
│ │ ├── file_0_6.txt
│ │ ├── file_0_7.txt
│ │ ├── file_0_8.txt
│ │ └── file_0_9.txt
│ └── batch_1
│ ├── file_1_0.txt
│ ├── file_1_1.txt
│ ├── file_1_2.txt
│ ├── file_1_3.txt
│ ├── fi...
Also, this might be a little stupid sorry, but your torch save command saves the model in the current folder, whereas you give clearml the 'model_folder/model' path instead. Could it be that the path is just incorrect?
Hey @<1539780305588588544:profile|ConvolutedLeopard95> , unfortunately this is not built-in into the YOLOv8 tracker. Would you mind opening an issue on the YOLOv8 github page and atting me? (I'm thepycoder on github)
I can then follow up the progress on it, because it makes sense to expose this parameter through the yaml.
That said, to help you right now, please change [this line](https://github.com/ultralytics/ultralytics/blob/fe61018975182f4d7645681b4ecc09266939dbfb/ultralytics/yolo/uti...
Could you use tags for that? In that case you can easily filter on which group you're interested in, or do you have a more impactful UI change in mind to implement groups? 🙂
With what error message did it fail? I would expect it to fail, because you finalized this version of your dataset by uploading it 🙂 You'll need a mutable copy of the dataset before you can remove files from it I think, or you could always remove the file on disk and create a new dataset with the uploaded one as a parent. In that way, clearml will keep track of what changed in between versions.
Hi ExasperatedCrocodile76 ,
You can try running the agent with these environment variables set to 1:
CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=1 CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=1
There's more env vars here: https://clear.ml/docs/latest/docs/clearml_agent/clearml_agent_env_var
Does that work for you?
I tried answering them as well, let us know what you end up choosing, we're always looking to make clearml better for everyone!
Does it help to also run docker login in the init bash script?
You should be able to access your AWS credentials from the environment (the agent will inject them based on your config)
Hey @<1539780272512307200:profile|GaudyPig83> !
Are you running a self-hosted server? Is this the only type of HTTP call that fails or does e.g. logging experiments also not work? A connection error usually means your docker containers can't reach each other.