Are you using the OSS version or the hosted one (app.clear.ml)? The ClearML enterprise offering has a built-in annotator. Please note that this was meant more for correcting annotations during the development process rather than mass annotating lots of images.

Hi Jax, I'm working on a few more examples of how to use clearml-data. should be released in a few weeks (with some other documentation updates). These however don't include the use case you're talking about. Would you care to elaborate more on that? Are you looking to store the code that created the data, in the execution part of the task that saves the data itself?

Did you try with function_kwargs?

In the installed pacakges I got:

• 'torch==1.14.0.dev20221205 # https://download.pytorch.org/whl/nightly/cu117/torch-1.14.0.dev20221205%2Bcu117-cp38-cp38-linux_x86_64.whl '
• torchtriton==2.0.0+0d7e753227
• 'torchvision==0.15.0.dev20221205 # https://download.pytorch.org/whl/nightly/cu117/torchvision-0.15.0.dev20221205%2Bcpu-cp38-cp38-linux_x86_64.whl '

Alrighty...let me check

Hi GrittyHawk31 , maybe I'm missing something, but what stops you from using Dataset.get() in the preprocessing script? Is there a limitation on it?

Hi EnviousStarfish54 If you want to not send info to the server, I suggest you to set an environment variable, this way as long as the machine has this envvar set it won't send to the server

Don't worry! It's all good 🙂

And as for clearml-data I would love to have more examples but not 100% sure what to focus on as using clearml-data is a bit...simple? In my, completely biased, eyes. I assume you're looking for workflow examples, and would love to get some inspiration 🙂

LOL Love this Thread and sorry I didn't answer earlier!

VivaciousPenguin66 EnviousStarfish54 I totally agree with you. We do have answers to "how do you do X or Y" but we don't have workflows really.

What would be a logical place to start? Would something like "training a Yolo V3 person detector on COCO dataset and how you enable continuous training (let's say adding PASCAL dataset afterwords) be something interesting?

The only problem is the friction between atomic and big picture. In...

If these indices tend to grow large, I think it would be cool if there was a flag that would periodically remove them. probably a lot of users aren't aware that these take up so much space

BTW! can you elaborate on the need for elevated privileges? What can't he do that you want him to?

https://clear.ml/docs/latest/docs/guides/set_offline/

Hi ZanyPig66 , do you want to have an agent per GPU? If so just add --gpus and specify the GPU number (0 or 1) that would be associated with this worker

Not sure I follow your suggestion 🙂
This is how my code compare looks, it's ok because I see the tags:

Hi JumpyPig73 , I reproduced the OOM issue but for me it's failing. Are you handling the error in python somehow so the script exists gracefully? otherwise it looks like a regular python exception...

Hi FierceHamster54 , this should be doable, actually we are adding this to other parts of the system so I'll make sure we update the autoscalers to be support this

That would simplify things 🙂

Hi FierceHamster54 can you try another instance type? I just tried with n1 and it works. We are looking to see if it's instance type related

Hi Jevgeni! September is always a slow month in Israel as it's holiday season 🙂 So progress is slower than usual and we didn't have an update!
Next week will be the next community talk and publishing of the next version of the roadmap, a separate message will follow

ReassuredTiger98 Nice digging and Ouch...that isn't fun. Let me see how quickly I can get eyes on this 🙂

Yeah, with pleasure 🙂

Yeah, makes sense. We actually thought that the "best practice" would be to launch the "actual code" (as opposed to the pipeline controller) from agents. But obviously we were wrong, or at least it doesn't cover the fact that a lot of the time, code is being written for debugging. So yeah, that's where we're at, ATM

That's how I see the scalar comparison, no idea which is the "good" and which is the "bad"

Yes, a preview is how you add the slice

Parent task in a dataset is basically an indication of lineage + sharing content.

Yeah I guess that's the culprit. I'm not sure clearml and wandb were planned to work together and we are probably interfering with each other. Can you try removing the wandb model save callback and try again with output_uri=True?

Also, I'd be happy to learn of your use-case that uses both clearml and wandb. Is it for eval purposes or anything else?

Yes! you can call this

Hi ScaryBluewhale66 , I believe the new server that's about the be released soon (this \ next week), we'll allow you to report a "single value metric". so if you want to report just a number per experiment you can, then you can also compare between runs.

Yes