This happened again 🤔
How many files does ClearML touch? :shocked_face_with_exploding_head:

Always great to find a bug! I'll make relevant SDK updates then.

Opened this - https://github.com/allegroai/clearml/issues/594

Thanks Alon. In the full/official documentation the clearml-data CLI is not mentioned anywhere, so perhaps it should be refreshed 😉

I think we're referring to different things here.
I won't be using the UI (and neither will my team).
But as mentioned, we've used DVC before and it adds a lot of junk metadata files to each GitHub PR (many dvc.yaml , dvc.lock and .gitignore files). We're trying to avoid that as much as possible, hence my question about GitHub pull...

and I don't think it's in the docs - we'll add that

Very welcome update, please use some highlighting for it too, it's so important for a complete understanding of how the remote execution works

packages an entire folder as zip

What if I have multiple files that are not in the same folder? (That is the current use-case)

It otherwise makes sense I think 🙂
Our workaround now for using a Dataset as we do, is to store the dataset ID as a configuration parameter, so it's always included too 😉

Any updates @<1523701087100473344:profile|SuccessfulKoala55> ? 🫣

TimelyPenguin76 that would have been nice but I'd like to upload files as artifacts (rather than parameters).

AgitatedDove14 I mean like a grouping in the artifact. If I add e.g. foo/bar to my artifact name, it will be uploaded as foo/bar .

Erm, one sec, let me find that thread...

Yes, exactly. I have not yet had a chance to try this out -- should it work?

It is

I'm not too worried about the dataset appearing (or not) in the Datasets tab. I would like it (the original task ) to to not disappear from the original project I assigned it to

No worries @<1537605940121964544:profile|EnthusiasticShrimp49> ! I made some headway by using Task.create , writing a temporary Python script, and using task.update in a similar way to how pipeline steps are created.

I'll try and create an MVC to reproduce the issue, though I may have strayed from your original suggestion because I need to be able to use classes and not just functions.

Sure, for example when reporting HTML files:

Still; anyone? 🥹 @<1523701070390366208:profile|CostlyOstrich36> @<1523701205467926528:profile|AgitatedDove14>

I'll try upgrading to 1.1.5, one moment

We have a read-only user with personal access token for these things, works seamlessly throughout and in our current on premise servers... So perhaps something missing in the autoscaler definitions?

SuccessfulKoala55 could this be related to the monkey patching for logging platform? We have our own logging handlers that we use in this case

Bump?

Woot! What about clearml-agent 1.2.0? 🙂

The results from searching in the "Add Experiment" view (can't resize column widths -> can't see project name ...)

We have a more complicated case but I'll work around it 😄

Follow up though - can configuration objects refer to one-another internally in ClearML?

Last but not least - can I cancel the offline zip creation if I'm not interested in it 🤔
EDIT: I see not, guess one has to patch ZipFile ...

https://clear.ml/docs/latest/docs/references/sdk/services_monitor

Then you can run this as a task, see also this example https://clear.ml/docs/latest/docs/guides/services/slack_alerts

After the task was initialized? 🤔

Thanks LazyFish41 , that explains it 👍

I think you're interested in the Monitor class:)

Using the PipelineController with add_function_step

Thanks David! I appreciate that, it would be very nice to have a consistent pattern in this!

I'm trying to decide if ClearML is a good use case for my team 🙂
Right now we're not looking for a complete overhaul into new tools, just some enhancements (specifically, model repository, data versioning).

We've been burnt by DVC and the likes before, so I'm trying to minimize the pain for my team before we set out to explore ClearML.