As for DAG workflow, I saw someone who integrated Clearml with Luigi but couldn't find the post anywhere! 😄

EnviousStarfish54 VivaciousPenguin66 Another question if we're in a sharing mood 😉 Do you think a video \ audio session with one of our experts, where you present a problem you're having (let's say large size of artifacts) and he tries to help you, or even can give some example code \ code skeleton. Would something like that be of interest? Would you spend some time in such monthly session?

To add to Natan's answer, you can run on the services docker anything depending on the HW. We don't recommend training with it as the server's machine might get overloaded. What you can do is simple stuff like cleanup or any other routines 🙂

And as for clearml-data I would love to have more examples but not 100% sure what to focus on as using clearml-data is a bit...simple? In my, completely biased, eyes. I assume you're looking for workflow examples, and would love to get some inspiration 🙂

Hi SubstantialElk6 We moved to a new docs site here: https://clear.ml/docs/latest/docs/

As for the git, I'm no git expert but having your own git server is doable. I can't tell you what it means in terms of how does it work in your organization though as every one has their own limitations and rules. And as I said, you can use SVN but the connection between it and ClearML won't be as good as with git.

Hi SpicyCrab51 , Thanks for the warm words 😄 Happy you enjoy our product!
As for your needs, I suggest you explore our https://clear.ml/docs/latest/docs/hyperdatasets/overview , they indeed were made to solve issues similar to what you're facing!
You can see a talk we gave that cover the Hyperdatasets https://www.youtube.com/watch?v=CcL8NNZfHlY !
Note that it is an enterprise feature, and is not part of the open source.
Contact me if you need more info 🙂

Hi TenseOstrich47 Yup 🙂 You can check our scheduler module:
https://github.com/allegroai/clearml/tree/master/examples/scheduler
It supports time-events as well as triggers to external events

A new version should be available in a week and a half or so 😄

As for Kedro, first I'll say it's a great tool! If you use it and love it, keep on keeping on 🙂 I think these guys did a great job! While ClearML doesn't have all the features of Kedro (And obviously Kedro either, they are 2 different tools with 2 different goals), we do have the pipelining feature. The UI is still in the works and Kedro does look better! But if you use CleraML agent, you can probably build better automations. That said, if I had to "advise" I'd say start with ClearML pipe...

Yes definitely. As I said, if you like kedro continue using it. Both tools live happily side by side.

We can't officially confirm nor deny this but yes :sleuth_or_spy:

Thanks! 😄 As i've mentioned above, these features were chosen because of users feedback so keep it up and Thanks again!

The ClearML team appreciates bitching anywhere you feel like it (especially the memes section).
In the absence of some UI \ UX channel I suggest to just write here. I can promise you the people who's responsibility it is to fix \ improve the UI are roaming here and will see the request 😄

And some real pipeline (As real as our tests get 😄 )

KindGiraffe71 We're working on a new docs version, it'll be there as well!

Hmm, I actually think there isn't a way. Once you'll have more projects in the system the project will be pushed down and you won't see it in the front page. Is there any specific reason why you want it removed?

JitteryCoyote63 Fair point 😅 , I'll be lying to say we haven't been slow on documenting new features 🙂 That being said, since what you're looking for seems REALLY straightforward (at least to people who know how it works internally 😛 ) we can probably do something about it rather quickly 🙂

Yeah I get what you're saying, but when developing ClearML we did not view it like that. when we run it locally or debug it, we thought of it more "This is running just on my local computer without an agent to make sure everything works before I use an agent to run the parts". Sorry that it confuses you 🙂

FiercePenguin76 JitteryCoyote63 are you guys using clearml-serving triton at the moment? If not, would be happy to hear what are the barriers for usage 🙂

As for your question, yes, our effort was diverted into other avenues and not a lot of public progress has been made.
That said, what is your plan for integration of the tools? automatically promote models to be served from within clearml?

BTW! can you elaborate on the need for elevated privileges? What can't he do that you want him to?

If these indices tend to grow large, I think it would be cool if there was a flag that would periodically remove them. probably a lot of users aren't aware that these take up so much space

Hi Jax, I'm working on a few more examples of how to use clearml-data. should be released in a few weeks (with some other documentation updates). These however don't include the use case you're talking about. Would you care to elaborate more on that? Are you looking to store the code that created the data, in the execution part of the task that saves the data itself?

Hi OutrageousSheep60 , The plan is to release this week \ early next week a version that solves this.

CLEARML_NO_DEFAULT_SERVER=1

Hi! check this page out

Hey There Jamie! I'm Erez from the ClearML team and I'd be happy to touch on some points that you mentioned.
First and foremost, I agree with the first answer that was given to you on reddit. There's no "right" tool. most tools are right for the right people and if a tool is too much of a burden, then maybe it isn't right!
Second, I have to say the use of SVN is a "bit" of a hassle. the MLOps space HEAVILY leans towards git. We interface with git and so does every other tool I know of. That ...

Hi MelancholyElk85 , documentation is a soft spot for us, we're trying to be better but we aren't always 🙂 https://github.com/allegroai/clearml/blob/master/examples/pipeline/pipeline_from_decorator.py this is an example, if you check the pipeline folder you'll see others (like pipeline_from_function.py

I think that's a hydra issue 🙂 I was able to reproduce this locally. I'll see what can be done