Care to share a link? 😄

Now in step2, I add a pre_execute_callback

Yeah I totally get what you're saying. Basically you want the same code to run locally or remotely, and something external would control whether it runs locally or enqueued to a worker. Am I right?

The new welcome screen to pipelines and our fancy new icon on the left sidebar 😄

JitteryParrot8 in the new SDK we'll have dataset.add_description() which will do the same as KindChimpanzee37 provided but with a nicer interface 😄

VivaciousPenguin66 This is very true! We are trying to explain the benefits of this method. Some people like it and some people like the flexibility. We do have our philosophy in mind when we create "best practices" and obviously features to ClearML but ultimately people should do what makes them the most productive!
If we are getting philosophical, I think it's the state of the industry and as it progresses, these standard methods would become more prominent.
also, to add to what you wrote,...

I'm also curious about using external orchestrators as opposed to the ClearML's built-in ones

` pipe = PipelineController(
project='examples',
name='Pipeline demo',
version='1.1',
add_pipeline_tags=False,
)

set the default execution queue to be used (per step we can override the execution)

pipe.set_default_execution_queue('default')

add pipeline components

pipe.add_parameter(
name='url',
description='url to pickle file',
default=' '
)
pipe.add_function_step(
name='step_one',
function=step_one,
function_kwargs=dict(pickle_data_url='${pi...

got it 🙂

That's true 🙂 Our SDK is a python based and your code needs to be python code for us to integrate with

Can you check again? It works for me. If you're still not able to reach it, can you send an image of the error you're getting?

To add to Natan's answer, you can run on the services docker anything depending on the HW. We don't recommend training with it as the server's machine might get overloaded. What you can do is simple stuff like cleanup or any other routines 🙂

We plan to expand our model object and have searchable key:value dicts associated with it, and maybe metric graphs. What you ask is for us to also add artifacts to it. These artifacts are going to be datasets (or something else?)? If I understand correctly, a key:value would be enough as you're not saving data, but only a links to where the data is. Am I right?

very weird

MelancholyElk85 , yes nesting pipelines is possible. as for flatting it afterwords, maybe AgitatedDove14 knows? I'm pretty sure it can't be done though

Are you talking about consecutive pipeline steps? Or parallel?

Yes definitely. As I said, if you like kedro continue using it. Both tools live happily side by side.

Hi IcyJellyfish61 , while spinning up and down EKS is not supported (albeit very cool 😄 ) we have an autoscaler in the applications section that does exactly what you need, spin up and down EC2 instances according to demand 🙂
If you're using http://app.clear.ml as you server, you can find it at https://app.clear.ml/applications .
Unfortunately, it is unavailable for the opensource server and only to paid tiers.

Just randomly check if there's a new version...every day 😉

Hi TenseOstrich47 What you can do is report the metric to clearml, then use the Taskscheduler to listen on a specific project. If a task in this project reports a metric below \ above a certain TH (Or I think if it's the highest \ lowest as well) you can trigger an event (Task \ function). That's how you do it with the Taskscheduler object

MelancholyElk85 I'll talk to the guy in charge and try! You can find main points here 🙂 https://docs.google.com/document/d/1QlPiDO2EzDq_HRvuVhYwPh5F_IsjgmmRTlPNdHik10k/edit

Hi @<1529271098653282304:profile|WorriedRabbit94> My name is Erez and I'm from ClearML's product team. Thanks for highlighting this! We are aware of the marketplace, and it is in our roadmap to add it there, but no firm date yet.
If you have any comments on adding the plugin to pycharm, or any method that makes it difficult to use, I'm very happy to hear feedback!

This is what I'm seeing the > is the title - series relation. I'm not 100% clear why the iteration is a problem, could you elaborate?

You mean add some list on top of all experiments with tags and their ID?

https://clear.ml/docs/latest/docs/clearml_data#datasetfinalize

Yes, any need for anything else?

Cool!

KindBlackbird59 I think you are looking for something like a "git repository" (which, IIRC, is how dvc sees "projects" or models),
that gives you a clear lineage (This model came first, then I got this model with this code and this data).
The way ClearML works is slightly different, and each "repo" is shown as a project which is flat. The way we envision users marking things like "model V2" is by adding tags.
The reason behind this design is that git, while has clear lineage, is harder to wo...

To organize work, we designate a special task type for datasets (so it's easy to search and browse through them) as well as tags that help you get finer granularity search capabilities.

WickedElephant66 while there is no way to tell an Agent which CPU cores it's associated with, you can work in docker mode and do the separation there (IE, each docker container sees only X cores). I know other clearml users use it that way