Yes, from archived experiments

Sure! That definitely makes sense. Where can I specify callbacks in the PipelineDecorator API?

It worked! Thank you very much

But this path actually does not exist in my system, so how should I fix that?

Hi Martin,

Actually Task.add_requirements behaves as I expect, since that part of the code is in the preprocessing script and for that task it does install all the specified packages. So, my question could be rephrased as the following: when working with PipelineController , is there any way to avoid creating a new development environment for each step of the pipeline?

According to the https://clear.ml/docs/latest/docs/clearml_agent provided in the official ClearML documentatio...

I currently deal with that by skipping the first 5 characters of the path, i. e. the 'file:' part. But I'm sure there is a cleaner way to proceed.

Done!

Great, thank you very much for the info! I just spotted the get_logger classmethod. As for the initial question, that's just the behavior I expected!

Makes sense, thanks!

Exactly, at first I was trying to call a component from another component, but it didn't work. Then I thought it would be more natural to do this using a pipeline, but it didn't recognize the user_config_creation function despite I imported it as I would do under PipelineDecorator.component . I really like the idea of enabling an argument to specify the components you are going to use in the pipeline so they are in the step's context! I will be eagerly waiting for that feature :D

By adding the slash I have been able to see that indeed the dataset is stored in output_url . However, when calling finalize , I get the same error. And yes, I have installed the version corresponding to the last commit :/

My idea is to take advantage of the capability of getting parameters connected to a task from another task to read the path where the artifacts are stored locally, so I don't have to define it again in each script corresponding to a different task.

Hi ExasperatedCrab78 ,
Sure! Sorry for the delay. I'm using Chrome Version 98.0.4758.102 (Official Build) (64-bit)

Having the ability to clone and modify the same task over and over again, in principle I would no longer need the multi_instance support feature from PipelineDecorator.pipeline. Is this correct, or are they different things?

That' s right, I don't know why I was trying to make it so complicated 😅

Thanks for helping. You and your team are doing a great job for the ML community.

Mmm that's weird. Because I can see the type hints in the function's arguments of the automatically generated script. So, maybe I'm doing something wrong or it's a bug, since they have been passed to the created step (I'm using clearml version 1.1.2 and clearml-agent version 1.1.0).

Ok, so it doesn't follow the exact same rules as Task.init ? I was afraid all the logs and outputs of a hyperparameter optimization task would be deleted just because no artifacts were created.

Hi AnxiousSeal95 !
That's it. My idea is that artifacts can be linked to the model. Typically these artifacts are often links to serialized objects (such as datasets or scalers). They are usually directories or temporary files in mount units that I want to be loaded as artifacts of the task, removed (as they are temporary) and later I can get a new local path via task.artifacts["scalers"].get_local_copy() . I think this way the model's dependence on the task that created it could be re...

Beautiful. I have tested the new functionality with several use cases and it works just as I expected. Excellent work, as usual :D

Thanks for the background. I now have a big picture of the process ClearML goes through. It was helpful in clarifying some of the questions that I didn't know how to ask properly. So, the idea is that a base task is already stored on the ClearML server for later use in a production environment. This is because such a task will always be created during the model development process.

Going back to my initial question, as far as I understood, if the environment caching option is ena...

Or maybe you could bundle some parameters that belongs to PipelineDecorator.component into high-level configuration variable (something like PipelineDecorator.global_config (?))

I'm getting a NameError because 'Optional' type hint is not defined in the global scope

Well the 'state.json' file is actually removed after the exception is raised

I'm using the last commit. I'm just fitting a scikit-learn MinMaxScaler object to a dataset of type tf.data.Dataset inside a function (which represents the model training step) decorated with PipelineDecorator.component . The function does not even return the scaler object as an artifact. However, the scaler object is logged as an artifact of the task, as shown in the image below.

Sure, I will post a mock example in a while

But how could I know whether an agent is up or not? Is it from the CLI or SDK?

Brilliant, that worked like a charm!

Glad to hear that! Thanks!

Yeah, but after doing that a message pops up showing a list of artifacts from the task that could not be deleted