That's right. There is no such package, it's just a custom module.
But that module uses tensorflow , and ClearML does not add it to the list of packages to install. The only solution available so far is to include it manually via Task.add_requirements ?

Perfect, that's exactly what I was looking for 🙂 Thanks!

If I try to connect a dictionary of type dict[str, list] with task.connect , when retrieving this dictionary with task.get_parameter I get another dictionary dict[str, str] . Therefore, I see the same behavior using task.connect :/

Sure, just by changing a few things from the previous example:
` from clearml import Task

task = Task.init()
task.connect({"metrics": ["nmae", "bias", "r2"]})

metrics_names = task.get_parameter("General/metrics")

print(metrics_names)
print(type(metrics_names)) `

Hi AgitatedDove14 ,
Any updates on the new ClearML release that fixes the bugs we mentioned in this thread? :)

Anyway, is there any way to retrieve the information stored in the RESULTS tab of ClearML Web UI?

I have also tried with type hints and it still broadcasts to string. Very weird...

BTW I would really appreciate it if you let me know when you get it fixed 🙏

Oh, I see. I guess somehow I can retrieve that information via Task.logger , since it is stored in JSON format? Thanks!

I think it could be a convenient approach. The new parameter abort_on_failed_steps could be a list containing the name of the steps for which the pipeline will stop its execution if any of them fail (so that we can ignore other steps that are not crucial to continue the pipeline execution)

But maybe another solution would be to pass the configuration files paths as function arguments, then read and parse them inside the pipeline

AgitatedDove14 I ended up with two pipelines being executed until they completed the workflow but duplicating each of their steps. You can check it here:
https://clearml.slack.com/files/U02A5DGPMPU/F02SR3G9RDK/image.png

Hey AgitatedDove14 ! Any news on this? 🙂

AgitatedDove14 I have the strong feeling it must be an agent issue, because when I place PipelineDecorator.run_locally() before calling the pipeline, everything works perfectly. See:

Well, this is just a mock example 🙂 . In the real application I'm working on there will be more than one configuration file (in principle one for the data and one for the DL model). Regarding the fix, I am not in a hurry at the moment. I'll happily wait for tomorrow (or the day after) when the commit is pushed!

Great! Thank you 👍

Hi CostlyOstrich36 AgitatedDove14
Oh no, I am not trying to say that I am using each agent to run a single task. I have several agents listening to a number of queues so that they are busy most of the time. As we talked about, it is not possible to run multiple pipelines ( PipelineDecorator.pipeline ) simultaneously in a single process. That's why I had been testing locally launching pipelines in different subprocesses and this way I have managed to run several pipelines concurrently....

AgitatedDove14 After checking, I discovered that apparently it doesn't matter if each pipeline is executed by a different worker, the error persists. Honestly this has me puzzled. I'm really looking forward to getting this functionality right because it's an aspect that would make ClearML shine even more.

AgitatedDove14 So did you get the same results without step duplication?

AgitatedDove14 Exactly, I've run into the same problem

Great AgitatedDove14 , I tested it on the mock example and it worked just as I expected 🎉

That's right! run_locally() does just what I was expecting

Yes, although I use both terms interchangeably. The information will actually be contained in JSON files.

Hi AgitatedDove14 , it's nice to know you've already pinpointed the problem! I think the solution you propose is a good one, but does that mean I have to unpack all the dictionary values as parameters of the pipeline function? Wouldn't that make the function too "dirty"? Or do you mean you will soon push a commit that will allow me to keep passing a dictionary and ClearML automatically flatten it?

AgitatedDove14 The pipelines are executed by the agents that are listening to the queue given by pipeline_execution_queue="controllers"

But how can I reference that exact daemon execution? I tried with the ID but it fails:

clearml-agent daemon AGENT_ID --stop

Sure, it would be very intuitive if the command to stop an agent would be as easy as:
clearml-agent daemon --stop AGENT_PID

Hi AgitatedDove14 , great, glad it was fixed quickly!

By the way, before releasing version 1.1.3 you might want to take a look at this mock example. I'm trying to run the same pipeline (with different configurations) in a single for loop, as you can see below:
` from clearml import Task
from clearml.automation.controller import PipelineDecorator

@PipelineDecorator.component(return_values=["msg"], execution_queue="myqueue1")
def step_1(msg: str):
msg += "\nI've survived step 1!"
re...

Hi AgitatedDove14
Using task.get_parameters I get the parameters in a dictionary, but the values are still of type 'string'. The quickest solution I can think of is parsing with eval built-in. wdyt?

For sure! Excluding some parts related to preprocessing, this is the code I would like to parallelize with dask.distributed.Client .

` from typing import Any, Dict, List, Tuple, Union
from pathlib import Path

import xarray as xr
from clearml import Task
from dask.distributed import Client, LocalCluster

def start_dask_client(
n_workers: int = None, threads_per_worker: int = None, memory_limit: str = "2Gb"
) -> Client:
cluster = LocalCluster(
n_workers=n_workers,
...