If I try to connect a dictionary of type dict[str, list] with task.connect , when retrieving this dictionary with task.get_parameter I get another dictionary dict[str, str] . Therefore, I see the same behavior using task.connect :/

Perfect, that's exactly what I was looking for 🙂 Thanks!

I mean what should I write in a script to import the APIClient? (sorry if I'm not explaining myself properly 😅 )

But how could I know whether an agent is up or not? Is it from the CLI or SDK?

My guess is to manually read and parse the string that clearml-agent list returns, but I'm pretty sure there's a cleaner way to do it, isn't there?

But what is the name of that API library in order to have access to those commands from Python SDK?

Where can I find this documentation?

Hi AgitatedDove14 Yes, I think so. When I have more time next week I will take a closer look at it and elaborate an example.

Hi ExasperatedCrab78 ,
Sure! Sorry for the delay. I'm using Chrome Version 98.0.4758.102 (Official Build) (64-bit)

Oddly enough I didn't run into this problem today 🤔 If it happens to me again, I'll return to this thread 🙂

By adding the slash I have been able to see that indeed the dataset is stored in output_url . However, when calling finalize , I get the same error. And yes, I have installed the version corresponding to the last commit :/

AgitatedDove14 In the 'status.json' file I could see the 'is_dirty' flag is set to True

Well the 'state.json' file is actually removed after the exception is raised

I can't figure out what might be going on

Thanks, I'd appreciate it if you let me know when it's fixed :D

AgitatedDove14 Oops, something still seems to be wrong. When trying to retrieve the dataset using get_local_copy() I get the following error:
` Traceback (most recent call last):
File "/home/user/myproject/lab.py", line 27, in <module>
print(dataset.get_local_copy())
File "/home/user/.conda/envs/myenv/lib/python3.9/site-packages/clearml/datasets/dataset.py", line 554, in get_local_copy
target_folder = self._merge_datasets(
File "/home/user/.conda/envs/myenv/lib/python3.9/site-p...

Well I tried several things but none of them have worked. I'm a bit lost

Yes, I'm working with the latest commit. Anyway, I have tried to run dataset.get_local_copy() on another machine and it works. I have no idea why this happens. However, on the new machine get_local_copy() does not return the path I expect. If I have this code:
dataset.upload( output_url="/home/user/server_local_storage/mock_storage" )I would expect the dataset to be stored under the path specified in output_url . But what I get with get_local_copy() is the follo...

Indeed it does! But what still puzzles me so badly is why I get below path when running dataset.get_local_copy() on one of the machines of my cluster:
/home/user/.clearml/cache/storage_manager/datasets/.lock.000.ds_61ff8d4335dd4b74bd78c3576fa44131.clearml
Why is it pointing to a .lock file?

Sure! That would be great too

Mmm what would be the implications of not being part of the DAG? I mean, how could that step be launched if it is not part of the execution graph?

Hi AgitatedDove14 ,
Any updates on the new ClearML release that fixes the bugs we mentioned in this thread? :)

Mmm well, I can think of a pipeline that could save its state in the instant before the error occurred. So that using some crontab/scheduler the pipeline could be resumed at the point where it was stopped in the case of not having been completed. Is there any functionality like this? Something like PipelineDecorator/PipelineController.resume_from(state_filepath) ?

is there any git redundancy on your network ? maybe you could configure a fallback server ?

I will ask this to the IT team

Thanks! I will try it

Exactly, at first I was trying to call a component from another component, but it didn't work. Then I thought it would be more natural to do this using a pipeline, but it didn't recognize the user_config_creation function despite I imported it as I would do under PipelineDecorator.component . I really like the idea of enabling an argument to specify the components you are going to use in the pipeline so they are in the step's context! I will be eagerly waiting for that feature :D

I mean to use a function decorated with PipelineDecorator.pipeline inside another pipeline decorated in the same way.
In the traceback attached below you can see that I am trying to use a component named user_config_creation inside the create_user_configs sub-pipeline. I have imported user_config_creation inside create_user_configs but a KeyError is raised (however I assume the function has been imported correctly because no ImportError or ` ModuleNo...

I don't know if you remember the need I had some time ago to launch the same pipeline through configuration. I've been thinking about it and I think PipelineController fits my needs better than PipelineDecorator in that respect.

Having the ability to clone and modify the same task over and over again, in principle I would no longer need the multi_instance support feature from PipelineDecorator.pipeline. Is this correct, or are they different things?

Hi AgitatedDove14 , so isn't it ClearML best practice to create a draft pipeline to have the task on the server so that it can be cloned, modified and executed at any time?