Badges 1183 × Eureka!
If I try to connect a dictionary of type
dict[str, list] with
task.connect , when retrieving this dictionary with
task.get_parameter I get another dictionary
dict[str, str] . Therefore, I see the same behavior using
Perfect, that's exactly what I was looking for 🙂 Thanks!
I mean what should I write in a script to import the APIClient? (sorry if I'm not explaining myself properly 😅 )
But how could I know whether an agent is up or not? Is it from the CLI or SDK?
My guess is to manually read and parse the string that
clearml-agent list returns, but I'm pretty sure there's a cleaner way to do it, isn't there?
But what is the name of that API library in order to have access to those commands from Python SDK?
Where can I find this documentation?
Hi AgitatedDove14 Yes, I think so. When I have more time next week I will take a closer look at it and elaborate an example.
Hi ExasperatedCrab78 ,
Sure! Sorry for the delay. I'm using Chrome Version 98.0.4758.102 (Official Build) (64-bit)
Oddly enough I didn't run into this problem today 🤔 If it happens to me again, I'll return to this thread 🙂
By adding the slash I have been able to see that indeed the dataset is stored in
output_url . However, when calling
finalize , I get the same error. And yes, I have installed the version corresponding to the last commit :/
AgitatedDove14 In the 'status.json' file I could see the 'is_dirty' flag is set to True
Well the 'state.json' file is actually removed after the exception is raised
I can't figure out what might be going on
Thanks, I'd appreciate it if you let me know when it's fixed :D
AgitatedDove14 Oops, something still seems to be wrong. When trying to retrieve the dataset using get_local_copy() I get the following error:
` Traceback (most recent call last):
File "/home/user/myproject/lab.py", line 27, in <module>
File "/home/user/.conda/envs/myenv/lib/python3.9/site-packages/clearml/datasets/dataset.py", line 554, in get_local_copy
target_folder = self._merge_datasets(
Well I tried several things but none of them have worked. I'm a bit lost
Yes, I'm working with the latest commit. Anyway, I have tried to run
dataset.get_local_copy() on another machine and it works. I have no idea why this happens. However, on the new machine
get_local_copy() does not return the path I expect. If I have this code:
dataset.upload( output_url="/home/user/server_local_storage/mock_storage" )I would expect the dataset to be stored under the path specified in
output_url . But what I get with
get_local_copy() is the follo...
Indeed it does! But what still puzzles me so badly is why I get below path when running
dataset.get_local_copy() on one of the machines of my cluster:
Why is it pointing to a .lock file?
Mmm what would be the implications of not being part of the DAG? I mean, how could that step be launched if it is not part of the execution graph?
Hi AgitatedDove14 ,
Any updates on the new ClearML release that fixes the bugs we mentioned in this thread? :)
Mmm well, I can think of a pipeline that could save its state in the instant before the error occurred. So that using some crontab/scheduler the pipeline could be resumed at the point where it was stopped in the case of not having been completed. Is there any functionality like this? Something like
is there any git redundancy on your network ? maybe you could configure a fallback server ?
I will ask this to the IT team
Exactly, at first I was trying to call a component from another component, but it didn't work. Then I thought it would be more natural to do this using a pipeline, but it didn't recognize the
user_config_creation function despite I imported it as I would do under
PipelineDecorator.component . I really like the idea of enabling an argument to specify the components you are going to use in the pipeline so they are in the step's context! I will be eagerly waiting for that feature :D
I mean to use a function decorated with
PipelineDecorator.pipeline inside another pipeline decorated in the same way.
In the traceback attached below you can see that I am trying to use a component named
user_config_creation inside the
create_user_configs sub-pipeline. I have imported
create_user_configs but a
KeyError is raised (however I assume the function has been imported correctly because no
ImportError or ` ModuleNo...
I don't know if you remember the need I had some time ago to launch the same pipeline through configuration. I've been thinking about it and I think PipelineController fits my needs better than PipelineDecorator in that respect.
Having the ability to clone and modify the same task over and over again, in principle I would no longer need the multi_instance support feature from PipelineDecorator.pipeline. Is this correct, or are they different things?
Hi AgitatedDove14 , so isn't it ClearML best practice to create a draft pipeline to have the task on the server so that it can be cloned, modified and executed at any time?