What are you seeing in the Task that was cloned (i.e. the one the HPO created not the original training task)?
by that I mean, configuration section, do you have the Args there ? (seems like the pic you attached, but I just want to make sure)
Also in the train.py file, do you also have Task.init ?
@<1566959357147484160:profile|LazyCat94>
I found the issue, the import of clearml should be before anything else, this way it patch the Argparser before using it
from clearml import Task
Move it to the first line, everything should work 🙂
@<1566959357147484160:profile|LazyCat94> I think this issue happens because you're calling parse_args() before calling task.init
This is odd, can you send th full log of the failed Task and if possible the code?
Hi @<1566959357147484160:profile|LazyCat94>
So it seems the arg parser is detecting the configuration YAML
The first thing I would suggest is changing it to a relative path (so that when launched on remote machines it will find the YAML file)
Regardless how are you launching the HPO ? are you spinning a new agent ?
(as background, argparser arguments are injected in realtime by the agent or the HPO when running as subprocesses)
Thank you for giving me the advice.
To answer your question, here is my workflow.
First, I create the task by running the below code
python3 train.py config object_detection.yaml
And in the same docker image, I run the below command to executing an agent
clearml-agent daemon --queue default --forground
After that, use this task id created above, I run the code I shared clearml_hyper.py
So I think argparser arguments are injected in the task itself before HPO