I think my mistake AgitatedDove14 was to run the experiment task in the same script.py as the hyper parameter task. I was creating and closing individual tasks which I assumed would work. It seems that it did not. However when I split the experiment task out completely it seems to have built the cloned task correctly. Perhaps it is the imports at the start of the script only being assigned to the first task that is created?
Hey TimelyPenguin76 , I am actually trying to perform hyper param optimization. In the parent task, the libs appear installed. The training task (child), there are only a few libraries installed. Incidentally, it is in training task that fails. The parent task runs to completion initially. Perhaps I am doing something incorrect with the hyper param set up.
I suspect that the agent may be pulling in the requirements from the base env and not the Research env. How is that controlled and can I specify that explicitly?
Yes, that is right, the cloned task. In the INSTALLED PACKAGES section it only shows 5 libs, which can't be right.
azure_storage_blob == 12.5.0
clearml == 1.1.4
pandas == 1.1.4
plotly == 4.14.2
pytz == 2021.3
The training task (child)
this is the task the HPO is cloning?
Do you have the packages in this task?
. Perhaps it is the imports at the start of the script only being assigned to the first task that is created?
Correct!
owever when I split the experiment task out completely it seems to have built the cloned task correctly.
Nice!!
the packages should be list in this section. if they arent, it wont try to install them
FlatStarfish45
In the parent task, the libs appear installed.
What do you mean by "parent Task"? Is this the base task we are optimizing (i.e. the experiment / model we are optimizing) ?
Or is it the "Optimization Task" itself?
Hi FlatStarfish45 ,
The HPO task will control the HPO process, means it will clone the base task (the one we are optimizing), change the parameters, enqueue it and collect the results.
The base task is the task we want to optimize.
Each one of those two tasks, have different requirements.
You can look at the https://clear.ml/docs/latest/docs/guides/optimization/hyper-parameter-optimization/examples_hyperparam_opt#set-up-the-arguments for how set the base task in the HPO task.
can this be the case?
Hi FlatStarfish45 ,
In the UI you can view the packages that the task will use (under EXECUTION tab), what do you get? Does it contains your packages in the INSTALLED PACKAGES section?
Thanks guys, working well now. Cheers for the assistance 🙂
The console output for the cloned task also does not appear to install any of the private packages. Seems like a minimal install, possibly from the conda base env.