Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Clm Community, I Am Having An Issue With A Private Package Install When Using Clearml-Agent By Env (Not Docker). Things I Have Tried: Adding The Package Repository To "Extra_Index_Url" Adding The Conda Env To "Python_Binary" Listing The Libraries To I

Hi CLM community, I am having an issue with a private package install when using clearml-agent by env (not docker). Things I have tried:

Adding the package repository to "extra_index_url"
Adding the conda env to "python_binary"
Listing the libraries to install in "post_packages"
Setting "detect_with_pip_freeze: true"
Setting "force_analyze_entire_repo: true"

The env builds correctly in the agent but does not install the packages by any of the methods listed above. Could someone look at the log and suggest where I am going wrong?

  
  
Posted 2 years ago
Votes Newest

Answers 12


. Perhaps it is the imports at the start of the script only being assigned to the first task that is created?

Correct!

owever when I split the experiment task out completely it seems to have built the cloned task correctly.

Nice!!

  
  
Posted 2 years ago

FlatStarfish45

In the parent task, the libs appear installed.

What do you mean by "parent Task"? Is this the base task we are optimizing (i.e. the experiment / model we are optimizing) ?
Or is it the "Optimization Task" itself?

  
  
Posted 2 years ago

The training task (child)

this is the task the HPO is cloning?
Do you have the packages in this task?

  
  
Posted 2 years ago

the packages should be list in this section. if they arent, it wont try to install them

  
  
Posted 2 years ago

Thanks guys, working well now. Cheers for the assistance 🙂

  
  
Posted 2 years ago

Hey TimelyPenguin76 , I am actually trying to perform hyper param optimization. In the parent task, the libs appear installed. The training task (child), there are only a few libraries installed. Incidentally, it is in training task that fails. The parent task runs to completion initially. Perhaps I am doing something incorrect with the hyper param set up.

  
  
Posted 2 years ago

Yes, that is right, the cloned task. In the INSTALLED PACKAGES section it only shows 5 libs, which can't be right.

azure_storage_blob == 12.5.0
clearml == 1.1.4
pandas == 1.1.4
plotly == 4.14.2
pytz == 2021.3

  
  
Posted 2 years ago

The console output for the cloned task also does not appear to install any of the private packages. Seems like a minimal install, possibly from the conda base env.

  
  
Posted 2 years ago

Hi FlatStarfish45 ,

In the UI you can view the packages that the task will use (under EXECUTION tab), what do you get? Does it contains your packages in the INSTALLED PACKAGES section?

  
  
Posted 2 years ago

I think my mistake AgitatedDove14 was to run the experiment task in the same script.py as the hyper parameter task. I was creating and closing individual tasks which I assumed would work. It seems that it did not. However when I split the experiment task out completely it seems to have built the cloned task correctly. Perhaps it is the imports at the start of the script only being assigned to the first task that is created?

  
  
Posted 2 years ago

Hi FlatStarfish45 ,

The HPO task will control the HPO process, means it will clone the base task (the one we are optimizing), change the parameters, enqueue it and collect the results.
The base task is the task we want to optimize.
Each one of those two tasks, have different requirements.
You can look at the https://clear.ml/docs/latest/docs/guides/optimization/hyper-parameter-optimization/examples_hyperparam_opt#set-up-the-arguments for how set the base task in the HPO task.
can this be the case?

  
  
Posted 2 years ago

I suspect that the agent may be pulling in the requirements from the base env and not the Research env. How is that controlled and can I specify that explicitly?

  
  
Posted 2 years ago
693 Views
12 Answers
2 years ago
one year ago
Tags
Similar posts