Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, When Running A Training Script From Pycharm, It Seems That Clearml Logs Only Those Packages That Are Explicitly Imported By My .Py Files; It Seems To Not Take The Pacakges That Are In The Requirements.Txt My Training Uses Keras

hi,
When running a training script from pycharm, it seems that clearml logs only those packages that are explicitly imported by my .py files; it seems to not take the pacakges that are in the requirements.txt

My training uses keras ImageDataGenerator , and to this end I added scipy in requirements.txt (but I do not use it directly in imports).

The result is that the agent doesn’t install scipy when I clone tasks, and the tasks fail.

Is there a way to force the the task.init to include all packages from the requirements.txt? Is there a way to force the agent to use the requirements.txt from the repo?Seems like there used to be a way ( agenrt.package_manager.force_repo_requirements_txt: true ) but now I can’t find evidence of it in the docs

I am using clearml 1.0.5
Thanks!

  
  
Posted 2 years ago
Votes Newest

Answers 4


RoughTiger69 , you can also use Task.add_requirements  for a specific package through the script

Example: Task.add_requirements('tensorflow', '2.4.0') Example: Task.add_requirements('tensorflow', '>=2.4') Example: Task.add_requirements('tensorflow') -> use the installed tensorflow version Example: Task.add_requirements('tensorflow', '') -> no version limit

  
  
Posted 2 years ago

Hi RoughTiger69

seems to not take the pacakges that are in the requirements.txt

The reason for not taking the entire python packages, it will most likely break when trying to run inside the agent.
The directly imported packages aill essentially pull their required packages, and thus create a stable env on the remote machine. The agent then will store the Entire env, as it assumes it will be able to fully replicate it the next time it runs.
If the "Installed Packages" section is empty (notice on hover you have a button to clear it), the agent will revert to using the requirement.txt inside the repo.
Make sense?

  
  
Posted 2 years ago

not the most intuitive approach but I’ll give it a go

  
  
Posted 2 years ago

RoughTiger69 Hi!

Regarding your questions:
You can use the following: Task.force_requirements_env_freeze(requirements_file='repo/some_folder/requirements.txt') before your task=Task.init(...) You can configure sdk.development.detect_with_pip_freeze=true in your ~/clearml.conf file for full env detection from the environment you're running from

  
  
Posted 2 years ago
563 Views
4 Answers
2 years ago
one year ago
Tags