Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Base_Template_Keras_Simply.Py

https://github.com/allegroai/clearml/blob/master/examples/optimization/hyper-parameter-optimization/base_template_keras_simple.py
Hi, I am running this code above using clearml-task command. I modified to requirements.txt to just include clearml and excluded tensorflow libraries. However, for the base docker image , I used this. tensorflow/tensorflow:latest-devel-gpu. So this definitely contains tensorflow. However, I see this error when running the clearml-agent in k8s.
Seems like the docker image wasnt read in.
File "base_template_keras_simple.py", line 15, in <module> import tensorflow as tf # noqa: F401 File "/root/.clearml/venvs-builds/3.6/lib/python3.6/site-packages/clearml/binding/import_bind.py", line 59, in __patched_import3 level=level) ModuleNotFoundError: No module named 'tensorflow'

  
  
Posted 3 years ago
Votes Newest

Answers 12


DeliciousBluewhale87 could you send the full log of the Task?

  
  
Posted 3 years ago

As I suspected, from your log:
agent.package_manager.system_site_packages = falseWhich is exactly the problem of the missing tensorflow (basically it creates a new venv inside the docker, but without the flag On, it does not inherit the docker preinstalled packages)
This flag should have been true.
Could it be that the clearml.conf you are providing for the glue includes this value?
(basically you should only have the sections that are either credentials or missing from the default, there is no need to pass full conf file)

  
  
Posted 3 years ago

Just figured out..
Seems like the docker image below, didnt have tensorflow package.. 😮
tensorflow/tensorflow:latest-devel-gpuI shld have checked prior... My Bad..
Thanks for the help

  
  
Posted 3 years ago

DeliciousBluewhale87 this is exactly how it works,
The glue puts a k8s job with the requested docker image (the one on the Task), the job itself (k8s job) starts the agent inside the requested docker, then the agent inside the docker will install all the required packages.

  
  
Posted 3 years ago

Hi AgitatedDove14 , Just updated that flag, but the problem continues..
` agent.package_manager.system_site_packages = true
.....
Environment setup completed successfully

Starting Task Execution:

ClearML results page: files_server:
Traceback (most recent call last):
File "base_template_keras_simple.py", line 15, in <module>
import tensorflow as tf # noqa: F401
File "/root/.clearml/venvs-builds/3.6/lib/python3.6/site-packages/clearml/binding/import_bind.py", line 59, in __patched_import3
level=level)
ModuleNotFoundError: No module named 'tensorflow' `

  
  
Posted 3 years ago

DeliciousBluewhale87 great we have progress, this look slike it is inheriting from the system packages:
For example you can see in the log,
Requirement already satisfied: future>=0.16.0 in /usr/local/lib/python3.6/dist-packagesNow the question is which docker it is running, because as you can see at the bottom of the log, tensorflow is not listed as installed, but other packages installed inside the docker are listed.
wdyt?

  
  
Posted 3 years ago

Assuming from previous threads this is run on K8s , I think a configuration is missing, use system packages:
https://github.com/allegroai/clearml-agent/blob/cb6bdece39751eaef975287609b8bab603f116e5/docs/clearml.conf#L57

  
  
Posted 3 years ago

Essentially, while running on k8s_glue, I want to pull the docker image/container, then pip install the additional requirements.txt into them...

  
  
Posted 3 years ago

No worries 🙂

  
  
Posted 3 years ago

DeliciousBluewhale87 could you send the new log?

  
  
Posted 3 years ago

AgitatedDove14 Full Log as requested.

  
  
Posted 3 years ago
743 Views
12 Answers
3 years ago
one year ago
Tags