Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Guys, Suppose I Have The Following Script:

Hi guys,

Suppose I have the following script:
` import numpy as np
import pandas as pd
from clearml import Task

Import required project dependencies.

from tf_funcs import some_keras_func

def some_numpy_func():
ones_tensor = np.ones((10, 10))
return ones_tensor

def some_pandas_func():
dates_range = pd.date_range("2021-01-01 00:00", "2021-01-31 18:00", freq="3H")
return dates_range

if name == "main":
task = Task.init(
project_name="project_example",
task_name="example task"
)

# Enqueue the task for execution. An agent will actually execute it later.
task.execute_remotely(queue_name="default")
print("Experiment id={} enqueue for execution".format(task.id))

np_var = some_numpy_func()
pd_var = some_pandas_func()

empty_model = some_keras_func() `

And this is the code of the tf_funcs module imported in previous script:
` import tensorflow as tf

def some_keras_func():
empty_model = tf.keras.Model()
return empty_model Because I'm using some_keras_func in the main script, I will need tensorflow installed. But somehow clearml is not able to catch the tensorflow dependency in the package collection stage. Is there any reason for this behavior? How can I make clearml to install tensorflow like it does with numpy and pandas ` ?

BTW the problem only happens when a clearml-agent executes the task, since it has to create the environment from scratch.

  
  
Posted 3 years ago
Votes Newest

Answers 11


Then ClearML should also detect the dependencies of the imported scripts? In this case shouldn't it detect that I am going to use tensorflow and install it as well? Because it is not actually recognizing it.

  
  
Posted 3 years ago

Thanks for helping. You and your team are doing a great job for the ML community.

  
  
Posted 3 years ago

So ClearML will scan all the repository code searching for package dependencies? Is that right?

  
  
Posted 3 years ago

My pleasure 💗

  
  
Posted 3 years ago

It's my own module (called 'tf_funcs.py')

  
  
Posted 3 years ago

Correct.
It starts with the initial script (entry point), if it is self contained (i.e. does not interact with the rest of the repo) it will only analyze it, otherwise it will analyze the entire repo code.

  
  
Posted 3 years ago

GiganticTurtle0 is it in the same repository ?
If it is it should have detected the fact that it needs to analyze the entire repository (not just the standalone script, and then discover tensorflow)

  
  
Posted 3 years ago

, it's just a custom module.

Is this your own module ? Is this a local folder we import from ?

  
  
Posted 3 years ago

That's right. There is no such package, it's just a custom module.
But that module uses tensorflow , and ClearML does not add it to the list of packages to install. The only solution available so far is to include it manually via Task.add_requirements ?

  
  
Posted 3 years ago

Hi GiganticTurtle0
ClearML will only list the directly imported packaged (not their requirements), meaning in your case it will only list "tf_funcs" (which you imported).
But I do not think there is a package named "tf_funcs" right ?

  
  
Posted 3 years ago

Hi GiganticTurtle0 ,

All the packages you are using should be under installed packages section in your task (in the UI). ClearML analyze and the full report should be under this section.

You can add any package you like with Task.add_requirements('tensorflow', '2.4.0') for tensorflow version 2.4.0 (or Task.add_requirements('tensorflow', '') for no limit).

If you dont want the package analyzer, you can configure in your ~/clearml.conf file: sdk.development.detect_with_pip_freeze: true and the installed packages section will be the same as you python environment pip freeze

  
  
Posted 3 years ago
1K Views
11 Answers
3 years ago
one year ago
Tags
Similar posts