Answered

Hi Guys, Suppose I Have The Following Script:

Hi guys,

Suppose I have the following script:
` import numpy as np
import pandas as pd
from clearml import Task

Import required project dependencies.

from tf_funcs import some_keras_func

def some_numpy_func():
ones_tensor = np.ones((10, 10))
return ones_tensor

def some_pandas_func():
dates_range = pd.date_range("2021-01-01 00:00", "2021-01-31 18:00", freq="3H")
return dates_range

if name == "main":
task = Task.init(
project_name="project_example",
task_name="example task"
)

# Enqueue the task for execution. An agent will actually execute it later.
task.execute_remotely(queue_name="default")
print("Experiment id={} enqueue for execution".format(task.id))

np_var = some_numpy_func()
pd_var = some_pandas_func()

empty_model = some_keras_func() `

And this is the code of the tf_funcs module imported in previous script:
` import tensorflow as tf

def some_keras_func():
empty_model = tf.keras.Model()
return empty_model Because I'm using some_keras_func in the main script, I will need tensorflow installed. But somehow clearml is not able to catch the tensorflow dependency in the package collection stage. Is there any reason for this behavior? How can I make clearml to install tensorflow like it does with numpy and pandas ` ?

BTW the problem only happens when a clearml-agent executes the task, since it has to create the environment from scratch.

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					GiganticTurtle0
				
					0
					 × 1

Votes Newest

Answers 11

So ClearML will scan all the repository code searching for package dependencies? Is that right?

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					GiganticTurtle0
				
					0
					 × 1

GiganticTurtle0 is it in the same repository ?
If it is it should have detected the fact that it needs to analyze the entire repository (not just the standalone script, and then discover tensorflow)

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

My pleasure 💗

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Correct.
It starts with the initial script (entry point), if it is self contained (i.e. does not interact with the rest of the repo) it will only analyze it, otherwise it will analyze the entire repo code.

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Hi GiganticTurtle0 ,

All the packages you are using should be under installed packages section in your task (in the UI). ClearML analyze and the full report should be under this section.

You can add any package you like with Task.add_requirements('tensorflow', '2.4.0') for tensorflow version 2.4.0 (or Task.add_requirements('tensorflow', '') for no limit).

If you dont want the package analyzer, you can configure in your ~/clearml.conf file: sdk.development.detect_with_pip_freeze: true and the installed packages section will be the same as you python environment pip freeze

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					TimelyPenguin76
				
					0
					 Administrator

, it's just a custom module.

Is this your own module ? Is this a local folder we import from ?

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

It's my own module (called 'tf_funcs.py')

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					GiganticTurtle0
				
					0
					 × 1

That's right. There is no such package, it's just a custom module.
But that module uses tensorflow , and ClearML does not add it to the list of packages to install. The only solution available so far is to include it manually via Task.add_requirements ?

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					GiganticTurtle0
				
					0
					 × 1

Thanks for helping. You and your team are doing a great job for the ML community.

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					GiganticTurtle0
				
					0
					 × 1

Then ClearML should also detect the dependencies of the imported scripts? In this case shouldn't it detect that I am going to use tensorflow and install it as well? Because it is not actually recognizing it.

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					GiganticTurtle0
				
					0
					 × 1

Hi GiganticTurtle0
ClearML will only list the directly imported packaged (not their requirements), meaning in your case it will only list "tf_funcs" (which you imported).
But I do not think there is a package named "tf_funcs" right ?

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Write your answer

1K Views

11 Answers

3 years ago

one year ago