Clearnl Failed To Detect Custom Packages

Answered

clearnl failed to detect custom packages livsdk installed from git.
I have a dev venv where I start initial clearml task. In this dev venv, I have custom package installed with pip install -e .
The initial task running in my dev venv run find and get this list of Installed package:

# Python 3.10.10 (main, Mar 05 2023, 19:07:49) [GCC]

# Local modules found - skipping:
# livsdk == ../livsdk/livsdk/__init__.py

Augmentor == 0.2.10
Pillow == 9.2.0
PyYAML == 6.0
albumentations == 1.2.1
azure_storage_blob == 12.16.0
boto3 == 1.21.21
clearml == 1.11.0
efficientnet_pytorch == 0.7.1
fastai == 2.7.11
hyperopt == 0.2.7
ipython == 8.13.1
matplotlib == 3.5.2
ml_collections == 0.1.1
numpy == 1.24.3
onnx == 1.12.0
onnxruntime == 1.14.1
onnxruntime_gpu == 1.14.1
onnxsim == 0.4.8
opencv_python == 4.7.0.72
optuna == 2.10.1
pandas == 1.4.3
protobuf == 3.20.1
psutil == 5.9.5
requests == 2.28.1
scikit_learn == 1.1.1
scipy == 1.8.1
seaborn == 0.12.1
setuptools == 67.6.0
six == 1.16.0
thop == 0.1.1.post2209072238
timm == 0.6.13
torch == 1.13.0
torchvision == 0.14.0
tqdm == 4.64.0
xgboost == 1.7.1

When I clone the task and enqueue it as is:

[...]
Environment setup completed successfully
Starting Task Execution:
Traceback (most recent call last):
  File "[redacted_path]/debug.py", line 2, in <module>
    import livsdk.livbatch
ModuleNotFoundError: No module named 'livsdk'

Question: how do you add private package that should be installed via git ??

  				
Posted 
	9 months ago

					More
				  		
  Report
		
					ManiacalLizard2
				
					0
					 × 1

Votes Newest

Answers 4

I think this is what you need:
None

  				
Posted 
	9 months ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

What should I put in there? What is the syntax for git package?

  				
Posted 
	9 months ago

					More
				  		
  Report
		
					ManiacalLizard2
				
					0
					 × 1

Please read the documentation, there is an example there

  				
Posted 
	9 months ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

So I tried:

import livsdk.livbatch

import clearml

clearml.Task.add_requirements("livsdk","

")
task = clearml.Task.init(project_name="hieu-test", task_name='base_config')

print("Done")

Which give me this list of Packages Installed:

# Python 3.10.10 (main, Mar 05 2023, 19:07:49) [GCC]

# Local modules found - skipping:
# livsdk == ../[REDACTED]/livsdk/__init__.py

Augmentor == 0.2.10
Pillow == 9.2.0
PyYAML == 6.0
albumentations == 1.2.1
azure_storage_blob == 12.16.0
boto3 == 1.21.21
clearml == 1.11.0
efficientnet_pytorch == 0.7.1
fastai == 2.7.11
hyperopt == 0.2.7
ipython == 8.13.1
matplotlib == 3.5.2
ml_collections == 0.1.1
numpy == 1.24.3
onnx == 1.12.0
onnxruntime == 1.14.1
onnxruntime_gpu == 1.14.1
onnxsim == 0.4.8
opencv_python == 4.7.0.72
optuna == 2.10.1
pandas == 1.4.3
protobuf == 3.20.1
psutil == 5.9.5
requests == 2.28.1
scikit_learn == 1.1.1
scipy == 1.8.1
seaborn == 0.12.1
setuptools == 67.6.0
six == 1.16.0
thop == 0.1.1.post2209072238
timm == 0.6.13
torch == 1.13.0
torchvision == 0.14.0
tqdm == 4.64.0
xgboost == 1.7.1
livsdk

And when I clone and enqueue the task:

[...]
Collecting timm==0.6.13
  Using cached timm-0.6.13-py3-none-any.whl (549 kB)

1692001815617 hieuOpensuse:0 DEBUG Collecting torch==1.13.0.*
  Using cached torch-1.13.0-cp310-cp310-manylinux1_x86_64.whl (890.1 MB)

1692001820764 hieuOpensuse:0 DEBUG Collecting torchvision==0.14.0.*
  Using cached torchvision-0.14.0-cp310-cp310-manylinux1_x86_64.whl (24.3 MB)
Collecting tqdm==4.64.0
  Using cached tqdm-4.64.0-py2.py3-none-any.whl (78 kB)
Collecting xgboost==1.7.1
  Using cached xgboost-1.7.1-py3-none-manylinux2014_x86_64.whl (193.6 MB)
ERROR: Ignored the following versions that require a different python version: 1.6.2 Requires-Python >=3.7,<3.10; 1.6.3 Requires-Python >=3.7,<3.10; 1.7.0 Requires-Python >=3.7,<3.10; 1.7.1 Requires-Python >=3.7,<3.10
ERROR: Could not find a version that satisfies the requirement livsdk (from versions: none)
ERROR: No matching distribution found for livsdk

1692001825912 hieuOpensuse:0 DEBUG 
clearml_agent: ERROR: Could not install task requirements!
Command '['/home/mhtrinh/.clearml/venvs-builds/3.10/bin/python', '-m', 'pip', '--disable-pip-version-check', 'install', '-r', '/tmp/cached-reqsa47z0j6r.txt', '--extra-index-url', '

 returned non-zero exit status 1.


1692001826035 hieuOpensuse:0 DEBUG Process failed, exit code 1

  				
Posted 
	9 months ago

					More
				  		
  Report
		
					ManiacalLizard2
				
					0
					 × 1

Write your answer

505 Views

4 Answers

9 months ago