Hi! I'M Running A Clearml Task Remotely. It Runs A Python Machine Learning Evaluation Script In Repo A. The Script Depends On A Module Which Is Defined In Another One Of My Company'S Repos: Repo B. The File Structure Of The Module In Repo B Is As Follows

Answered

Hi! I'm running a ClearML Task remotely. It runs a Python machine learning evaluation script in repo A. The script depends on a module which is defined in another one of my company's repos: repo B.

The file structure of the module in repo B is as follows:

-module_b.py
-templates
--my_template.yml

The evaluation script calls into module_b, which tries to load my_template.yml

When I run this locally without using ClearML, the YAML file can be successfully loaded. But when I run this remotely on ClearML, I get the following error:

FileNotFoundError: [Errno 2] No such file or directory: '/root/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/module_b/templates/my_template.yml'

Any idea why this might be happening? I originally thought the static template files weren't being bundled up properly with module B, but I checked the setup.py file and the call to setuptools.setup() takes in this parameter: include_package_data=True

So I'd expect this to be working fine 🤔

  				
Posted 
	one year ago

					More
				  		
  Report
		
					FlutteringFrog26
				
					0
					 × 1

Votes Newest

Answers 6

Yes, it’s running in Docker and thanks for the suggestion. I’ll try a manual install 👍

  				
Posted 
	one year ago

					More
				  		
  Report
		
					FlutteringFrog26
				
					0
					 × 1

once you install manually your package inside the docker container, check that your file module_b/templates/my_template.yml is where it should be

  				
Posted 
	one year ago

					More
				  		
  Report
		
					ManiacalLizard2
				
					0
					 × 1

so the issue is that for some reason, the pip install by the agent don't behave the same way as your local pip install ?
Have you tried to manually install your module_b with pip install inside the machine that is running clearml-agent ? Seeing your example, looks like you are even running inside docker ?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					ManiacalLizard2
				
					0
					 × 1

Hi, @<1523701070390366208:profile|CostlyOstrich36> I think I could have explained myself better. So the Task is running a script in repo A. But that script depends on module B, which is defined in repo B. I have a clearml_requirements.txt file in repo A. When the remote . ClearML Task runs, it installs the dependencies listed in that .txt file, including module B.

The problem is that the remote ClearML server is unable to find the static files that should come bundled along with module B.

Is that a bit clearer? 🤞

  				
Posted 
	one year ago

					More
				  		
  Report
		
					FlutteringFrog26
				
					0
					 × 1

Hi @<1614069770586427392:profile|FlutteringFrog26> , if I'm not mistaken ClearML doesn't support running from different repoes. You can only clone one code repository per task. Is there a specific reason these repoes are separate?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

or simply create a new venv in your local PC, then install your package with pip install from repo url and see if your file is deployed properly in that venv

  				
Posted 
	one year ago

					More
				  		
  Report
		
					ManiacalLizard2
				
					0
					 × 1

Write your answer

888 Views

6 Answers

one year ago