Hi, I Am Exploring The Following Workflow To Submit A Task To Remote Server:

Answered

Hi, i am exploring the following workflow to submit a task to remote server:

task = Task.create(
        branch=branch,
        repo=repo,
        script=script,
        requirements_file=requirements_file,
        ...
    )

# local repo typical setup follows this:
pip install -r requirements.txt
pip install -e .

So, Task.create(requirements_file=str) would account for the first step "pip install -r requirements.txt", how would I let clearml task know that it need to install my repo (in development mode) to run the task?

  				
Posted 
	6 months ago

					More
				  		
  Report
		
					ScrawnyCrocodile51
				
					0
					 × 1

Votes Newest

Answers 3

Hi @<1797800418953138176:profile|ScrawnyCrocodile51> , not sure what you're trying to do. Can you please elaborate?

  				
Posted 
	6 months ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

install in dev mode is the easiest without having to publish it first

  				
Posted 
	6 months ago

					More
				  		
  Report
		
					ScrawnyCrocodile51
				
					0
					 × 1

Sure, essentially my local python project organized using "src layout", look like this:

foo/
    |--src/
        |--module.py
    |--pyproject.toml
    |--clearml_tasks/
        |--task1.py

in the project, it would use absolute import like from foo import module , and I would install foo project in a editable mode during setup.

When I trying to create clearml task and send it to remote server using above way (leverage requirements.txt to configure library dependencies, and provide it where to clone my foo repo), I couldn't find a way to ask clearml to install my repo as editable mode. So it would install every other libraries I need but throw error ModuleNotFoundError: No module named 'foo'

  				
Posted 
	6 months ago

					More
				  		
  Report
		
					ScrawnyCrocodile51
				
					0
					 × 1

Write your answer

550 Views

3 Answers

6 months ago