Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi All! I Want To Run My Task Remotely On An Agent, But I'M Having Trouble With The Requirements Setup. I Have A

Hi all!

I want to run my task remotely on an agent, but I'm having trouble with the requirements setup.

I have a requirements.txt with many packages to install, and the last line is " . " (which means 'install my package from this repo').
My package install involves a torch.utils.cpp_extension.CUDAExtension called, lets say, 'cuda_ext'

But it seems the agent is not correctly installing my packages.

The logs show "Successfully built package" and "Successfully installed package"; but then "Summary - instaled python packages : ...." does not show my package on the list
The task then fails due to an ImportError: 'cuda_ext' not found

Any tips? I've spent a lot more time than I would like on this 😞

  
  
Posted 2 years ago
Votes Newest

Answers 14


Thanks for the help anyway!

  
  
Posted 2 years ago

Looks like it was a python thing, not a clearml thing!

Clearml correctly installs the . from requirements.txt , but the project from the working directory was conflicting with the installed package, so python couldn't find the compiled extension.

With some small changes to my repo, everything works

  
  
Posted 2 years ago

Also, if you check the logs my package is actually built at step 4:

2023-05-03 10:07:58
Building wheels for collected packages: softgroup
  Building wheel for softgroup (setup.py) ... ?25l-
2023-05-03 10:08:14
 \ |
2023-05-03 10:08:19
 / - \

Looks like the -e flag is ignored. But it should work either way 🤔

  
  
Posted 2 years ago

in what order does the agent do things?
I assumed it was

  • Start the docker container
  • Run the docker setup bash script
  • Pull the repo , checkout the commit, apply changes
  • Install pip requirementsIn this case, i wouldn't have the correct version of the repo at the time the setup bash script runs
  
  
Posted 2 years ago

Can't you do that in the docker bash script?

  
  
Posted 2 years ago

I ned to pip-install the package because i need to build some Cuda extensions

  
  
Posted 2 years ago

Is there any way i can do something equivalent to -e . in the agent context?

  
  
Posted 2 years ago

I don't think -e . will work when running from the agent context

  
  
Posted 2 years ago

Basically: locally, when i run pip install -r requirements.txt , the softgroup.ops package is installed correctly. But not on the remote worker

I install the softgroup.ops package via the last line in requirements.txt , i.e. pip install -e .

  
  
Posted 2 years ago

And here is the repo: None

  
  
Posted 2 years ago

I attached three logs:

  • local_console_output : how i setup my local task. Important commands: apt-install that installs the same dependencies that are on the docker_setup_bash_script ; and pip install -r requirements.txt
  • local_task_output: clearml experiment console log. The error "the following arguments are required: config" is the expected behavior
  • remote_task_output: clearml experiment console log obtained when i clone the local task and enqueue it for remote execution. Notice that the behavior is different: i get ImportError: cannot import name 'ops' from 'softgroup.ops' (/root/.clearml/venvs-builds/3.7/task_repository/SoftGroup.git/softgroup/ops/__init__.py)
  
  
Posted 2 years ago

👍

  
  
Posted 2 years ago

Not sure if i can because of some proprietary stuff on the code.

But i'll try writing a minimum working example on monday!

  
  
Posted 2 years ago

Hi @<1556450111259676672:profile|PlainSeaurchin97> , can you share the full log and an example of how the requirements file looks?

  
  
Posted 2 years ago