Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hey All, We Are Trying To Clone A Task That Uses Custom Pip Installed Packages And Run It Via An Agent. When Running Locally, We Simply “

Hey all,

We are trying to clone a Task that uses custom pip installed packages and run it via an Agent.

When running locally, we simply “ pip install ./path/to/package ” for each package in our repo before running the Task.

The custom packages are theoretically accessible to ClearML, as they exist within the same github repo as the Task’s code, and this github repo is successfully recognised by ClearML - it appears in the “details” of the Task.

However ClearML does not seem to know that the custom python packages exist/are being used by the Task: the custom packages do not appear in the “installed packages” of the Task, despite the custom packages being used successfully when running said Task locally.

What is the recommended method to use custom python packages with ClearML Tasks?

Thanks

  
  
Posted one year ago
Votes Newest

Answers 7


my colleague, @<1534706830800850944:profile|ZealousCoyote89> has been looking at this – I think he has used the relevant kwarg in the component decorator to specify the packages, and I think it worked but I’m not 100%. Connah?

  
  
Posted one year ago

you can either add it manually to the installed packages, or remove the installed packages and use a setup.py file to manage the installation process

  
  
Posted one year ago

@<1523701079223570432:profile|ReassuredOwl55> did you try adding manually ?

./path/to/package

You can also do that from code:

Task.add_requirements("./path/to/package")
# notice you need to call Task.add_requirements before Task.init
task = Task.init(...)
  
  
Posted one year ago

I managed to get it running with:

task.set_packages('./package/requirements.txt')

where one of the lines in ./package/requirements.txt points to the package within our repo. E.g:

-e git+

I'll try pointing it directly to the package, that would be much easier to work with!

  
  
Posted one year ago

Similar error if I set the package after cloning the task:


Task.clone(...)
task.set_packages('./package')


  File "/home/ec2-user/miniconda3/envs/ml/lib/python3.10/site-packages/clearml/backend_interface/task/task.py", line 1414, in set_packages
    with open(packages) as f:
IsADirectoryError: [Errno 21] Is a directory: './ml'
  
  
Posted one year ago

To illustrate, here’s an example repo:

repo/
    package1/
    package2/ # installed separately to package1
    task_script.py # requires package1 and package2 to have been pip installed
    
  
  
Posted one year ago

We are cloning an existing task (pipeline). Adding Task.add_requirements("./path/to/package") before .Task.clone(...) gives:

2023-02-22 14:08:31,508 - clearml.task - WARNING - Requirement ignored, Task.add_requirements() must be called before Task.init()

Followed by this further down:

    with Path(package_name).open() as requirements_txt:
  File "/home/ec2-user/miniconda3/envs/ml/lib/python3.10/site-packages/pathlib2/__init__.py", line 1548, in open
    return io.open(
IsADirectoryError: [Errno 21] Is a directory: 'package'

Because we're cloning a task, I've tried specifying the package path in the pipeline decorator we're using and the steps which require the package. E.g.:

@PipelineDecorator.component(
    cache=True, 
    task_type=TaskTypes.data_processing,
    packages='./ppackage',
)
def step(...

@PipelineDecorator.pipeline(
    packages='./package',
)
def pipe(...

When running this pipeline, I get the following which has made me think that packages cannot be specified via paths?

    with Path(package_name).open() as requirements_txt:
  File "/home/ec2-user/miniconda3/envs/ml/lib/python3.10/site-packages/pathlib2/__init__.py", line 1548, in open
    return io.open(
IsADirectoryError: [Errno 21] Is a directory: 'package'
  
  
Posted one year ago
1K Views
7 Answers
one year ago
one year ago
Tags
Similar posts