To illustrate, here’s an example repo:

repo/
    package1/
    package2/ # installed separately to package1
    task_script.py # requires package1 and package2 to have been pip installed
    

you can either add it manually to the installed packages, or remove the installed packages and use a setup.py file to manage the installation process

@<1523701079223570432:profile|ReassuredOwl55> did you try adding manually ?

./path/to/package

You can also do that from code:

Task.add_requirements("./path/to/package")
# notice you need to call Task.add_requirements before Task.init
task = Task.init(...)

my colleague, @<1534706830800850944:profile|ZealousCoyote89> has been looking at this – I think he has used the relevant kwarg in the component decorator to specify the packages, and I think it worked but I’m not 100%. Connah?

We are cloning an existing task (pipeline). Adding Task.add_requirements("./path/to/package") before .Task.clone(...) gives:

2023-02-22 14:08:31,508 - clearml.task - WARNING - Requirement ignored, Task.add_requirements() must be called before Task.init()

Followed by this further down:

    with Path(package_name).open() as requirements_txt:
  File "/home/ec2-user/miniconda3/envs/ml/lib/python3.10/site-packages/pathlib2/__init__.py", line 1548, in open
    return io.open(
IsADirectoryError: [Errno 21] Is a directory: 'package'

Because we're cloning a task, I've tried specifying the package path in the pipeline decorator we're using and the steps which require the package. E.g.:

@PipelineDecorator.component(
    cache=True, 
    task_type=TaskTypes.data_processing,
    packages='./ppackage',
)
def step(...

@PipelineDecorator.pipeline(
    packages='./package',
)
def pipe(...

When running this pipeline, I get the following which has made me think that packages cannot be specified via paths?

    with Path(package_name).open() as requirements_txt:
  File "/home/ec2-user/miniconda3/envs/ml/lib/python3.10/site-packages/pathlib2/__init__.py", line 1548, in open
    return io.open(
IsADirectoryError: [Errno 21] Is a directory: 'package'

Similar error if I set the package after cloning the task:

Task.clone(...)
task.set_packages('./package')


  File "/home/ec2-user/miniconda3/envs/ml/lib/python3.10/site-packages/clearml/backend_interface/task/task.py", line 1414, in set_packages
    with open(packages) as f:
IsADirectoryError: [Errno 21] Is a directory: './ml'

I managed to get it running with:

task.set_packages('./package/requirements.txt')

where one of the lines in ./package/requirements.txt points to the package within our repo. E.g:

-e git+

I'll try pointing it directly to the package, that would be much easier to work with!