Hi everyone,
I am working on a script that gets enqueued, and executes remotely on clearml-agent workers.
I got a "Hello World" version of the script running, the python env gets set up, and installs some basic packages I need from PyPI.
However, I need to include some private packages that are hosted on our GitLab Package Registry.
These can be installed by pip on the cli or from a requirements.txt file by specifying the URL which looks like:package_name==0.1.0 --index-url https://<username>:<token>@gitlab.com/api/v4/projects/<project_id>/packages/pypi/simple
But so far I have not had success with the automatic package detection.
I've also tried the manual methods like: task.set_packages , Task.force_requirements_env_freeze(False, requirements_file="requirements.txt")
But I get errors during the environment setup once its running remotely:
ERROR: Invalid requirement: 'package==0.1.0 --index-url https://<username>:<token>@gitlab.com/api/v4/projects/<project_id>/packages/pypi/simple'
Hint: It looks like a path. File 'package==0.1.0 --index-url https://<username>:<token>@gitlab.com/api/v4/projects/<project_id>/packages/pypi/simple' does not exist.
RequirementsManager handler <clearml_agent.helper.package.external_req.ExternalRequirements object at 0x7f13e6d4ab90> raised exception: Failed installing GIT/HTTPs package 'package==0.1.0 --index-url https://<username>:<token>@gitlab.com/api/v4/projects/<project_id>/packages/pypi/simple'
clearml_agent: ERROR: Could not install task requirements!
Failed installing GIT/HTTPs package 'package==0.1.0 --index-url https://<username>:<token>@gitlab.com/api/v4/projects/<project_id>/packages/pypi/simple'
Hi @<1590152201068613632:profile|StaleLeopard22> , you can simply add the extra index url as part of the agent requirements as such:
agent.package_manager.extra_index_url=["<extra_index_url>",...]
Yes this solved it for me, although this is not ideal as pip will look in our private package repo and in PyPI for each dependency that is installed, slowing the environment setup process significantly
