Hi 🙂

How do you provide the package path to the agent? Also can you attach the log of the agent?

Only downside, which is not related to clearml, is that codeartifact authorisation tokens have to have a minimum lifespan of 15 mins. Usually, setting up envs before task execution takes less than a couple minutes, so the token lingers in the background. Nonetheless, all works as expected!

I since installed the package from the private pypi and it all works as expected now 

👍

Another update - the tasks run fine and installs the packages from the correct index URL. However, by default, py_db @ git .. is added in the installed packages panel. Could this be from a requirements.txt file somewhere? To get it to work, I have to remove the @ git part, and then it works. Just very strange that it defaults to git pip install 🤔

This is included as part of the config file at ~/clearml.conf on the clearml-agent

extra_docker_shell_script: [ "apt-get install -y awscli", "aws codeartifact login --tool pip --repository data-live --domain ds-15gifts-code", ]
Not sure how to get a log from the CLI but I can get the error from the clearml server UI, one sec

Nope, from a remote server. It was that I had installed the package from git locally, so when pushing the task, clearml assumed it should also install from git. I since installed the package from the private pypi and it all works as expected now 🙂

Only downside, which is not related to clearml, is that codeartifact authorisation tokens have to have a minimum lifespan of 15 mins

Yeah, that's always annoying with AWS APIs 🙂

I don't think we explicitly pass the package path to the agent. I expect it to run a regular pip install but it seems to be doing it via git somehow

Using SSH credentials - replacing https url ' ' with ssh url ' ' Replacing original pip vcs 'git+ ' with ' ' Collecting py_db Cloning ssh://****@github.com/15gifts/py-db.git (to revision 851daa87317e73b4602bc1bddeca7ff16e1ac865) to /tmp/pip-install-zpiar1hv/py-db Running command git clone -q 'ssh://****@github.com/15gifts/py-db.git' /tmp/pip-install-zpiar1hv/py-db 2021-12-08 15:56:31 ERROR: Repository not found. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. ERROR: Command errored out with exit status 128: git clone -q 'ssh://****@github.com/15gifts/py-db.git' /tmp/pip-install-zpiar1hv/py-db Check the logs for full command output. RequirementsManager handler <clearml_agent.helper.package.external_req.ExternalRequirements object at 0x7fad9123c3d0> raised exception: Failed installing GIT/HTTPs package ' ' clearml_agent: ERROR: Could not install task requirements! Failed installing GIT/HTTPs package ' '

I can authorise CodeArtifact if I ssh into the server, and install the private package with no issues. Seems like something is forcing clearml-agent to use github cloning to install, rather than directly pip. Not sure if this is a configuration I have set up myself, or whether the server is configured to do this

Are you running the agent from your laptop as well?

Okay solved the problem. It is using the version that is locally installed (on my laptop). Is there a way to prevent this? Perhaps a requirements.txt or something like that>