Are you running the agent from your laptop as well?

This is included as part of the config file at ~/clearml.conf on the clearml-agent

extra_docker_shell_script: [ "apt-get install -y awscli", "aws codeartifact login --tool pip --repository data-live --domain ds-15gifts-code", ]
Not sure how to get a log from the CLI but I can get the error from the clearml server UI, one sec

I don't think we explicitly pass the package path to the agent. I expect it to run a regular pip install but it seems to be doing it via git somehow

Hi 🙂

How do you provide the package path to the agent? Also can you attach the log of the agent?

Another update - the tasks run fine and installs the packages from the correct index URL. However, by default, py_db @ git .. is added in the installed packages panel. Could this be from a requirements.txt file somewhere? To get it to work, I have to remove the @ git part, and then it works. Just very strange that it defaults to git pip install 🤔

I can authorise CodeArtifact if I ssh into the server, and install the private package with no issues. Seems like something is forcing clearml-agent to use github cloning to install, rather than directly pip. Not sure if this is a configuration I have set up myself, or whether the server is configured to do this

Using SSH credentials - replacing https url ' ' with ssh url ' ' Replacing original pip vcs 'git+ ' with ' ' Collecting py_db Cloning ssh://****@github.com/15gifts/py-db.git (to revision 851daa87317e73b4602bc1bddeca7ff16e1ac865) to /tmp/pip-install-zpiar1hv/py-db Running command git clone -q 'ssh://****@github.com/15gifts/py-db.git' /tmp/pip-install-zpiar1hv/py-db 2021-12-08 15:56:31 ERROR: Repository not found. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. ERROR: Command errored out with exit status 128: git clone -q 'ssh://****@github.com/15gifts/py-db.git' /tmp/pip-install-zpiar1hv/py-db Check the logs for full command output. RequirementsManager handler <clearml_agent.helper.package.external_req.ExternalRequirements object at 0x7fad9123c3d0> raised exception: Failed installing GIT/HTTPs package ' ' clearml_agent: ERROR: Could not install task requirements! Failed installing GIT/HTTPs package ' '

Nope, from a remote server. It was that I had installed the package from git locally, so when pushing the task, clearml assumed it should also install from git. I since installed the package from the private pypi and it all works as expected now 🙂

Okay solved the problem. It is using the version that is locally installed (on my laptop). Is there a way to prevent this? Perhaps a requirements.txt or something like that>

Only downside, which is not related to clearml, is that codeartifact authorisation tokens have to have a minimum lifespan of 15 mins. Usually, setting up envs before task execution takes less than a couple minutes, so the token lingers in the background. Nonetheless, all works as expected!

Only downside, which is not related to clearml, is that codeartifact authorisation tokens have to have a minimum lifespan of 15 mins

Yeah, that's always annoying with AWS APIs 🙂

I since installed the package from the private pypi and it all works as expected now 

👍