Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hey. Seeing A Strange Error When Trying To Install A Private Python Package When The Agent Executes The Task. We Use Aws Codeartifact As The Private Pypi Repository. The Agent Attempts To Install The Package From Git, Rather Than A Regular Pip Install And

Hey. Seeing a strange error when trying to install a private python package when the agent executes the task. We use AWS CodeArtifact as the private PyPi repository. The agent attempts to install the package from git, rather than a regular pip install and I cannot figure out why. Any ideas?

  
  
Posted 2 years ago
Votes Newest

Answers 12


Hi 🙂

How do you provide the package path to the agent? Also can you attach the log of the agent?

  
  
Posted 2 years ago

This is included as part of the config file at ~/clearml.conf on the clearml-agent

extra_docker_shell_script: [ "apt-get install -y awscli", "aws codeartifact login --tool pip --repository data-live --domain ds-15gifts-code", ]
Not sure how to get a log from the CLI but I can get the error from the clearml server UI, one sec

  
  
Posted 2 years ago

I don't think we explicitly pass the package path to the agent. I expect it to run a regular pip install but it seems to be doing it via git somehow

  
  
Posted 2 years ago

Using SSH credentials - replacing https url ' ' with ssh url ' ' Replacing original pip vcs 'git+ ' with ' ' Collecting py_db Cloning ssh://****@github.com/15gifts/py-db.git (to revision 851daa87317e73b4602bc1bddeca7ff16e1ac865) to /tmp/pip-install-zpiar1hv/py-db Running command git clone -q 'ssh://****@github.com/15gifts/py-db.git' /tmp/pip-install-zpiar1hv/py-db 2021-12-08 15:56:31 ERROR: Repository not found. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. ERROR: Command errored out with exit status 128: git clone -q 'ssh://****@github.com/15gifts/py-db.git' /tmp/pip-install-zpiar1hv/py-db Check the logs for full command output. RequirementsManager handler <clearml_agent.helper.package.external_req.ExternalRequirements object at 0x7fad9123c3d0> raised exception: Failed installing GIT/HTTPs package ' ' clearml_agent: ERROR: Could not install task requirements! Failed installing GIT/HTTPs package ' '

  
  
Posted 2 years ago

I can authorise CodeArtifact if I ssh into the server, and install the private package with no issues. Seems like something is forcing clearml-agent to use github cloning to install, rather than directly pip. Not sure if this is a configuration I have set up myself, or whether the server is configured to do this

  
  
Posted 2 years ago

Another update - the tasks run fine and installs the packages from the correct index URL. However, by default, py_db @ git .. is added in the installed packages panel. Could this be from a requirements.txt file somewhere? To get it to work, I have to remove the @ git part, and then it works. Just very strange that it defaults to git pip install 🤔

  
  
Posted 2 years ago

Okay solved the problem. It is using the version that is locally installed (on my laptop). Is there a way to prevent this? Perhaps a requirements.txt or something like that>

  
  
Posted 2 years ago

Are you running the agent from your laptop as well?

  
  
Posted 2 years ago

Nope, from a remote server. It was that I had installed the package from git locally, so when pushing the task, clearml assumed it should also install from git. I since installed the package from the private pypi and it all works as expected now 🙂

  
  
Posted 2 years ago

Only downside, which is not related to clearml, is that codeartifact authorisation tokens have to have a minimum lifespan of 15 mins. Usually, setting up envs before task execution takes less than a couple minutes, so the token lingers in the background. Nonetheless, all works as expected!

  
  
Posted 2 years ago

I since installed the package from the private pypi and it all works as expected now 

👍

  
  
Posted 2 years ago

Only downside, which is not related to clearml, is that codeartifact authorisation tokens have to have a minimum lifespan of 15 mins

Yeah, that's always annoying with AWS APIs 🙂

  
  
Posted 2 years ago