Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi!

Hi! 👋 I am still quite new to ClearML but we are currently assessing how useful it is at the place where I work - and so far, it looks pretty cool! 🙂
We have one issue with git repo URLs and ssh key authentication . I found an issue on it on GitHub, also posted there but now I've been thinking about it more and guess a discussion here might be better than a bloated, non-focused issue on GitHub. Details in the thread.

  
  
Posted 2 years ago
Votes Newest

Answers 16


Hm I tried it again (even cleaning up the vcs cache before since that caused an issue before) but it still does not work. Looking at the code, I also could not find the place where this should happen. For all I can tell, there are only translations from https->ssh and ssh->https , but not ssh->ssh .

To add that, I quickly coded up this PR:
https://github.com/allegroai/clearml-agent/pull/72

Could you take a look at it? On our installation here, it shows the desired behavior, I can later see in the log
Using SSH credentials - ssh url ' ' with ssh url ' 'and the cloning works 🙂

  
  
Posted 2 years ago

generally speaking the agent will convert the repo url to the auth scheme it is configured with, ssh->hhtp if using user/pass, and http->ssh if using ssh

  
  
Posted 2 years ago

should work as well 🙂

  
  
Posted 2 years ago

OK but ssh->ssh?

  
  
Posted 2 years ago

My only point is, if we have no force_git_ssh_port or force_git_ssh_user we should not touch the SSH link (i.e. less chance of us messing with the original URL if no one asked us to)

  
  
Posted 2 years ago

Ohh I see now the force SSH did not replace the user in the SSH link (only if the original was http), right ?

  
  
Posted 2 years ago

You are awesome !!!

  
  
Posted 2 years ago

Hi EagerOtter28
The agent knows how to do the http->ssh conversion on the fly, in your cleaml.conf (on the agent's machine) set force_git_ssh_protocol: true
https://github.com/allegroai/clearml-agent/blob/42606d9247afbbd510dc93eeee966ddf34bb0312/docs/clearml.conf#L25

  
  
Posted 2 years ago

Found a https://github.com/allegroai/clearml-agent/issues/42#issuecomment-887331420 . Though would any of the above proposed solutions be feasible?

  
  
Posted 2 years ago

Ah OK 🤔 So should I maybe update the PR to not touch the URL if neither user nor port are 'force-set'?

  
  
Posted 2 years ago

Alright will do so later 🙂 👍

  
  
Posted 2 years ago

Hey AgitatedDove14 - thank you for the help! 🙂 Though in our case, most developers have the repo setup with ssh key authentication. Thus the task gets a 'ssh url' like ssh:// and not https:// . Consequently, the conversion is never called. Or is it already expected behavior that ClearML agent rewrites ssh://mydomain.com:2022/ ... to ssh://git@mydomain.com:2022/ ... if I have force_git_ssh_protocol: true and force_git_ssh_user: "git" ?

  
  
Posted 2 years ago

When we run a script containing Task.init from within our repo, it creates a repo URL that looks like this:

Now the agents trying to execute this task fail with:
cloning: agent_user@git.mycompany.com: Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.
An easy fix is to change the URL in the UI to include the user, e.g. ssh://git@git.mycompany.com:2022/myuser/repo.git , but as mentioned by AgitatedDove14 https://github.com/allegroai/clearml-agent/issues/42#issuecomment-757045256 , this does not scale well.

Using a config
I also tried creating a config, like:
Host git.mycompany.com HostName 123.123.123.123 IdentityFile ~/.ssh/id_rsa IdentitiesOnly yes User git Port 2022Running the clone command outside of the docker container ( git clone ssh://git.mycompany.com:2022/myuser/repo.git ) work with this - but within the docker container, SSH refuses to use the config because the owner of the config does not match ( agent_user vs root ).

I see many potential ways to solve this, but I am not sure what is the best. Maybe I am also missing an obvious solution?
Ideally, we could set the repo user in the config on developer machines when we create the task, thereby having the correct URL directly stored in the backend. Alternatively, could we use Task.update() or something to update the repo after the call to Task.init() ? Would that propagate into the backend? And would you have an example snippet? Could the option agent.force_git_ssh_user = "git" be extended from the use-case of rewriting https to ssh URLs to also rewrite ssh URLs?
Any input would be greatly appreciated! 🙂

  
  
Posted 2 years ago

Or is it already expected behavior that ClearML agent rewrites ...

Yep, that should work

  
  
Posted 2 years ago