Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Clearml-Session Fails Ssh Tunneling. It Does Not Use Key Auth, Instead Sets Up Some Weird Password And Then Fails To Auth:

clearml-session fails ssh tunneling. It does not use key auth, instead sets up some weird password and then fails to auth:
` Remote machine is ready
Setting up connection to remote session
Starting SSH tunnel
Warning: Permanently added '[192.168.1.55]:10022' (ED25519) to the list of known hosts.
boris@192.168.1.55's password: 63ed50ec899321256fb3c20d783494bb489e55dd1ba0a7498b238fe0acfc5d28

Permission denied, please try again.
boris@192.168.1.55's password: Error: incorrect password `

  
  
Posted 2 years ago
Votes Newest

Answers 22


AgitatedDove14
made a new one:
https://pastebin.com/LxLFk7py

  
  
Posted 2 years ago

I can telnet the port from my mac:
(base) *[main][~/Documents/plant_age]$ telnet 192.168.1.55 10022 Trying 192.168.1.55... Connected to 192.168.1.55. Escape character is '^]'. SSH-2.0-OpenSSH_8.4p1 Debian-5+deb11u1 ^C

  
  
Posted 2 years ago

The agent is started with this command:
clearml-agent --debug daemon --queue gpu --gpus 0 --foreground --docker <gitlab org registry>/project-precog/clearml_config

  
  
Posted 2 years ago

Btw it seems the docker runs in

network=host

Yes, this is so if you have multiple agents running on the same machine they can find a new open port 🙂

I can telnet the port from my mac:

Okay this seems like it is working

  
  
Posted 2 years ago

Is there a way to check if the port is accessible from my local machine?

  
  
Posted 2 years ago

Btw it seems the docker runs in network=host

  
  
Posted 2 years ago

I mean if I enter my host machine ssh password it works. But we will disable password auth in future, so it’s not an option

  
  
Posted 2 years ago

Ok, it makes sense. But it’s running in docker mode and it is trying to ssh into the host machine and failing

  
  
Posted 2 years ago

The task log is here:
the log on my local machine is here:

  
  
Posted 2 years ago

Sure, will send in a few min when it executes

  
  
Posted 2 years ago

It does not use key auth, instead sets up some weird password and then fails to auth:

AdventurousButterfly15 it ssh Into the container inside the container it sets new daemon with new random very long password
It will Not ssh to the host machine (i.e. the agent needs to run in docker mode, not venv mode), make sense ?

  
  
Posted 2 years ago

hmm can you share the log of the Task? (the clearml-session created Task)

  
  
Posted 2 years ago

All ports are open (both agent machine and client machine are working within same VPN)

  
  
Posted 2 years ago

no available 😞

  
  
Posted 2 years ago

The agent is started from a non-root user if that matters

  
  
Posted 2 years ago

I start clearml-session on my mac this way:
clearml-session --queue gpu --docker registry.gitlab.com/periplo-innovation/project-precog/clearml_config

  
  
Posted 2 years ago

I mean if I enter my host machine ssh password it works. But we will disable password auth in future, so it’s not an option

To clarify, it should not allow users to ssh into the host machine (if you can do that this means you own it), it only allows users to SSH into the container the host machine spins, make sense ?

  
  
Posted 2 years ago

But it’s running in docker mode and it is trying to ssh into the host machine and failing

It is Not sshing to the machine it is sshing directly Into the container.
Notice the port is is sshing to is 10022 which is mapped into the container

  
  
Posted 2 years ago

But what should I do? It does not work, it says incorrect password as you can see

  
  
Posted 2 years ago

But what should I do? It does not work, it says incorrect password as you can see

How are you spinning the agent machine ?
Basically 10022 port from the host (agent machine) is routed into the container, but it still needs to be open on the host machine, could it be it is behind a firewall? Are you (client side runnign clearml-session) on the same network as the machien runnign the agent ?

  
  
Posted 2 years ago

set the following:
CLEARML_AGENT_DISABLE_SSH_MOUNT=1 clearml-agent daemon ...The issue is, it will automatically mount the .ssh of the host into the container, so that if you are using SSH to clone git you have credentials, in your case, it also mounts the configuration, hence failing to login.
I will make sure we add it to the configuration file, so it is more visible

  
  
Posted 2 years ago

same: Not Found (#404)
May I suggest to DM it to me (so it is not public)

  
  
Posted 2 years ago
1K Views
22 Answers
2 years ago
one year ago
Tags