Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi! I Have A Gpu Workstation At The Office (No Public Ip) With Latest Clearml-Agent Installed. When I Was In The Same Network - I Was Able To Use Clearml-Session From My Laptop. Now I Work From Home, And Clearml-Session Fails With

Hi! I have a GPU workstation at the office (no public ip) with latest clearml-agent installed. When I was in the same network - I was able to use clearml-session from my laptop. Now I work from home, and clearml-session fails with
` Waiting for environment setup to complete [usually about 20-30 seconds]

Remote machine is ready
Setting up connection to remote session
Starting SSH tunnel

SSH tunneling failed, retrying in 3 seconds
Starting SSH tunnel

SSH tunneling failed, retrying in 3 seconds
Starting SSH tunnel `I also have ngrok running on the workstation, so when I need to ssh into it - I use the public hostname provided by ngrok.
Any advice on how can I make clearml-session to work?

  
  
Posted 3 years ago
Votes Newest

Answers 20


Here’s my workaround - ignore the fail messages, and manually create an SSH connection to the server with Jupyter port forwarded.

  
  
Posted 3 years ago

Hi FiercePenguin76

Here’s my workaround - ignore the fail messages, and manually create an SSH connection to the server with Jupyter port forwarded.

You are correct, clearml-session assumes it can SSH into the remote agent machine, from that point it automatically tunnels all other connections on top of the original SSH (well with some fancy SSH keep-alive proxy).
I'm assuming that from home you cannot connect to the SSH machine at the office, which makes sense, but out of curiosity how did you manage to SSH to it manually ?

  
  
Posted 3 years ago

I use ngrok to connect to it

  
  
Posted 3 years ago

So I thought, maybe I can tell clearml-session to use hostname from ngrok

  
  
Posted 3 years ago

ngrok to connect to the remote server at the office?
That makes sense, I guess this is the equivalent of using a VPN, from that point onward clearml-session can directly access the remote machine, right?

  
  
Posted 3 years ago

well, I first run clearml-session to start everything on the remote machine, then I close the local process (while Interactive is still running on the remote machine)

  
  
Posted 3 years ago

Then I ssh into the remote machine using ngrok hostname and tunnel the port for Jupyter

  
  
Posted 3 years ago

For others, who haven’t heard about ngrok:
Ngrok exposes local servers behind NATs and firewalls to the public internet over secure tunnels.

  
  
Posted 3 years ago

Oh in that case add --remote-gateway <external_ip> It will connect to the provided address instead of the local one. (you can also add --public-ip which will automatically resolve the public IP of the server

  
  
Posted 3 years ago

I tried both, didn’t work

  
  
Posted 3 years ago

I guess that’s because ngrok is not like a Dynamic DNS

  
  
Posted 3 years ago

I think the clearml-session CLI is missing the ability to add cutom port to the external address, does that make sense ?

  
  
Posted 3 years ago

it is missing in CLI, but I was able to set external_ssh_port and external_address in GUI. It was certainly a step forward, but still failed

  
  
Posted 3 years ago

` Remote machine is ready
Setting up connection to remote session
Starting SSH tunnel
Warning: Permanently added '<CENSORED>' (ECDSA) to the list of known hosts.
Enter passphrase for key '/Users/jevgenimartjushev/.ssh/id_rsa': <CENSORED>

SSH tunneling failed, retrying in 3 seconds `

  
  
Posted 3 years ago

to know why exactly ssh tunneling fails

  
  
Posted 3 years ago

Hmmm, yes we should definitely add --debug (if you can, please add a GitHub issue so it is not forgotten).
FiercePenguin76 Specifically are you able to ssh manually to <external_address>:<external_ssh_port> ?

  
  
Posted 3 years ago

yes I can.

  
  
Posted 3 years ago

Many thanks FiercePenguin76 !

  
  
Posted 3 years ago
999 Views
20 Answers
3 years ago
one year ago
Tags