Here’s my workaround - ignore the fail messages, and manually create an SSH connection to the server with Jupyter port forwarded.

Hi FiercePenguin76

Here’s my workaround - ignore the fail messages, and manually create an SSH connection to the server with Jupyter port forwarded.

You are correct, clearml-session assumes it can SSH into the remote agent machine, from that point it automatically tunnels all other connections on top of the original SSH (well with some fancy SSH keep-alive proxy).
I'm assuming that from home you cannot connect to the SSH machine at the office, which makes sense, but out of curiosity how did you manage to SSH to it manually ?

I use ngrok to connect to it

So I thought, maybe I can tell clearml-session to use hostname from ngrok

ngrok to connect to the remote server at the office?
That makes sense, I guess this is the equivalent of using a VPN, from that point onward clearml-session can directly access the remote machine, right?

well, I first run clearml-session to start everything on the remote machine, then I close the local process (while Interactive is still running on the remote machine)

Then I ssh into the remote machine using ngrok hostname and tunnel the port for Jupyter

For others, who haven’t heard about ngrok:
Ngrok exposes local servers behind NATs and firewalls to the public internet over secure tunnels.

Oh in that case add --remote-gateway <external_ip> It will connect to the provided address instead of the local one. (you can also add --public-ip which will automatically resolve the public IP of the server

I tried both, didn’t work

I guess that’s because ngrok is not like a Dynamic DNS

I think the clearml-session CLI is missing the ability to add cutom port to the external address, does that make sense ?

it is missing in CLI, but I was able to set external_ssh_port and external_address in GUI. It was certainly a step forward, but still failed

` Remote machine is ready
Setting up connection to remote session
Starting SSH tunnel
Warning: Permanently added '<CENSORED>' (ECDSA) to the list of known hosts.
Enter passphrase for key '/Users/jevgenimartjushev/.ssh/id_rsa': <CENSORED>

SSH tunneling failed, retrying in 3 seconds `

I think we need logging here: https://github.com/allegroai/clearml-session/blob/bf1851cd3831c19cc0eadd9b2ffc0613f97f16e1/clearml_session/main.py#L564

to know why exactly ssh tunneling fails

Hmmm, yes we should definitely add --debug (if you can, please add a GitHub issue so it is not forgotten).
FiercePenguin76 Specifically are you able to ssh manually to <external_address>:<external_ssh_port> ?

yes I can.

https://github.com/allegroai/clearml-session/issues/1

Many thanks FiercePenguin76 !