@<1523701087100473344:profile|SuccessfulKoala55> I solved it by other settings, thanks.

I also tried to run ssh -P 31919 root@10.190.253.18 , but got error message: ssh: connect to host 0.0.124.175 port 22: Unknown error .

Hi @<1523701087100473344:profile|SuccessfulKoala55> , Even I run the clearml-session with command line option --remote-ssh-port and --remote-gateway the SSH tunneling still failed.
Following is my complete step:

• set k8s service with the following yml:

kind: Service
apiVersion: v1
metadata:
  name: clearml-agent-1-nodeprot
  namespace: clearml
spec:
  ports:
  - name: clearml-agent-ssh
    port: 10022
    targetPort: 10022
    nodePort: 31919
  type: NodePort
  selector:
    ai.allegro.agent.serial: pod-1

2. run python k8s_glue_example.py --queue gpu-1 --ports-mode --template-yaml gpu-1.yml on k8s node.
3. run clearml-session --docker nvidia/cuda:11.0.3-runtime-ubuntu20.04 --remote-gateway 10.190.253.18 --remote-ssh-port 31919 on my PC. 10.190.253.18 is the node IP that session pod running.
4. the clearml-session log on my PC:

Remote machine is ready
Setting up connection to remote session
Starting SSH tunnel to root@10.190.253.18, port 31919

SSH tunneling failed, retrying in 3 seconds
Starting SSH tunnel to root@10.190.253.18, port 31919
.......

Could you provide a complete example or tutorial?

Hi @<1589065012880609280:profile|PunySquid51> , clearml-session uses port 10022 by default. you can use the --remote-ssh-port command line option to specify a different port to use