Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello, I Have Been Using Clearml Interactive Session For More Than 3 Months And I Am Facing With Random Ssh Disconnection Errors In Vscode Once In A While After Creating The Session. Sometimes Reconnecting Works, If It Does Not Work I Reconnect The Clear

Hello,

I have been using ClearML interactive session for more than 3 months and I am facing with random SSH disconnection errors in VSCode once in a while after creating the session. Sometimes reconnecting works, if it does not work I reconnect the clearml-session by pressing "r" from the client console, then try to reconnect via VSCode. Yet in worst case, sometimes reconnecting through interactive session also does not work, in that case I need to shutdown the interactive session and start again and the problem is gone. My workflow is I create interactive session with a docker image and then develop the code in local VSCode, my local OS is Windows 11. What could be the problem, and if there is a possible solution, for this SSH connection error?

  
  
Posted 2 months ago
Votes Newest

Answers 7


Hello Martin, thanks for the response. My clearml-session version is 0.9.0 and clearml version is 1.14.1. The remote machine is in Azure behind the load-balancer, we are using docker images, so directly connecting to pods.

  
  
Posted 2 months ago

@<1699955693882183680:profile|UpsetSeaturtle37> can you try with the latest clearml-session (0.14.0) I remember a few improvements there

The remote machine is in Azure behind the load-balancer, we are using docker images, so directly connecting to pods.

yeah LB in the middle might be introducing SSH hiccups, first upgrade to the latest clearml-session it better ocnfigures the SSH client/server to support longer timeout connection, if that does not work try the -- keepalive=true
Let me know how it goes

  
  
Posted 2 months ago

@<1699955693882183680:profile|UpsetSeaturtle37> good progress, regrading the error, 0.15.0 is supposed to be out tomorrow, it includes a fix to that one.
BTW: can you run with --debug

  
  
Posted 2 months ago

clearml-session versions from 0.14.0 to 0.11.0 are all failed (raised the same error as above), I can only use 0.10.0 (the main clearml version is 1.15.1).

  
  
Posted 2 months ago

Thanks for the response will update you if anything happens 👍

  
  
Posted 2 months ago

Hi @<1699955693882183680:profile|UpsetSeaturtle37>
What's your clearml-session version? where is the remote machine ?
And yes if the network connection is bad we have seen this behavior you can try with --keepalive=true
Notice that these are SSH networking issue, not something to do with the clearml-session layer the --keepalive is trying to automatically detect these disconnects and make sure it reconnects for you.

  
  
Posted 2 months ago

I updated the clearml-session to 0.14.0. When I start the interactive session it seems successful at first, I can see it on the ClearML Web UI, however console raises error and stops the connection:
image

  
  
Posted 2 months ago
214 Views
7 Answers
2 months ago
2 months ago
Tags