Hi @<1523701070390366208:profile|CostlyOstrich36> , sure (a little anonymised). I took the output from the "console" tab because somehow wsl overwrites the lines in my terminal

clearml-session --project session_test2 --remote-ssh-port 10023 --queue my_queue --docker my_image:310.0
clearml-session - CLI for launching JupyterLab / VSCode on a remote machine
Verifying credentials
Use previous queue (resource) 'my_queue' [Y]/n? y

Interactive session config:
{
    "base_task_id": null,
    "docker": "my_image:310.0",
    "git_credentials": false,
    "jupyter_lab": true,
    "keepalive": false,
    "password": "19da0fc81721616ca0f8e2368ee2fa913f311d144c805722fb4ab5b4be4a4d4f",
    "project": "session_test2",
    "queue": "my_queue",
    "remote_ssh_port": "10023",
    "vscode_server": true,
    "yes": false
}

Launch interactive session [Y]/n? y
Removing stale interactive sessions
Creating new session
Configuring new session
New session created [id=7484669b0ff14fa3bf1abb618c70d903]
Waiting for remote machine allocation [id=7484669b0ff14fa3bf1abb618c70d903]
.Status [queued]
..Status [in_progress] - queued pulled by agent
Remote machine allocated
Setting remote environment [Task id=7484669b0ff14fa3bf1abb618c70d903]
Setup process details: 

Waiting for environment setup to complete [usually about 20-30 seconds, see last log line/s below]
> Executing: ['docker', 'run', '-t', '--gpus', '"device=2"', '--network', 'host', '--shm-size=8G', '-l', 'clearml-worker-id=my_server:gpu2', '-l', 'clearml-parent-worker-id=my_server:gpu2', '-e', 'CLEARML_WORKER_ID=my_server:gpu2', '-e', 'CLEARML_DOCKER_IMAGE=my_image:310.0 --network host', '-e', 'CLEARML_TASK_ID=7484669b0ff14fa3bf1abb618c70d903', '--name', 'clearml-id-7484669b0ff14fa3bf1abb618c70d903-xnvpchpo', '-v', '/tmp/.clearml_agent.qysuc32g.cfg:/tmp/clearml.conf', '-e', 'CLEARML_CONFIG_FILE=/tmp/clearml.conf', '-v', '/home/my_folder/ai-clearml/.clearml/apt-cache.4:/var/cache/apt/archives', '-v', '/home/my_folder/ai-clearml/.clearml/pip-cache:/root/.cache/pip', '-v', '/home/my_folder/ai-clearml/.clearml/pip-download-cache:/root/.clearml/pip-download-cache', '-v', '/home/my_folder/ai-clearml/.clearml/cache:/clearml_agent_cache', '-v', '/home/my_folder/ai-clearml/.clearml/vcs-cache:/root/.clearml/vcs-cache', '-v', '/home/my_folder/ai-clearml/.clearml/venvs-cache:/root/.clearml/venvs-cache', '--rm', 'my_image:310.0', 'bash', '-c', 'echo \'Binary::apt::APT::Keep-Downloaded-Packages "true";\' > /etc/apt/apt.conf.d/docker-clean ; chown -R root /root/.cache/pip ; export DEBIAN_FRONTEND=noninteractive ; export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL libsm6 libxext6 libxrender-dev libglib2.0-0" ; [ ! -z $(which git) ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL git" ; declare LOCAL_PYTHON ; [ ! -z $LOCAL_PYTHON ] || for i in {15..5}; do which python3.$i && python3.$i -m pip --version && export LOCAL_PYTHON=$(which python3.$i) && break ; done ; [ ! -z $LOCAL_PYTHON ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL python3-pip" ; [ -z "$CLEARML_APT_INSTALL" ] || (apt-get update -y ; apt-get install -y $CLEARML_APT_INSTALL) ; [ ! -z $LOCAL_PYTHON ] || export LOCAL_PYTHON=python3 ; $LOCAL_PYTHON -m pip install -U "pip<20.2 ; python_version < \'3.10\'" "pip<22.3 ; python_version >= \'3.10\'" ; $LOCAL_PYTHON -m pip install -U clearml-agent ; cp /tmp/clearml.conf ~/default_clearml.conf ; NVIDIA_VISIBLE_DEVI> Requirement already satisfied: zipp>=0.5 in /mmdet/my-venv/lib/python3.9/site-packages (from importlib-metadata>=4.> Installing collected packages: pyzmq, traitlets, pickleshare, exceptiongroup, backcall, matplotlib-inline, executing, asttokens, pure-eval, stack-data, ptyprocess, pexpect, decorator, prompt-toolkit, parso, jedi, ipython, nest-asyncio, debugpy, jupyter-core, comm, tornado, jupyter-client, psutil, ipykernel, jupyter-console, webencodings, bleach, rpds-py, attrs, referencing, jsonschema-specifications, jsonschema, fastjsonschema, nbformat, nbclient, tinycss2, defusedxml, pandocfilters, mistune, jupyterlab-pygments, soupsieve, beautifulsoup4, nbconvert, argon2-cffi-bindings, argon2-cffi, terminado, jupyter-server-terminals, sniffio, anyio, websocket-client, python-json-logger, rfc3986-validator, rfc3339-validator, jupyter-events, send2trash, overrides, prometheus-client, jupyter-server, notebook-shim, json5, babel, jupyterlab-server, jupyter-lsp, async-lru, jupyterlab, notebook, ipython-genutils, qtpy, qtconsole, widgetsnbextension, jupyterlab-widgets, ipywidgets, jupyter, tomlkit, mccabe, lazy-object-proxy, wrapt, astroid, isort, pylint, pathlib2, pyjwt, orderedmu     cessing triggers for libc-bin (2.35-0ubuntu3.1) ...


Remote machine is ready
Setting up connection to remote session
Starting SSH tunnel to root@172.24.6.50, port 10025

root@172.24.6.50's password: 19da0fc81721616ca0f8e2368ee2fa913f311d144c805722fb4ab5b4be4a4d4f

Welcome to Ubuntu 22.04.2 LTS (GNU/Linux 5.4.0-58-generic x86_64)

 * Documentation:  

 * Management:     

 * Support:        

This system has been minimized by removing packages and content that are
not required on a system that users do not log into.

To restore this content, you can run the 'unminimize' command.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

(my-venv) root@myserver:~#

Interactive session is running:
SSH: ssh root@localhost -p 8022 [password: 19da0fc81721616ca0f8e2368ee2fa913f311d144c805722fb4ab5b4be4a4d4f]
Jupyter Lab URL: 

VSCode server available at 

Connection is up and running
Enter "r" (or "reconnect") to reconnect the session (for example after suspend)
Ctrl-C (or "quit") to abort (remote session remains active)
or "Shutdown" to shutdown remote interactive session

Sometimes the port param works, sometimes it doesn't. I did not find any differences in the logs so far.

we want to use a session/a port per gpu on the machine but are behind a firewall. currently ports 10020-10025 are allowed, but even with the port param set it sometimes jumps out of this range. I cannot always reproduce it, but I suspect it somehow depends on other sessions running at the time

Hi @<1595225628804648960:profile|TroubledLion34> , can you please add a log of what you're inputting/getting in clearml-session ?