sh-4.2$ cat /var/log/studio/kernel_gateway.log | head -n10
{"__timestamp__": "2023-02-23T21:48:28.036559Z", "__schema__": "sagemaker.kg.request.schema", "__schema_version__": 1, "__metadata_version__": 1, "account_id": "", "duration": 0.0012829303741455078, "method": "GET", "uri": "/api", "status": 200}
{"__timestamp__": "2023-02-23T21:48:39.111068Z", "__schema__": "sagemaker.kg.request.schema", "__schema_version__": 1, "__metadata_version__": 1, "account_id": "", "duration": 0.0012879371643066406, "method": "GET", "uri": "/api/kernels", "status": 200}
{"__timestamp__": "2023-02-23T21:48:39.116324Z", "__schema__": "sagemaker.kg.request.schema", "__schema_version__": 1, "__metadata_version__": 1, "account_id": "", "duration": 0.0007715225219726562, "method": "GET", "uri": "/api/terminals", "status": 200}
{"__timestamp__": "2023-02-23T21:48:39.272822Z", "__schema__": "sagemaker.kg.request.schema", "__schema_version__": 1, "__metadata_version__": 1, "account_id": "", "duration": 0.0007491111755371094, "method": "GET", "uri": "/api/terminals", "status": 200}
{"__timestamp__": "2023-02-23T21:48:43.000795Z", "__schema__": "sagemaker.kg.request.schema", "__schema_version__": 1, "__metadata_version__": 1, "account_id": "", "duration": 2.539133071899414, "method": "POST", "uri": "/api/kernels", "status": 201}
{"__timestamp__": "2023-02-23T21:48:43.073568Z", "__schema__": "sagemaker.kg.request.schema", "__schema_version__": 1, "__metadata_version__": 1, "account_id": "", "duration": 0.0013430118560791016, "method": "GET", "uri": "/api/kernels/6ba227af-ff2c-4b20-89ac-86dcac95e2b2", "status": 200}
{"__timestamp__": "2023-02-23T21:48:43.469751Z", "__schema__": "sagemaker.kg.request.schema", "__schema_version__": 1, "__metadata_version__": 1, "account_id": "", "duration": 0.0013761520385742188, "method": "GET", "uri": "/api/kernels/6ba227af-ff2c-4b20-89ac-86dcac95e2b2", "status": 200}
{"__timestamp__": "2023-02-23T21:48:43.702549Z", "__schema__": "sagemaker.kg.request.schema", "__schema_version__": 1, "__metadata_version__": 1, "account_id": "", "duration": 0.0013780593872070312, "method": "GET", "uri": "/api/kernels/6ba227af-ff2c-4b20-89ac-86dcac95e2b2", "status": 200}
{"__timestamp__": "2023-02-23T21:48:43.986808Z", "__schema__": "sagemaker.kg.request.schema", "__schema_version__": 1, "__metadata_version__": 1, "account_id": "", "duration": 0.0007445812225341797, "method": "GET", "uri": "/api/kernels/6ba227af-ff2c-4b20-89ac-86dcac95e2b2", "status": 200}
{"__timestamp__": "2023-02-23T21:48:43.992860Z", "__schema__": "sagemaker.kg.request.schema", "__schema_version__": 1, "__metadata_version__": 1, "account_id": "", "duration": 0.001028299331665039, "method": "GET", "uri": "/api/kernels", "status": 200}

api/kernels does report back the active kernel, but doesn't give notebook paths or anything

Yes, I'm running a notebook in Studio. Where should it be captured?

lots of things like {"__timestamp__": "2023-02-23T23:49:23.285946Z", "__schema__": "sagemaker.kg.request.schema", "__schema_version__": 1, "__metadata_version__": 1, "account_id": "", "duration": 0.0007679462432861328, "method": "GET", "uri": "/api/kernels/6ba227af-ff2c-4b20-89ac-86dcac95e2b2", "status": 200}

Try to add here:
None

server_info['url'] = f"http://{server_info['hostname']}:{server_info['port']}/"

curious whether it impacts anything besides sagemaker. I'm thinking it's generically a kernel gateway issue, but I'm not sure if other platforms are using that yet

but the call to jupyter_server.serverapp.list_running_servers() does return the server

This is very odd ... let me check something

yep

if there are any tests/debugging you'd like me to try, just let me know

Hi @<1532532498972545024:profile|LittleReindeer37>
Yes you are correct it should capture the entire jupyter notebook in sagemaker studio.
Just verifying this is the use case, correct ?

but the only exception handler is for requests.exceptions.SSLError

Hmm and you are getting empty list for thi one:

server_info['url'] = f"http://{server_info['hostname']}:{server_info['port']}/"

but even then the sessions endpoint is still empty

but one possible workaround is to try to figure out if it's running in a gateway and then find the only notebook running on that server

so notebooks ends up empty

if I add the base_url it's not found

if I change it to 0.0.0.0 it works

nope, that's wrong

environ{'PYTHONNOUSERSITE': '0',
        'HOSTNAME': 'gfp-science-ml-t3-medium-d579233e8c4b53bc5ad626f2b385',
        'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI': '/_sagemaker-instance-credentials/xxx',
        'JUPYTER_PATH': '/usr/share/jupyter/',
        'SAGEMAKER_LOG_FILE': '/var/log/studio/kernel_gateway.log',
        'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/tmp/miniconda3/condabin:/tmp/anaconda3/condabin:/tmp/miniconda2/condabin:/tmp/anaconda2/condabin',
        'REGION_NAME': 'us-east-1',
        'AWS_INTERNAL_IMAGE_OWNER': 'Custom',
        'AWS_DEFAULT_REGION': 'us-east-1',
        'PWD': '/home/sagemaker-user',
        'AWS_REGION': 'us-east-1',
        'SHLVL': '1',
        'HOME': '/home/sagemaker-user',
        'AWS_SAGEMAKER_PYTHONNOUSERSITE': '0',
        'AWS_ACCOUNT_ID': 'xxx',
        '_': '/opt/.sagemakerinternal/conda/bin/jupyter-kernelgateway',
        'LC_CTYPE': 'C.UTF-8',
        'KERNEL_LAUNCH_TIMEOUT': '40',
        'KERNEL_WORKING_PATH': '',
        'KERNEL_GATEWAY': '1',
        'JPY_PARENT_PID': '9',
        'PYDEVD_USE_FRAME_EVAL': 'NO',
        'TERM': 'xterm-color',
        'CLICOLOR': '1',
        'FORCE_COLOR': '1',
        'CLICOLOR_FORCE': '1',
        'PAGER': 'cat',
        'GIT_PAGER': 'cat',
        'MPLBACKEND': '

_inline'}

I will once I figure out the fix!

I think it just ends up in /home/sagemaker-user/{notebook}.ipynb every time

so notebook path is empty

What happens when you call:

from clearml.backend_interface.task.repo import ScriptInfo

print(ScriptInfo._ScriptInfo__legacy_jupyter_notebook_server_json_parsing(None))

and cat /var/log/studio/kernel_gateway.log | grep ipynb comes up empty

sadly no

print(os.environ)

if I instead change the request url to f"http://{server_info['hostname']}:{server_info['port']}/api/sessions" then it gets a 200 response... however , the response is an empty list

still empty

it does return kernels, just not sessions