
Reputation
Badges 1
75 × Eureka!but the only exception handler is for requests.exceptions.SSLError
nice! Just tested it on my end as well, looks like it works!
sh-4.2$ cat /var/log/studio/kernel_gateway.log | head -n10
{"__timestamp__": "2023-02-23T21:48:28.036559Z", "__schema__": "sagemaker.kg.request.schema", "__schema_version__": 1, "__metadata_version__": 1, "account_id": "", "duration": 0.0012829303741455078, "method": "GET", "uri": "/api", "status": 200}
{"__timestamp__": "2023-02-23T21:48:39.111068Z", "__schema__": "sagemaker.kg.request.schema", "__schema_version__": 1, "__metadata_version__": 1, "account_id": "", "duration": 0.00128793...
but r.json()
is an empty list
the problem is here: None
here's my script:
#!/bin/bash
echo "******************** Starting Agent ********************"
echo "******************** Getting ENV Variables ********************"
source /etc/profile.d/env-vars.sh
# test that we can access the API
echo "******************** Waiting for ${CLEARML_API_HOST} connectivity ********************"
curl --retry 10 --retry-delay 10 --retry-connrefused ${CLEARML_API_HOST}/debug.ping
# start the agent
for i in $(seq 1 ${NUM_WORKERS})
do
export CLEARML_WORK...
and cat /var/log/studio/kernel_gateway.log | grep ipynb
comes up empty
I could just loop through and create separate pipelines with different parameters, but seems sort of inefficient. the hyperparameter optimization might actually work in this case utilizing grid search, but seems like kind of a hack
curious whether it impacts anything besides sagemaker. I'm thinking it's generically a kernel gateway issue, but I'm not sure if other platforms are using that yet
I'm doing that and it's working well
Just ran the same notebook in a local Jupyter Lab session and it worked as I expected it might, saving a copy to Artifacts
if I instead change the request url to f"http://{server_info['hostname']}:{server_info['port']}/api/sessions"
then it gets a 200 response... however , the response is an empty list
I'm not sure if Subprojects will work for that - can you use the Web UI to compare the artifacts from two separate subprojects?
but maybe that doesn't matter, actually - it might be one session per host I guess
cool, thanks! the first one was what I had thought of but seemed unpythonic, so I'll give the second a shot
but even then the sessions endpoint is still empty
I think it just ends up in /home/sagemaker-user/{notebook}.ipynb
every time
which I looked at previously to see if I could import sagemaker.kg or kernelgateway or something, but no luck
if I use the same kernel there'll be two
looks like the same as in server_info
poking around a little bit, and clearml.backend_interface.task.repo.scriptinfo.ScriptInfo._get_jupyter_notebook_filename()
returns None
and that requests.get()
throws an exception:
ConnectionError: HTTPConnectionPool(host='default', port=8888): Max retries exceeded with url: /jupyter/default/api/sessions (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7ba9cadc30>: Failed to establish a new connection: [Errno -2] Name or service not known'))