![Profile picture](https://clearml-web-assets.s3.amazonaws.com/scoold/avatars/LittleReindeer37.png)
Reputation
Badges 1
75 × Eureka!But we're also testing out new models all the time, which are typically implemented as git branches - they run on the same set of inputs but don't output their results into production
if there are any tests/debugging you'd like me to try, just let me know
api/kernels
does report back the active kernel, but doesn't give notebook paths or anything
cool, if I call clearml.binding.hydra_bind.PatchHydra._register_omegaconf(cfg)
directly inside the notebook I can manually cause it to log the Hydra config, it just doesn't seem to autodetect if you're doing a manual call to Hydra Compose
I could just loop through and create separate pipelines with different parameters, but seems sort of inefficient. the hyperparameter optimization might actually work in this case utilizing grid search, but seems like kind of a hack
I can get it to run up to here: None
if I use the same kernel there'll be two
and that requests.get()
throws an exception:
ConnectionError: HTTPConnectionPool(host='default', port=8888): Max retries exceeded with url: /jupyter/default/api/sessions (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7ba9cadc30>: Failed to establish a new connection: [Errno -2] Name or service not known'))
but the only exception handler is for requests.exceptions.SSLError
I think it just ends up in /home/sagemaker-user/{notebook}.ipynb
every time
environ{'PYTHONNOUSERSITE': '0',
'HOSTNAME': 'gfp-science-ml-t3-medium-d579233e8c4b53bc5ad626f2b385',
'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI': '/_sagemaker-instance-credentials/xxx',
'JUPYTER_PATH': '/usr/share/jupyter/',
'SAGEMAKER_LOG_FILE': '/var/log/studio/kernel_gateway.log',
'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/tmp/miniconda3/condabin:/tmp/anaconda3/condabin:/tmp/miniconda2/condabin:/tmp/anaconda2/condabin'...
looks like the same as in server_info
those look like linear DAGs to me, but maybe I'm missing something. I'm thinking something like the map operator in Prefect where I can provide an array of ["A", "B", "C"]
and run the steps outlined with dotted lines independently for each of those are arguments
the CLEARML_*
variables are all explained here: None
Yes, I'm running a notebook in Studio. Where should it be captured?
and is there any way to capture hydra from a notebook as a Configuration? you don't use the typical @hydra.main()
but rather call the compose API , and so far in my testing that doesn't capture the OmegaConf in ClearML
poking around a little bit, and clearml.backend_interface.task.repo.scriptinfo.ScriptInfo._get_jupyter_notebook_filename()
returns None
I will once I figure out the fix!
which I looked at previously to see if I could import sagemaker.kg or kernelgateway or something, but no luck
but even then the sessions endpoint is still empty
I'm not sure if Subprojects will work for that - can you use the Web UI to compare the artifacts from two separate subprojects?
I've poked around both the internal URL that Jupyter kernel is running on and some of the files in /sagemaker/.jupyter
but no luck so far - I can find plenty of kernel info, but not session
if I add the base_url it's not found
but the call to jupyter_server.serverapp.list_running_servers()
does return the server