Hi @<1523701118159294464:profile|ExasperatedCrab78> , I have a sad update on this issue. It does not seem to be completely solved yet. 😕 But I think I can at least describe it a bit better now:
- Models which are located on the clearML servers (created by
Task.init(..., output_uri=True)
) still run perfectly. - Models which are located on azure blob storage make different problems in different scenarios (which made me think we resolved this issue):- When I start the docker container add a model from the clearML server and afterwards add a model located on azure (on the same endpoint) I get no error and all my http requests are answered properly.
- When I start the docker container with no model added and first add a model from azure, I get this error
poll failed for model directory 'test_model_pytorch': failed to open text file for read /models/test_model_pytorch/config.pbtxt: No such file or directory
. - When I start a docker container where a model from azure was already added before I get this error:
clearml-serving-triton | Updating local model folder: /models
clearml-serving-triton | Error retrieving model ID ca186e8440b84049971a0b623df36783 []
clearml-serving-triton | Starting server: ['tritonserver', '--model-control-mode=poll', '--model-repository=/models', '--repository-poll-secs=60.0', '--metrics-port=8002', '--allow-metrics=true', '--allow-gpu-metrics=true']
clearml-serving-triton | Traceback (most recent call last):
clearml-serving-triton | File "clearml_serving/engines/triton/triton_helper.py", line 540, in <module>
clearml-serving-triton | main()
clearml-serving-triton | File "clearml_serving/engines/triton/triton_helper.py", line 532, in main
clearml-serving-triton | helper.maintenance_daemon(
clearml-serving-triton | File "clearml_serving/engines/triton/triton_helper.py", line 274, in maintenance_daemon
clearml-serving-triton | raise ValueError("triton-server process ended with error code {}".format(error_code))
clearml-serving-triton | ValueError: triton-server process ended with error code 1
Side note: In the meantime I also set up the docker container on a linux server and get the same error as on my windows computer with docker desktop.
I am not sure if this is really about passing the azure credentials because I feel like I have tried all possibilities which are suggested online. For a final try I directly wrote my azure account and storage key into the docker-compose-triton.yml with this syntax: AZURE_STORAGE_ACCOUNT: $(AZURE_STORAGE_ACCOUNT:-myazureaccount}
At least this should work, or am I wrong? I would really appreciate hearing your thoughts on this.