DefiantHippopotamus88HTTPConnectionPool(host='localhost', port=8081):This will not work because inside the container of the second docker compose "fileserver" is not definedCLEARML_FILES_HOST=" "
You have two options:
configure to the docker compose to use the networkhost on all conrtainers (as oppsed to the isolated mode they are now running ing)2. Configure all of the CLEARML_* to point to the Host IP address (e.g. 192.168.1.55) , then rerun the entire thing.
the url above i accessible from the container:$ docker-compose exec clearml-serving-inference bash root@a041497a554d:~/clearml# curl `
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title> <h1>Method Not Allowed</h1> <p>The method is not allowed for the requested URL.</p> root@a041497a554d:~/clearml# `
I don't know why it requests localhost
clearml-serving-inference | 2022-07-03 22:06:26,893 - clearml.storage - ERROR - Could not download , err: HTTPConnectionPool(host='localhost', port=8081): Max retries exceeded with url: /DevOps/serving%20example%2010.0a76d264e30940c2b600375fa839f1a2/artifacts/py_code_test_model_pytorch2/preprocess.py (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc3f41b1790>: Failed to establish a new connection: [Errno 111] Connection refused'))
curl -X POST " " -H "accept: application/json" -H "Content-Type: application/json" -d '{"url": " "}' {"detail":"Error processing request: Error: Failed loading preprocess code for 'py_code_test_model_pytorch2': 'NoneType' object has no attribute 'loader'"}
I made it working with full port reassignment to 9090 in clearml-serving-inference
which still send me an error that the format of my request is somehow wrong
but then I started from scratch by creating completely new project and new endpoint
In my understanding requests still go through
clearml-server
which configuration I left
DefiantHippopotamus88 actually this is Not correct.
clearml-server only acts as a control plane, no actual requests are routed to it, it is used to sync model state, stats etc. not part of the request processing flow itself.curl: (56) Recv failure: Connection reset by peerThis actually indicates 9090 port is not being listened to...
What's the final docker-compose you are using?
and what are you seeing when running netstat -natp | grep LISTEN ?
In my understanding requests still go through clearml-server which configuration I left intact. Maybe due to the port change in clearml-serving I need to adjust smth.
curl -X POST " " -H "accept: application/json" -H "Content-Type: application/json" -d '{"url": " "}' curl: (56) Recv failure: Connection reset by peer
DefiantHippopotamus88 you are sending the curl to the wrong port , it should be 9090 (based on what remember from the unified docker compose) on your setup
How can I clean database or whatever to get to the beginning?
I tried to switch off auto-refresh, but it doesn't help
I tried, step by step from here
https://github.com/allegroai/clearml-serving/tree/main/examples/pytorch
and result is the same.
Then I tried to remove my old serving examples to start checking from scratch by the immediately restart after stopping
First let's try to test if everything works as expected. Since 405 really feels odd to me here. Can I suggest following one of the examples start to end to test the setup, before adding your model?
basicaly I don't want to train a new model and I try to create an endpoint following the example but I finally get$ curl -X POST " " -H "accept: application/json" -H "Content-Type: application/json" -d '{"url": " ` "}'
<html> <head><title>405 Not Allowed</title></head> <body> <center><h1>405 Not Allowed</h1></center> <hr><center>nginx/1.20.1</center> </body> </html> `
DefiantHippopotamus88 you can create a custom endpoint and do that, but it will be running I the same instance , is this what you are after? Notice that Triton actually supports it already, you can check the pytorch example