curl inside clearml-apiserver works just fine 

And from outside the docker, on the machine running the docker, it fails? That's very strange...

oh it's not a problem.. if you can fling up the logs of ES after startup that's probably the next step.. along with a 'docker network list' output 👍

so I am not entirely sure what else you have changed Sir

Possibly...

here is my network return after docker inspect

Actually, it uses the apiserver.listen.ip configuration value (which is 0.0.0.0 by default)

I think because clearml-apiserver is binded to 127.0.0.1 inside docker container instead of 0.0.0.0

HugePelican43 did you try docker ps and see that the clearml-apiserver container is up? if so, can you docker exec -it cleaml-apiserver /bin/bash and try to do the curl http://localhost:8008 from inside it?

yeah from the host curl return curl: (56) Recv failure: Connection reset by peer.

cannot fix it

do you think it happened because of container firewall?

SuccessfulKoala55

You can look at the entrypoint inside the docker (it's called wrapper.sh )

SuccessfulKoala55

You mean for the agent-services?

BTW, from the netstat output you sent it seems like the apiserver is binded correctly inside the docker

Seems to be some system/docker configuration issue - maybe related to you now having admin permissions?

😢 ah it's strange. I tried all the solutions I came up with

"NetworkSettings": {
"Bridge": "",
"SandboxID": "xx",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"8008/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "8008"
}
],
"8080/tcp": null,
"8081/tcp": null
},
"SandboxKey": "/var/run/docker/netns/xx",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"mlops_backend": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"apiserver",
"f86e5d0835f4"
],
"NetworkID": "xx",
"EndpointID": "xx",
"Gateway": "172.25.0.1",
"IPAddress": "172.25.0.6",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "xx",
"DriverOpts": null
},
"mlops_frontend": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"apiserver",
"f86e5d0835f4"
],
"NetworkID": "xx",
"EndpointID": "xx",
"Gateway": "172.26.0.1",
"IPAddress": "172.26.0.3",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "xxx",
"DriverOpts": null
}
}
}
}
]

What am I supposed to set CLEARML_HOST_IP value?

since this is an enterprise machine, and you don't have sudo/root, I am wondering if there is already other docker networks/composer setups running/in use

humhh I will ask the server admin

👍

I will get back to you tmr 😄 cheers

I believe host address of apiserver is not set to 0.0.0.0

curl inside clearml-apiserver works just fine 😄

thank you so much for your effort

I also run netstat inside the container:
netstat -an | grep LISTEN | grep 8008
tcp 0 0 0.0.0.0:8008 0.0.0.0:* LISTEN

but the code binds it to 0.0.0.0 by default...

I assign 0.0.0.0