since this is an enterprise machine, and you don't have sudo/root, I am wondering if there is already other docker networks/composer setups running/in use

so I am not entirely sure what else you have changed Sir

thank you so much for your effort

oh it's not a problem.. if you can fling up the logs of ES after startup that's probably the next step.. along with a 'docker network list' output 👍

I will get back to you tmr 😄 cheers

👍

HugePelican43 did you try docker ps and see that the clearml-apiserver container is up? if so, can you docker exec -it cleaml-apiserver /bin/bash and try to do the curl http://localhost:8008 from inside it?

curl inside clearml-apiserver works just fine 😄

"NetworkSettings": {
"Bridge": "",
"SandboxID": "xx",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"8008/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "8008"
}
],
"8080/tcp": null,
"8081/tcp": null
},
"SandboxKey": "/var/run/docker/netns/xx",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"mlops_backend": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"apiserver",
"f86e5d0835f4"
],
"NetworkID": "xx",
"EndpointID": "xx",
"Gateway": "172.25.0.1",
"IPAddress": "172.25.0.6",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "xx",
"DriverOpts": null
},
"mlops_frontend": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"apiserver",
"f86e5d0835f4"
],
"NetworkID": "xx",
"EndpointID": "xx",
"Gateway": "172.26.0.1",
"IPAddress": "172.26.0.3",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "xxx",
"DriverOpts": null
}
}
}
}
]

here is my network return after docker inspect

SuccessfulKoala55

I also run netstat inside the container:
netstat -an | grep LISTEN | grep 8008
tcp 0 0 0.0.0.0:8008 0.0.0.0:* LISTEN

What am I supposed to set CLEARML_HOST_IP value?

I assign 0.0.0.0

I believe host address of apiserver is not set to 0.0.0.0

You mean for the agent-services?

curl inside clearml-apiserver works just fine 

And from outside the docker, on the machine running the docker, it fails? That's very strange...

yeah from the host curl return curl: (56) Recv failure: Connection reset by peer.

I think because clearml-apiserver is binded to 127.0.0.1 inside docker container instead of 0.0.0.0

SuccessfulKoala55

but the code binds it to 0.0.0.0 by default...

You can look at the entrypoint inside the docker (it's called wrapper.sh )

Actually, it uses the apiserver.listen.ip configuration value (which is 0.0.0.0 by default)

😢 ah it's strange. I tried all the solutions I came up with

cannot fix it

BTW, from the netstat output you sent it seems like the apiserver is binded correctly inside the docker

Seems to be some system/docker configuration issue - maybe related to you now having admin permissions?

humhh I will ask the server admin

do you think it happened because of container firewall?

Possibly...