What am I supposed to set CLEARML_HOST_IP value?

I will get back to you tmr 😄 cheers

here is my network return after docker inspect

I assign 0.0.0.0

humhh I will ask the server admin

Actually, it uses the apiserver.listen.ip configuration value (which is 0.0.0.0 by default)

SuccessfulKoala55

Possibly...

curl inside clearml-apiserver works just fine 😄

oh it's not a problem.. if you can fling up the logs of ES after startup that's probably the next step.. along with a 'docker network list' output 👍

since this is an enterprise machine, and you don't have sudo/root, I am wondering if there is already other docker networks/composer setups running/in use

Seems to be some system/docker configuration issue - maybe related to you now having admin permissions?

SuccessfulKoala55

thank you so much for your effort

curl inside clearml-apiserver works just fine 

And from outside the docker, on the machine running the docker, it fails? That's very strange...

BTW, from the netstat output you sent it seems like the apiserver is binded correctly inside the docker

"NetworkSettings": {
"Bridge": "",
"SandboxID": "xx",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"8008/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "8008"
}
],
"8080/tcp": null,
"8081/tcp": null
},
"SandboxKey": "/var/run/docker/netns/xx",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"mlops_backend": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"apiserver",
"f86e5d0835f4"
],
"NetworkID": "xx",
"EndpointID": "xx",
"Gateway": "172.25.0.1",
"IPAddress": "172.25.0.6",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "xx",
"DriverOpts": null
},
"mlops_frontend": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"apiserver",
"f86e5d0835f4"
],
"NetworkID": "xx",
"EndpointID": "xx",
"Gateway": "172.26.0.1",
"IPAddress": "172.26.0.3",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "xxx",
"DriverOpts": null
}
}
}
}
]

I think because clearml-apiserver is binded to 127.0.0.1 inside docker container instead of 0.0.0.0

HugePelican43 did you try docker ps and see that the clearml-apiserver container is up? if so, can you docker exec -it cleaml-apiserver /bin/bash and try to do the curl http://localhost:8008 from inside it?

do you think it happened because of container firewall?

I also run netstat inside the container:
netstat -an | grep LISTEN | grep 8008
tcp 0 0 0.0.0.0:8008 0.0.0.0:* LISTEN

yeah from the host curl return curl: (56) Recv failure: Connection reset by peer.

👍

but the code binds it to 0.0.0.0 by default...

😢 ah it's strange. I tried all the solutions I came up with

You can look at the entrypoint inside the docker (it's called wrapper.sh )

I believe host address of apiserver is not set to 0.0.0.0

so I am not entirely sure what else you have changed Sir

You mean for the agent-services?

cannot fix it