As a side note, I attempted to debug the issue using strace by tracing the connect system call with the command:
$ strace -e connect python ./t1.py
connect(3, {sa_family=AF_INET, sin_port=htons(8008), sin_addr=inet_addr("XXX.XXX.XXX.XXX")}, 16) = -1 EINPROGRESS (Operation now in progress)
connect(3, {sa_family=AF_INET, sin_port=htons(8015), sin_addr=inet_addr("XXX.XXX.XXX.XXX")}, 16) = -1 EINPROGRESS (Operation now in progress)
The output showed that the program first tried to connect to port 8008, which is open on the server and also defined in the clearml.conf file. However, the connect call to port 8008 occurred only once, while all other connect calls were made to port 8015 for unknown reasons.
I am unsure why the program suddenly attempted to connect to port 8015. When I tested it on a different computer from a different network, this issue did not occur, and all connect calls were made to the same 8008 port, which is open and defined.
Do you have any idea why the program attempted to connect to the 8015 port?
Is there a firewall in between or something stopping the connection?
@<1546303269423288320:profile|MinuteStork43> the clearml server will not redirect any call - is this is happening, it's probably some proxy, FW or a load balancer in between the client and the server (which makes sense since calls from different networks work just fine)
Yes there is a firewall , but it open on this port (8008). the clearml suddenly attempted to connect to port 8015...
@<1523701070390366208:profile|CostlyOstrich36>
Hi @<1546303269423288320:profile|MinuteStork43> , how did you set the apiserver in clearml.conf ?
After digging deeper into the starce log, I found the following:
Due to some unknown reason, the clearml server has directed me to port 8015 (and this is only happening on one network, I have tested two different computers on this network, both behave the same). Outside the network, everything works correctly.
connect(3, {sa_family=AF_INET, sin_port=htons(8008), sin_addr=inet_addr(…..
poll([{fd=3, events=POLLOUT|POLLERR}], 1, 3000) = 1 ([{fd=3, revents=POLLOUT}])
getsockopt(3, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
poll([{fd=3, events=POLLOUT}], 1, 3000) = 1 ([{fd=3, revents=POLLOUT}])
sendto(3, "GET /auth.login ……) = 402
ioctl(3, FIONBIO, [1]) = 0
poll([{fd=3, events=POLLIN}], 1, 10000) = 1 ([{fd=3, revents=POLLIN}])
recvfrom(3, "HTTP/1.1 302 Found\r\nLocation: None \r\nConnection: close\r\nX-Frame-Options: SAMEORIGIN\r\nX-XSS-Protection: 1; mode=block\r\nX-Content-Type-Options: nosniff\r\nContent-Security-Policy: frame-ancestors 'self'\r\n\r\n", 8192, 0, NULL, NULL) = 232