@<1523701087100473344:profile|SuccessfulKoala55> I see the api_server is misconfigured error in the logs for the clearml-fileserver . Unfortunately I can't copy and paste the logs for policy reasons, but I'll paraphrase them below:
- config files loaded from
/opt/clearml/fileserver - [ERROR] [werkzeug] (ip address) code 400, message bad request version
urllib3.connectionpoolRetrying after connection broken by NewConnectionError- Traceback: urllib3/connection.py: ConnectionRefusedError [Errno 111] Connection refused- During the handling of above exception, another error occured: MaxRetryError- During the handling of above exception: ConnectionError: HTTPConnectionPool(host='apiserver', port=8008): Max retries exceeded with url:
/auth.login- During the handling of above exception:ValueError: Connection Error: it seems api_server is misconfigured
Perhaps your container is not part of the correct docker networks as the fileserver contains is?
Hi @<1722786133810614272:profile|LazyClams7> , if you see api_server is misconfigured error in the logs for the clearml-fileserver , this basically means the filesserver can't reach the apiserver (it needs to), this doesn't seem to be a client issue. The filessererver tried to reach the apiserver by default using the None endpoint, which should be available inside the docker backend network - you can try to exec into the fileserver container and see if curl to that address works
Also, how are you configuring clearml.conf inside your container?
Just wanted to add that the above error api_server is misconfigured seems to happen regardless of whether I modify the standard docker_compose.yml
@<1523701087100473344:profile|SuccessfulKoala55> right now it is configured with:
network_mode: service:apiserver
in docker-compose-win-10.yml . I have also tried:
networks:
- frontend
- backend
But with this configuration, I was not even able to communicate with the apiserver.
Correction: I am able to authenticate with the api server with both configurations. However, with the first configuration, I can use the task cli to do for instance:
clearml-task --project examples --name remote_test --repo
--branch master --script /webinar-0620/keras_mnist.py --args batch_size=64 epochs=1 --queue default
However, this does not work with the second configuration (I get a connection refused error)
I also tried using the ip address of the files_server container, but that didn't seem to make a difference
Hi @<1722786133810614272:profile|LazyClams7> where exactly do you see the error? Can you send the complete logs?
@<1523701087100473344:profile|SuccessfulKoala55> I did as you suggested: from the fileserver container, I was able to authenticate with the API server and receive a token.
however, from my development container if I doclearml-data add --files Affairs.csv
I still get a connetion refused error.
@<1523701087100473344:profile|SuccessfulKoala55> No, I am not able to upload a file with either configuration. However, the first configuration works better in that I can use the task CLI.