Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Folks,

Hi Folks,

  • I'm experimenting with a local deployment of ClearML server on my laptop. I'm developing on a docker container anyway, so I think it makes sense to add my dev container to the clearml docker-compose.yml so that my dev container can communicate with the ClearML server.
  • What is the best way to do this? - I tried:
services:
  devcontainer:
    build: Dockerfile
    ...
    network_mode: service:apiserver
    depends_on:
    - apiserver
    - redis
    - mongo
    - elasticsearch
    - fileserver

  apiserver:
...

With this approach I seem to be able to communicate with the apiserver. for instance, I can create a task with the python SDK. However, I don't seem to be able to communicate with the fileserver. if I do the dataset Hello World example I get connection refused when adding data to a dataset. If I inspect the logs for the fileserver I see an error similar to this one: None

  • I tried modifying the docker-compose.yml as follows:
services:
  devcontainer:
    build: Dockerfile
    ...
    networks: 
      - frontend
      - backend
...
  apiserver:
...

But with this configuration I can't even connect to the apiserver.

Thanks so much for your help

  
  
Posted 4 months ago
Votes Newest

Answers 12


Just wanted to add that the above error api_server is misconfigured seems to happen regardless of whether I modify the standard docker_compose.yml

  
  
Posted 4 months ago

Hi @<1722786133810614272:profile|LazyClams7> where exactly do you see the error? Can you send the complete logs?

  
  
Posted 4 months ago

Also, how are you configuring clearml.conf inside your container?

  
  
Posted 4 months ago

@<1523701087100473344:profile|SuccessfulKoala55> I see the api_server is misconfigured error in the logs for the clearml-fileserver . Unfortunately I can't copy and paste the logs for policy reasons, but I'll paraphrase them below:

  • config files loaded from /opt/clearml/fileserver
  • [ERROR] [werkzeug] (ip address) code 400, message bad request version
  • urllib3.connectionpool Retrying after connection broken by NewConnectionError
  • Traceback: urllib3/connection.py: ConnectionRefusedError [Errno 111] Connection refused- During the handling of above exception, another error occured: MaxRetryError- During the handling of above exception: ConnectionError: HTTPConnectionPool(host='apiserver', port=8008): Max retries exceeded with url: /auth.login- During the handling of above exception: ValueError : Connection Error: it seems api_server is misconfigured
  
  
Posted 4 months ago

I used clearml-init to configure the clearml.conf . This used the following:

  • api_server : None
  • web_server: None
  • files_server : NoneJust to see if it would help, I tried changing localhost to host.docker.internal but it didn't help.
  
  
Posted 4 months ago

I also tried using the ip address of the files_server container, but that didn't seem to make a difference

  
  
Posted 4 months ago

Hi @<1722786133810614272:profile|LazyClams7> , if you see api_server is misconfigured error in the logs for the clearml-fileserver , this basically means the filesserver can't reach the apiserver (it needs to), this doesn't seem to be a client issue. The filessererver tried to reach the apiserver by default using the None endpoint, which should be available inside the docker backend network - you can try to exec into the fileserver container and see if curl to that address works

  
  
Posted 3 months ago

@<1523701087100473344:profile|SuccessfulKoala55> I did as you suggested: from the fileserver container, I was able to authenticate with the API server and receive a token.
however, from my development container if I do
clearml-data add --files Affairs.csv
I still get a connetion refused error.

  
  
Posted 3 months ago

Perhaps your container is not part of the correct docker networks as the fileserver contains is?

  
  
Posted 3 months ago

@<1523701087100473344:profile|SuccessfulKoala55> right now it is configured with:

network_mode: service:apiserver

in docker-compose-win-10.yml . I have also tried:

networks:
   - frontend
   - backend

But with this configuration, I was not even able to communicate with the apiserver.
Correction: I am able to authenticate with the api server with both configurations. However, with the first configuration, I can use the task cli to do for instance:

clearml-task --project examples --name remote_test --repo 
 --branch master --script /webinar-0620/keras_mnist.py --args batch_size=64 epochs=1 --queue default

However, this does not work with the second configuration (I get a connection refused error)

  
  
Posted 3 months ago

So the first configuration works?

  
  
Posted 3 months ago

@<1523701087100473344:profile|SuccessfulKoala55> No, I am not able to upload a file with either configuration. However, the first configuration works better in that I can use the task CLI.

  
  
Posted 3 months ago
362 Views
12 Answers
4 months ago
3 months ago
Tags
Similar posts