Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello ! When Running

Hello !

When running Dataset.get the wrong file_server api is being used.
This is the content of my clearml.conf
api { # Notice: 'host' is the api server (default port 8008), not the web server. api_server: web_server: files_server: # Credentials are generated using the webapp, /profile # Override with os environment: CLEARML_API_ACCESS_KEY / CLEARML_API_SECRET_KEY credentials {"access_key": "XX", "secret_key": "XX"} }but when trying to retrieve a Dataset I am getting this error :
ERROR - Could not download , err: HTTPConnectionPool(host='localhost', port=8081): Max retries exceeded with url: /annotation_dataset/annotation.ce2abe847e004ac282cc435bfa9c4bd5/artifacts/state/state.json (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1cca4eb1c0>: Failed to establish a new connection: [Errno 111] Connection refused'))So it is using localhost instead of fileserver .
What could I be doing wrong ? The code is being run inside a docker container, maybe that somehow impacts the process ?

  
  
Posted 2 years ago
Votes Newest

Answers 16


I am very confused now, I tried switch to my local machine and change the clearml.conf.
It only partly worked :
Dataset.list_datasets() returns the correct list (from the remote server).
But Dataset.get(dataset_id="ce2abe847e004ac282cc435bfa9c4bd5")
gives me :
2021-12-20 13:46:39,404 - clearml.storage - ERROR - Could not download ` , err: Failed getting object localhost:8081/annotation_dataset/annotation.ce2abe847e004ac282cc435bfa9c4bd5/artifacts/state/state.json (404): <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">

<title>404 Not Found</title> <h1>Not Found</h1> <p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p> `

  
  
Posted 2 years ago

I am very confused now, I tried switch to my local machine and change the clearml.conf.
It only partly worked :

Notice that the Dataset.get (...) is downloading an artifact that was uploaded before, basically it gets the full URL and downloads the data. it seems the original dataset uploaded to "localhost:8081", could that be the case?

  
  
Posted 2 years ago

What is the proper way to change a clearml.conf ?

  
  
Posted 2 years ago

Ah! That's it, thank you very much ! I did not know this was an issue. I though the dataset was only linked to the fileserver and not to the specific url used to upload it.

  
  
Posted 2 years ago

here is the command I am using :
sudo docker run -it -v /home/ubuntu/app/:/app/ -v /home/ubuntu/folder/clearml.conf:/root/clearml.conf --network "clearml_backend" my_image bash

  
  
Posted 2 years ago

I though the dataset was only linked to the fileserver and not to the specific url used to upload it. (

ShinyRabbit94 yep exactly! the idea is that you can actually do the storage on any solution (S3/GS etc.) the file server is just the default one 🙂

  
  
Posted 2 years ago

Can you try going into the docker and verifying you have the same clearml.conf inside?

  
  
Posted 2 years ago

Check the environment variables, maybe test with export maybe there's some env var hiding there 🙂

  
  
Posted 2 years ago

I did, I copy pasted the config from within the docker

  
  
Posted 2 years ago

Using export?

  
  
Posted 2 years ago

What is the proper way to change a clearml.conf ?

inside a container you can mount an external clearml.conf, or override everything with OS environment
https://clear.ml/docs/latest/docs/configs/env_vars#server-connection

  
  
Posted 2 years ago

of course, I am checking using the env command

  
  
Posted 2 years ago

Inside the docker, that is

  
  
Posted 2 years ago

there is nothing in the env

  
  
Posted 2 years ago

How is the docker being raised and which volumes are being shared?

  
  
Posted 2 years ago

I was looking at the code of the Dataset class, but I could not find where the files_server is retrieved.

  
  
Posted 2 years ago