I though the dataset was only linked to the fileserver and not to the specific url used to upload it. (
ShinyRabbit94 yep exactly! the idea is that you can actually do the storage on any solution (S3/GS etc.) the file server is just the default one 🙂
Ah! That's it, thank you very much ! I did not know this was an issue. I though the dataset was only linked to the fileserver and not to the specific url used to upload it.
What is the proper way to change a clearml.conf ?
inside a container you can mount an external clearml.conf, or override everything with OS environment
https://clear.ml/docs/latest/docs/configs/env_vars#server-connection
I am very confused now, I tried switch to my local machine and change the clearml.conf.
It only partly worked :
Notice that the Dataset.get (...) is downloading an artifact that was uploaded before, basically it gets the full URL and downloads the data. it seems the original dataset uploaded to "localhost:8081", could that be the case?
I was looking at the code of the Dataset class, but I could not find where the files_server is retrieved.
What is the proper way to change a clearml.conf ?
I am very confused now, I tried switch to my local machine and change the clearml.conf.
It only partly worked :Dataset.list_datasets() returns the correct list (from the remote server).
But Dataset.get(dataset_id="ce2abe847e004ac282cc435bfa9c4bd5")
gives me :2021-12-20 13:46:39,404 - clearml.storage - ERROR - Could not download ` , err: Failed getting object localhost:8081/annotation_dataset/annotation.ce2abe847e004ac282cc435bfa9c4bd5/artifacts/state/state.json (404): <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>404 Not Found</title> <h1>Not Found</h1> <p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p> `
of course, I am checking using the env command
Check the environment variables, maybe test with export maybe there's some env var hiding there 🙂
I did, I copy pasted the config from within the docker
Can you try going into the docker and verifying you have the same clearml.conf inside?
here is the command I am using :sudo docker run -it -v /home/ubuntu/app/:/app/ -v /home/ubuntu/folder/clearml.conf:/root/clearml.conf --network "clearml_backend" my_image bash
How is the docker being raised and which volumes are being shared?