![Profile picture](https://clearml-web-assets.s3.amazonaws.com/scoold/avatars/ThickSeaurchin47.png)
Reputation
Badges 1
41 × Eureka!From running it with the credentials I got with the non-self-hosted clearml instance
clearml-init
ClearML SDK setup process
Please create new clearml credentials through the settings page in your clearml-server
web app (e.g. None )
Or create a free account at None
In settings page, press "Create new credentials", then press "Copy to clipboard".
Paste copied configuratio...
If a external storage is not specified it will use internal correct?
So if I give up on having specific folder paths for each project within onedrive I could just treat it as a alternative path for internal artifact storage.
Nevermind, 😆 as you thought, seems I was just needing a few more f5s on my dashboard. Thank you very much for this
I’ll put in the actual copy paste later tonight thank you for the help
And again thank you for the help with this.
I discovered part of the problem. I did not have boto3 installed on this conda env.
No error, just failure to upload it seems
It gave no import error, and I'm still having problems. I returned to my original script and it shows some file transfer print statements, but I don't see the files appearing in minio
The exact error I am getting is:
line 1095, in output_uri
raise ValueError("Could not get access credentials for '{}' "
Hold on should host be
` s3://ipaddr:9000?
>>> print(json.dumps(config_obj.get("sdk"), indent=2))
{
"storage": {
"cache": {
"default_base_dir": "~/.clearml/cache"
},
"direct_access": [
{
"url": "file://*"
}
]
},
"metrics": {
"file_history_size": 100,
"matplotlib_untitled_history_size": 100,
"images": {
"format": "JPEG",
"quality": 87,
"subsampling": 0
},
"tensorboard_single_series_per_graph": false
},
"network": {
"file_upload_retries":...
I am able to capture clearml experiments on the clearml server running on the same machine as the minio.
Yes I will give it a try and get back to you.
Thank you!
I wasn’t getting my hopes up on storing tasks elsewhere :)
For the models & artifacts is there a parameter to change the default directory of saving and loading to something else?
As I’ve used it, rsync is akin to scp, but provides more control.
There are two things I want to accomplish with this:
- I want a reliable location to store my tasks without needing to worry about running out of space.
- I want to be able to access these stored tasks in the clearml ui when they are in this storage
While rsync would let me backup the file server contents, like cp’ing them into a rclone of onedrive. I don’t think it would let me view tasks that I have put on onedrive.
with the filepath leading to /clearml/task.py
This answered my question, I ended up setting up minio running off of a external hard drive which backups to onedrive.
Though I am still having some problems getting the client to connect to minio (see recent thread)
clearml-init
ClearML SDK setup process
Please create new clearml credentials through the settings page in your clearml-server
web app (e.g. None )
Or create a free account at None
In settings page, press "Create new credentials", then press "Copy to clipboard".
Paste copied configuration here:
api {
web_server: http://:8080
api_server: http://:8008
file...
For clarification, I want to store artifacts and models (maybe tasks too) like how the webui lets you specify a external storage s3 bucket when you make a folder, but with onedrive.
I.e in such a way I can still see them on the ui, but don’t necessarily have them stored on the server
@<1523701205467926528:profile|AgitatedDove14>
clearml python version: 1.91
python version: 3.9.15
the server is running the docker-compose on RHEL
Minio is on the same server and the 9000 and 9001 ports are open for tcp
I changed the default address space from 172.xxx.xxx.xx for docker to another space. This is not the issue as I can replicate this issue without this modified address space.
See configuration file below, I'm running the global section test now
aws {
s3 {...
credentials: [
specifies key/secret credentials to use when handli$
{
#
This will apply to all buckets in this host ($
host: "...:9000”
key: "*********"
secret: "******************"
multipart: false
secure: false
}
]
}
boto3 {
pool_connections: 512
max_ multipart_concurrency: 16
Is that being used as a dictionary key?
This error is thrown by a failed .get()
function call on the StorageHandler
object I looked at the ._ _ dict _ _.keys() parameter list of the StorageHandler, and I don't see anyway to access the dictionary directly.
And I’m still having my other issues I mentioned. I’m just ignoring the unreachable machines for the moment.
I placed the same key and secret in the global locations under s3{ } and this did not change anything
Bare with the spacing, I ocr’d this. The quotes and spacing is right
I upgraded to 1.9.3 and that didn’t change my error.
I created a new bucket with the name testbucket which didn’t change anything (I only updated this name in the output_uri parameter)
I tried curl on the minio:9000 which returns some html with AccessDenied as content
I tried curl on minio:9001 which returns the minio console html
Sorry for the delay on responding to this, I had some family craziness.