You can check out boto3 python client (This is what we use to download / upload all S3 stuff), but minio-client probably already uses it under the hood.
We also use aws cli to do some downloading, it is way faster than python.
Regarding pdfs, yes, you have no choice but to preprocess it
What you want is to have a service script that cleans up archived tasks, here is what we used: None
ok, I found it.
Are S3 links supported?
How can I do that?
I need to save the original hash, otherwise I lose all trackability to about 2k experiments
We fixed the issue, thanks, had to update everything to latest.
will it be appended in clearml?
"s3" is part of domain to the host
This is what I see on fresh clearml
Where all my mounts are on /mnt/data/clearml-server instead of /opt/clearml
Specifying it like this, gets me different error:
Exception has occurred: ValueError
- Insufficient permissions (delete failed) for None
botocore.exceptions.ClientError: An error occurred (IllegalLocationConstraintException) when calling the DeleteObject operation: The me-south-1 location constraint is incompatible for the region specific endpoint this request was sent to.
During handling of the above exception, another exception occurred:
File "/home/ma...
WebApp: 1.14.1-451 • Server: 1.14.1-451 • API: 2.28
7 out of 30 GB is currently used and is quite stable
Adding bucket in clearml.conf causes the same error: clearml.storage - ERROR - Failed uploading: Could not connect to the endpoint URL: " None "
I purged all docker images and it still doesnt seem right
I see no side panel and it doesnt ask for login name
im also batch uploading, maybe thats the problem?
- The dataset is about 1TB containing 1 million files
- I dont have the SSD space locally to do the upload
- So i download a part of the dataset, use add_files() and then upload() to that batch
- Upload the dataset
I noticed that each batch is slower and slower
I tried it with port, but still having the same issue
Tried it with/without secure and multipart
Im doing all of this because there isnt (or im not aware of) any good way understand what datasets are on workers
In which ui? Because there are two ways to do it. When clicking on artifacti url there is a popup (but has no way to change host url)
Our s3 host doesnt have port (didnt specify port in clearml.conf anywhere and upload works)
![image](https://clearml-web-assets.s3.amazonaws.com/scoold/images/TT9A...
we use Ceph Storage Cluster, interface to it is the same as S3
I dont get what I have misconfigured.
The only thing I have not added is "region" field in clearml.conf because we literally dont have, its a self hosted cluster.
You can try and replicate this s3 config I have posted earlier.
how to get rid of this auto appended line
I see in clearml-agent that it is created here
Where can i override this so that it uses uv instead of trying to install python with apt