I already found the source code and i modified it as needed.
How can I now get this info from Task that is created when Dataset is created?
Couldnt find anything like clearml.Dataset(id=id).get_size()

ClearML team should really write up some tutorial about this. I see this question weekly now. The short answer on what we did when we migrated servers was to wite a python script that takes data from clearml mongodb(stores tasks and datasets) and elastic (stores debug image urls, logs, scalars) and migrate them to other clearml instance databases

It is also possible to just make a copy of all the database files and move them to another server

Is is even known if the bug is fixed on that version?

Is fileserver folder needed for successful backup?

This is the link generated

You can check out boto3 python client (This is what we use to download / upload all S3 stuff), but minio-client probably already uses it under the hood.
We also use aws cli to do some downloading, it is way faster than python.

Regarding pdfs, yes, you have no choice but to preprocess it

removing it doesnt fix the problem

also, if I try to set the url to None it auto replaces it with None : None

What you want is to have a service script that cleans up archived tasks, here is what we used: None

ok, I found it.
Are S3 links supported?

How can I do that?
I need to save the original hash, otherwise I lose all trackability to about 2k experiments

We fixed the issue, thanks, had to update everything to latest.

will it be appended in clearml?
"s3" is part of domain to the host

s

This is what I see on fresh clearml
Where all my mounts are on /mnt/data/clearml-server instead of /opt/clearml

Specifying it like this, gets me different error:

Exception has occurred: ValueError

• Insufficient permissions (delete failed) for None
  botocore.exceptions.ClientError: An error occurred (IllegalLocationConstraintException) when calling the DeleteObject operation: The me-south-1 location constraint is incompatible for the region specific endpoint this request was sent to.

During handling of the above exception, another exception occurred:

File "/home/ma...

WebApp: 1.14.1-451 • Server: 1.14.1-451 • API: 2.28

File is written

7 out of 30 GB is currently used and is quite stable

Adding bucket in clearml.conf causes the same error: clearml.storage - ERROR - Failed uploading: Could not connect to the endpoint URL: " None "


![image](https://clearml-web-assets.s3.amazonaws.com/scoold/images/TT...

I purged all docker images and it still doesnt seem right
I see no side panel and it doesnt ask for login name

im also batch uploading, maybe thats the problem?

• The dataset is about 1TB containing 1 million files
• I dont have the SSD space locally to do the upload
• So i download a part of the dataset, use add_files() and then upload() to that batch
• Upload the dataset
  I noticed that each batch is slower and slower

I tried it with port, but still having the same issue
Tried it with/without secure and multipart

I dont have a region. I guess I will wait till tomarrow then?

@<1523701070390366208:profile|CostlyOstrich36> Updated webserver and the problem still persists
This is the new stack:
WebApp: 1.15.1-478 • Server: 1.14.1-451 • API: 2.28

notice, we didnt update API (we had running experiments)

clearml.conf is a fresh one i did clearml-init to make sure

Im doing all of this because there isnt (or im not aware of) any good way understand what datasets are on workers