only GPU and machine

restarted ES container. same. no scalars

This happened after we backed up and restored the server

it is selfhosted

it means they are not in ES?

Thanks. We solved it by copying all ES data again.

yes

can upgrading to 1.11.0 help?

clearml-data create --name [Dataset Name] --project [Project Name] --output-uri clearml-data add --files [FILE_PATH] --id [Id] clearml-data close

nothing special in developer tools

AgitatedDove14

256GB in total of data

all metadata that standard for clearml dataset: hashes , tempstamps and names of the 1M uploaded files

so correct numbers are:

or we create parent - child 2 datasets splitting the set to two parts

chunksize: 512 Mb

Martin you didn't get me right. We have 1 million small files which we upload in chunks of 512 mb

{"type": "server", "timestamp": "2023-06-21T08:26:03,816Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "${sys:es.logs.cluster_name}", "node.name": "clearml", "message": "uncaught exception in thread [process reaper (pid 218)]", "cluster.uuid": "9l7JSn6ES0upH0UZK2C8Tw", "node.id": "x3JpeJiKSWSNsz8s7Y0Kuw" ,
uncaught exception in thread [process reaper (pid 218)]
java.security.AccessControlException: access denied ("java.lang.RuntimePermi...

this was Running experiment while outage happened. Plots of the other tasks are OK.

clearml==1.8.3

@<1523701070390366208:profile|CostlyOstrich36> yes. empty

as we see it the only way is to split this dataset to smaller sub-datasets

78GB

~2000 files in each chunk

the files are uploaded but metadata is absent 😞

Might it be that we lost it in migration process?

what I meant is that we have 1,000,000 small files in the dataset

500 chunks in total