what I meant is that we have 1,000,000 small files in the dataset

AgitatedDove14

Might it be that we lost it in migration process?

it means they are not in ES?

{"type": "server", "timestamp": "2023-06-21T08:26:03,816Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "${sys:es.logs.cluster_name}", "node.name": "clearml", "message": "uncaught exception in thread [process reaper (pid 218)]", "cluster.uuid": "9l7JSn6ES0upH0UZK2C8Tw", "node.id": "x3JpeJiKSWSNsz8s7Y0Kuw" ,
uncaught exception in thread [process reaper (pid 218)]
java.security.AccessControlException: access denied ("java.lang.RuntimePermi...

this was Running experiment while outage happened. Plots of the other tasks are OK.

This happened after we backed up and restored the server

clearml-data create --name [Dataset Name] --project [Project Name] --output-uri clearml-data add --files [FILE_PATH] --id [Id] clearml-data close

it is selfhosted

yes

Thanks. We solved it by copying all ES data again.

only GPU and machine

or we create parent - child 2 datasets splitting the set to two parts

can upgrading to 1.11.0 help?

chunksize: 512 Mb

78GB

nothing special in developer tools

~2000 files in each chunk

the files are uploaded but metadata is absent 😞

500 chunks in total

all metadata that standard for clearml dataset: hashes , tempstamps and names of the 1M uploaded files

Martin you didn't get me right. We have 1 million small files which we upload in chunks of 512 mb

restarted ES container. same. no scalars

so correct numbers are:

256GB in total of data

clearml==1.8.3

@<1523701070390366208:profile|CostlyOstrich36> yes. empty

as we see it the only way is to split this dataset to smaller sub-datasets