{"type": "server", "timestamp": "2023-06-21T08:26:03,816Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "${sys:es.logs.cluster_name}", "node.name": "clearml", "message": "uncaught exception in thread [process reaper (pid 218)]", "cluster.uuid": "9l7JSn6ES0upH0UZK2C8Tw", "node.id": "x3JpeJiKSWSNsz8s7Y0Kuw" ,
uncaught exception in thread [process reaper (pid 218)]
java.security.AccessControlException: access denied ("java.lang.RuntimePermi...

yes

chunksize: 512 Mb

256GB in total of data

~2000 files in each chunk

so correct numbers are:

500 chunks in total

Martin you didn't get me right. We have 1 million small files which we upload in chunks of 512 mb

AgitatedDove14

78GB

what I meant is that we have 1,000,000 small files in the dataset

all metadata that standard for clearml dataset: hashes , tempstamps and names of the 1M uploaded files

as we see it the only way is to split this dataset to smaller sub-datasets

or we create parent - child 2 datasets splitting the set to two parts

clearml-data create --name [Dataset Name] --project [Project Name] --output-uri clearml-data add --files [FILE_PATH] --id [Id] clearml-data close

restarted ES container. same. no scalars

can upgrading to 1.11.0 help?

only GPU and machine

the files are uploaded but metadata is absent 😞

clearml==1.8.3