 
			Reputation
Badges 1
30 × Eureka!all metadata that standard for clearml dataset: hashes , tempstamps and names of the 1M uploaded files
Might it be that we lost it in migration process?
nothing special in developer tools
it means they are not in ES?
only GPU and machine
Thanks. We solved it by copying all ES data again.
or we create parent - child 2 datasets splitting the set to two parts
Martin you didn't get me right. We have 1 million small files which we upload in chunks of 512 mb
what I meant is that we have 1,000,000 small files in the dataset
the files are uploaded but metadata is absent 😞
{"type": "server", "timestamp": "2023-06-21T08:26:03,816Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "${sys:es.logs.cluster_name}", "node.name": "clearml", "message": "uncaught exception in thread [process reaper (pid 218)]", "cluster.uuid": "9l7JSn6ES0upH0UZK2C8Tw", "node.id": "x3JpeJiKSWSNsz8s7Y0Kuw" ,
uncaught exception in thread [process reaper (pid 218)]
java.security.AccessControlException: access denied ("java.lang.RuntimePermi...this was Running experiment while outage happened. Plots of the other tasks are OK.
clearml-data create --name [Dataset Name] --project [Project Name] --output-uri clearml-data add --files [FILE_PATH] --id [Id] clearml-data close
as we see it the only way is to split this dataset to smaller sub-datasets
This happened after we backed up and restored the server
can upgrading to 1.11.0 help?
@<1523701070390366208:profile|CostlyOstrich36> yes. empty
restarted ES container. same. no scalars

