Hi RoughTiger69 ! Can you try adding the files using a python script such that we could get an exception traceback, something like this:
` from clearml import Dataset

or just use the ID of the dataset you previously created instead of creating a new one

parent_dataset = Dataset.create(dataset_name="xxxx", dataset_project="yyyyy", output_uri=" ")
parent_dataset.add_files("folder1")
parent_dataset.upload()
parent_dataset.finalize()

child_dataset = Dataset.create(dataset_name="xxxx", dataset_project="yyyyy", output_uri=" ", parent_datasets=[parent_dataset.id]) # or just use the ID of the dataset you previously created
child_dataset.add_files("folder2")
child_dataset.upload()
child_dataset.finalize() `Also, how many files are in the parent dataset?
Thanks

no, I tried either with very small files or with 20GB as the parent

It seems to work fine when the parent is on clear.ml storage (tried with toy example of data)

Tried with 1.6.0, doesn’t work

#this is the parent clearml-data create --project xxx --name yyy --output-uri `
clearml-data add folder1
clearml-data close

#this is the child, where XYZ is the parent's id
clearml-data create --project xxx --name yyy1 --parents XYZ --output-uri
clearml-data add folder2
clearml-data close
#now I get the error above `

Can you try it with clearml==1.6.0 please?
Also, can you list the exact commands you ran?

RoughTiger69 , do you have a rough estimate on the size that breaks it?

quick update, still trying to reproduce ...

I tested it again with much smaller data and it seems to work.
I am not sure what is the difference between the use-cases. it seems like something specifically about the particular (big) parent doesn’t agree with clearml…