And another question regarding squashing: sometimes I get the following error: FileNotFoundError: [Errno 2] No such file or directory: '/home/vscode/.clearml/cache/storage_manager/datasets/ds_4f3436f7b3ef484f8148a9c25a444ee5/file.ann
— why is there an attempt to access the file locally?
Ah, I wasn’t aware this is possible! Yes, perfect, thanks a lot!
Hi SmallGiraffe94 ! Dataset.squash
doesn't set as parents the ids you specify in dataset_ids
. Also, notice that the current behaviour of squash
is pulling the files from all the datasetes from a temp folder and re-uploading them. How about creating a new dataset with id1, id2, id3
as parents Dataset.create(..., parent_datasets=[id1, id2, id3])
instead? Would this fit your usecase?