but can it NOT use /tmp for this i’m merging about 100GB
You mean to configure your Temp folder for when squashing ?
you can do hack the following:
` import tempfile
tempfile.tempdir = "/my/new/temp"
tempfile.tempdir = None `But regradless I think this is worth a GitHub issue with feature request, to set the temp folder///
creating a dataset with parents worked very well and produced great visuals on the UI!
I tried the squash solution, however this somehow caused a download of all the datasets into my
so this actually works, kind or like git squash, bottom line it will repackage the data from all the different versions into one new version. This means downloading the data from all squashed versions, then repackaging it into a single new version. Make sense ?
"Hi, love what you guys did with the new datasets!" Thanks 🙂 !
you can squash the datasets together : it will result in the creation of a child dataset, that will contain its parents data merged together. Note that there will be no duplicate upload of the parents data : when a dataset inherits from parents datasets, it receives the references to the data uploaded by the parents.
you can also create a new dataset and specify some parents dataset using the -- parents parameter. the behavior will be the same