you can pass use the compression
parameter in dataset.upload
. The supported values are:ZipFile.ZIP_STORED (no compression) ZipFile.ZIP_DEFLATED (requires zlib) ZipFile.ZIP_BZIP2 (requires bz2) ZipFile.ZIP_LZMA (requires lzma)
Note that you need to import ZipFile
beforehand: from zipfile import ZipFile
You're probably looking for ZIP_BZIP2
, but I'm not sure about that.
For pipelines there's currently no way to use different compressions. You can still use it when explicitly uploading https://clear.ml/docs/latest/docs/references/sdk/dataset/#upload
It's models not datasets in our case...
But we can also just tar the folder and return that... Was just hoping to avoid doing that
PricklyRaven28 I think then you're looking for: ZipFile.ZIP_STORED
Ok, tnx (:
We just see that taring and untaring is much faster than zip for big models
I know zip and tar.gz are supported for auto extraction. But you're looking for a setting to have artifacts compressed with tar instead of zip?