I know zip and tar.gz are supported for auto extraction. But you're looking for a setting to have artifacts compressed with tar instead of zip?
For pipelines there's currently no way to use different compressions. You can still use it when explicitly uploading https://clear.ml/docs/latest/docs/references/sdk/dataset/#upload
Ok, tnx (:
We just see that taring and untaring is much faster than zip for big models
you can pass use the compression
parameter in dataset.upload
. The supported values are:ZipFile.ZIP_STORED (no compression) ZipFile.ZIP_DEFLATED (requires zlib) ZipFile.ZIP_BZIP2 (requires bz2) ZipFile.ZIP_LZMA (requires lzma)
Note that you need to import ZipFile
beforehand: from zipfile import ZipFile
You're probably looking for ZIP_BZIP2
, but I'm not sure about that.
It's models not datasets in our case...
But we can also just tar the folder and return that... Was just hoping to avoid doing that
PricklyRaven28 I think then you're looking for: ZipFile.ZIP_STORED