Hi, Is There Any Way To Upload Data To A Clearml Dataset Without Compression At All? I Have Very Small Text Files That Make Up A Dataset And Compression Seems To Take Most Of The Upload Time And It Provide Almost No Benefits W.R.T Size

Unanswered

AgitatedDove14 thanks for the tip. I tried it but it still compresses the data to zip format. I suspect it is since ZIP_STORED is a constant that equals 0:

This is problematic due to line 689 in dataset.py ( https://github.com/allegroai/clearml/blob/d17903d4e9f404593ffc1bdb7b4e710baae54662/clearml/datasets/dataset.py#L689 ):
compression=compression or ZIP_DEFLATED

Since ZIP_DEFLATED=8, passing compression=0 still causes data to be compressed

  				
Posted 
	one year ago

					More
				  		
  Report
		
					HugeArcticwolf77
				
					0
					 × 1

116 Views

0 Answers

one year ago