Hi All!
I'm trying to easily show and query (with SQL) Parquet files stored by ClearML in an S3 storage.
I've tried to integrate DuckDB, but for large chunked files of several GB, it's really slow compared to the efficiency DuckDB has in directly querying standard Parquet files (Maybe because in ClearML storage, they are zipped?).
Do you know if an official integration exists, or if there is another tool/method to do that?
Thanks!
Thanks for the reply, so I have to download the file before, I hoped it would be read on the fly.
Hi @<1874989039501709312:profile|LividDragonfly0> , are you simply reading the files from your code? There is not specific integration, but simply accessing downloaded files should work
