Well - that will convert it to a binary pickle format but not as parquet -
since the artifact will be accessed from other platforms we want to use parquet
Hi OutrageousSheep60 , can you try with auto_pickle=True when uploading the artifact?
CostlyOstrich36 - but we will use any method that will allow us to save the files as parquet.
We are not yet using clearml Dataset - i'm not sure if this is a solution
Using the https://allegro.ai/clearml/docs/rst/references/clearml_python_ref/task_module/task_task.html?highlight=upload_artifact#clearml.task.Task.upload_artifact method. It works well, but only saves it as a csv (which is very problematic since when loading the artifact none of the data types of the columns are preserved...)
Thx CostlyOstrich36 for your reply
Can't see the reverence to parquet . we are currently using the above functionality , but the pd.DataFrame is only saved as csv compressed by gz
Hi!
I think the example here should help you.
https://github.com/allegroai/clearml/blob/master/examples/reporting/pandas_reporting.py#L19
Together with this
https://github.com/allegroai/clearml/blob/master/examples/reporting/artifacts.py
Tell me if it helped 🙂