Hi!
I think the example here should help you.
https://github.com/allegroai/clearml/blob/master/examples/reporting/pandas_reporting.py#L19
Together with this
https://github.com/allegroai/clearml/blob/master/examples/reporting/artifacts.py
Tell me if it helped 🙂
Thx CostlyOstrich36 for your reply
Can't see the reverence to parquet . we are currently using the above functionality , but the pd.DataFrame is only saved as csv compressed by gz
Using the https://allegro.ai/clearml/docs/rst/references/clearml_python_ref/task_module/task_task.html?highlight=upload_artifact#clearml.task.Task.upload_artifact method. It works well, but only saves it as a csv (which is very problematic since when loading the artifact none of the data types of the columns are preserved...)
CostlyOstrich36 - but we will use any method that will allow us to save the files as parquet.
We are not yet using clearml Dataset - i'm not sure if this is a solution
Hi OutrageousSheep60 , can you try with auto_pickle=True when uploading the artifact?
Well - that will convert it to a binary pickle format but not as parquet -
since the artifact will be accessed from other platforms we want to use parquet