Hi,

do you know how to upload pyspark dataframes with clearml as artifact?

For example, i have code:

task = Task.init(
	project_name="Try to upload pyspark df",
	task_name="ExampleTask",
)

df = spark.read.csv(path)

task.upload_artifact("my_df_name", df)

It dosen't work, but if i make toPandas() everything works fine:

pdf = df.toPandas()
task.upload_artifact("my_df_name", pdf)

The problem with this is that toPandas() is very slow and overloads RAM. We use pyspark for processing big data, exactly for this reasons: can parallelize the calculations and speed up the pipeline (or even make it calculable).
Lack of ability to upload pyspark dataframes makes clearML unusable for larger data sets and real problems :(

Do you have any solution for that?

Note:
task.upload_artifact("my_df_name", df.coalesce(1)) also dosen't work

Thanks for help!