Hi WorriedSwan6 , ClearML uses ElasticSearch & MongoDB as databases for all of that information. I suggest checking online for backup procedures of these databases in K8s

Hey CostlyOstrich36 , thx for the reply.
Can you advice if you think this will be the right path to go:

• Mongodb backup via mongodump to s3,
• Elasticsearch Install the S3 repository plugin and create a snapshot.The above can be triggered with cronjob once a week.

On restore to a fresh cluster, I will install clearml helm chart, and:

• Mongodb will download the file from S3 and use mongorestore
• Elasticsearch will download the snapshot from s3 and use:

curl -X POST "localhost:9200/_snapshot/snapshot-repo/<snapshot_name>/_restore" -H "Content-Type: application/json" -d '{
  "indices": "<index_pattern>",
  "ignore_unavailable": true,
  "include_global_state": false
}'

?

At the moment I tried to just copy-past the volumes from one clearml to another (for both mongodb and elasticsearch), and I can see the projects and tasks, but not the CONFIGURATION, ARTIFACTS, SCALARS or PLOTS (shows as "no log data" \ no chart data")