Hi there ClearML experts
I've been trying to delete heavy project (with ~4K experiments) using the API and getting the following error:
curl \ --header "Content-Type: application/json" \ --header "Authorization: Basic $auth" \ --request POST \ --data '{"project":"f4b6cb47bb3940c9a3bf79c5c78e983d", "delete_contents":"true", "force":"true"}' \ https://<clearml-server>:8008/projects.delete
"meta": { "id": "0025d551945a4f9bb4a7d569eefb373f", "trx": "0025d551945a4f9bb4a7d569eefb373f", "endpoint": { "name": "projects.delete", "requested_version": "2.14", "actual_version": "1.0" }, "result_code": 500, "result_subcode": 100, "result_msg": "General data error (ConflictError(409, '{\"took\":941,\"timed_out\":false,\"total\":4231780,\"deleted\":0,\"batches\":1,\"version_conflicts\":1000,\"noops\":0,\"retries\":{\"bulk\":0,\"search\":0},\"throttled_millis\":0,\"requests_per_second\":-1.0,\"throttled_until_millis\":0,\"failures\":[{\"index\":\"events-log-d1bd92a3b039400cbafc60a7a5b1e52b\",\"type\":\"_doc\",\"id\":\"da2c8742125049769f4fd0fc3218c8af\",\"cause\":{\"type\":\"version_conflict_engine_exception\",\"reason\":\"[da2c8742125049769f4fd0fc3218c8af]: version conflict, required seqNo [22951076], primary term [33]. but no document was found\",\"index_uuid\":\"-awMmqjBQSGMUTEqFmglUQ\",\"shard\":\"0\", ....... {\"type\":\"version_conflict_engine_exception\",\"reason\":\"[76da0c0029d4ca408f1680b67174c44e]: version conflict, required seqNo [214671292], primary term [33]. but no document was found\",\"index_uuid\":\"Cek9t2y1Tyilh4sO5V6Aog\",\"shard\":\"0\",\"index\":\"events-training_stats_scalar-d1bd92a3b039400cbafc60a7a5b1e52b\"},\"status\":409},{\"index\":\"events-training_stats_scalar-d1bd92a3b039400cbafc60a7a5b1e52b\",\"type\":\"_doc\",\"id\":\"86ade76899afa5e09d21307ffe22d432\",\"cause\": "error_stack": null, "error_data": {} }, "data": {}
When trying to delete much smaller sub project (inside same parent project ~4K project, different project ID) getting back result 200 and all works
Any idea?