Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Fam! Sorry For The Potential Dumb Question, But I Couldn’T Find Anything On The Interwebs About It. I’M Hosting A Clearml Server On Aws, Using S3 As A Backend For Artifact Storage. I Find That Whenever I Delete Archived Artifacts In The Web App, I Get

Hi fam! Sorry for the potential dumb question, but I couldn’t find anything on the interwebs about it.

I’m hosting a ClearML server on AWS, using S3 as a backend for artifact storage. I find that whenever I delete archived artifacts in the web app, I get a popup saying that the actual files weren’t deleted from S3 (so presumably only the metadata on the server gets deleted).

I’ve confirmed that the EC2 instance hosting the ClearML server has permissions to delete files from the relevant bucket, and if I remote into the instance I can delete files using the AWS CLI. Any ideas what the problem could be? Maybe the AWS credentials aren’t propagating into the relevant docker container?

  
  
Posted 2 years ago
Votes Newest

Answers 20


(It seems like the web server doesn’t log the call to AWS, I just see this:
{SERVER IP} - - [22/Dec/2021:23:58:37 +0000] "POST /api/v2.13/models.delete_many HTTP/1.1" 200 348 " ID}/models/{MODEL ID}/general?{QUERY STRING PARAMS THAT DETERMINE TABLE APPEARANCE} {BROWSER INFO} "-"

  
  
Posted 2 years ago

Hi QuaintPelican38
We already tried to implement deleting S3 resources in the UI but got blocked by CORS.
we will implement deleting these resources through the server in a near version.
in the mean time you can use the SDK to delete entities with their artifacts.

  
  
Posted 2 years ago

. Would you have any suggestions about where I could look to debug? Maybe the docker logs of the web server?

Let me check, we had the same issue reported today, Let me double check with front-end people and get back to you

  
  
Posted 2 years ago

Ok great! I’ve actually provided a key and secret so I guess it should be working. Would you have any suggestions about where I could look to debug? Maybe the docker logs of the web server?

  
  
Posted 2 years ago

It seems like the web server doesn’t log the call to AWS, I just see this:

This points to the browser actually sending the AWS delete command. Let me check with FE tomorrow

  
  
Posted 2 years ago

QuaintPelican38 did you have a workaround for this then? Some cleanup service or similar?

  
  
Posted 2 years ago

Thanks Shay, good to know i just hadn't configured something correctly!

  
  
Posted 2 years ago

🙏 You folks are the best

  
  
Posted 2 years ago

This seems to be more complicated than what it looks like (ui/backend combination), not are not working on it, just that it might take some time as it passes control to the backend (which by design does not touch external storage points).
Maybe we should create an S3 cleanup service, listing buckets and removing if the Task ID does not exist any longer. wdyt?

  
  
Posted 2 years ago

I don't mind running the cleanup service periodically so long i have an avenue to do so. Actually, if you can build such a service, then wouldn't it make sense to just call this service whenever a task ID is deleted?

  
  
Posted 2 years ago

Hey AgitatedDove14 is there any update on this?

  
  
Posted 2 years ago

The default cleanup service should work with S3 with a correctly configured clearml service agent if I understand the workings correctly.

Yes I think you are correct

I am referring to the UI.

In that case, no 😞 . This is actually a backend server change (from the UI it should be relatively simple). Is this somehow a showstopper ?

  
  
Posted 2 years ago

No, it is just a pain to find files that have been deleted by a user, but are actually not deleted in the fileserver/s3 🙂
But no worries, nothing that is crucial.

I created this issue today, which can alleviate the pain temporarily: https://github.com/allegroai/clearml-server/issues/133

  
  
Posted 2 years ago

Thanks AgitatedDove14 ! Just to make sure I’m understanding correctly, do you mean that the ClearML Web server in https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server issues a delete command to the ClearML API server, which is then responsible for trying to delete the files in S3? And that I need to enter an AWS key/secret in the profile page of the web app here?

  
  
Posted 2 years ago

I get a popup saying that the actual files weren’t deleted from S3 (so presumably only the metadata on the server gets deleted).

Hi QuaintPelican38
The browser client actual issues the delete "command", (the idea is separation of the meta-data and data, e.g. artifacts). That means you have to provide the key/secret to the UI (see profile page)

  
  
Posted 2 years ago

in 

 issues a delete command to the ClearML API server,...

almost, it issues the boto S3 delete commands (directly to the S3 server, not through the cleaml-server)

And that I need to enter an AWS key/secret in the profile page of the web app here? (edited)

correct

  
  
Posted 2 years ago

Hi UnevenDolphin73 sorry for the slow reply, been on leave!

We don’t have a solution right now, but if there’s no fix to the frontend in the near future we’ll probably try to write a script that queries the ClearML API for all artefacts, queries S3 for all artefacts, and figures out orphaned artefacts to delete.

  
  
Posted 2 years ago

hi DeterminedCrab71 AgitatedDove14 do you have any updates on this? I have accumulated a lot of data (artifacts and datasets) on my bucket and I'll need a way to delete them whenever I remove my tasks

  
  
Posted 2 years ago

I am referring to the UI. The default cleanup service should work with S3 with a correctly configured clearml service agent if I understand the workings correctly.

  
  
Posted 2 years ago

Hi ReassuredTiger98
Are you referring to the UI (as much as I understand there was an improvement, but generally speaking, it still needs the users to have the S3 credentials in the UI client, not backend)
Or are you asking on the cleanup service ?

  
  
Posted 2 years ago
1K Views
20 Answers
2 years ago
one year ago
Tags