Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Quick Question: Does Overriding A Task With Task.Init() Do A Proper Delete Of Debug Samples At An Fileserver Or S3 Host?

Quick question: Does overriding a task with Task.init() do a proper delete of debug samples at an fileserver or S3 host?

Posted 9 months ago
Votes Newest

Answers 12

Hi @<1523701868901961728:profile|ReassuredTiger98> , I'm not sure I understand - overriding meaning adding Task.init() at the start of the code?

Posted 9 months ago

As far as ClearML system objects are concerned, they are overwritten and older objects are usually replaced. In come cases, they can still exist in the system, but are still associated with the task (so the same thing will apply to them as is done for all objects related to this task when you delete it, for example)

Posted 9 months ago

Yes, from the documentation:

Creates a new Task (experiment) if:

    The Task never ran before. No Task with the same task_name and project_name is stored in ClearML Server.

    The Task has run before (the same task_name and project_name), and (a) it stored models and / or artifacts, or (b) its status is Published , or (c) it is Archived.

    A new Task is forced by calling Task.init with reuse_last_task_id=False.

Otherwise, the already initialized Task object for the same task_name and project_name is returned, or, when being executed remotely on a clearml-agent, the task returned is the existing task from the backend.

So if I start my script twice, the scalars/debug samples etc. are overriden. But what happens to the previous ones?

Posted 9 months ago

@<1523701087100473344:profile|SuccessfulKoala55> I just did the following (everything locally, not with clearml-agent)

  • Set my credentials and S3 endpoint to A
  • Run a task with Task.init() and save a debug sample to S3
  • Abort the task
  • Change my credentials and S3 endpoint to B
  • Restart the taskThe result are lingering files in A that seem not to be associated with the task. I would expect ClearML to instead error the task or to track the lingering files somewhere, so they can marked for later deletion.

For reference: I tested this by looking at the object browser of MinIO.

Posted 9 months ago

Thank you very much!

Posted 9 months ago

If you simply aborted the task and than re-run, overriding previous uploads, than the system will not delete these objects currently. However, this seems like a pretty edge-case to me - why would you do that on a regular basis?

Posted 9 months ago

Or does MinIO delay deletion somehow? Deleting a task via the web interface also does not result in deletion of debug samples on MinIO

Posted 9 months ago

What are A and B? servers? where do you update the URI for the SDK to store to?

Posted 9 months ago

However, this seems like a pretty edge-case to me - why would you do that on a regular basis?

For me this is how I use ClearML as tensorboard replacement. To start some debug runs before adding it to a clearml-agent queue. For me this seems like the most common usage case or am I missunderstanding ClearML?

Posted 9 months ago

I use this snippet:

" # or 

Posted 9 months ago

In that case, you will need to delete these in your code

Posted 9 months ago

Alright, that s unfortunate. But thank you very much!

Posted 9 months ago
12 Answers
9 months ago
9 months ago