@<1523701087100473344:profile|SuccessfulKoala55> I just did the following (everything locally, not with clearml-agent)
- Set my credentials and S3 endpoint to A
- Run a task with Task.init() and save a debug sample to S3
- Abort the task
- Change my credentials and S3 endpoint to B
- Restart the taskThe result are lingering files in A that seem not to be associated with the task. I would expect ClearML to instead error the task or to track the lingering files somewhere, so they can marked for later deletion.
For reference: I tested this by looking at the object browser of MinIO.
Or does MinIO delay deletion somehow? Deleting a task via the web interface also does not result in deletion of debug samples on MinIO
Hi @<1523701868901961728:profile|ReassuredTiger98> , I'm not sure I understand - overriding meaning adding Task.init() at the start of the code?
However, this seems like a pretty edge-case to me - why would you do that on a regular basis?
For me this is how I use ClearML as tensorboard replacement. To start some debug runs before adding it to a clearml-agent queue. For me this seems like the most common usage case or am I missunderstanding ClearML?
What are A and B? servers? where do you update the URI for the SDK to store to?
Alright, that s unfortunate. But thank you very much!
As far as ClearML system objects are concerned, they are overwritten and older objects are usually replaced. In come cases, they can still exist in the system, but are still associated with the task (so the same thing will apply to them as is done for all objects related to this task when you delete it, for example)
In that case, you will need to delete these in your code
Yes, from the documentation:
Creates a new Task (experiment) if:
The Task never ran before. No Task with the same task_name and project_name is stored in ClearML Server.
The Task has run before (the same task_name and project_name), and (a) it stored models and / or artifacts, or (b) its status is Published , or (c) it is Archived.
A new Task is forced by calling Task.init with reuse_last_task_id=False.
Otherwise, the already initialized Task object for the same task_name and project_name is returned, or, when being executed remotely on a clearml-agent, the task returned is the existing task from the backend.
So if I start my script twice, the scalars/debug samples etc. are overriden. But what happens to the previous ones?
If you simply aborted the task and than re-run, overriding previous uploads, than the system will not delete these objects currently. However, this seems like a pretty edge-case to me - why would you do that on a regular basis?
I use this snippet:
Logger.current_logger().set_default_upload_destination(
"
" # or
)