Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Everyone I Am Experiencing Issues With

Hi everyone
I am experiencing issues with delete_artifacts method. When I remove artifacts with default flag delete_from_storage=True the file actually disappears from storage but I still can see it in UI and what is more important when calling Task.get_task('my_task_id').artifacts I can see the full list of artifacts including those that were removed. All this removal process is done on running task during experiment. Would appreciate any help with that. Thanks in advance.

  
  
Posted 4 months ago
Votes Newest

Answers 8


Hi @<1643785593177509888:profile|FrustratingSeagull27> , do you have some sample code that recreates this behavior?

  
  
Posted 4 months ago

image

  
  
Posted 4 months ago

I have found an ugly workaround to finally remove artifacts but I doubt that things should work like this:

# simulate training process
for epoch in range(NUM_EPOCHS):

    # task.mark_started(force=True)

    artifact_name = f'artifact_epoch_{epoch}'

    # upload
    task.upload_artifact(
        name=artifact_name,
        artifact_object=PATH_TO_ARTIFACT,
        wait_on_upload=True
    )

    # remove
    task.delete_artifacts(artifact_names=[artifact_name])

    # remove for sure
    task = Task.get_task(task.id)
    task.delete_artifacts(artifact_names=[artifact_name], delete_from_storage=False)
  
  
Posted 4 months ago

BTW, super weird that only two of them. I expect to have all of them removed or remain but actually only the last one was removed completely. And as I said before actual files were deleted so trying to download them will result in 404 error.

  
  
Posted 4 months ago

Hi. Thanks for your response. Here is a snippet:

import os

from clearml import Task

# basic parameters
PATH_TO_ARTIFACT = '/app/artifact'
PROJECT_NAME = 'my_awesome_project'
TASK_NAME = 'my_awesome_task'
OUTPUT_URI = '
'
NUM_EPOCHS = 3


# create something to log in ClearML
if not os.path.exists(PATH_TO_ARTIFACT):
    os.makedirs(PATH_TO_ARTIFACT)

with open(os.path.join(PATH_TO_ARTIFACT, 'test.txt'), 'w') as f:
    f.write('very important data')


# create new task
task = Task.init(
    project_name=PROJECT_NAME,
    task_name=TASK_NAME,
    output_uri=OUTPUT_URI,
)

# simulate training process
for epoch in range(NUM_EPOCHS):

    artifact_name = f'artifact_epoch_{epoch}'

    # upload
    task.upload_artifact(
        name=artifact_name,
        artifact_object=PATH_TO_ARTIFACT,
        wait_on_upload=True
    )

    # remove
    task.delete_artifacts(artifact_names=[artifact_name])


# check what artifacts we have
print(task.artifacts)
  
  
Posted 4 months ago

After task completion I still can see these two files:

  
  
Posted 4 months ago

Besides, when I monitor manually the process I can see that artifacts also temporarily disappear in UI but then appear again. Looks like they are somehow automatically restored from cache or so. Can this be true and if yes how to disable such behaviour? Thanks again

  
  
Posted 4 months ago

PPS. My artifacts are folders with bunch of files that are getting archived. These doesn't happen when logging/removing pure Model objects

  
  
Posted 4 months ago