After task completion I still can see these two files:
Hi. Thanks for your response. Here is a snippet:
import os
from clearml import Task
# basic parameters
PATH_TO_ARTIFACT = '/app/artifact'
PROJECT_NAME = 'my_awesome_project'
TASK_NAME = 'my_awesome_task'
OUTPUT_URI = '
'
NUM_EPOCHS = 3
# create something to log in ClearML
if not os.path.exists(PATH_TO_ARTIFACT):
os.makedirs(PATH_TO_ARTIFACT)
with open(os.path.join(PATH_TO_ARTIFACT, 'test.txt'), 'w') as f:
f.write('very important data')
# create new task
task = Task.init(
project_name=PROJECT_NAME,
task_name=TASK_NAME,
output_uri=OUTPUT_URI,
)
# simulate training process
for epoch in range(NUM_EPOCHS):
artifact_name = f'artifact_epoch_{epoch}'
# upload
task.upload_artifact(
name=artifact_name,
artifact_object=PATH_TO_ARTIFACT,
wait_on_upload=True
)
# remove
task.delete_artifacts(artifact_names=[artifact_name])
# check what artifacts we have
print(task.artifacts)
I have found an ugly workaround to finally remove artifacts but I doubt that things should work like this:
# simulate training process
for epoch in range(NUM_EPOCHS):
# task.mark_started(force=True)
artifact_name = f'artifact_epoch_{epoch}'
# upload
task.upload_artifact(
name=artifact_name,
artifact_object=PATH_TO_ARTIFACT,
wait_on_upload=True
)
# remove
task.delete_artifacts(artifact_names=[artifact_name])
# remove for sure
task = Task.get_task(task.id)
task.delete_artifacts(artifact_names=[artifact_name], delete_from_storage=False)
BTW, super weird that only two of them. I expect to have all of them removed or remain but actually only the last one was removed completely. And as I said before actual files were deleted so trying to download them will result in 404 error.
PPS. My artifacts are folders with bunch of files that are getting archived. These doesn't happen when logging/removing pure Model
objects
Hi @<1643785593177509888:profile|FrustratingSeagull27> , do you have some sample code that recreates this behavior?
Besides, when I monitor manually the process I can see that artifacts also temporarily disappear in UI but then appear again. Looks like they are somehow automatically restored from cache or so. Can this be true and if yes how to disable such behaviour? Thanks again