Okay, I am working with medical images. And when running a testing script I want to save the predictions (also big medical images of another modality). What happens when I do logger.upload_artifact(..). Then a file is copied to this folder?

Is it possible to get the folder with the artifacts/models? (edited)

You can directly get the artifacts/models url then deduce the folder
task = Task.get_task('my_task_id') print(task.artifacts['my artifact'].url)

Old legacy code that has its own folder structure per experiment. I can also do it the other way around. Does task.get_output_destination() return the folder including project name and <task_name>.<task_id>?

I see task.get_output_destination() returns a url like http://localhost:8081 . Is it possible to get the folder with the artifacts/models?

. It is not possible to specify the full output destination right?

Correct 😞

It is the folder the clearml creates and the folder we create ourself to store the predictions

Yes I see:

"The default location for output models and other artifacts. If True is passed, the default files_server will be used for model storage. In the default location, ClearML creates a subfolder for the output. The subfolder structure is the following: <output destination name> / <project name> / <task name>.<Task ID>"

So it makes a folder in the output destination <project_name>/<task name>.<Task ID>. It is not possible to specify the full output destination right?

It is for storing the predictions a trained model makes, so two different models do create slightly different images

That actually makes sense.
So how would you create exactly the same file (i.e. why do you need to manually control the upload folder, wouldn't creating a new unique folder suffice ?)

So two folders with artifacts per experiment. I was wondering if there was a more efficient solution and if it could be combined.

Not sure I follow, is two subfolders for two different things are not they it is supposed to be ?

It is for storing the predictions a trained model makes, so two different models do create slightly different images

We use nifty images, except for an 3D array the image also contains voxel spacing, and origin and direction in a world frame

Yep, make sense ... you can just upload them as debug samples from local files.
I guess the main difference is the context, debug samples (used for debugging) vs artifacts (might be useful from other Tasks / context)
https://github.com/allegroai/clearml/blob/6b9297660e0ed83a77bce3da2fab384c552206fd/examples/reporting/image_reporting.py#L36

I see, will it be possible in the future to directly write custom/not supported formats to the folder? Because we are working with very big files, having them stored at multiple locations is something we try to avoid

Yes

yep

Correct (copied == uploaded)

BTW: GreasyPenguin14 you can also upload them as debug samples (when setting the output_uri, the debug samples will be uploaded to the same destination)
https://github.com/allegroai/clearml/blob/6b9297660e0ed83a77bce3da2fab384c552206fd/examples/reporting/image_reporting.py#L21

Task.init(..., output_uri='s3://...')

You can however change the prefix, and you can always have access to these links.
Any reason for controlling the exact output destination ?
(BTW: You can manually upload via StorageManager, and then register the uploaded link)

I see, however it looks like medical images are not supported. We use nifty images, except for an 3D array the image also contains voxel spacing, and origin and direction in a world frame

Yes, now I new unique folder is created per experiment where the predictions are saved. That works. The only thing is that now there is the folder that clearml makes for an experiment and the folder that saves the resuts. So two folders with artifacts per experiment. I was wondering if there was a more efficient solution and if it could be combined.

It is the folder the clearml creates and the folder we create ourself to store the predictions

I see... If that is the case, the only solution I can think of is manually uploading the files with StorageManager(...) then get the url, and register it as debug_media or artifact:
logger.report_media("image", "type a", iteration=iteration, url=" ") task.upload_artifact('a link', artifact_object=' ')

GreasyPenguin14 you mean the artifacts/models ?

Because we are working with very big files, having them stored at multiple locations is something we try to avoid

Just so I better understand, is this for storing files as part of a dataset, or as debug samples ?
In other words can two diff processes create the exact same file (image) ?

Okay, so I have to first save the generated image somewhere and then with logger.report_media it is copied to the folder?