BTW: GreasyPenguin14 you can also upload them as debug samples (when setting the output_uri, the debug samples will be uploaded to the same destination)
https://github.com/allegroai/clearml/blob/6b9297660e0ed83a77bce3da2fab384c552206fd/examples/reporting/image_reporting.py#L21
GreasyPenguin14 you mean the artifacts/models ?
Okay, so I have to first save the generated image somewhere and then with logger.report_media it is copied to the folder?
Yes, now I new unique folder is created per experiment where the predictions are saved. That works. The only thing is that now there is the folder that clearml makes for an experiment and the folder that saves the resuts. So two folders with artifacts per experiment. I was wondering if there was a more efficient solution and if it could be combined.
Yes I see:
"The default location for output models and other artifacts. If True is passed, the default files_server will be used for model storage. In the default location, ClearML creates a subfolder for the output. The subfolder structure is the following: <output destination name> / <project name> / <task name>.<Task ID>"
So it makes a folder in the output destination <project_name>/<task name>.<Task ID>. It is not possible to specify the full output destination right?
Is it possible to get the folder with the artifacts/models? (edited)
You can directly get the artifacts/models url then deduce the foldertask = Task.get_task('my_task_id') print(task.artifacts['my artifact'].url)
You can however change the prefix, and you can always have access to these links.
Any reason for controlling the exact output destination ?
(BTW: You can manually upload via StorageManager, and then register the uploaded link)
I see, will it be possible in the future to directly write custom/not supported formats to the folder? Because we are working with very big files, having them stored at multiple locations is something we try to avoid
It is for storing the predictions a trained model makes, so two different models do create slightly different images
It is for storing the predictions a trained model makes, so two different models do create slightly different images
That actually makes sense.
So how would you create exactly the same file (i.e. why do you need to manually control the upload folder, wouldn't creating a new unique folder suffice ?)
It is the folder the clearml creates and the folder we create ourself to store the predictions
Because we are working with very big files, having them stored at multiple locations is something we try to avoid
Just so I better understand, is this for storing files as part of a dataset, or as debug samples ?
In other words can two diff processes create the exact same file (image) ?
. It is not possible to specify the full output destination right?
Correct 😞
Okay, I am working with medical images. And when running a testing script I want to save the predictions (also big medical images of another modality). What happens when I do logger.upload_artifact(..). Then a file is copied to this folder?
We use nifty images, except for an 3D array the image also contains voxel spacing, and origin and direction in a world frame
Yep, make sense ... you can just upload them as debug samples from local files.
I guess the main difference is the context, debug samples (used for debugging) vs artifacts (might be useful from other Tasks / context)
https://github.com/allegroai/clearml/blob/6b9297660e0ed83a77bce3da2fab384c552206fd/examples/reporting/image_reporting.py#L36
It is the folder the clearml creates and the folder we create ourself to store the predictions
I see... If that is the case, the only solution I can think of is manually uploading the files with StorageManager(...) then get the url, and register it as debug_media or artifact:logger.report_media("image", "type a", iteration=iteration, url="
") task.upload_artifact('a link', artifact_object='
')
I see, however it looks like medical images are not supported. We use nifty images, except for an 3D array the image also contains voxel spacing, and origin and direction in a world frame
I see task.get_output_destination() returns a url like http://localhost:8081 . Is it possible to get the folder with the artifacts/models?
So two folders with artifacts per experiment. I was wondering if there was a more efficient solution and if it could be combined.
Not sure I follow, is two subfolders for two different things are not they it is supposed to be ?
Old legacy code that has its own folder structure per experiment. I can also do it the other way around. Does task.get_output_destination() return the folder including project name and <task_name>.<task_id>?