Yes I see:
"The default location for output models and other artifacts. If True is passed, the default files_server will be used for model storage. In the default location, ClearML creates a subfolder for the output. The subfolder structure is the following: <output destination name> / <project name> / <task name>.<Task ID>"
So it makes a folder in the output destination <project_name>/<task name>.<Task ID>. It is not possible to specify the full output destination right?
Okay, I am working with medical images. And when running a testing script I want to save the predictions (also big medical images of another modality). What happens when I do logger.upload_artifact(..). Then a file is copied to this folder?
We use nifty images, except for an 3D array the image also contains voxel spacing, and origin and direction in a world frame
Yep, make sense ... you can just upload them as debug samples from local files.
I guess the main difference is the context, debug samples (used for debugging) vs artifacts (might be useful from other Tasks / context)
BTW: GreasyPenguin14 you can also upload them as debug samples (when setting the output_uri, the debug samples will be uploaded to the same destination)
It is the folder the clearml creates and the folder we create ourself to store the predictions
I see... If that is the case, the only solution I can think of is manually uploading the files with StorageManager(...) then get the url, and register it as debug_media or artifact:
logger.report_media("image", "type a", iteration=iteration, url="
") task.upload_artifact('a link', artifact_object='
Yes, now I new unique folder is created per experiment where the predictions are saved. That works. The only thing is that now there is the folder that clearml makes for an experiment and the folder that saves the resuts. So two folders with artifacts per experiment. I was wondering if there was a more efficient solution and if it could be combined.
Because we are working with very big files, having them stored at multiple locations is something we try to avoid
Just so I better understand, is this for storing files as part of a dataset, or as debug samples ?
In other words can two diff processes create the exact same file (image) ?
It is for storing the predictions a trained model makes, so two different models do create slightly different images
That actually makes sense.
So how would you create exactly the same file (i.e. why do you need to manually control the upload folder, wouldn't creating a new unique folder suffice ?)