Thanks for the reply CostlyOstrich36 !
Does the task read/use the cache_dir directly? It's fine for it to be a cache and then removed from the fileserver; if users want the data to stay they will use the ClearML Dataset 🙂

The S3 solution is bad for us since we have to create a folder for each task (before the task is created), and hope it doesn't get overwritten by the time it executes.

Argument augmentation - say I run my code with python train.py my_config.yaml -e admin.env , then ClearML logs the my_config.yaml and -e admin.env as arguments for execute_remotely . I'd like to perhaps augment these to be config.yaml and remove the -e admin.env flag.