As the meme goes, well yes but actually no, since the input path is provided via argparse? I'm also not sure how this would help debug from the WebUI - you can't really see the contents of a zipped file/the configuration tab is too messy for such a nested configuration as the one we have. It's best suited as an artifact.
EDIT: Or am I missing something? Point being, when the remote execution begins, the entry point tries to run e.g. python train.py --config_file path/to/local/file.yaml
which then fails, since that file does not exist.
Even if we override argparse
with some arguments from ClearML (I suppose this is the idea behind the autoconnect feature?), then our yaml file contains !include
instructions, referring to other yaml files, using a relative path. Then, our config file may or may not refer to additional files (again relative path).
As of now, we internally analyze the configuration file, use StorageManager
to upload everything, check if we're running as a remote execution before doing anything, and if so, download using StorageManager
and cleanup using boto3
.