Hi, I'm conceptually struggling to understand how to get my models to be saved permanently by my self-hosted ClearML server. As it stands now, when I train a model using a clearml-agent, the model checkpoints are being logged by ClearML in the dashboard, but the checkpoint path is just the path to the checkpoint file within the cached venv that the agent was running in (eg. /home/username/.clearml/venvs-builds/3.8/task_repository/wl-emd.git/logs/WordSpanEMD6_32/version_0/checkpoints/epoch=9-step=23499.ckpt). Files within that venv are are not persistent, and my models are disappearing every time I rerun a task that reuses that venv build. So when I attempt to load the model for inference, I get an error that the model path doesn't exist (eg. InputModel(model_id=model_id).get_weights(True) raises missing file error). Surely I'm missing something simple here right? Thanks for any help!
Try adding the following to your Task.init()
task = Task.init(project_name='<PROJECT_NAME>', task_name='<TASK_NAME>', output_uri=True)
Another option that will solve the issue as well is editing your ~/clearml.conf and changing the following section sdk.development.default_output_uri to point to your fileserver/hosting
Buried in documentation, I missed that completely. Thanks!
