Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Hi, I'M Conceptually Struggling To Understand How To Get My Models To Be Saved Permanently By My Self-Hosted Clearml Server. As It Stands Now, When I Train A Model Using A Clearml-Agent, The Model Checkpoints Are Being Logged By Clearml In The Dashboard,

Hi, I'm conceptually struggling to understand how to get my models to be saved permanently by my self-hosted ClearML server. As it stands now, when I train a model using a clearml-agent, the model checkpoints are being logged by ClearML in the dashboard, but the checkpoint path is just the path to the checkpoint file within the cached venv that the agent was running in (eg. /home/username/.clearml/venvs-builds/3.8/task_repository/wl-emd.git/logs/WordSpanEMD6_32/version_0/checkpoints/epoch=9-step=23499.ckpt). Files within that venv are are not persistent, and my models are disappearing every time I rerun a task that reuses that venv build. So when I attempt to load the model for inference, I get an error that the model path doesn't exist (eg. InputModel(model_id=model_id).get_weights(True) raises missing file error). Surely I'm missing something simple here right? Thanks for any help!

Posted 2 years ago
Votes Newest

Answers 2

Try adding the following to your Task.init()

task = Task.init(project_name='<PROJECT_NAME>', task_name='<TASK_NAME>', output_uri=True)
Another option that will solve the issue as well is editing your ~/clearml.conf and changing the following section sdk.development.default_output_uri to point to your fileserver/hosting

Posted 2 years ago

Buried in documentation, I missed that completely. Thanks!

Posted 2 years ago
2 Answers
2 years ago
one year ago