Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hey, I’M Getting The Following Error When Loading A Model Using Model.Get_Local_Copy()

Hey, I’m getting the following error when loading a model using model.get_local_copy()
… raise ValueError("Could not retrieve a local copy of model weights {}, " ValueError: Could not retrieve a local copy of model weights 4984cc389e3642b79c5819561bdeb47c, failed downloading file:///root/.clearml/venvs-builds/3.8/task_repository/my-repo.git/path/checkpoints/epoch=4.ckptWhere should this .ckpt be located? I thought the agent will automatically store those ckpts on the clearML fileserver, but it seems like it stored it inside the agents docker container? Are those ckpt files lost?

I typically can access and download artifacts of any task but didn’t try it for output_models. Now I tried to download a few of my output models and realise that I can’t load them. Any idea why?

  
  
Posted 2 years ago
Votes Newest

Answers 9


Hi ClumsyElephant70 , by default, the agent will not send these to the server. In order to do that, you'll need to set the output_uri parameter in Task.init() (or set the agent's default_output_uri in the clearml.conf file) and point it to the ClearML file server's address

  
  
Posted 2 years ago

thanks for the info, thats really bad 😬 I thought that the output_uri defaults to the fileserver 🙄

  
  
Posted 2 years ago

I can see the following using docker ps:
d5330ec8c47d allegroai/clearml-agent "/usr/agent/entrypoi…" 3 weeks ago  Up 3 weeks  clearml

I execute the following to access the container
docker exec -u root -t -i clearml /bin/bash

I went to /root/.clearml/venv-builds but it is empty

  
  
Posted 2 years ago

It's probably the daemon agent docker, not the one that was spun for your experiment

  
  
Posted 2 years ago

I assume you've executed the agent in docker mode?

  
  
Posted 2 years ago

SuccessfulKoala55 I'm currently inside the docker container to recover the ckpt files. But /root/.clearml/venvs-builds seems to be empty. Any idea where I could then find the ckpt files?

  
  
Posted 2 years ago

thanks a lot, yes it was the daemon :man-facepalming: I already could recover one checkpoint!

  
  
Posted 2 years ago

which docker did you exec into?

  
  
Posted 2 years ago

I assume it was run using a ClearML agent?

  
  
Posted 2 years ago
552 Views
9 Answers
2 years ago
one year ago
Tags