Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello! I'M Trying To Make A Simple Eval.Py Script That Will Go Pull The Best Model Of A Given Experiment, Load It Locally And Evaluate It On Whatever Data I Give. Question 1: Is There A Standard Way Documented Somewhere To Do This? Question 2: I'M Loadin

Hello!
I'm trying to make a simple eval.py script that will go pull the best model of a given experiment, load it locally and evaluate it on whatever data I give.
Question 1: Is there a standard way documented somewhere to do this?
Question 2: I'm loading the previous config to rebuild the model as follows,
` task = Task.get_task(task_id=args.task_id)

# Get config
cfg = task.get_parameters_as_dict()['General'] `but this command casts everything in the config as strings, so I can't use it without casting everything back .... Is there a better way to keep the original typing on the config?

Thanks in advance!

  
  
Posted one year ago
Votes Newest

Answers 13


Hi MistakenDragonfly51 , regarding your questions:
ClearML has a model repository built in. You can load an input model using InputModel module ( https://clear.ml/docs/latest/docs/references/sdk/model_inputmodel ). Also, you can fetch the models of an experiment using Task.get_models() - https://clear.ml/docs/latest/docs/references/sdk/task#get_models Can you elaborate on how this config looks in the UI when you view it?

  
  
Posted one year ago

Hi MistakenDragonfly51
Notice that Models are their own entity, you can query them based on tags/projects/names etc.
Querying and getting Models is done by Model class:
https://clear.ml/docs/latest/docs/references/sdk/model_model#modelquery_models

task.get_models()

is always empty. (edited)

How come there are no Models on the Task? (in other words how come this is empty?)

  
  
Posted one year ago

For now, I retrieve and load the model as follows (pytorch lightning)clearml_model = task.models['output'][-1] model_path = clearml_model.get_local_copy() main_loop = LightningModel.load_from_checkpoint(checkpoint_path=model_path)
Not sure how InputModel would help and task.get_models() is always empty.

  
  
Posted one year ago

Wait, that makes no sense to me. The API from python and the API from the UI are getting the same data from the backend ...
What are you getting with?
from clearml import Task task = Task.get_task(task_id=<put task id here>) print(task.models)

  
  
Posted one year ago

That's a very good question.
They are visible in the UI.
And accessible in the way I mentioned above.

Any idea?

  
  
Posted one year ago

sorry, they both return the same. was a typo in my test for task.models vs task.get_models()

  
  
Posted one year ago

  1. in the UI it does have the proper types.
  
  
Posted one year ago

Oh that makes sense.
So now you can just get the models as dict as well (basically clearml allows you to access them both as a list, so it is easy to get the last created, and as dict so you can match the filenames)
This one will get the list of models
print(task.models["output"].keys())Now you can just pick the best one
model = task.models["output"]["epoch13-..."] my_model_file = model.get_local_copy()

  
  
Posted one year ago

Well it seems we forgot that one 😞 I'll quickly make sure it is there.
As a quick solution (no need to upgrade)
task.models["output"]._models.keys()

  
  
Posted one year ago

Fixed in pip install clearml==1.8.1rc0 🙂

  
  
Posted one year ago

That might be me, let me check...

  
  
Posted one year ago

héhé thanks

  
  
Posted one year ago

I'm assuming it's in a later version than 1.7.1 (the one I'm running).

But I guess for my question #1 I was doing it fine. Do you have any idea for #2?
Old bug and I should update maybe?

  
  
Posted one year ago
655 Views
13 Answers
one year ago
one year ago
Tags