Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello! I'M Trying To Make A Simple Eval.Py Script That Will Go Pull The Best Model Of A Given Experiment, Load It Locally And Evaluate It On Whatever Data I Give. Question 1: Is There A Standard Way Documented Somewhere To Do This? Question 2: I'M Loadin

Hello!
I'm trying to make a simple eval.py script that will go pull the best model of a given experiment, load it locally and evaluate it on whatever data I give.
Question 1: Is there a standard way documented somewhere to do this?
Question 2: I'm loading the previous config to rebuild the model as follows,
` task = Task.get_task(task_id=args.task_id)

# Get config
cfg = task.get_parameters_as_dict()['General'] `but this command casts everything in the config as strings, so I can't use it without casting everything back .... Is there a better way to keep the original typing on the config?

Thanks in advance!

  
  
Posted 2 years ago
Votes Newest

Answers 13


sorry, they both return the same. was a typo in my test for task.models vs task.get_models()

  
  
Posted 2 years ago

I'm assuming it's in a later version than 1.7.1 (the one I'm running).

But I guess for my question #1 I was doing it fine. Do you have any idea for #2?
Old bug and I should update maybe?

  
  
Posted 2 years ago

Hi MistakenDragonfly51 , regarding your questions:
ClearML has a model repository built in. You can load an input model using InputModel module ( https://clear.ml/docs/latest/docs/references/sdk/model_inputmodel ). Also, you can fetch the models of an experiment using Task.get_models() - https://clear.ml/docs/latest/docs/references/sdk/task#get_models Can you elaborate on how this config looks in the UI when you view it?

  
  
Posted 2 years ago

Fixed in pip install clearml==1.8.1rc0 🙂

  
  
Posted 2 years ago

  1. in the UI it does have the proper types.
  
  
Posted 2 years ago

Oh that makes sense.
So now you can just get the models as dict as well (basically clearml allows you to access them both as a list, so it is easy to get the last created, and as dict so you can match the filenames)
This one will get the list of models
print(task.models["output"].keys())Now you can just pick the best one
model = task.models["output"]["epoch13-..."] my_model_file = model.get_local_copy()

  
  
Posted 2 years ago

Hi MistakenDragonfly51
Notice that Models are their own entity, you can query them based on tags/projects/names etc.
Querying and getting Models is done by Model class:
https://clear.ml/docs/latest/docs/references/sdk/model_model#modelquery_models

task.get_models()

is always empty. (edited)

How come there are no Models on the Task? (in other words how come this is empty?)

  
  
Posted 2 years ago

héhé thanks

  
  
Posted 2 years ago

Well it seems we forgot that one 😞 I'll quickly make sure it is there.
As a quick solution (no need to upgrade)
task.models["output"]._models.keys()

  
  
Posted 2 years ago

That might be me, let me check...

  
  
Posted 2 years ago

For now, I retrieve and load the model as follows (pytorch lightning)clearml_model = task.models['output'][-1] model_path = clearml_model.get_local_copy() main_loop = LightningModel.load_from_checkpoint(checkpoint_path=model_path)
Not sure how InputModel would help and task.get_models() is always empty.

  
  
Posted 2 years ago

Wait, that makes no sense to me. The API from python and the API from the UI are getting the same data from the backend ...
What are you getting with?
from clearml import Task task = Task.get_task(task_id=<put task id here>) print(task.models)

  
  
Posted 2 years ago

That's a very good question.
They are visible in the UI.
And accessible in the way I mentioned above.

Any idea?

  
  
Posted 2 years ago
1K Views
13 Answers
2 years ago
one year ago
Tags