Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
I Am Not Familiar With Pytorch, But Is It Expected That So Many “Models” Are Created? These Are Being Repeated As Well For A Single Task (This Is Training A T5_Model With Transformers):

I am not familiar with pytorch, but is it expected that so many “models” are created? These are being repeated as well for a single task (this is training a t5_model with transformers):

  
  
Posted 2 years ago
Votes Newest

Answers 13


Can you point me at relevant code in ClearML for the autoconnect so that I can understand exactly what's happening

  
  
Posted 2 years ago

these are being repeated as well for a single task (this is training a t5_model with transformers): (edited)

Seems like someone is storing lots of files with torch.save that ClearML automatically logs.
You can disable the autolog:
task = Task.init(..., auto_connect_frameworks={'pytorch': False})

  
  
Posted 2 years ago

Yeah concerns make sense.

The underlying root issue is unnecessary models being added or at least what I think are unnecessary and even happening when you load a model to test.

Do people use ClearML with huggingface transformers? The code is std transformers code.

Will create an issue.

  
  
Posted 2 years ago

Does a pipeline step behave differently?

  
  
Posted 2 years ago

Maybe two thing here:

If Task.init() is called in an already running task, don’t reset auto_connect_frameworks? (if i am understanding the behaviour right) Option to disable these in the clearml.conf

  
  
Posted 2 years ago

` if project_name is None and Task.current_task() is not None:
project_name = Task.current_task().get_project_name()

if project_name is None and not Task.running_locally():
task = Task.init()
project_name = task.get_project_name() `

  
  
Posted 2 years ago

OK i found what’s happening:

I had an additional Task.init() - just the blank one, to get the project name. Adding the disable to that as well fixed the issue

  
  
Posted 2 years ago

If Task.init() is called in an already running task, don’t reset auto_connect_frameworks? (if i am understanding the behaviour right)

Hmm we might need to somehow store the state of it ...

Option to disable these in the clearml.conf

I think this will be to general, as this is code specific , no?

  
  
Posted 2 years ago

Do people use ClearML with huggingface transformers? The code is std transformers code.

I believe they do 🙂
There is no real way to differentiate between, "storing model" using torch.save and storing configuration ...

  
  
Posted 2 years ago

Does a pipeline step behave differently?

Are you disabling it in the pipeline step ?
(disabling it for the pipeline Task has no effect on the pipeline steps themselves)

  
  
Posted 2 years ago

AgitatedDove14 - yeah wanted to see what’s happening before disabling as I wasn’t sure if this is what’s expected.

  
  
Posted 2 years ago

Also is there a way to disable pytorch like this from clearml.conf?

  
  
Posted 2 years ago

AgitatedDove14 - i am disabling pytorch like above but still see auto models . I even see model import when running evaluation from a loaded model

  
  
Posted 2 years ago
660 Views
13 Answers
2 years ago
one year ago
Tags