Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
When Running Jobs, My Pipeline Controller Always Updates To The Latest Git Commit Id But Sometimes My Pipeline Steps Do Not. This Appears To Be Somewhat Random So I Believe It Is Due To Caching. Has Anyone Else Encountered This Or Have Any Idea How To Fix

When running jobs, my pipeline controller always updates to the latest git commit id but sometimes my pipeline steps do not. This appears to be somewhat random so I believe it is due to caching. Has anyone else encountered this or have any idea how to fix it?

  
  
Posted 3 years ago
Votes Newest

Answers 8


Clarification question: was cache_executed_step present in clearml 0.17 (which we are running)? I am trying to add it in manually and it is giving me an unexpected keyword argument error

  
  
Posted 3 years ago

Hi Martin, this is indeed the case. I am not passing cache_executed_step=false to the controller but I assume this is the case as it is false by default

  
  
Posted 3 years ago

Notice there is no need to upgrade the server, only the ClearML python package

  
  
Posted 3 years ago

Hi AdventurousRabbit79 , I haven't seen this behaviour before - is there a specific use-case or example you're using?

  
  
Posted 3 years ago

For usage examples, it's difficult as the caching seems to be sporadic (works 90% of the time)

  
  
Posted 3 years ago

AdventurousRabbit79 are you passing cache_executed_step=False to the PipelineController ?
https://github.com/allegroai/clearml/blob/332ceab3eadef4997e897d171957975a247a6dc1/clearml/automation/controller.py#L129
Could you send a usage example ?

my pipeline controller always updates to the latest git commit id

This will only happen if the Task the pipeline creates has no specific commit ID, and instead just uses the latest from the git repo. Is this the case ?

  
  
Posted 3 years ago

Hi Jake, we aren't using any specific use-case or example but this has been occurring sporadically throughout the last month or so. Would it be possible to hop on a 15 or so minute zoom call some time to show you what has been happening?

  
  
Posted 3 years ago

AdventurousRabbit79 you are correct, caching was introduced in v1.0 , also notice the default is no caching, you have to specify that you want caching per step.

  
  
Posted 3 years ago
914 Views
8 Answers
3 years ago
one year ago
Tags