Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello, What Happens To Running Experiments When The Clearml Agent Is Killed (And Restarted A Few Minutes Later)?

Hello, what happens to running experiments when the ClearML Agent is killed (and restarted a few minutes later)?

  
  
Posted one month ago
Votes Newest

Answers 3


Thanks for your reply. Would you be able to point me towards a section in the documentation or maybe even a piece of code that handles the killing of the tasks?

  
  
Posted one month ago

I don;t think there is a "kill task" code. By principle, in Linux, as a parent process, ClearML agent launch the training process. When a parent process is terminated, the linux kernel will, in most of the case, kill all child processes, including your training process.
There may be some way to resume a task from ClearML agent when it restart, but I don;t think that is the default behavior

  
  
Posted one month ago

If the agent is the one running the experiment, very likely that your task will be killed.
And when the agent come back, immediately or later, probably nothing will happen. It won't resume ...

  
  
Posted one month ago
131 Views
3 Answers
one month ago
one month ago
Tags