Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, I Have A Question About How To Get Notified When A Task Finishes Running. We Are Creating Training Tasks Programatically On Clearml Server. It Is Relatively Easy To Do Through Directly Accessing The Clearml Server Api. However, We'D Like Some Way T

Hi, I have a question about how to get notified when a task finishes running.

We are creating training tasks programatically on ClearML Server. It is relatively easy to do through directly accessing the ClearML Server API.

However, we'd like some way to be notified when the training task has finished. Is there some way to register a callback URL when creating a task? I'm imaging that ClearML server would send a GET HTTP request to this URL when training has finished. (Looking through the ClearML Server documentation, I couldn't find anything like this.)

Alternatively, since we are creating the training tasks programatically, in theory we could modify the training script to ping a given URL when training is finished. However, ideally we wouldn't need the training script to know anything about our server setup (which machine to ping, etc).

Another possibility would be to run a process somewhere that periodically polls ClearML Server for tasks that have recently finished. I'm imagining something similar to the Slack Notification example: https://github.com/allegroai/clearml/blob/0c017a73310cac0e699f3b1a61c462a5397706b6/examples/services/monitoring/slack_alerts.py . The downside of this is that we need another process running somewhere specifically doing this.

  
  
Posted 3 years ago
Votes Newest

Answers 3


Thanks for the suggestions. I'll have to look into creating my own notification task.

  
  
Posted 3 years ago

Hi JoyousElephant80

Another possibility would be to run a process somewhere that periodically polls ClearML Server for tasks that have recently finished

this is the easiest way to implement what you are after, and have full control over the logic itself.
Basically you inherit from the Monitor class
And implement the callback function:
https://github.com/allegroai/clearml/blob/c58e8a4c6a1294f8acec6ed9cba81c3b91aa2abd/clearml/automation/monitor.py#L128

WDYT?

  
  
Posted 3 years ago

you are right about the Slack monitoring service, this is exactly its purpose.

You don’t really need to run it as a separate process, you can have a ClearML agent in services mode ( https://allegro.ai/clearml/docs/docs/use_cases/clearml_agent_use_case_examples.html#launching-clearml-agent-in-services-mode ) and enqueue the Slack monitoring task (among with other services in parallel).

  
  
Posted 3 years ago