Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
I'M Trying To Read The Runtimes Of Tasks After The Pipeline Finished, For Analytics. Currently, I'M Doing Something Like This:

I'm trying to read the runtimes of tasks after the pipeline finished, for analytics. currently, I'm doing something like this:

def collect_task_data(pipe: PipelineController):
    pipe_dag = pipe.get_pipeline_dag()

    def process_task(node_task):
        node, task = node_task
        exp_type = _extract_experiment_type_from_task_name(node)
        runtime = None
        if task.job is not None:
            runtime = task.job.task.data.active_duration
        return exp_type, task.status, runtime

    processed_data = list(map(process_task, pipe_dag.items()))

I'm getting None values for active_duration, job_started and job_ended in all tasks (also on completed ones).
what am I missing?

  
  
Posted 6 months ago
Votes Newest

Answers 6


I'd suggest using the API directly, fetch a task and compare it's start to end time.

  
  
Posted 6 months ago

I also noticed the datetime strings for last_update and started to be None

  
  
Posted 6 months ago

ok, how would you approach all tasks in a pipeline and get their data?

  
  
Posted 6 months ago

Hi @<1639799308809146368:profile|TritePigeon86> , where in the documentation did you see these parameters active_duration, job_started and job_ended ?

  
  
Posted 6 months ago

the last two were found by debugging, but active_duration (which i'm using) is in tha api and derived by them

  
  
Posted 6 months ago

The webUI uses the API for everything, I'd suggest using the webUI as a reference to how to approach this.

  
  
Posted 6 months ago
476 Views
6 Answers
6 months ago
6 months ago
Tags