Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello. Please Tell Me How To Make Sure That When You Start The Pipeline, Nothing Superfluous Is Installed In The Service Queue?

Hello. Please tell me how to make sure that when you start the pipeline, nothing superfluous is installed in the service queue?

@PipelineDecorator.pipeline(
    project=CLEARML_CONFIG.pipeline.project_name,
    name=CLEARML_CONFIG.pipeline.pipeline_name,
)
def executing_pipeline(dataset_id: str):
    print(f"{dataset_id = }")
    weights_path = train_step(dataset_id)
    print(f"{weights_path = }")
    evaluate_step(weights_path)


@PipelineDecorator.component(
    execution_queue=CLEARML_CONFIG.queue,
    docker=CLEARML_CONFIG.docker_image,
    docker_args=CLEARML_CONFIG.docker_arguments,
    return_values=["weights_path"],
    cache=False,
    task_type=TaskTypes.training,
    repo="",
    repo_branch=BRANCH,
)
def train_step(dataset_id: str):
    ...
    return weights_path


@PipelineDecorator.component(
    execution_queue=CLEARML_CONFIG.queue,
    cache=False,
    task_type=TaskTypes.testing,
    docker=CLEARML_CONFIG.docker_image,
    docker_args=CLEARML_CONFIG.docker_arguments,
    parents=["train_step"],
    repo="",
    repo_branch=BRANCH,
)
def evaluate_step(weights_path):
    ...


if __name__ == "__main__":
    PipelineDecorator.set_default_execution_queue("services")
    executing_pipeline(dataset_id="...")
  
  
Posted 11 months ago
Votes Newest

Answers 3


Hi, Can you please elaborate on what you mean and what is happening?

  
  
Posted 11 months ago

Can you please provide a stand alone snippet that reproduces this behavior? Can you provide a log of the run?

  
  
Posted 11 months ago

For example, when I start the pipeline, pytorch starts to be installed in the service queue. But I would like it to be installed only inside the queue that train step will run on.

  
  
Posted 11 months ago
571 Views
3 Answers
11 months ago
11 months ago
Tags