Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello. Please Tell Me How To Make Sure That When You Start The Pipeline, Nothing Superfluous Is Installed In The Service Queue?

Hello. Please tell me how to make sure that when you start the pipeline, nothing superfluous is installed in the service queue?

@PipelineDecorator.pipeline(
    project=CLEARML_CONFIG.pipeline.project_name,
    name=CLEARML_CONFIG.pipeline.pipeline_name,
)
def executing_pipeline(dataset_id: str):
    print(f"{dataset_id = }")
    weights_path = train_step(dataset_id)
    print(f"{weights_path = }")
    evaluate_step(weights_path)


@PipelineDecorator.component(
    execution_queue=CLEARML_CONFIG.queue,
    docker=CLEARML_CONFIG.docker_image,
    docker_args=CLEARML_CONFIG.docker_arguments,
    return_values=["weights_path"],
    cache=False,
    task_type=TaskTypes.training,
    repo="",
    repo_branch=BRANCH,
)
def train_step(dataset_id: str):
    ...
    return weights_path


@PipelineDecorator.component(
    execution_queue=CLEARML_CONFIG.queue,
    cache=False,
    task_type=TaskTypes.testing,
    docker=CLEARML_CONFIG.docker_image,
    docker_args=CLEARML_CONFIG.docker_arguments,
    parents=["train_step"],
    repo="",
    repo_branch=BRANCH,
)
def evaluate_step(weights_path):
    ...


if __name__ == "__main__":
    PipelineDecorator.set_default_execution_queue("services")
    executing_pipeline(dataset_id="...")
  
  
Posted one year ago
Votes Newest

Answers 3


Hi, Can you please elaborate on what you mean and what is happening?

  
  
Posted one year ago

For example, when I start the pipeline, pytorch starts to be installed in the service queue. But I would like it to be installed only inside the queue that train step will run on.

  
  
Posted one year ago

Can you please provide a stand alone snippet that reproduces this behavior? Can you provide a log of the run?

  
  
Posted one year ago
851 Views
3 Answers
one year ago
one year ago
Tags