Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello. Please Tell Me How To Make Sure That When You Start The Pipeline, Nothing Superfluous Is Installed In The Service Queue?

Hello. Please tell me how to make sure that when you start the pipeline, nothing superfluous is installed in the service queue?

@PipelineDecorator.pipeline(
    project=CLEARML_CONFIG.pipeline.project_name,
    name=CLEARML_CONFIG.pipeline.pipeline_name,
)
def executing_pipeline(dataset_id: str):
    print(f"{dataset_id = }")
    weights_path = train_step(dataset_id)
    print(f"{weights_path = }")
    evaluate_step(weights_path)


@PipelineDecorator.component(
    execution_queue=CLEARML_CONFIG.queue,
    docker=CLEARML_CONFIG.docker_image,
    docker_args=CLEARML_CONFIG.docker_arguments,
    return_values=["weights_path"],
    cache=False,
    task_type=TaskTypes.training,
    repo="",
    repo_branch=BRANCH,
)
def train_step(dataset_id: str):
    ...
    return weights_path


@PipelineDecorator.component(
    execution_queue=CLEARML_CONFIG.queue,
    cache=False,
    task_type=TaskTypes.testing,
    docker=CLEARML_CONFIG.docker_image,
    docker_args=CLEARML_CONFIG.docker_arguments,
    parents=["train_step"],
    repo="",
    repo_branch=BRANCH,
)
def evaluate_step(weights_path):
    ...


if __name__ == "__main__":
    PipelineDecorator.set_default_execution_queue("services")
    executing_pipeline(dataset_id="...")
  
  
Posted 6 months ago
Votes Newest

Answers 3


For example, when I start the pipeline, pytorch starts to be installed in the service queue. But I would like it to be installed only inside the queue that train step will run on.

  
  
Posted 6 months ago

Can you please provide a stand alone snippet that reproduces this behavior? Can you provide a log of the run?

  
  
Posted 6 months ago

Hi, Can you please elaborate on what you mean and what is happening?

  
  
Posted 6 months ago
377 Views
3 Answers
6 months ago
6 months ago
Tags