Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hey Having An Issue Passing Parameters To A Component In A Pipeline, The Parameters Appear To Be

Hey having an issue passing parameters to a component in a pipeline, the parameters appear to be None inside the component function:
` @PipelineDecorator.component(
return_values=['dataset_id'],
cache=False,
task_type=TaskTypes.data_processing,
execution_queue='Quad_VCPU_16GB',
)
def generate_dataset(start_date: str, end_date: str, input_aws_credentials_profile: str = 'default'):
"""
Convert autocut logs from a specified time window into usable dataset in generic format.
"""
print('[STEP 1/4] Generating dataset from autocut logs...')
import os
import cv2
import sys
import srsly
import boto3
import shutil
import numpy as np
import pandas as pd
from clearml import Dataset
from zipfile import ZipFile

print(f"start_date: {start_date} end_date: {end_date}") `

Prints start_date: None end_date: None

Here is the controller:
` @PipelineDecorator.pipeline(
name="VINZ Auto-Retrain",
project="VINZ",
version="0.0.1",
pipeline_execution_queue="Quad_VCPU_16GB"
)
def executing_pipeline(start_date, end_date):
print("Starting VINZ Auto-Retrain pipeline...")
print(f"Start date: {start_date}")
print(f"End date: {end_date}")

window_dataset_id = generate_dataset(start_date, end_date)

if name == 'main':
PipelineDecorator.debug_pipeline()

executing_pipeline(
    start_date="2022-01-01",
    end_date="2022-03-02"
) `

It's clearly coming from CLearML because when replacing Pipeline.run_locally() by Pipeline.debug_pipeline() which results in the components to be executed as classic python function instead of tasks (cf debug_pipeline() and run_locally() methods docstrings inside controller.py ) the parameters are properly passed and the component prints: start_date: 2022-01-01 end_date: 2022-03-02

Is it possible the pipeline example on github is not up to date ?

PS: same issue when running on ClearML workers

  
  
Posted 2 years ago
Votes Newest

Answers 15


From what I've seen, another fix would be to pass a default value to your parameters, for example: def executing_pipeline(start_date=0, end_date=0):

  
  
Posted 2 years ago

I had the same issue, they fixed it in 1.7.2rc1 I think. You can try pip install clearml==1.7.2rc1

  
  
Posted 2 years ago

Yeah me too, I had no idea what was happening 😅

  
  
Posted 2 years ago

pip package clearml-agent is version 1.3.0

  
  
Posted 2 years ago

Okay I confirm having default parameters fixes that issue, but kinda sad to have lost 3 days into that super weird behavior

  
  
Posted 2 years ago

Apparently, the issue is that it will always pass None if there's no default value

  
  
Posted 2 years ago

Hi FierceHamster54 , which clearml version are you using?

  
  
Posted 2 years ago

And remote workers are managed by the GCP auto-scaler app so I presume those clearml-agent are up-to-date too

  
  
Posted 2 years ago

Nice it works 😍
I'll try to update the version in the image I provide to the workers of th autoscaler app (but sadly I don't control the version of those in itself since it's CLearML managed)

  
  
Posted 2 years ago

And running with a Python 3.10 interpreter

  
  
Posted 2 years ago

I'm looking for the github issue so I can check the PR fixing that, do you have any info on that ?

  
  
Posted 2 years ago

Seems pretty critical tho, SuccessfulKoala55 is the 1.7.2 releasing soon ? ( I see the rc2 has been built 3 days ago)

  
  
Posted 2 years ago

Hey SuccessfulKoala55 currently using the clearml package version 1.7.1 and my server is a PRO SaaS deployment

  
  
Posted 2 years ago

Nope sorry I don't have that, but this was our discussion about it: https://clearml.slack.com/archives/CTK20V944/p1665477602050259

  
  
Posted 2 years ago

FierceHamster54 1.7.2 will be released very soon 🙂

  
  
Posted 2 years ago
973 Views
15 Answers
2 years ago
one year ago
Tags