Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hey Having An Issue Passing Parameters To A Component In A Pipeline, The Parameters Appear To Be

Hey having an issue passing parameters to a component in a pipeline, the parameters appear to be None inside the component function:
` @PipelineDecorator.component(
return_values=['dataset_id'],
cache=False,
task_type=TaskTypes.data_processing,
execution_queue='Quad_VCPU_16GB',
)
def generate_dataset(start_date: str, end_date: str, input_aws_credentials_profile: str = 'default'):
"""
Convert autocut logs from a specified time window into usable dataset in generic format.
"""
print('[STEP 1/4] Generating dataset from autocut logs...')
import os
import cv2
import sys
import srsly
import boto3
import shutil
import numpy as np
import pandas as pd
from clearml import Dataset
from zipfile import ZipFile

print(f"start_date: {start_date} end_date: {end_date}") `

Prints start_date: None end_date: None

Here is the controller:
` @PipelineDecorator.pipeline(
name="VINZ Auto-Retrain",
project="VINZ",
version="0.0.1",
pipeline_execution_queue="Quad_VCPU_16GB"
)
def executing_pipeline(start_date, end_date):
print("Starting VINZ Auto-Retrain pipeline...")
print(f"Start date: {start_date}")
print(f"End date: {end_date}")

window_dataset_id = generate_dataset(start_date, end_date)

if name == 'main':
PipelineDecorator.debug_pipeline()

executing_pipeline(
    start_date="2022-01-01",
    end_date="2022-03-02"
) `

It's clearly coming from CLearML because when replacing Pipeline.run_locally() by Pipeline.debug_pipeline() which results in the components to be executed as classic python function instead of tasks (cf debug_pipeline() and run_locally() methods docstrings inside controller.py ) the parameters are properly passed and the component prints: start_date: 2022-01-01 end_date: 2022-03-02

Is it possible the pipeline example on github is not up to date ?

PS: same issue when running on ClearML workers

  
  
Posted one year ago
Votes Newest

Answers 15


pip package clearml-agent is version 1.3.0

  
  
Posted one year ago

Nice it works 😍
I'll try to update the version in the image I provide to the workers of th autoscaler app (but sadly I don't control the version of those in itself since it's CLearML managed)

  
  
Posted one year ago

Seems pretty critical tho, SuccessfulKoala55 is the 1.7.2 releasing soon ? ( I see the rc2 has been built 3 days ago)

  
  
Posted one year ago

I had the same issue, they fixed it in 1.7.2rc1 I think. You can try pip install clearml==1.7.2rc1

  
  
Posted one year ago

Hi FierceHamster54 , which clearml version are you using?

  
  
Posted one year ago

From what I've seen, another fix would be to pass a default value to your parameters, for example: def executing_pipeline(start_date=0, end_date=0):

  
  
Posted one year ago

Okay I confirm having default parameters fixes that issue, but kinda sad to have lost 3 days into that super weird behavior

  
  
Posted one year ago

And remote workers are managed by the GCP auto-scaler app so I presume those clearml-agent are up-to-date too

  
  
Posted one year ago

I'm looking for the github issue so I can check the PR fixing that, do you have any info on that ?

  
  
Posted one year ago

Apparently, the issue is that it will always pass None if there's no default value

  
  
Posted one year ago

Nope sorry I don't have that, but this was our discussion about it: https://clearml.slack.com/archives/CTK20V944/p1665477602050259

  
  
Posted one year ago

Hey SuccessfulKoala55 currently using the clearml package version 1.7.1 and my server is a PRO SaaS deployment

  
  
Posted one year ago

Yeah me too, I had no idea what was happening 😅

  
  
Posted one year ago

FierceHamster54 1.7.2 will be released very soon 🙂

  
  
Posted one year ago

And running with a Python 3.10 interpreter

  
  
Posted one year ago
624 Views
15 Answers
one year ago
one year ago
Tags