Well given a file architecture looking like this:|_ __init__.py |_ my_pipeline.py |_ my_utils.py
With the content of my_pipeline.py
being:
` from clearml.automation.controller import PipelineDecorator
from clearml import Task, TaskTypes
from my_utils import do_thing
Task.force_store_standalone_script()
@PipelineDecorator.component(...)
def my_component(dataset_id: str):
import pandas as pd
from clearml import Dataset
dataset = Dataset.get(dataset_id=input_dataset_id)
dataset_path = dataset.get_local_copy()
dataset = do_thing(dataset)
... `
And the content of my_utils.py
being:def do_thing(df: pd.DataFrame) -> pd.DataFrame: """Just a simple shuffle this is an example not a CS course""" df = df.sample(frac=1) return df
Should I do an import pandas as pd
in my_utils.py
given than the call of do_thing()
is done within my component and thus in the scope of the component's pandas import ? Will clearML resolve that function, upload it and propagate the component from which it is called depdendencies to it?