Hello everyone,
I'm working on a flexible machine learning pipeline using ClearML, covering data retrieval, preprocessing, training, and evaluation. I want to add hyperparameter optimization (HPO) as a pipeline step that could optimize the preprocessing and training steps (I'm using add_function_step), but I'm facing some challenges:
- How can I integrate HPO into the ClearML pipeline in a generic way?
- What's the best way to pass optimized parameters between steps in an adaptable pipeline?
- How can I ensure I'm following ClearML best practices for this kind of integration?Has anyone successfully tackled this before? I'd love to hear about your experiences, particularly:
- How you integrated HPO into a ClearML pipeline without tying it to specific parameters
- Examples of setting up HPO as a pipeline step that works well with other steps
- Methods for passing optimized hyperparameters between steps in a versatile pipelineAny advice or examples would be greatly appreciated. Thanks in advance for your help!
