CostlyOstrich36 If I run the pipeline with the same input parameters, all the steps will also be re-run, nothing will be taken from the cache

CostlyOstrich36 Any ideas?

How are you writing your pipelines?

I create a pipeline via PipelineController with adding a step as a function

pipe = PipelineController(
        name=cfg.clearml.pipeline_name,
        project=cfg.clearml.project_name,
        target_project=True,
        version=cfg.clearml.version,
        add_pipeline_tags=True,
        docker=cfg.clearml.dockerfile,
        docker_args=DefaultMLPLATparam().docker_arg,
        packages=packages,
        retry_on_failure=3
    )

for parameter in cfg.clearml.params:
        pipe.add_parameter(
            name=parameter["name"],
            default=parameter["default"],
            # description=parameter["description"],
        )

for step in cfg.clearml.steps:
        pipe.add_function_step(
            name=step["name"],
            function=function,
            function_kwargs=step["function_kwargs"],
            function_return=(
                list(step["return_params"]) if step["return_params"] else None
            ),
            project_name=cfg.clearml.project_name,
            repo=cfg.clearml.repo,
            repo_branch=cfg.clearml.repo_branch,
            docker=step["dockerfile"],
            docker_args=docker_arg,
            execution_queue=task_queue_name,
            cache_executed_step=True,
            packages=f"{os.getcwd()}/{step['path_to_reqs']}",
            parents=list(step["parents"]) if step["parents"] else None,
            retry_on_failure=3
        )
    pipe.start(queue=None)

CostlyOstrich36

Hi GrievingKoala83 , please provide a standalone code snippet. I mean a single stand alone piece of code that I can just run without modifications, even imports

Can you please provide a standalone code snippet that reproduces this behaviour?

Hi GrievingKoala83 , what happens if you rerun this via the webUI ?

CostlyOstrich36 Above, I provided the code for this pipeline, I specify cache_executed_step=True for each pipeline step , but it doesn't work.

CostlyOstrich36 can you help me?