Thanks a lot @<1523701070390366208:profile|CostlyOstrich36> your comment boosted my confidence and i'll deep dive into clearML from now on.
What I wanted is already there in clearML, my worry was why to waste time in learning other tool if clearML is already serving the whole purpose.

May be I can start posting few enhancements which might be good for whole clearML community.

Thats nice to hear.
Thanks a lot @<1523701205467926528:profile|AgitatedDove14> :-)

Please see this code snippet:
reviews_dataset_path = Dataset.get(
dataset_id=config["reviews_data_file_id"],
only_completed=True,
only_published=False,
). get_local_copy()

is there a way we can still read the data without using ". get_local_copy()"

Hi @<1523701205467926528:profile|AgitatedDove14>
first of all many thanks to you for always replying promptly 🙂

Initially I thought function is defined outside that is why I was getting that error. However I am using this way and that also gives error:

@PipelineDecorator.component(return_values=["op"])
def step1():
op = "I am in step-1"
return op

@PipelineDecorator.component(return_values=["op"])
def step2(op):
return ( lambda op:step1(op))

While executing step2, I g...

Hi @<1523701205467926528:profile|AgitatedDove14>
What I mean is, dataset is loaded in server, I want to call that dataset in local PC without downloading its copy and do the analysis.

Hi @<1523701205467926528:profile|AgitatedDove14>
Thanks alot for pointing out "dso" variable 🙂 . I did not realize that same name i was using for create and get dataset function.
It was my mistake, i changed the variable and everything is functional as expected.

Thank you again.

Hi @<1523701070390366208:profile|CostlyOstrich36>
here is the full code:

import os
import sys,shutil
import clearml
from clearml import Task, Dataset, Logger
from clearml import PipelineDecorator, PipelineController

project_name = "Titanic Project"
dataset_name = "titanic_data"

datasets = Dataset.list_datasets()
for dataset in datasets:
if dataset["project"] == project_name and dataset["name"] == dataset_name:
parent_datasets_id = dataset["id"]

print(parent_datasets_id)

dso...