Hi @<1720249421582569472:profile|NonchalantSeaanemone34> , can you please provide a full log of a run? Also do you have a full snippet that reproduces this behaviour?

Hi @<1523701070390366208:profile|CostlyOstrich36>
here is the full code:

import os
import sys,shutil
import clearml
from clearml import Task, Dataset, Logger
from clearml import PipelineDecorator, PipelineController

project_name = "Titanic Project"
dataset_name = "titanic_data"

datasets = Dataset.list_datasets()
for dataset in datasets:
if dataset["project"] == project_name and dataset["name"] == dataset_name:
parent_datasets_id = dataset["id"]

print(parent_datasets_id)

dso = Dataset.create(
dataset_project= project_name,
dataset_name= dataset_name,
parent_datasets=[parent_datasets_id],
)

dso = Dataset.get(
dataset_project= project_name,
dataset_name= dataset_name,
only_completed=True,
only_published=False,
alias='latest',
)

if os.path.exists("./data"):
shutil.rmtree("./data")

local_path = dso.get_mutable_local_copy("./data")

print(local_path)

with open("./data/titanic.csv","a+") as fh:
fh.write('\n885,0,3,"Sutehaasll, Mr. Henry Jr.",male,45,0,0,SOTON/OQ 392076,7.05,,S\n')

op = dso.sync_folder(local_path="./data",verbose=True)
print(op)
dso.finalize(auto_upload=True,verbose=True)

@<1720249421582569472:profile|NonchalantSeaanemone34>

dso = Dataset.create(
        dataset_project= project_name,
        dataset_name= dataset_name,
        parent_datasets=[parent_datasets_id],
)
dso = Dataset.get(
        dataset_project= project_name,
        dataset_name= dataset_name,
        only_completed=True,
        only_published=False,
        alias='latest',
)

why are you creating a dataset then getting a dataset on the same object?
it seems you are trying to upload to the existing dataset and not the newly created one notice in both cases dso is the variable name

Hi @<1523701205467926528:profile|AgitatedDove14>
Thanks alot for pointing out "dso" variable 🙂 . I did not realize that same name i was using for create and get dataset function.
It was my mistake, i changed the variable and everything is functional as expected.

Thank you again.