Hi
i'm trying to finalize a dataset, but although the finalize(auto_upload=True) completes successfully (see image) - But the dataset is still in an uploading status (resulting with is_fianl() as False)
The dataset has a single parquet file
I notices it when trying to get a local copy
I'm running clearml==1.11.1
ds.get_local_copy()
ValueError Traceback (most recent call last)
Cell In[37], line 1
----> 1 ds.get_local_copy()
File ~/..../.direnv/python-3.10.7/lib/python3.10/site-packages/clearml/datasets/dataset.py:940, in Dataset.get_local_copy(self, use_soft_links, part, num_parts, raise_on_error, max_workers)
938 self._task = Task.get_task(task_id=self._id)
939 if not self.is_final():
--> 940 raise ValueError("Cannot get a local copy of a dataset that was not finalized/closed")
941 max_workers = max_workers or psutil.cpu_count()
943 # now let's merge the parents
ValueError: Cannot get a local copy of a dataset that was not finalized/closed
Any suggestions?
Hi @<1523701323046850560:profile|OutrageousSheep60> , we just released v1.12.1 - can you please check with that version?
upgrading to 1.12.1 didn't help
I think the issue is that when I create the dataset - i used
use_current_task=True,
If I change it to
use_current_task=False,
then it finalizes
will do
A work around that worked for me is to explicitly complete the task, seems like the flush has some bug
task = Task.get_task('...')
task.close()
task.mark_completed()
ds.is_final()
True
