UpsetCrow72

2 Questions, 2 Answers

Active since 10 January 2023

Last activity one year ago

Reputation

Badges 1

2 × Eureka!

Questions 2
Answers 2

0 Votes

1 Answers

946 Views

0 Votes 1 Answers 946 Views

Hey, Is There Some Way / Workaround To Speed Up Working With Datasets With Large Number Of Files? Getting A Local Copy Of One Of Our Dataset With 70K Files Already Takes Longer Than Expected, But Working With A Dataset Of Around 100K Files That Has Multip

Hey, is there some way / workaround to speed up working with datasets with large number of files? Getting a local copy of one of our dataset with 70k files a...

clearml

one year ago

0 Votes

5 Answers

1K Views

0 Votes 5 Answers 1K Views

Hi 👋 I'm trying to remove a few files from a dataset, which has been finalized already with .finalize() . Both with $ clearml-data sync and with $clearml-da...

dataset

2 years ago

0 Hi

I'm getting it by $ clearml-data get --id <id> --copy <local_path> and then this is the log output of $ clearml-data remove --id <id> --files <files>

` clearml-data - Dataset Management & Versioning CLI
Removing files/folder from dataset id 2a0eb9ab619c442abc204775f217d0b9
2022-09-19 13:44:12,964 - clearml.Task - ERROR - Action failed <400/110: tasks.add_or_update_artifacts/v2.10 (Invalid task status: expected=created, status=completed)> (task=2a0eb9ab619c442abc204775f217d0b9, ar...

2 years ago

0 Hi

hi CostlyOstrich36 , sorry, let me make it a bit more clear.

I simply upload a bunch of files as a new dataset using the Python API. Then using the CLI I get a local copy where I remove a few of the files. At this step, I tried both simply removing them from the file-system and then using $ clearml-data sync , and also using $ clearml-data remove . I get an error, invalid status, but the result is the same as I described above: file count was updated, but when I get a new local...

2 years ago