ClearML FAQ | Dataset Uploading Failed, But Task Finished Successfully. As A Result - Dataset Is In Inconsistent State, Where It Thinks That There'S A File Inside, But There Isn'T:

Answered

Dataset Uploading Failed, But Task Finished Successfully. As A Result - Dataset Is In Inconsistent State, Where It Thinks That There'S A File Inside, But There Isn'T:

Dataset uploading failed, but task finished successfully. As a result - dataset is in inconsistent state, where it thinks that there's a file inside, but there isn't:


Add ltv_0.1_training_mode_1d.parquet
Compressing /root/.clearml/venvs-builds/3.10/code/73151e68e6a64cdeb8c17b8ad45a0ab2/ltv_0.1_training_mode_1d.parquet
2024-12-02 16:23:04
Uploading dataset changes (1 files compressed to 33.49 MiB) to


2024-12-02 14:23:00,465 - clearml.storage - ERROR - Failed uploading: [Errno 24] Too many open files: '/tmp/dataset.de905c3622a8445c911c14202f1b6427.8r313hcg.zip'
2024-12-02 14:23:00,504 - clearml.storage - ERROR - Failed uploading: [Errno 24] Too many open files: '/tmp/dataset.de905c3622a8445c911c14202f1b6427.8r313hcg.zip'
2024-12-02 14:23:00,543 - clearml.storage - ERROR - Failed uploading: [Errno 24] Too many open files: '/tmp/dataset.de905c3622a8445c911c14202f1b6427.8r313hcg.zip'
2024-12-02 14:23:00,543 - clearml.storage - ERROR - Exception encountered while uploading Upload failed
[2024-12-02 14:23:00] {task.py:2561} WARNING - Failed uploading artifact 'data'. Retrying... (1/3)

--retry logs omitted--

File compression and upload completed: total size 33.49 MiB, 1 chunk(s) stored (average size 33.49 MiB)
2024-12-02 16:23:10
Updating statistics and genealogy
2024-12-02 16:23:53
Process completed successfully

  				
Posted 
	16 days ago

					More
				  		
  Report
		
					FancyOtter74
				
					0
					 × 1

Votes Newest

Answers 4

Is there any way to make sure that task would fail?

  				
Posted 
	16 days ago

					More
				  		
  Report
		
					FancyOtter74
				
					0
					 × 1

my code:

    dataset = Dataset.create(
        dataset_project=PROJECT_NAME,
        dataset_name=f"processed_{mode}",
        dataset_tags=task.get_tags(),
        parent_datasets=None,
        use_current_task=False,
        output_uri=BUCKET,
    )

dataset.add_files(path, verbose=True)
dataset.upload(verbose=True)    dataset.finalize(verbose=True)

  				
Posted 
	16 days ago

					More
				  		
  Report
		
					FancyOtter74
				
					0
					 × 1

Hi @<1613344994104446976:profile|FancyOtter74> , you can use Task.mark_failed for this - None

  				
Posted 
	15 days ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

but what's the best way to catch the exception? All high-level clearml function calls return normally

  				
Posted 
	15 days ago

					More
				  		
  Report
		
					FancyOtter74
				
					0
					 × 1

Write your answer

74 Views

4 Answers

16 days ago

15 days ago