Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Dataset Uploading Failed, But Task Finished Successfully. As A Result - Dataset Is In Inconsistent State, Where It Thinks That There'S A File Inside, But There Isn'T:

Dataset uploading failed, but task finished successfully. As a result - dataset is in inconsistent state, where it thinks that there's a file inside, but there isn't:


Add ltv_0.1_training_mode_1d.parquet
Compressing /root/.clearml/venvs-builds/3.10/code/73151e68e6a64cdeb8c17b8ad45a0ab2/ltv_0.1_training_mode_1d.parquet
2024-12-02 16:23:04
Uploading dataset changes (1 files compressed to 33.49 MiB) to 

2024-12-02 14:23:00,465 - clearml.storage - ERROR - Failed uploading: [Errno 24] Too many open files: '/tmp/dataset.de905c3622a8445c911c14202f1b6427.8r313hcg.zip'
2024-12-02 14:23:00,504 - clearml.storage - ERROR - Failed uploading: [Errno 24] Too many open files: '/tmp/dataset.de905c3622a8445c911c14202f1b6427.8r313hcg.zip'
2024-12-02 14:23:00,543 - clearml.storage - ERROR - Failed uploading: [Errno 24] Too many open files: '/tmp/dataset.de905c3622a8445c911c14202f1b6427.8r313hcg.zip'
2024-12-02 14:23:00,543 - clearml.storage - ERROR - Exception encountered while uploading Upload failed
[2024-12-02 14:23:00] {task.py:2561} WARNING - Failed uploading artifact 'data'. Retrying... (1/3)

--retry logs omitted--

File compression and upload completed: total size 33.49 MiB, 1 chunk(s) stored (average size 33.49 MiB)
2024-12-02 16:23:10
Updating statistics and genealogy
2024-12-02 16:23:53
Process completed successfully
  
  
Posted 20 days ago
Votes Newest

Answers 4


my code:

    dataset = Dataset.create(
        dataset_project=PROJECT_NAME,
        dataset_name=f"processed_{mode}",
        dataset_tags=task.get_tags(),
        parent_datasets=None,
        use_current_task=False,
        output_uri=BUCKET,
    )

dataset.add_files(path, verbose=True)
dataset.upload(verbose=True)    dataset.finalize(verbose=True)
  
  
Posted 20 days ago

Hi FancyOtter74 , you can use Task.mark_failed for this - None

  
  
Posted 19 days ago

but what's the best way to catch the exception? All high-level clearml function calls return normally

  
  
Posted 19 days ago

Is there any way to make sure that task would fail?

  
  
Posted 20 days ago