Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, I Am Experiencing Issues When Uploading Artifacts To The Dataset Task With Clearml Version V1.1.4Rc0. The Problem Is The Artifacts Are Uploaded To The Default Clearml Server, Even Though I Have Specified The Path To Our Storage Medium. The Code To Dem

Hi, I am experiencing issues when uploading artifacts to the Dataset task with ClearML version v1.1.4rc0. The problem is the artifacts are uploaded to the default ClearML server, even though I have specified the path to our storage medium. The code to demonstrate this is as follows:
` from pathlib import Path
from clearml import Task, Dataset

dataset_dir = Path("data")
dataset_dir.mkdir(exist_ok=True)

for i in range(1, 6):
new_filepath = dataset_dir / f"data_file_{i}.txt"
new_filepath.touch()

dataset = Dataset.create(dataset_name="Mock dataset", dataset_project="Mocks")
dataset.add_files(path=dataset_dir, verbose=True)
dataset.upload()

print("Dataset default storage:", dataset.get_default_storage())

output: Dataset default storage: file:///home/mount/user/server_local_storage

dataset_task = Task.get_task(task_id=dataset.id)

print("Dataset task default storage:", dataset_task.get_output_destination())

output: Dataset task default storage: file:///home/mount/user/server_local_storage

import numpy as np

some_artifact = np.ones(shape=(5, 5))

dataset_task.upload_artifact(name="Numpy data", artifact_object=some_artifact)

dataset.finalize(verbose=True) `In the image attached the "data" artifact has been uploaded to the correct path, however, "Numpy data" is on the community file server, but in the code I show the output destinations point to my storage media. Can someone reproduce this example? Is this the expected behavior?

  
  
Posted 2 years ago
Votes Newest

Answers 12


Hi GiganticTurtle0

dataset_task = Task.get_task(task_id=dataset.id)

Hmmm I think that when it gets the Task "output_uri" is not updated from the predefined Task (you can obviously set it again).
This seems like a bug that is unrelated to Datasets.
Basically any Task that you retrieve will default to the default ouput_uri (not the stored one)

  
  
Posted 2 years ago

Hi AgitatedDove14 , gotcha. So how can I temporarily fix it? I'm not able to find something like task.set_output_uri() in the official docs. Or maybe do you plan to solve this problem in the very short term?

  
  
Posted 2 years ago

AgitatedDove14 It's in the configuration file where I specified that information. But I think this error has only appeared since I upgraded to version 1.1.4rc0

  
  
Posted 2 years ago

GiganticTurtle0 where in the code you set the output destination to "file:///home/mount/user/server_local_storage" ?

  
  
Posted 2 years ago

BTW: in your code, you should probably replace
dataset_task = Task.get_task(task_id=dataset.id)with:
dataset_task = dataset._task

  
  
Posted 2 years ago

Or maybe do you plan to solve this problem in the very short term? (edited)

Yes we will 🙂

  
  
Posted 2 years ago

So how can I temporarily fix it?

Try:
task.output_uri = task.get_output_destination()

  
  
Posted 2 years ago

But I think this error has only appeared since I upgraded to version 1.1.4rc0

Hmm let me check something

  
  
Posted 2 years ago

But I cannot go back to version v1.1.3 because there is another bug related to the Dataset tags

  
  
Posted 2 years ago

GiganticTurtle0 found it, fix will be pushed tomorrow 🙂

  
  
Posted 2 years ago

Fix was pushed to GitHub 🙂

  
  
Posted 2 years ago

So happy to hear that!

  
  
Posted 2 years ago
605 Views
12 Answers
2 years ago
one year ago
Tags
Similar posts