Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi All, After Upgrading To Sdk 1.8.0 We Are Having Issue Adding External Files To Dataset From Gcs. This Is The Code We Use:

HI all,
After upgrading to sdk 1.8.0 we are having issue adding external files to dataset from gcs.
This is the code we use:
dataset_id = dataset = Dataset.create(....) dataset.add_external_files(source_url=" ` ")
dataset.upload()
dataset.finalize()

d = Dataset.get(dataset_id=dataset_id)
dataset_path = d.get_local_copy() We are getting the below error: clearml - INFO - Failed downloading test1/file1.csv Error is 'Requested path does not exist: /home.... `
Looking at the dataset.link_entries it seems the “ gs://somebucket ” was removed from the original link.

Any idea what we are doing wrong?

  
  
Posted 2 years ago
Votes Newest

Answers 9


Hi ApprehensiveSeahorse83 ! Looks like this is a bug. We will fix it ASAP

  
  
Posted 2 years ago

For now we store only one file in a dataset.

  
  
Posted 2 years ago

I just downgraded to 1.7.2, will wait for the fix.
Thanks for the workaround.

  
  
Posted 2 years ago

Immediately after i use “add_external_files” i see it changed the original file link and removed from it the “ gs://bucket_name

  
  
Posted 2 years ago

not getting any error when uploading.
I use “add_external_files” so it is not really uploading the file just the dataset info

  
  
Posted 2 years ago

Thanks

  
  
Posted 2 years ago

You could try this in the meantime if you don't mind temporary workarounds:
dataset.add_external_files(source_url=" ", wildcard=["file1.csv"], recursive=False)

  
  
Posted 2 years ago

this only affects single files, if you wish to add directories (with wildcards as well) you should be able to

  
  
Posted 2 years ago

Do you get any error when uploading?
It looks like it can upload but can't download afterwards.

  
  
Posted 2 years ago
1K Views
9 Answers
2 years ago
one year ago
Tags
Similar posts