Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi All, After Upgrading To Sdk 1.8.0 We Are Having Issue Adding External Files To Dataset From Gcs. This Is The Code We Use:

HI all,
After upgrading to sdk 1.8.0 we are having issue adding external files to dataset from gcs.
This is the code we use:
dataset_id = dataset = Dataset.create(....) dataset.add_external_files(source_url=" ` ")
dataset.upload()
dataset.finalize()

d = Dataset.get(dataset_id=dataset_id)
dataset_path = d.get_local_copy() We are getting the below error: clearml - INFO - Failed downloading test1/file1.csv Error is 'Requested path does not exist: /home.... `
Looking at the dataset.link_entries it seems the “ gs://somebucket ” was removed from the original link.

Any idea what we are doing wrong?

  
  
Posted one year ago
Votes Newest

Answers 9


Immediately after i use “add_external_files” i see it changed the original file link and removed from it the “ gs://bucket_name

  
  
Posted one year ago

not getting any error when uploading.
I use “add_external_files” so it is not really uploading the file just the dataset info

  
  
Posted one year ago

this only affects single files, if you wish to add directories (with wildcards as well) you should be able to

  
  
Posted one year ago

Do you get any error when uploading?
It looks like it can upload but can't download afterwards.

  
  
Posted one year ago

Thanks

  
  
Posted one year ago

You could try this in the meantime if you don't mind temporary workarounds:
dataset.add_external_files(source_url=" ", wildcard=["file1.csv"], recursive=False)

  
  
Posted one year ago

I just downgraded to 1.7.2, will wait for the fix.
Thanks for the workaround.

  
  
Posted one year ago

For now we store only one file in a dataset.

  
  
Posted one year ago

Hi ApprehensiveSeahorse83 ! Looks like this is a bug. We will fix it ASAP

  
  
Posted one year ago
692 Views
9 Answers
one year ago
one year ago
Tags
Similar posts