Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello! When Trying To Use Clearml Datasets With Google Cloud Storage With The Authorized User Credentials It Will Fail And Say Some Fields Are Missing From The Json. This Isn'T An Issue If The User Is Using A Service Account Json Key, Is A Service Account

Hello! When trying to use ClearML Datasets with Google Cloud Storage with the authorized user credentials it will fail and say some fields are missing from the JSON. This isn't an issue if the user is using a Service Account JSON Key, is a service account account absolutely necessary to do uploads to Google Cloud Storage?

  
  
Posted one year ago
Votes Newest

Answers 7


For context, the google cloud storage SDK allows an authorized user credentials. This makes it a bit awkward for our developers already using their creds.

  
  
Posted one year ago

The error in question:
2022-12-01 17:33:03,687 - clearml.storage - ERROR - Failed creating storage object gs://<my_bucket> Reason: Service account info was not in the expected format, missing fields token_uri, client_email.

  
  
Posted one year ago

Hi ShortElephant92

This isn't an issue if the user is using a Service Account JSON Key,

Are you saying that when you are using GS python sdk directly it works?

For context, the google cloud storage SDK allows an authorized user credentials.

ClearML actually uses the google python SDK, the JSON is just a way to pass the credentials to the google SDK, I'm not sure it points to "service account"? where did that requirement came from ?
is it from here Service account info was not in the expected format, missing fields token_uri, client_email. ?
Could that be the same issue?
https://stackoverflow.com/questions/43004904/accessing-gae-log-files-using-google-cloud-logging-python

  
  
Posted one year ago

Hi AgitatedDove14 I found the possible bug, I'll open a PR for it and we can discuss! It has to do with how we pass credentials to the GCS client.

  
  
Posted one year ago

I opened a PR for it, it has a bug attached to it too: https://github.com/allegroai/clearml/pull/841

  
  
Posted one year ago

nice !

  
  
Posted one year ago

Thanks ShortElephant92 ! PR looks good, I'll ask the guts to take a look

  
  
Posted one year ago
1K Views
7 Answers
one year ago
one year ago
Tags