Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
We Have A Environment Variables Definitions.Py File Which Every User Configures On Their Local Machine. This File Includes Local Paths As Well As Aws/Api Credentials. This Is An Issue When Spinning Up Clearml Tasks Since It Is Not Included In The Git Repo

We have a environment variables definitions.py file which every user configures on their local machine. This file includes local paths as well as AWS/API credentials. This is an issue when spinning up clearml tasks since it is not included in the git repo.

I see a few possible solutions
uploading a predefined definitions.py file from the machine which spun up the job (hacky but will work) using something like dotenv and an .env environmental variables file
What is the recommended way to deal with this?

Thanks!

  
  
Posted 2 years ago
Votes Newest

Answers 15


Hmm I cannot think of something that will provide something a per user basis.
Wouldn't a global set of credentials that the agent is using be enough ?
(on the local machine, user can keep using the "definitions.py")

  
  
Posted 2 years ago

UpsetTurkey67 The single set of online documentation ( https://clear.ml/docs/latest/docs ), denotes OSS/Free-SaaS/Paid features as such. For example: https://clear.ml/docs/latest/docs/configs/clearml_conf#configuration-vault

  
  
Posted 2 years ago

CloudySwallow27 okay essentially this defs file is kind of a user "secret vault" for access credentials, is that correct?

  
  
Posted 2 years ago

that's the downside

  
  
Posted 2 years ago

exactly

  
  
Posted 2 years ago

you could also use:
https://github.com/allegroai/clearml/blob/ce7e77a00e869a2690f31cbc578636ce88bc4613/docs/clearml.conf#L188
and setup the clearml.conf on the users machine to automatically log the environment variables at run time (stored under the Configuration tab).
Then the agent will pull these same variables at execution time and set them

  
  
Posted 2 years ago

AgitatedDove14 np
FrothyDog40 thanks!

  
  
Posted 2 years ago

gotit. any solution where the credentials wouldnt be visible to everyone?

  
  
Posted 2 years ago

yes 😞

  
  
Posted 2 years ago

AgitatedDove14 do you have two sets of documentations: for the free tier and paid? (our team is on the pro plan, but I might have been using the documentation of the open sourced version, so want to double check)

  
  
Posted 2 years ago

UpsetTurkey67 my apologies I just notices the message

  
  
Posted 2 years ago

sure thing 🙂

  
  
Posted 2 years ago

Right, so this "vault" design is built into the paid tiers of ClearML to achieve exactly that. Long story short, users can put their credentials/configs on the clearml-server and the agent (or the clients) will pull and merge them into the execution.
It's very cool and works really nice, but not part of the open source (or the SaaS tier).
What you could do is store these configurations on the Task itself (one way o r another). Maybe for example have an empty definitions.py file part of the git repository, and then the local changes (i.e. the users credentials) are stored as part pf the uncommitted changes section. This would mean that the agent will replicate the same user's definitions/py into the remote machine, and pass the credentials.
wdyt?

  
  
Posted 2 years ago

ok, i suppose that will have to do for now. thank you!

  
  
Posted 2 years ago

the 2nd option looks good. would everyone's credentials be displayed on the server though?

  
  
Posted 2 years ago
1K Views
15 Answers
2 years ago
one year ago
Tags