Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, Clearml Console Leaks Credentials Passed In As Env Vars. The Issue Remains With Clearml Version==1.1.1.135 - 1.1.1 - 2.1.4 (As Listed On The Profile Page) I Am Using K8S Glue And The Clearml.Conf Has The Following In The Agent Section.

Hi, ClearML console leaks credentials passed in as Env Vars. The issue remains with clearml version==1.1.1.135 - 1.1.1 - 2.1.4 (As listed on the profile page)
I am using K8S Glue and the clearml.conf has the following in the agent section.

hide_docker_command_env_vars { enabled: true extra_keys: ['TRAINS_AGENT_GIT_USER','TRAINS_AGENT_GIT_PASS','AWS_ACCESS_KEY','AWS_SECRET_ACCESS',''] }
This is an extract of my codes run_remote.py
` import os
TRAINS_AGENT_GIT_USER="gituser"
TRAINS_AGENT_GIT_PASS="gitpass"
AWS_ACCESS_KEY=os.environ.get("AWS_ACCESS_KEY")
AWS_SECRET_ACCESS=os.environ.get("AWS_SECRET_ACCESS")
print("AWS_ACCESS_KEY: ", AWS_ACCESS_KEY)
print("AWS_SECRET_ACCESS: ", AWS_SECRET_ACCESS)

from clearml import Task, Logger
task = Task.init(project_name='DETECTRON2',task_name='DefaultModelArchitecture',task_type='training')
task.set_base_docker("harbor.ai/public/detectron2:v3 --env GIT_SSL_NO_VERIFY=true --env TRAINS_AGENT_GIT_USER=" +TRAINS_AGENT_GIT_USER+" --env TRAINS_AGENT_GIT_PASS="+TRAINS_AGENT_GIT_PASS + " --env AWS_ACCESS_KEY="+AWS_ACCESS_KEY + " --env AWS_SECRET_ACCESS="+AWS_SECRET_ACCESS)
task._update_requirements({})
task.execute_remotely(queue_name="2xV100-32ram", exit_process=True) I run my codes in the following manner on my client AWS_ACCESS_KEY=mykey AWS_ACCESS_ACCESS=myaccess python run_remote.py `
What happens is on my ClearML Server, i still see the secrets printed in various portions of the web display.
See screenshots.
https://user-images.githubusercontent.com/2820927/131271328-8df37da6-a0ad-4c54-af37-3b2fc1dc6ced.png
https://user-images.githubusercontent.com/2820927/131271336-c1993b17-dd62-4227-a826-d20df3b54ea9.png

  
  
Posted 3 years ago
Votes Newest

Answers 22


ok, i'll wait till i get my hands on vault then. thanks.

  
  
Posted 3 years ago

👍

  
  
Posted 3 years ago

Ah ok. So it will be fixed on the ClearML server web UI as well? (See screenshots).

  
  
Posted 3 years ago

Can this issue be solved with vault? It doesn't make sense to expose secrets like that.

  
  
Posted 3 years ago

SubstantialElk6 this feature was only introduced in the 1.0.1 RC releases (i.e. it is not part of v1.0.0) - you can either use one of the RC releases, or wait for the official v1.0.1 (will be released very soon).

I also see this on my logs, noting that the config is read in but its still printing the supposedly hidden keys on the logs and UI.

agent.hide_docker_command_env_vars.enabled = true

agent.hide_docker_command_env_vars.extra_keys.0='TRAINS_AGENT_GIT_USER'

The agent prints all of the configuration it loads, regardless of whether any part of the code uses it or not...

  
  
Posted 3 years ago

I also see this on my logs, noting that the config is read in but its still printing the supposedly hidden keys on the logs and UI.
agent.hide_docker_command_env_vars.enabled = true agent.hide_docker_command_env_vars.extra_keys.0='TRAINS_AGENT_GIT_USER' ..... docker_cmd=harbor.ai/public/detectron2:v3 --env TRAINS_AGENT_GIT_USER=gituser

  
  
Posted 3 years ago

Hi SuccessfulKoala55 ,i managed to install clearml-agent==1.0.1rc5. However, the same issues occur.

  
  
Posted 3 years ago

Its 1.0.0. As printed on the top of the logs in ClearML Server UI.

  
  
Posted 3 years ago

Hi SubstantialElk6 , no worries, the reason it's not working for you is probably due to the fact that it's still in RC, and the agent version installed in the K8s POS is probably of the previous official version (or some older version). Can you share the more of the agents log (from the start)?

  
  
Posted 3 years ago

I didn't realize we were talking about the config printout, since you've added these env vars to the default docker...

  
  
Posted 3 years ago

This is the log i extracted.

  
  
Posted 3 years ago

Oh! I see now! 🙂

  
  
Posted 3 years ago

Also, in the UI we can't really know your agent setting (also, you probably have multiple agents), so we can't know which env var should be masked...

  
  
Posted 3 years ago

As for the UI - that's a good question - how can we hide it there if it's also editable? Does it make sense to let you edit it but not see it?

  
  
Posted 3 years ago

That's an easy fix, you'll see it in the next RC 🙂

  
  
Posted 3 years ago

The vault (as part of the pair version) can certainly solve it, without the need for env vars that can be visible in certain scenarios (like you showed)

  
  
Posted 3 years ago

Hi SubstantialElk6 , what's your agent / ask version?

  
  
Posted 3 years ago

Hi just wondering if I did something wrong here. Would k8s-glue be the reason is not working? I'm purchasing the enterprise version and if vault has the same problem it'll be a big issue.

  
  
Posted 3 years ago

For some reason, I thought it was when the agent was executing the task, not when it prints out the configuration...

  
  
Posted 3 years ago

and out of curiosity, what did you think we were talking about? cos i didn't see anywhere else that might print the secrets.

  
  
Posted 3 years ago

Well, you're indeed running 1.0.1rc5 in there

  
  
Posted 3 years ago

Hi, by agent logs i suppose you meant the logs from the ClearML server console panel?

  
  
Posted 3 years ago
1K Views
22 Answers
3 years ago
one year ago
Tags