Hi SubstantialElk6
I think you are absolutely correct, it seems the glue pops all the arguments, when in fact it should maybe process them a,d convert the --env/-e
What do you think?
Aloso I assume if these are the default arguments they should actually be part of the k8s apply.yaml template no ?
Ohh, I see now, yes that should be fixed as well 🙂
Hi SubstantialElk6
We can't seem to find a way for the end user to pass in their git credentials when they run their codes in both agent and non-agent scenarios. Any advice here?
The bottom line is the agent needs to have read-only access to all the repositories so it can launch any Task. I would recommend to create an agent git user with read-only credentials and configure the agent to use it. wdyt?
AgitatedDove14 , will these be fixed?
Passing env via the code Passing env via template yaml
These are both specific cases of the glue, and yes both need to be fixed.
(1) I think is actually a feature, nonetheless we should support it.
FriendlySquid61 could you verify specifically on (2)
Hmm that sounds like the agent needs to access a vault with credentials per user, unfortunately this is not covered in the open-source 😞 I "think" this is supported in the enterprise version as part of the permission management
The apply.yaml template is not working (E.g. the arguments env is not passed to the container), this is why i tried the code approaach instead.
Hi AgitatedDove14 , i was refering totask.set_base_docker("nvcr.io/nvidia/tensorflow:19.11-tf2-py3 --env TRAINS_AGENT_GIT_USER=git_username_here --env TRAINS_AGENT_GIT_PASS=git_password_here")The above will give errorskipping docker argument TRAINS_AGENT_GIT_USER=git_username_here (only -e --env supported) TRAINS_AGENT_GIT_PASS=git_username_here (only -e --env supported)
Hi, just wondering if this 'feature: Passing env via the code' is in the works?
https://clearml.slack.com/archives/CTK20V944/p1616677400127900?thread_ts=1616585832.098200&cid=CTK20V944
Thanks SubstantialElk6 !
I believe an initial a fix was pushed 😉 A full one (merging Task --env with k8s template) will be added soon
SubstantialElk6 (2) yes definitely will be fixed
Regrading (1), what do you mean by "via the code" ? Do you mean like as a Task docker cmd ?
I can't think of any hack that will satisfy your IT other than than an actual vault...
wdyt?
Hi AgitatedDove14 . I'm trying out passing env via the code instead.task.set_base_docker("nvcr.io/nvidia/tensorflow:19.11-tf2-py3 --env TRAINS_AGENT_GIT_USER=git_username_here --env TRAINS_AGENT_GIT_PASS=git_password_here")So the strange thing is when my k8sglue pulls a task, this happens.Pulling task xxxxxxxxxx launching on kubernetes cluster Pushing task xxxxxxxxxx into temporary pending queue Kubernetes scheduling task id=xxxxxxxxxxxx skipping docker argument TRAINS_AGENT_GIT_USER=git_username_here (only -e --env supported) TRAINS_AGENT_GIT_PASS=git_username_here (only -e --env supported) FATAL ERROR: blah blah.. clear_agent/glue/k8s.py env=docker_args.pop().strip() IndexError: pop from empty listAfter looking through, i noted that the pop method pops from the last item (instead of first item), so by the time it reaches the --env, there's nothing else to pop hence the error.
I fixed that and got another error pertaining to a key in this line [ https://github.com/allegroai/clearml-agent/blob/cb6bdece39751eaef975287609b8bab603f116e5/clearml_agent/glue/k8s.py#L401 ]
If i set the key to 'env', i get some kubectl errors instead.
By the way, this is linked to [ https://clearml.slack.com/archives/CTK20V944/p1616398936061300 ]
No i didn't indicate this particular issue on the git issue. Only the apply template.yml is on the issue.
Could you please add it, I really do not want to miss it 🙂
unfortunately, our security posture is so strict that we cannot have an agent git user that have unfettered read access to all repos.
Hi FriendlySquid61 , AgitatedDove14 , the issue and possible fix is in this issue raise. https://github.com/allegroai/clearml-agent/issues/51