i meant I should have thought to check there earlier! anyway, thanks again for your attention and help! 🙂

We always aim to improve 🙂

thanks much for your help. should have thought to check there earlier, but kind of forgot that was a thing.

That's probably an issue we should look at

i don’t get why the agent init log would list the username from clearml.conf but then use the env vars

there were env variables set in .zshrc

:face_palm:

no change in behavior and enable_git_ask_pass does not show up in the agent init output

so now i have

git_pass: "[NEW KEY]"
enable_git_ask_pass: false

in my clearml.conf file

the VCS cache was empty before that run. then, even with the VCS cache being disabled in the config, there was a new lock file and directory after running.

will try the git ask pass thing.

Just checking again, can you delete the vcs cache?

~/.clearml/vcs-cache

Also, make sure this is turned on:
None

yes. had to sanitize it a bit, but left the git username/key intact (since the key is invalid now)

sorry for the delay, had work and personal emergencies 😕

Can you share the agent log, specifically when it's trying to clone?

❯ cat ~/clearml.conf | grep git_user
    git_user: "aaaaaaaaaaaaa"
❯ cat ~/clearml.conf | grep -A 2 vcs_cache
    vcs_cache: {
        enabled: false,
        path: ~/.clearml/vcs-cache

further, there’s now data in the VCS cache, even though i disabled it

• stopped agent
• updated clearml.conf to have different username, wrote file
• verified the vcs-cache is empty
• started the agent, which resulted in this output

...
agent.custom_build_script =
agent.disable_task_docker_override = false
agent.git_user = aaaaaaaaaaaaa
agent.default_python = 3.9
...

(that’s the username I changed it to)

• reset and enqueued the task
  checkout failed, it’s still attempting to use the old creds

Can you try to change the username as well and see if the agent uses the new one?

yes

Yes, it does. What setting in the file did you use? git_pass ?

if i run clearml-agent daemon that reads from ~/clearml.conf , right?

yes—am running the agent on a workstation. am sshed into that workstation and verified the change in the conf by explicitly disabling the VCS cache and then looking for that in the agent’s startup output

the clearml.conf used by the agent?

i updated the token in ~/clearml.conf , was careful to ensure it was only specified in one place

Also, are you sure you cleaned the VCS cache? There's no other place the agent can get the old token from...

How did you update the new token?

restarted the server on the off chance that had anything to do with it, and no. VCS is disabled, and the task is trying to pull the correct/latest commit.

Hmmm. Just tried cloning a brand new task and the agent is still using the expired github access token.

weird. will move forward with manually recreating the task.

agent version is

❯ clearml-agent --version
CLEARML-AGENT version 1.5.2