thanks for that tip. i cleared out the vcs cache and was already using the latest version of the agent, same problem persists.
there’s a python version mismatch, i will make a different env for the agent to run in that has a matching python version
agent version is
❯ clearml-agent --version
CLEARML-AGENT version 1.5.2
so now i have
git_pass: "[NEW KEY]"
enable_git_ask_pass: false
in my clearml.conf file
yes. had to sanitize it a bit, but left the git username/key intact (since the key is invalid now)
sorry for the delay, had work and personal emergencies 😕
the VCS cache was empty before that run. then, even with the VCS cache being disabled in the config, there was a new lock file and directory after running.
i don’t get why the agent init log would list the username from clearml.conf
but then use the env vars
That's probably an issue we should look at
weird. will move forward with manually recreating the task.
Can you try to change the username as well and see if the agent uses the new one?
further, there’s now data in the VCS cache, even though i disabled it
- stopped agent
- updated clearml.conf to have different username, wrote file
- verified the vcs-cache is empty
- started the agent, which resulted in this output
...
agent.custom_build_script =
agent.disable_task_docker_override = false
agent.git_user = aaaaaaaaaaaaa
agent.default_python = 3.9
...
(that’s the username I changed it to)
- reset and enqueued the task
checkout failed, it’s still attempting to use the old creds
i meant I should have thought to check there earlier! anyway, thanks again for your attention and help! 🙂
Can you share the agent log, specifically when it's trying to clone?
Also, are you sure you cleaned the VCS cache? There's no other place the agent can get the old token from...
Hmmm. Just tried cloning a brand new task and the agent is still using the expired github access token.
Yes, it does. What setting in the file did you use? git_pass
?
thanks much for your help. should have thought to check there earlier, but kind of forgot that was a thing.
also tried disabling the VCS cache in the config—pull this from the output of the agent’s startup output:
agent.vcs_cache.enabled = false
❯ cat ~/clearml.conf | grep git_user
git_user: "aaaaaaaaaaaaa"
❯ cat ~/clearml.conf | grep -A 2 vcs_cache
vcs_cache: {
enabled: false,
path: ~/.clearml/vcs-cache
if i run clearml-agent daemon
that reads from ~/clearml.conf
, right?
Hi @<1523712386849050624:profile|NastyFox63> , I assume this is the vcs cache storing the old token (this is probably an older agent version as this has been fixed in recent versions). I suggest clearing the vcs cache of the agent (should be under ~/.clearml) and upgrading the agent version
yes—am running the agent on a workstation. am sshed into that workstation and verified the change in the conf by explicitly disabling the VCS cache and then looking for that in the agent’s startup output
restarted the server on the off chance that had anything to do with it, and no. VCS is disabled, and the task is trying to pull the correct/latest commit.
Just checking again, can you delete the vcs cache?
~/.clearml/vcs-cache
Also, make sure this is turned on:
None