further, there’s now data in the VCS cache, even though i disabled it

• stopped agent
• updated clearml.conf to have different username, wrote file
• verified the vcs-cache is empty
• started the agent, which resulted in this output

...
agent.custom_build_script =
agent.disable_task_docker_override = false
agent.git_user = aaaaaaaaaaaaa
agent.default_python = 3.9
...

(that’s the username I changed it to)

• reset and enqueued the task
  checkout failed, it’s still attempting to use the old creds

i meant I should have thought to check there earlier! anyway, thanks again for your attention and help! 🙂

so now i have

git_pass: "[NEW KEY]"
enable_git_ask_pass: false

in my clearml.conf file

if i run clearml-agent daemon that reads from ~/clearml.conf , right?

also tried disabling the VCS cache in the config—pull this from the output of the agent’s startup output:

agent.vcs_cache.enabled = false

Hmmm. Just tried cloning a brand new task and the agent is still using the expired github access token.

i don’t get why the agent init log would list the username from clearml.conf but then use the env vars

agent version is

❯ clearml-agent --version
CLEARML-AGENT version 1.5.2

yes—am running the agent on a workstation. am sshed into that workstation and verified the change in the conf by explicitly disabling the VCS cache and then looking for that in the agent’s startup output

yes. had to sanitize it a bit, but left the git username/key intact (since the key is invalid now)

sorry for the delay, had work and personal emergencies 😕

will try the git ask pass thing.

okay, that’s a fresh install, and the backend is agg:
` Python 3.8.8 (default, Feb 24 2021, 21:46:12)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.

import matplotlib
matplotlib.get_backend()
'agg' `
the machine is headless, and there’s no window server running.

that sounds like all good news to me! thanks for the info 🙂

hmmm, looks like maybe i should set it directly: https://clearml.slack.com/archives/CTK20V944/p1603369102359500?thread_ts=1603362214.350500&cid=CTK20V944

yep, that was it. thanks for all your help and sorry to bother 🙂

installed all of those using pip

of course, SERVER_IP_ADDRESS is the actual IP address of the server, AND i made sure that CLEARML_HOST_IP was set correctly before issuing the docker-compose command

here’s a the file with the keys and IP redacted: https://clearml.slack.com/files/U01PN0S6Y67/F0231N0GZ19/clearml.conf

okay, they are somehow set as environment variables. let me figure out how they were set.

well, as generated by clearml-agent init —i pasted the text directly from the web app into the CLI interface, and it generated clearml.conf

i did this and have the same error.

hmm, it was confusing to me, but it’s kind of an edge case where I was taking over a computer after a colleague left, seems like that might not be a common scenario

i’ll try changing the IP and look for a different error.

yes, that call appeared to be successful—had to wrap in quotes because of the contents of the key:
$ curl -u 'J9*****':'R2*****' `
{"meta":{"id":"6db9ae72249f417fa2b6b8705b44f38a","trx":"6db9ae72249f417fa2b6b8705b44f38a","endpoint":{"name":"users.get_current_user","requested_version":"2.13","actual_version":"1.0"},"result_code":200,"result_subcode":0,"result_msg":"OK","error_stack":null,"error_data":{}},"data":{"user":{"company":{"id":"d1bd92a3b039400cbafc60a7a5b1e52b","name":"trains"},...

no, it’s a key I don’t recognize

🤔

Media is uploaded to a preconfigured bucket (see setup_upload()) with a key (filename) describing the task ID, title, series and iteration.

ugh, turns out i had a plt.show() in there, that was causing blank figs.

that said, report_matplotlib_figure did not end up putting anything into “plots” or “debug samples”

` $ clearml-agent -d daemon --gpus 1 --foreground
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): DIFFERENT_IP_ADDRESS:8008
DEBUG:urllib3.util.retry:Incremented Retry for (url='/auth.login'): Retry(total=239, connect=3, read=240, redirect=240, status=240)
WARNING:urllib3.connectionpool:Retrying (Retry(total=239, connect=3, read=240, redirect=240, status=240)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff49318dd10>: Failed to ...

yes, i can do this again. i did use clearml-agent init to generate clearml.conf after generating a fresh set of keys