the VCS cache was empty before that run. then, even with the VCS cache being disabled in the config, there was a new lock file and directory after running.

yep

ugh, turns out i had a plt.show() in there, that was causing blank figs.

that said, report_matplotlib_figure did not end up putting anything into “plots” or “debug samples”

thanks for that tip. i cleared out the vcs cache and was already using the latest version of the agent, same problem persists.

there’s a python version mismatch, i will make a different env for the agent to run in that has a matching python version

hey Martin.B, wondering if you were able to find anything out about this?

hi SubstantialElk6 , not sure if you were successful on this but i struggled with it as well, and it looks like the information is not in the linked document anymore.

in the end i realized that i needed to download apiserver.conf from the clearml-server repo ( https://github.com/allegroai/clearml-server/blob/master/apiserver/config/default/apiserver.conf ) and then add a user/pass for myself (starting at line 82).

okay, so looks like the docs for report_media need an update.

thanks for doing that and thanks for your work on the project 🙂

❯ cat ~/clearml.conf | grep git_user
    git_user: "aaaaaaaaaaaaa"
❯ cat ~/clearml.conf | grep -A 2 vcs_cache
    vcs_cache: {
        enabled: false,
        path: ~/.clearml/vcs-cache

` $ clearml-agent -d daemon --gpus 1 --foreground
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): DIFFERENT_IP_ADDRESS:8008
DEBUG:urllib3.util.retry:Incremented Retry for (url='/auth.login'): Retry(total=239, connect=3, read=240, redirect=240, status=240)
WARNING:urllib3.connectionpool:Retrying (Retry(total=239, connect=3, read=240, redirect=240, status=240)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff49318dd10>: Failed to ...

here’s a the file with the keys and IP redacted: https://clearml.slack.com/files/U01PN0S6Y67/F0231N0GZ19/clearml.conf

no, it’s a key I don’t recognize

should api.credentials.access_key be the same as the access_key in clearml.conf ?

` $ clearml-agent config
Current configuration (clearml_agent v1.0.0, location: /home/username/clearml.conf):

agent.worker_id =
agent.worker_name = computer
agent.force_git_ssh_protocol = false
agent.python_binary =
agent.package_manager.type = pip
agent.package_manager.pip_version = <20.2
agent.package_manager.system_site_packages = false
agent.package_manager.force_upgrade = false
agent.package_manager.conda_channels.0 = pytorch
agent.package_manager.conda_channels.1 ...

that sounds like all good news to me! thanks for the info 🙂

okay, they are somehow set as environment variables. let me figure out how they were set.

also, i’m noticing the “last used” field does not update when I try to start an agent, but does change when I issue the curl command you gave earlier

of course, SERVER_IP_ADDRESS is the actual IP address of the server, AND i made sure that CLEARML_HOST_IP was set correctly before issuing the docker-compose command

yes—am running the agent on a workstation. am sshed into that workstation and verified the change in the conf by explicitly disabling the VCS cache and then looking for that in the agent’s startup output

looking in the web app, under the “App Credentials” section, it lists those credentials as “used” when I attempted the curl command.

i don’t get why the agent init log would list the username from clearml.conf but then use the env vars

yes, sorry for not catching that earlier—doesn’t seem to change anything

installed all of those using pip

it looks like that worked: https://demoapp.demo.clear.ml/projects/52eb5c9d938244daaa6fa460edce5e22/experiments/40ad94b6544144e38da78d051c6cee59/output/log

yep, i’d be glad to

https://demoapp.demo.clear.ml/projects/55d709bfcbc4429a8ebe4751d9fecd30/experiments/15f54924faa245f2baadc446e1315e39/output/metrics/plots

https://demoapp.demo.clear.ml/projects/55d709bfcbc4429a8ebe4751d9fecd30/experiments/22c5e2ee1d574cb2a453ec0050ddb29f/output/debugImages

i think the fix worked 🙂

yes. had to sanitize it a bit, but left the git username/key intact (since the key is invalid now)

sorry for the delay, had work and personal emergencies 😕

should be posted in the “uncommitted changes” section 🙂