the VCS cache was empty before that run. then, even with the VCS cache being disabled in the config, there was a new lock file and directory after running.

further, there’s now data in the VCS cache, even though i disabled it

will try the git ask pass thing.

good questions 🙂
they are plots. they have unique titles. i’m using the auto-logging mechanism—so set up the task, then plt.show() no more than 114 plots are shown in the plots tab.

since it’s probably relevant—i have to use the Agg backend since the machine is headless

yep, that was it. thanks for all your help and sorry to bother 🙂

yep, that’s what i’m seeing, they’re all PNGs in that folder.

yes, i see no more than 114 plots in the list on the left side in full screen mode—just checked and the behavior exists on safari and chrome

i’ve just verified that they’re all writen to /opt/clearml/data/fileserver/[PROJECT_NAME]/[DESCRIPTION]/metrics

running my own clearml server with a vanilla config (obtained from github), except i have one fixed user

wondering if there has been an update on this?

yes, sorry for not catching that earlier—doesn’t seem to change anything

installed all of those using pip

yep

okay, that’s a fresh install, and the backend is agg:
` Python 3.8.8 (default, Feb 24 2021, 21:46:12)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.

import matplotlib
matplotlib.get_backend()
'agg' `
the machine is headless, and there’s no window server running.

very strange

hey Martin.B, wondering if you were able to find anything out about this?

that sounds like all good news to me! thanks for the info 🙂

this also fixed a couple other bugs i was seeing. Thanks very much to you for your help and please pass my thanks on to the team as well.

i think the fix worked 🙂

we do use all those packages, and the version numbers are correct

in the main script, these are the first imports:
import argparse import time import json import pytorch_lightning as pl from pytorch_lightning.accelerators import acceleratorthen after that we import stuff from the repo, and the listed packages are imported in those files

i checked the exact same thing

$ conda list | grep pandas geopandas 0.9.0 pyhd8ed1ab_1 conda-forge geopandas-base 0.9.0 pyhd8ed1ab_1 conda-forge pandas 1.3.3 py39hde0f152_0 conda-forge

$ conda list | grep matplotlib matplotlib 3.4.3 py39hf3d152e_1 conda-forge matplotlib-base 3.4.3 py39h2fa2bec_1 conda-forge

yeah, it’s in one of the imports from the repo

and it’s in the “installed packages” from the child task:
` absl-py==0.14.0
aiohttp==3.7.4.post0
async-timeout==3.0.1
attrs==21.2.0
cachetools==4.2.2
certifi==2021.5.30
chardet==4.0.0
charset-normalizer==2.0.6
clearml==1.1.1
cycler==0.10.0
Cython==0.29.24
fsspec==2021.9.0
furl==2.1.2
future==0.18.2
google-auth==1.35.0
google-auth-oauthlib==0.4.6
grpcio==1.40.0
idna==3.2
joblib==1.0.1
jsonschema==3.2.0
kiwisolver==1.3.2
Markdown==3.3.4
matplotlib==3.4.3
multidict==5.1.0
numpy==1.21.2
oauthlib=...

getting different issues (torchvision vs. cuda compatibility, will work on that), but i’m betting that was the issue

i’ll clone and enqueue, but i’m guessing that’s the issue

but, the call used to start the script was python -m module.name --args

actually no