- name: CLEARML__APISERVER__PRE_POPULATE__ENABLED value: "false" - name: CLEARML__APISERVER__PRE_POPULATE__ZIP_FILES value: /opt/clearml/db-pre-populate - name: CLEARML_SERVER_DEPLOYMENT_TYPE value: helm-cloud
The rest are clearly credentials…
I'm not pretty sure multi-threading is the reason for this issue, and that restarting will solve it (but you will still need to move your new tasks from the trash collection to the normal collection).
However, I would like to understand the deployment changes you made since if you do not fix them, this might happen again...
SuccessfulKoala55 Hi Jake
We didn’t change anything related to gunicorn. Is there any specific thing I can check for?
Also I noticied that it’s not running the gunicorn as a command but loads it in the python code, I don’t think it’s possible to change the threading with env that way.
FiercePenguin76 I have a theory that this is cause by a thread-safety issue - the apiserver code-base is not designed to run in multiple threads right now and scale is handles by processes. Enabling threads in gunicorn may in theory cause this exact behavior
I assume it's only triggered in mass deletes for now, so your options either to wait for a patch server version (a new version is about to be released, so we'll either make it to this version or push a patch version immediately after), or change your deployment to use gunicorn (which is a change of behavior, I know)