Hi everyone,
I'm experiencing an issue with ClearML running on K8S. After upgrading the ClearML server helm chart from version 7.11.5, I'm seeing the following errors:

In the agent:

[2025-02-16 17:26:45,889] [9] [WARNING] [clearml.service_repo] Returned 400 for tasks.enqueue in 4ms, msg=Validation error (Cannot skip setting execution queue for a task that is not enqueued or does not have execution queue set)

In the clearml-server-api-pod:

[2025-02-16 19:13:45,658] [9] [WARNING] [clearml.service_repo] Returned 400 for queues.remove_task in 3ms, msg=Invalid queue id or task not in queue: task=37c6ce31c53d449994f7c9096c26d6f7, id=99d6dd77e67a4f12bb5de901596e0e1e, company=d1bd92a3b039400cbafc60a7a5b1e52b

[2025-02-16 19:13:45,669] [9] [WARNING] [clearml.service_repo] Returned 400 for tasks.enqueue in 3ms, msg=Validation error (Cannot skip setting execution queue for a task that is not enqueued or does not have execution queue set)

I've tried several versions, including the latest 7.14.2, but the error persists. For testing, I'm using a simple pipeline:

import clearml
from clearml import PipelineController

pipe = PipelineController(
    name='simple-pipeline',
    project='hello-world-project',
    version='1.0.0',
)

pipe.set_default_execution_queue('default')

def say_hello():
    print("Hello World!")
    return {"message": "Hello World!"}

pipe.add_function_step(
    name='hello-step',
    function=say_hello,
    function_return=['hello_result']
)

pipe.start(queue='default')

I believe the issue lies with the clearml-apiserver. When I downgrade the clearml-apiserver image in the helm chart back to version 1.16.2-502, the agent successfully picks up the job.

Additional information:

• My Kubernetes version is 29.2.10
• I've reproduced this issue on other K8s versions
• The problem persists even when using the default values.yaml