Also, in the link above there is the warning

Do not enqueue training or inference tasks into the services queue. They will put an unnecessary load on the server.

I am not using the dedicated

services

queue on the server but I am doing training and inference in the pipeline component.

Steps of a pipeline should have dedicated queues with relevant resources to them