is there a way to assign a job to a specific worker? or is it only working on queue level

Only on a queue level, but you can have as many as you like and spin the agent on it (notice you can have multiple queues on the same agent, pulling based on priority/order).