Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Hi All, I Have An Ongoing Issue With Queues.

Hi all, I have an ongoing issue with queues.

  • I send a task to my_queue which does not have any listening agents. This is done using task = Task.init(...) and task.execute_remotely("my_queue") .
  • When I look in Queues ( None ) I see my task and I see that the task is in Pending state in projects.
  • After a few minutes (it varies from a minute to 6 minutes), the task disappears from the queue. There are no listening agents so there is no reason for it to be gone. Furthermore, when I look at the Task, it is still in a Pending state. I have confirmed this using the python SDK.
  • This has led me to conclude that there is a bug in the backend for the queues.
    This is a major issue for us because we use the AWS Autoscaler for basically all our compute and when I send a job to the queue, the Autoscaler creates a new instance but by the time that instance is ready, the Task is no longer in the queue (but is still in Pending!) so it doesn't get picked up.

Has anyone else experienced this behaviour?


Posted 8 months ago
Votes Newest



I've managed to fix it.

Basically, I had a tracker running on our queues to ensure that none of them were lagging. This was using get_next_task from APIClient().queues .

If you call get_next_task it removes the task from the queue but does not put it into another state. I think because typically get_next_task is immediately followed by something to make the task run in the daemon or delete it.

Hence you end up in this weird state were the task thinks its queued because nothing modified the status but the queue doesn't think it has a task. A bit of an edge case but potentially worth looking at.

Posted 8 months ago
1 Answer
8 months ago
8 months ago
Similar posts