Follow-up on incident
An edge-case bug in our document parser caused it to stall while processing a job.
After a stall, every later document handled by that parser instance would hang until the instance restarted.
On Oct 28, this left many documents from other customers hanging. Each hung document held a slot in our shared file-processing queue until it timed out.
We have deployed permanent fixes for the identified bug, as well as additional measures to distribute jobs initially queued on a potentially stuck worker.
Resolved
Follow-up on incident
An edge-case bug in our document parser caused it to stall while processing a job.
After a stall, every later document handled by that parser instance would hang until the instance restarted.
On Oct 28, this left many documents from other customers hanging. Each hung document held a slot in our shared file-processing queue until it timed out.
We have deployed permanent fixes for the identified bug, as well as additional measures to distribute jobs initially queued on a potentially stuck worker.
Resolved
The incident is resolved, and queues were cleared up properly.
Identified
We've identified the issue as a combination of our organization rate limiting system, and a stuck processing queue for one of the document types we process.
A restart of the stuck processor started clearing the queue, and we've temporarily raised rate limits for all affected user accounts to help process the backlog faster.
Investigating
We're investigating reports of slow or incomplete parsing runs in our Paperwork API.
