This incident is resolved. Some customers saw slow or failed requests when they uploaded, opened or deleted files, and when they ran actions.

Async requests (default behaviour) eventually resolved, and all responses were delivered, even if under our usual service level. On some synchronous requests to our API, requests failed.

We increased database capacity and deployed a fix that speeds up file requests.

Since 15:42 EST, requests complete normally.

We apologize for the disruption.

Follow-up on incident
An edge-case bug in our document parser caused it to stall while processing a job.

After a stall, every later document handled by that parser instance would hang until the instance restarted.

On Oct 28, this left many documents from other customers hanging. Each hung document held a slot in our shared file-processing queue until it timed out.

We have deployed permanent fixes for the identified bug, as well as additional measures to distribute jobs initially queued on a potentially stuck worker.

On August 27 between 18:10 and 18:22 UTC, audio transcription processing was delayed for our customers leveraging our pooled inference infrastructure.

Jobs submitted during this window were queued and processed automatically once capacity was restored — no data was lost, no job was rejected, and no action is required from our customers and users.

The delay occurred during a routine scale-up operation when our cloud GPU provider had a temporary capacity shortage.

We have restored full capacity and are widening the range of hardware we can fall back to so this cannot recur.

We've started reporting as different metrics our uptime and incident timelines for our Enterprise Workspace API, our Paperwork API, and our different Inference Services.