Previous incidents

Customers who were exceeding their concurrency limit during this outage may be encountering strange errors in their crawls.

This is due to the nature of the outage affecting our concurrency limiting system, causing unique jobs to be inserted into a separate per-team concurrency queue multiple times. When these jobs are promoted from the concurrency queue into the main queue, they retain their IDs, which causes a violation of the unique ID constraint.

A fix for this issue is now deployed.

The issue is now resolved. As part of the hotfix, all ongoing concurrency queued jobs had to be discarded, causing ongoing crawls to freeze -- please reach out to support if you are affected. We apologize for the disturbance, and thank you for your patience.

The issue has been resolved. Thank you for your patience.

We've shipped a fix and the incident is now resolved. Thank you for your patience.

The elevated API timeouts have been fully resolved. All services are operating normally. We apologize for the disruption and appreciate your patience.

We have identified and pushed a hotfix for the root cause. We are working on a permanent fix.

The issue is now resolved. We apologize for the inconvenience. The maintenance we had previously scheduled for Sunday will address the root cause of this downtime.

Resolved

The incident has been resolved.

We are back!

The issue was resolved.

The system has recovered. We have escalated the root cause to our upstream provider.

We are back.

Upstream provider has resolved the issue.

firecrawl.dev recovered.

We are back. Job timeout metrics have recovered to pre-incident levels.

The issue came down to misconfigured pipeline queue limits on Dragonfly -- we configured them with high values expecting a high load on production, however, they ended up being ridiculously high. This caused Dragonfly's backpressure mechanisms to kick in way too late, only when the state of the instance is practically already unsalvageable. The configuration was tuned, and we will continue to monitor this.

The API has recovered.

We quickly restarted the workers to get them un-stuck again. After that, we revisited the core issue. A lot of jobs were finishing at the same time, started hammering the same sorted set in Redis at the same time, and ran into the same race conditions. We decided to break up these clumps of requests by applying a timeout of random length before retrying every time the anti-race mechanism activates. This spreads out the workers nicely.

After applying the fix and observ...

Service is restored. Crawls that appeared "stuck" should now resume.

The issue is fully resolved. Apologies for the disruption and thank you for your patience.

The issue was caused by a load spike, triggering a scale-up to a high amount of API and Worker pods. These pods interface heavily with our Dragonfly (Redis-equivalent) instance via BullMQ for job queueing. The increased connections and requests to Dragonfly caused it to start having to queue pipeline operations, which caused BullMQ operations to have a delay, making the system fail and scrape jobs accu...