Fail-open and fail-closed

Designed to fail open: if the gateway degrades, traffic goes straight to your provider. Armed caps refuse rather than guess; so does a kill switch the gateway can read.

That is the whole design in two sentences. The gateway sits in the path of your production traffic, so a fault in anything Recovea added must not become a failure your traffic sees. The exceptions are the stops you set: you asked for a stop, and a stop that gives way when something breaks is not a stop.

What fails open

• A fault in the gateway's own admission layer. If a node has not yet received the key table from the control plane, or a local cache or the rate limiter faults, the request is passed through instead of refused, unless the kill switch is on for its key or organization, which is checked first once the key table has been received and the node can read the switches it holds; a node that has the key table but cannot read them knows no stop. On that pass the gateway cannot tell which organization the request belongs to, so it has nothing to sign with: your Recovea key is removed before the request is forwarded, the provider key Recovea holds for you is not used, and no cap can be looked up for it. The pass is counted, and it raises an alarm on our side.
• The control plane going away. Every decision is made from state held in the node's own memory. If the feed of caps and keys from the control plane stops, the node keeps serving on the last state it received; the state grows stale, and traffic keeps flowing.
• Metering. The metering event is written after the response has been sent, on a path your request does not wait for. A metering fault never delays or fails a request.

What fails open is not metered. A fail-open pass has no organization to attribute a reading to, so it writes no receipt; a receipt covers what routed through the gateway and was admitted, and nothing else.

What fails closed

• An armed cap at its line. The request is refused with 402 before your provider is called (Cap).

• An armed cap that cannot confirm its spend. A node that has not confirmed its spend counters within the last sixty seconds will not enforce a figure it cannot trust, and it will not let the spend run unwatched either. It refuses:

  Spend could not be confirmedCopy

  HTTP/1.1 503 Service Unavailable
  retry-after: 1
  x-recovea-budget-scope: organization
  x-recovea-budget-window: month
  
  {
    "error": {
      "message": "Monthly spend cannot be confirmed right now. Retry in a moment.",
      "type": "insufficient_quota",
      "param": null,
      "code": "budget_unverifiable"
    }
  }

  The message names the window of the cap that could not be confirmed, and never says a budget was reached, because none was. It clears on its own as soon as the node's counters are confirmed again.

• The kill switch. While it is on for a key or for the organization, and the gateway holds the organization's current keys and switches, every request it covers is refused with 402 and kill_switch_on. Where a cap would answer 503 because its spend cannot be confirmed, or where a throttling fault would let a request through, the kill switch still refuses (The kill switch). A gateway node that cannot read them knows no stop (What fails open).

• A key the gateway does not know. An unknown or revoked key is refused 401, the same answer for both, and nothing is forwarded.

A cap with its hard stop off never refuses, so it never fails closed: it counts and alerts, and traffic flows whether or not its spend can be confirmed. Every cap on Free is the same once the organization's month has reached the Free ceiling (On Free).

The two refusals do not share a status

On one node, over one route, spend that has been reached answers 402 with no retry hint, and spend that cannot be confirmed answers 503 with Retry-After: 1. Until 2026-09-14 both answered 402, and a counter store that blipped for a second told a correct integration its caller was out of credit. Both bodies carry error.type insufficient_quota, and so does the kill switch's 402, so read the status and error.code together:

When the provider cannot be reached

When your provider cannot be reached at all, the gateway answers 502 with error.code upstream_unreachable and the message "Upstream provider unreachable." Every other error your provider returns is passed back to you unchanged.