Fail-open and fail-closed

Designed to fail open: if the gateway degrades, traffic goes straight to your provider. The only stop is a cap you armed.

That is the whole design in two sentences. The gateway sits in the path of your production traffic, so a fault in anything Recovea added must not become a failure your traffic sees. The one exception is a cap you armed: you asked for a stop, and a stop that gives way when something breaks is not a stop.

What fails open

• A fault in the gateway's own admission layer. If a node has not yet received the key table from the control plane, or a local cache or the rate limiter faults, the request is passed through instead of refused. On that pass the gateway cannot tell which organization the request belongs to, so it has nothing to sign with: your Recovea key is removed before the request is forwarded, the provider key Recovea holds for you is not used, and no cap can be looked up for it. The pass is counted, and it raises an alarm on our side.
• The control plane going away. Every decision is made from state held in the node's own memory. If the feed of caps and keys from the control plane stops, the node keeps serving on the last state it received; the state grows stale, and traffic keeps flowing.
• Metering. The metering event is written after the response has been sent, on a path your request does not wait for. A metering fault never delays or fails a request.

What fails open is not metered. A fail-open pass has no organization to attribute a reading to, so it writes no receipt; a receipt covers what routed through the gateway and was admitted, and nothing else.

What fails closed

• An armed cap at its line. The request is refused with 402 before your provider is called (Cap).

• An armed cap that cannot confirm its spend. A node that has not confirmed its spend counters within the last sixty seconds will not enforce a figure it cannot trust, and it will not let the spend run unwatched either. It refuses:

  Spend could not be confirmedCopy

  HTTP/1.1 503 Service Unavailable
  retry-after: 1
  x-recovea-budget-scope: organization
  x-recovea-budget-window: month
  
  {
    "error": {
      "message": "Monthly spend cannot be confirmed right now. Retry in a moment.",
      "type": "insufficient_quota",
      "param": null,
      "code": "budget_unverifiable"
    }
  }

  The message names the window of the cap that could not be confirmed, and never says a budget was reached, because none was. It clears on its own as soon as the node's counters are confirmed again.

• A key the gateway does not know. An unknown or revoked key is refused 401, the same answer for both, and nothing is forwarded.

A cap with its hard stop off never refuses, so it never fails closed: it counts and alerts, and traffic flows whether or not its spend can be confirmed.

The two refusals do not share a status

On one node, over one route, spend that has been reached answers 402 with no retry hint, and spend that cannot be confirmed answers 503 with Retry-After: 1. Until 2026-09-14 both answered 402, and a counter store that blipped for a second told a correct integration its caller was out of credit. Both bodies carry error.type insufficient_quota, so read the status and error.code together:

When the provider cannot be reached

When your provider cannot be reached at all, the gateway answers 502 with error.code upstream_unreachable and the message "Upstream provider unreachable." Every other error your provider returns is passed back to you unchanged.