Spend caps, and what happens at the edge

Most gateways reconcile a budget after the fact and fail the request when it is gone. This one clamps the response instead.

Reserve, clamp, settle

  1. The gateway estimates the prompt cost and reserves it against the key’s remaining budget.
  2. It works out how many output tokens the remaining budget affords and clamps max_tokens down to fit.
  3. When the response completes it releases the hold and settles the real cost.

Only when the budget cannot cover even a minimal response does the call fail, with 402.

No ceiling of our own

A request that names no max_tokens gets the model’s own headroom, its context length minus the prompt, held under a configured default. The cap is the limit, not an arbitrary number picked by the gateway.

Disconnects

Output is counted as it streams and settles on whichever comes first, the end of the response or the client hanging up. A cancelled stream bills what was generated, not a timeout’s worth of nothing.

NextBilling