The problem: runaway AI agents and surprise bills

There's a pattern that keeps showing up in developer communities: someone runs an AI agent overnight, something loops, and they wake up to a bill they didn't expect. A recent case that got attention involved a multi-agent LangChain setup — an analyzer and a verifier stuck in a loop for 11 days, burning $47,000 before anyone noticed. The cost curve looked like normal product growth until it very much wasn't.

The OpenAI platform has soft cap alerts. You can set a notification at 80% of a monthly limit. But that's reactive — it tells you after money is spent, not before. And in an agentic workflow where a loop can double your spend every few hours, "we'll notify you at 80%" is not a safety mechanism. It's a receipt.

What I wanted was simple: a hard stop. Not "warn me when I'm close." Stop the requests. Enforce the number. Make it impossible to overspend by more than one request worth.

So I built a proxy.

What I wanted: a kill-switch, not a dashboard

The distinction matters. A dashboard gives you visibility. A kill-switch gives you control.

Most cost management tools in the AI space are dashboards — they show you what you spent, maybe break it down by model or endpoint, maybe send an alert. All useful. None of them stop a runaway loop.

What I wanted was a single URL I could point my OpenAI calls at. I'd set a dollar cap when I create a token. Every request would check that cap before it forwarded. When it's hit, requests stop — not with a warning, with a 429. The loop fails. The bill stops.

No dashboard to babysit. No webhook to configure. No "you've reached 80% of your limit" email to check on a Saturday morning. Just: you set a number, and that number holds.

The architecture: one Rails app, one Postgres row

The whole thing is simpler than it sounds.

your app
   │
   ▼
POST /proxy          ← ai_budget proxy (Rails + Fly.io)
   │
   ├── 1. Authenticate caller (Bearer token → ProxyToken row)
   ├── 2. Check budget (row lock → exceeded? → 429 or continue)
   ├── 3. Forward to OpenAI (Faraday, caller's key in-memory)
   ├── 4. Record cost (atomic UPDATE on usage_millicents)
   └── 5. Return response

The data model is one table:

create_table "proxy_tokens" do |t|
  t.string  "token",            null: false
  t.string  "email"
  t.string  "label"
  t.integer "limit_millicents", default: 0, null: false
  t.integer "usage_millicents", default: 0, null: false
  t.boolean "revoked",          default: false, null: false
end

Amounts are stored in millicents (1/100,000 of a dollar) to avoid floating-point arithmetic entirely. A $10 cap is stored as 1_000_000_000. When you hit the limit, usage_millicents >= limit_millicents and the request stops.

No Redis. No Sidekiq. No pub/sub. One Postgres row per token, one request path, one machine on Fly.io in ams.

The two parts that mattered: atomic enforcement and key passthrough

Atomic enforcement

The tricky part of a budget proxy isn't checking the budget — it's checking it correctly under concurrent requests.

Here's the naive version of the problem: two requests arrive simultaneously. Both read usage_millicents = 9_950_000 against a limit_millicents = 10_000_000 cap. Both see "not exceeded." Both forward to OpenAI. Both come back and increment the counter. You end up at 10_050_000 — over budget, but the check already passed.

The fix is a row lock. Before checking, acquire a lock on the row. Any concurrent request for the same token has to wait until the lock releases before it can read. This serialises the check, so the second request sees the updated value:

def enforce_budget
  exceeded = @current_token.with_lock { @current_token.exceeded? }
  return unless exceeded

  render(
    json: { errors: [{ status: 429, detail: "Budget exceeded" }] },
    status: :too_many_requests
  )
end

For recording usage after a response comes back, I use update_counters — a single atomic SQL UPDATE that increments the column in the database rather than reading, modifying, and writing back in application code:

def record_usage!(millicents)
  self.class.update_counters(id, usage_millicents: millicents)
end

This means the increment is safe even if multiple requests finish at the same time.

Key passthrough

The second design decision worth explaining: I never store your OpenAI API key.

The proxy uses two separate authorization headers. The first authenticates you to the proxy:

Authorization: Bearer <your_proxy_token>

The second carries your OpenAI key:

X-Provider-Authorization: Bearer <your_openai_key>

The proxy reads X-Provider-Authorization from the incoming request, holds it in memory, and passes it directly to OpenAI on the forwarded request. It is never written to the database, never logged, never persisted anywhere. If you grep the codebase for where the provider key is stored — it isn't.

This has a real tradeoff: you have to send your OpenAI key on every request. That's a deliberate choice. The alternative — storing it server-side — is a security liability I wasn't willing to accept for a tool designed to proxy sensitive API credentials.

Production gotchas

The SSL crash

The first latency run I did locally gave sensible numbers. The first run against the Fly deployment failed immediately with a redirect loop.

The cause: force_ssl = true in Rails sends an HTTP-to-HTTPS redirect for every plain HTTP request. Fly.io terminates TLS at its edge and forwards traffic to the app over plain HTTP on the internal network. So the proxy was redirecting Fly's own healthcheck — and everything else — to HTTPS, which Fly had already handled.

The fix is two config lines:

# production.rb
config.assume_ssl = true   # treat incoming requests as HTTPS even though they arrive over HTTP
config.force_ssl = true    # still enforce HTTPS, but now correctly understands it's already on HTTPS
config.ssl_options = {
  redirect: { exclude: ->(request) { request.path == "/up" } }
}

assume_ssl tells Rails the connection is already secure. The health check exclusion stops Fly's /up checks getting a 301 instead of a 200.

open_timeout vs read_timeout

Faraday has two distinct timeout settings and they fail in completely different ways.

open_timeout is the time to establish the TCP connection and complete the SSL handshake. If OpenAI is unreachable, this fires and fails fast. Five seconds is plenty.

read_timeout (Faraday's timeout) is the time to receive the response body after the connection is open. For gpt-4o-mini, typical latency is 700–900ms. For a slower model or a long prompt, it can be several seconds. Thirty seconds is the right number here — tight enough to detect a hanging request, loose enough not to cut off legitimate calls.

CONNECTION = Faraday.new(
  request: { open_timeout: 5, timeout: 30 }
) do |conn|
  conn.adapter :net_http_persistent
end

Persistent connections

The net_http_persistent adapter reuses TCP and TLS sessions across requests. Without it, every proxy call pays the handshake overhead — around 50–100ms for TLS on a fresh connection. With it, that cost is paid once and amortised across all subsequent requests to the same host.

The CONNECTION object is a class-level constant so it's shared across Puma threads. The key detail: per-request headers (including the caller's OpenAI key) are passed to .post at call time, not configured on the connection. The connection is credential-agnostic — only the request is not.

Numbers

Measured today on gpt-4o-mini, single-turn chat completion, from Fly ams to OpenAI:

The proxy adds about 50ms. The rest is OpenAI.

What isn't measured: multi-turn conversations, large prompts, models other than gpt-4o-mini, or latency under concurrent load. At this scale — one machine, low traffic — the numbers are stable. At higher concurrency, the row-lock serialisation in enforce_budget will add measurable wait time for tokens with many simultaneous callers. That's a known tradeoff; at the scale this is useful for, it doesn't matter.

What I left out on purpose

Streaming. The proxy handles non-streaming chat completions only. Streaming budget enforcement is a different problem — you need to enforce the cap on token counts as they flow, which doesn't fit Rails' request/response model cleanly. That's a Bun sidecar I'll build when people start asking for it.

Dashboard. There's no UI. You create tokens with a curl command, check usage with another. For a tool aimed at developers, a console interface is fine at launch. A React dashboard is on the roadmap but not on the critical path.

Multi-region. The proxy runs in one Fly region (ams). Row-level locking on a single Postgres instance is the right choice for now — consistent, simple, auditable. Multi-region would require distributed coordination that adds real complexity for a marginal latency gain at this scale.

Soft-cap alerts. No "you've reached 80%" email. Deliberate omission — alerts require infrastructure (email sending, scheduling) and they don't solve the underlying problem. The hard cap is the product.

Each of these is a next decision, not a current one. Shipping a narrow tool that works cleanly is more useful than shipping a broad one that half-works.

Try it

The proxy core is 100% open source. You can read exactly how the budget logic works before routing your first request.

• Code: github.com/naurisSeglins/ai_budget_proxy?utm_source=hashnode

• Live: ai-budget-proxy.fly.dev/?utm_source=hashnode

• 30-second quickstart in the README

One URL, two headers, no SDK to install. Set a cap. When you hit it, requests stop.