Adding an LLM to Your Next.js App Without Blowing Up Your API Bill

The bill is an architecture problem, not a pricing problem

Most teams that get a scary LLM invoice didn't pick the wrong model — they wired it up naively. Every keystroke fires a call. Every request re-sends the whole conversation. The biggest model runs on requests a tiny model could handle. None of that is the vendor's fault; it's all fixable in your Next.js app. Here are the six patterns we use on every LLM feature we ship.

1. Stream from a Route Handler, don't block

Put the model call in a Route Handler (app/api/.../route.ts) and stream tokens back. Never call the LLM directly from a Client Component — you'd leak your key and lose control of the request. Streaming doesn't cut cost, but it cuts perceived latency to near zero, which stops users from mashing "regenerate" (which very much does cost you).

// app/api/chat/route.ts
export const runtime = 'nodejs';

export async function POST(req: Request) {
  const { messages } = await req.json();
  const stream = await llm.chat({ messages, stream: true });
  return new Response(stream.toReadableStream(), {
    headers: { 'Content-Type': 'text/event-stream' },
  });
}

2. Cache the deterministic calls

A huge share of LLM calls are the same call. Classification, extraction, tagging, "summarize this fixed document" — same input, same output. Hash the input and cache the result. In Next.js you have three good layers:

• unstable_cache for server-side function memoization with tags.
• Route segment caching / revalidate for whole responses that change on a schedule.
• A real KV store (Upstash, Redis) keyed by a hash of the prompt for cross-request hits.

The first cache hit on a classification endpoint you thought was "cheap" is often where the savings actually live.

3. Route to the cheapest model that can do the job

You do not need your flagship model for "is this message spam?" Build a tiny router: cheap/fast model by default, escalate to the expensive one only when the task genuinely needs it (long reasoning, code, nuanced tone). A two-line heuristic — token count, task type, or a confidence check on the cheap model's answer — routinely cuts spend by more than half with no quality loss on the easy majority.

4. Budget tokens like you budget money

Two rules end most runaway costs:

1. Cap the context you send. Don't ship the entire chat history every turn — send a rolling window plus a running summary. Retrieve only the chunks this turn needs; don't paste the whole knowledge base.
2. Cap the output. Set max_tokens. An unbounded response is an unbounded bill, and most of the time the model is padding anyway.

5. Rate-limit and debounce at the edge

Put a rate limiter in middleware.ts so a single user (or a scraper) can't run up your bill in a loop. On the client, debounce anything that fires on input — autocomplete and "as you type" features should wait for a pause, not fire per character. This is the cheapest fix on the list and the one most often skipped.

// middleware.ts — sketch
import { NextResponse } from 'next/server';
export async function middleware(req: Request) {
  const ok = await limiter.check(getIp(req)); // e.g. Upstash Ratelimit
  return ok ? NextResponse.next() : new NextResponse('Slow down', { status: 429 });
}
export const config = { matcher: '/api/(chat|generate)/:path*' };

6. Measure quality with evals before you optimize

Cost optimization that quietly wrecks quality is worse than a big bill. Before you swap models or trim context, build a small eval set: real inputs with known-good outputs, scored automatically. Now "route this to the cheaper model" becomes a measured decision — you keep the savings only where quality holds. Without evals you're optimizing blind and won't find out you broke something until a customer does.

Put together

A well-built LLM feature in Next.js looks like this: a streaming Route Handler, a cache in front of the deterministic calls, a router that reserves the expensive model for hard requests, hard caps on input and output tokens, an edge rate limiter, and an eval set gating changes. None of it is exotic. All of it is the difference between a feature that scales and an invoice that scares your CFO.

That's the exact checklist we run when a client says "we added AI and the bill exploded." Nine times out of ten, the fix is three of these six.