Building a Next.js Streaming Chat UI for Local LLMs

5 min read · Updated Jul 17, 2026

You build a Next.js streaming chat UI for a local LLM with an App Router route handler that opens a ReadableStream from Ollama’s /api/chat endpoint and pipes it straight to the client, and a client component using the Vercel AI SDK’s useChat hook to render each token as it arrives. The part most tutorials skip is what happens when the user navigates away mid-response, which without an abort handler keeps your local GPU generating a reply nobody will ever see.

Key takeaways

A Next.js route handler can return a raw ReadableStream directly, no extra library needed to proxy Ollama’s streaming response.
The Vercel AI SDK’s useChat hook handles token-by-token rendering, loading state, and abort out of the box once your route handler speaks its expected stream format.
Always propagate the client’s AbortSignal through to the fetch call against Ollama. Without it, closing the browser tab does not stop the model from generating.
Render partial markdown defensively. An incomplete code fence mid-stream will break a naive markdown renderer.
Local models stream at a speed that depends entirely on your GPU, not the network, so test your loading states against your actual hardware, not a cloud API’s speed.

Why the route handler is the part that actually matters

Most local LLM chat tutorials focus on the client component and treat the backend as an afterthought. For a local model, the route handler is doing real work: it is the only thing standing between your React component and a model server that speaks a slightly different streaming format than what most front-end libraries expect out of the box.

The route handler: proxying Ollama’s stream

typescript

export const runtime = 'nodejs';

export async function POST(req: Request) {
  const { messages } = await req.json();

  const ollamaRes = await fetch('http://localhost:11434/api/chat', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ model: 'llama3:8b', messages, stream: true }),
    signal: req.signal,
  });

  const stream = new ReadableStream({
    async start(controller) {
      const reader = ollamaRes.body!.getReader();
      const decoder = new TextDecoder();

      while (true) {
        const { done, value } = await reader.read();
        if (done) break;

        const lines = decoder.decode(value).split('\n').filter(Boolean);
        for (const line of lines) {
          const chunk = JSON.parse(line);
          controller.enqueue(new TextEncoder().encode(chunk.message.content));
        }
      }
      controller.close();
    },
  });

  return new Response(stream, {
    headers: { 'Content-Type': 'text/plain; charset=utf-8' },
  });
}

app/api/chat/route.ts: proxy Ollama’s streaming chat endpoint to the client.

Passing req.signal straight into the fetch call to Ollama is the single most important line in this handler. It means an abort on the client side propagates all the way down to the model server, not just to your own route handler.

Close-up of JavaScript code on a laptop screen in a code editor

The client: useChat and a manual abort wire-up

tsx

'use client';

import { useChat } from 'ai/react';

export function LocalChat() {
  const { messages, input, handleInputChange, handleSubmit, stop, isLoading } =
    useChat({ api: '/api/chat' });

  return (
    <div>
      {messages.map((m) => (
        <p key={m.id}>
          <strong>{m.role}:</strong> {m.content}
        </p>
      ))}

      {isLoading && (
        <button onClick={() => stop()}>Stop generating</button>
      )}

      <form onSubmit={handleSubmit}>
        <input value={input} onChange={handleInputChange} />
        <button type="submit">Send</button>
      </form>
    </div>
  );
}

A minimal chat component using the Vercel AI SDK against the route handler above.

The night I noticed the GPU fan would not stop

In November 2024 I shipped a first version of this exact chat UI without forwarding the abort signal into the Ollama fetch call. During testing I closed a browser tab mid-response and kept working. About ten minutes later I noticed the laptop fan still spinning at full speed. Ollama was still generating a reply to a request the browser had long since abandoned, because nothing had told it to stop. The fix was the one line, signal: req.signal, that I had left out to keep the first version "simple."

Rendering partial markdown without it looking broken mid-stream

A response streaming token by token will, for a moment, contain an opened but unclosed code fence or an incomplete markdown table. Rendering that raw through a strict markdown parser produces a flash of broken layout on every message. Use a streaming-tolerant renderer, or render plain text until you detect the closing fence, then swap to the rendered markdown component.

Local model streaming speed versus a cloud API, on consumer hardware.
Setup	Typical tokens/sec	What the user perceives
Llama 3 8B on RTX 3060 (local)	25 to 40 tok/s	Noticeably slower start, steady stream once running
Llama 3 8B on Apple M2 (local, unified memory)	15 to 25 tok/s	Comparable to a slow cloud connection
GPT-4o-mini (cloud)	80 to 120 tok/s	Feels closer to instant for short replies

I think most local-LLM chat UI tutorials borrow loading-state assumptions from cloud API demos and it shows. A spinner tuned for a 100ms time-to-first-token will look broken against a local model that takes 800ms to start streaming on consumer hardware. Test your actual loading UI against your actual GPU, not against a cloud API’s much faster baseline.

“The fan spinning on an empty tab is the local-LLM equivalent of a cloud API bill nobody reads until it is too big to ignore. Both come from the same missing line: propagate the abort.”

Frequently asked questions

Can I use the Vercel AI SDK with a local model instead of OpenAI?

Yes. Point useChat at your own route handler, and have that route handler call your local model server (Ollama, LM Studio) instead of a cloud provider. The SDK does not care what generates the tokens, only that the response stream matches its expected format.

Why does my local LLM keep generating after I close the browser tab?

Your route handler is not forwarding the abort signal to the request against your local model server. Pass the incoming request’s signal property into the fetch call to Ollama so a client-side abort actually stops generation on the GPU.

Should I use the Edge runtime or Node runtime for a local LLM route handler?

Node runtime. The Edge runtime cannot reach a model server running on localhost or your local network, since it executes in a separate isolated environment. Set export const runtime = "nodejs" explicitly in the route handler.

Why does the chat UI flash broken formatting while streaming?

A partial response mid-stream often contains an unclosed code fence or table, which a strict markdown parser renders incorrectly for a moment. Use a streaming-tolerant markdown renderer or delay rendering a block until its closing syntax arrives.

How fast should I expect a local LLM to stream compared to a cloud API?

Meaningfully slower on consumer hardware. Expect roughly 25 to 40 tokens per second for an 8B model on a mid-range GPU, versus 80 or more for a cloud API like GPT-4o-mini. Tune your loading states to that reality rather than a cloud API’s speed.