Handling Streaming Errors and Reconnection in LLM Chat UIs
7 min read · Updated Aug 5, 2026
When an LLM stream drops mid-response, the two things you must not do are silently show nothing and silently show a truncated answer as if it were complete. Detect the drop (the connection closes, or your own idle timeout fires before the stream’s normal end-of-message signal), keep whatever partial text already rendered, mark it visibly as interrupted, and offer the user one clear action: retry. Full automatic reconnection into the same in-progress generation is not usually possible, since most LLM APIs cannot resume a stream mid-token, so treat this as "preserve and offer retry," not "reconnect and continue."
Key takeaways
- Most LLM streaming APIs (OpenAI, Anthropic) have no resume capability. Once a connection drops mid-generation, you cannot pick back up mid-token, you can only start a new request.
- Detect a dropped stream two ways: the connection actually closes (an error or abort event), or an idle timeout you set yourself fires because no new token arrived in N seconds. A hung connection that never errors is just as broken as one that closes.
- Never discard the partial text a user has already seen. Keep it rendered, mark it as interrupted, and let the user decide whether to retry.
- Distinguish a genuine drop from the model’s normal end-of-stream signal (a
[DONE]marker or afinish_reasonfield). Treating normal completion as an error is the single most common bug in homemade streaming UIs. - Rate-limit automatic reconnect attempts. An unthrottled retry loop against a provider that is actively down just adds you to the pile of traffic slowing its recovery.
Why you cannot just "reconnect" an LLM stream like a WebSocket
A WebSocket or SSE connection dropping is usually a transport problem: the pipe broke, but the thing on the other end (a chat server tracking room state, say) is often still there and can pick up where it left off. An LLM generation is different. Once the request that triggered it is gone, mid-generation state on the provider’s side is gone with it, current provider APIs do not expose a way to resume token 47 of an interrupted 80-token response. Reconnecting the transport layer alone does nothing useful here. The only real recovery is starting a new generation request, which is why the UI decision matters more than the network code.
Telling a real error apart from a normal stream end
A stream ending is not automatically bad news. The model finishing its response and the connection dying mid-response look almost identical to careless code, both mean "no more data is coming," and the entire bug class here is failing to tell them apart.
async function readStream(response, onToken, onInterrupted) {
const reader = response.body.getReader();
const decoder = new TextDecoder();
let idleTimer;
const resetIdleTimer = () => {
clearTimeout(idleTimer);
idleTimer = setTimeout(() => {
reader.cancel();
onInterrupted('idle-timeout');
}, 12000);
};
resetIdleTimer();
try {
while (true) {
const { done, value } = await reader.read();
if (done) return; // connection closed cleanly, not necessarily an error
resetIdleTimer();
const chunk = decoder.decode(value);
for (const line of chunk.split('\n')) {
if (!line.startsWith('data: ')) continue;
const payload = line.replace('data: ', '');
if (payload === '[DONE]') {
clearTimeout(idleTimer);
return; // this is success, not a failure
}
onToken(payload);
}
}
} catch (err) {
clearTimeout(idleTimer);
onInterrupted('connection-error');
}
}
Preserve the partial response, do not erase it
When a stream genuinely does drop, resist the urge to clear the message and show a generic error instead. The user already read 40 words of a real, useful answer. Deleting it and replacing it with "Something went wrong, please try again" throws away information the user has already invested attention in.
- Streaming: tokens are actively arriving, show the normal typing/cursor indicator.
- Complete: the stream ended via
[DONE]or a stopfinish_reason, render the final message normally. - Interrupted: the connection errored, aborted, or the idle timeout fired. Keep the partial text visible, visually mark it (a dashed border or a small "Response interrupted" label works well), and show a single retry action.
The demo that stalled once every fifteen runs
In April 2024, I was demoing a support-bot pipeline to a client’s ops team, and over the two weeks leading up to launch, roughly one in every fifteen runs (I counted, mostly out of nerves) hit a silent stall: the connection stayed open, nothing threw an error, and the UI just sat there for upwards of a minute with no new tokens. A proxy between us and the model provider was buffering the entire response instead of passing it through token by token, so from the browser’s side, nothing had technically failed, it was just waiting. Adding a twelve-second idle timeout that aborted the fetch and triggered the retry UI turned that same hang into a two-second, clearly labeled "Response interrupted, retry?" instead of a support engineer refreshing the page mid-call, hoping nobody in the room noticed.
What each signal actually means
| Signal | What it means | What the UI should do |
|---|---|---|
| [DONE] or finish_reason: stop | The model finished normally | Render the final message, nothing else |
| fetch/EventSource error event | The connection actually broke | Keep partial text, show interrupted state, offer retry |
| Idle timeout fires | Connection is open but nothing is arriving | Abort manually, treat identically to a connection error |
| HTTP error status on initial request | The request never started streaming at all | Show a distinct error, retrying the same request is safe |
I think most "flaky streaming" bug reports are not actually about the network. They are about a UI that cannot tell the difference between the model finishing and the connection breaking. I have fixed more of these by adding one if-statement checking for [DONE] than by touching any retry or reconnection code at all.
OpenAI’s API reference (2024) documents its streaming responses ending with a literal data: [DONE] line, distinct from any connection-level close event. That line is the signal a parser should treat as success. Anything else that ends the stream, an error event, a timeout, an abort, is the actual failure case, and conflating the two is where most of this bug class comes from.
“A response that stops halfway is not gone. It is 40 good words and one missing ending, and the UI should treat it exactly that specifically.”
Frequently asked questions
Frequently asked questions
Can I resume an LLM stream after it disconnects?
Not with most current provider APIs. Once a generation request is interrupted, there is no way to resume mid-token. The only real recovery path is starting a new generation request, so the UI should be designed around retry rather than resume.
How do I tell a real error apart from the model finishing normally?
Check for the provider’s explicit end-of-stream signal, a literal [DONE] marker or a finish_reason field set to stop, and treat that as success. Only route to an error or interrupted state when you catch an actual thrown exception, an abort, or your own idle timeout firing.
What should the UI show when a stream drops mid-response?
Keep the partial text that already rendered, mark it visibly as interrupted (a dashed border or a small label works well), and offer a single clear retry action. Do not clear the message and replace it with a generic error.
Why do I need an idle timeout if I already handle the error event?
Because a hung connection, often caused by a proxy or CDN buffering the response, does not always throw an error at all. It just goes quiet. An idle timeout catches that case by treating a prolonged gap between tokens as a failure on its own.
Should reconnection retry automatically or wait for the user?
For a single dropped generation, wait for the user to click retry rather than retrying automatically, since automatic retries can pile up against a provider that is genuinely struggling. If you do retry automatically, rate-limit it with backoff rather than looping immediately.
What causes a stream to hang with no error at all?
Most commonly, an intermediate proxy or CDN buffering the full response before forwarding it, rather than passing tokens through as they arrive. From the browser’s perspective the connection is still open and nothing has technically failed, which is exactly why an idle timeout, not just error handling, is necessary.