HTTP 200 Doesn't Mean Success: Silent Failures in Streaming LLM APIs

A user sends a message. The assistant's reply comes back empty. Not an error banner, not a spinner that never resolves. An empty reply, saved to disk as a finished conversation turn: content: [], zero input tokens, zero output tokens, marked complete.

Savant stores every conversation as a YAML file on the user's machine, and in this case the user was me; the bug turned up during development, in a test scenario. That was the first stroke of luck in this story: the corpse was right there to examine. The assistant message was empty, and nothing anywhere said anything had gone wrong.

Every layer reported success.

Following the numbers down

The API log for that turn looked healthy at first glance. Request sent, response received, HTTP 200. Then the details: Model= blank, InputTokens=0, OutputTokens=0, Events received: 2.

A normal streaming reply from the Claude API arrives as dozens or hundreds of server-sent events (SSE, the line-based streaming format most LLM APIs use): a message_start, a stream of content_block_delta chunks carrying the text, a message_stop. Two events is not a small reply. It's the shape of something else entirely.

Even that number was lying to me, though I only learned it later. The counter incremented once when it saw an event: line and again when the event completed, so "2" was one event, counted twice. The instrument I was using to diagnose the failure had its own bug. (It's fixed now; it counts events.)

So: one event. I added temporary logging to capture the raw SSE lines, reproduced the failure, and the entire stream was three lines:

event: error
data: {"type":"error","error":{"type":"overloaded_error","message":"Overloaded"}}

The API was telling me it was overloaded and couldn't serve the request. Over a connection that had already returned HTTP 200.

Why nothing complained

Savant's streaming parser handled six event types: message_start, content_block_start, content_block_delta, content_block_stop, message_delta, message_stop. Anything else fell through and was ignored.

Ignoring unknown SSE events is not a bug by itself. It's what the docs tell you to do; providers add new event types, and a parser that crashes on novelty is worse than one that skips it. The bug was that error isn't an unknown event. It's a documented one, and it was missing from the list.

From there the failure propagated with perfect politeness. The stream ended, the scanner reported no error, and the parse function returned success with an empty response. The reply text is only finalized when a completion event fires, and none had, so the content stayed empty. Then the last line of defense waved it through: the completeness check treats empty content as complete. The conversation saved, the UI moved on, and the only witness was a YAML file with an empty message in it.

Read that chain again. HTTP said 200. The stream parser said done. The scanner said no error. The save said complete. Four layers, each honestly reporting that its own narrow job had succeeded, adding up to a user staring at a blank reply.

The fix

Three changes, all in the streaming client.

First, the parser now recognizes event: error, parses the payload into a typed error, and returns it instead of pretending the stream ended well:

if eventType == "error" {
    sseErr := parseSSEError(eventData)
    logger.Error("Claude API SSE error in stream: type=%s, message=%s",
        sseErr.Type, sseErr.Message)
    return sseErr
}

Second, overloaded_error is transient by nature, so it gets retried: up to three times, waiting 2s, 4s, then 8s between attempts. In the incident above the error was the first and only event in the stream, so a retry restarts cleanly, and the error only reaches the user after all retries are exhausted. Other stream errors aren't retried; they surface immediately. That includes api_error, the in-stream equivalent of a 500. That's a choice rather than an oversight: I'd rather have eyes on a provider's internal errors for a while before teaching the client to retry them automatically.

Third, the cleanup pass: the double-counting event counter, an allocated-but-unused debug buffer, a dead error check. Unit tests cover the error parsing, and an integration test now drives a real conversation end to end and asserts the saved content is actually there.

"This is documented behavior"

If you build against these APIs, you may already be drafting the comment: the streaming docs list the error event, overload errors inside 200-OK streams are known behavior, and this post is a changelog entry for reading the documentation.

Mostly, yes. I'm not claiming the API misbehaved; it did exactly what its documentation says.

There's a sharper version of the same comment, so let me say it for you: the official SDKs handle stream error events for you, so why is there a hand-rolled SSE parser here at all? Also fair. Savant talks to Claude, GPT, Gemini, Grok, and local Ollama models through one streaming abstraction, one callback shape shared by five thin provider clients, each written directly against the provider's HTTP API from its documentation. There's no vendor SDK anywhere in the stack. That keeps five providers uniform, and it means every parser bug is mine alone. Anthropic's SDK would have caught this one.

The narrower point survives both comments, and it's about which side of the parser the failure lands on. When a REST call fails, the failure is in your face: non-2xx status, exception, red log line. When a streaming call fails mid-stream, the failure arrives as data, and data you don't handle defaults to silence. The failure mode of a missing case isn't a crash. It's a success report.

That asymmetry is worth designing for. Status codes stopped being the answer the moment the answer started arriving in the body.

What I didn't fix

The completeness check that treats empty content as complete is still in the code. I fixed the one path I know of that fed it an empty response, but the assumption itself survives, one layer down, waiting for the next path I haven't imagined. A save-time guard that refuses to mark an empty assistant reply as complete is the honest fix, and I haven't written it yet. It isn't quite the one-liner it sounds like: the predicate decides when a turn is finished, so changing it means auditing every caller first, or you trade a silent bug for a loud one. But that's why it's sequenced behind a caller audit, not an excuse to skip it.

The retry also assumes the error arrives before any content. Here it did; the error was the whole stream. But nothing stops the API from failing after streaming half a reply, and a retry at that point would re-run the request after the user has watched partial text arrive. Handling that cleanly means telling the frontend to discard the partial turn before retrying, and I haven't built it.

And the test this bug really calls for doesn't exist yet either. The unit tests exercise the error parsing; the integration test exercises the happy path against the live API, which won't serve you an overload on cue. The right regression test is a stub server that replays that three-line stream through the parser and the retry loop. It's small, and it's fair to hold me to it.

What I took away

1. In a streaming API, the error channel is in-band. Enumerate the documented event types against your parser's switch statement; the gap is where your silent failure lives.
2. "Empty" is a result that deserves suspicion at every layer. A zero-token reply with a blank model name should not be savable as a completed turn. Cheap sanity checks at persistence boundaries catch whole classes of upstream bugs.
3. Trust your diagnostics, but verify them first. The event counter was wrong in a way that made the mystery look deeper than it was. The instruments need tests too.

The empty reply was the whole bug, start to finish: empty because an error went unread, saved because empty counted as done. If you've hit a silent failure shaped like this one, in Savant's stream handling or your own, write to me at support@usesavant.com. The comments on a post like this are usually sharper than the post.

Savant is a desktop AI workspace: one app for Claude, GPT, Gemini, Grok, and local models, with built-in tools for research, documents, images, email, and more. It's currently in an invite-only rollout. You can request access here.