The Bug Was in Stderr, but the Symptom Was in Stdout

At 16:35:16 the model created a three-slide presentation. At 16:35:20 it called a tool named preview_slide to look at slide one. At 16:35:23 the tool server had rendered the slide to a PNG and started returning it.

Then nothing happened for two minutes and forty-eight seconds.

At 16:38:11 Savant gave up, cancelled the call, and tore the server down. The user-visible result: a presentation stuck at three slides, a dead conversation task, and a tool server getting re-initialized as if it had crashed. It hadn't crashed. It was sitting there the whole time, alive and blocked, waiting for us.

The stdout response never arrived. But the evidence was on the other stream. The captured server log cut off mid-JSON, right in the middle of the image payload:

"type": "image",
[ERROR] Error reading stderr: bufio.Scanner: token too long

The bug was in stderr. The symptom was in stdout.

Two readers, one forgotten

Savant talks to MCP servers (MCP is the Model Context Protocol, the standard way to plug external tool servers into an AI app) by spawning them as child processes. The child speaks JSON-RPC over stdin and stdout, and writes its logs to stderr. So the transport drains two streams with two readers.

The stdout reader is the important one, and it was treated that way. It parses JSON-RPC responses, and because tool results can be large, its bufio.Scanner buffer was raised to 512 KB.

The stderr reader just copies log lines to a file. It got a default bufio.Scanner, and a default scanner gives up on any line longer than 64 KB. Worse, our read loop treated that as fatal: on token too long it logged the error and returned. Permanently. From that moment, nobody was reading the child's stderr.

That should be harmless. It's just logs.

The pipe doesn't care that it's "just logs"

A pipe between two processes is a small kernel buffer, 16 to 64 KB depending on the OS. When it fills, the writer doesn't get an error. It gets suspended, mid-write, until someone reads from the other end. (The smaller the buffer, the faster this bites; we ship on macOS and Windows, where pipes run small.)

Our tool server's framework logs every full request and response to stderr, including the response it was about to send on stdout: a base64-encoded PNG of a 1920×1080 slide, as one line, far past the 64 KB limit. That single line killed the stderr reader. The server kept logging, the stderr pipe filled, and the kernel put the server to sleep on its next stderr write.

And a process that's asleep on stderr never finishes writing to stdout.

That's the whole deadlock. The JSON-RPC response Savant was waiting for existed; the server was partway through producing it. It just needed to finish a log line first, and the log line needed a reader that had already quit. Savant waited on stdout, the server waited on stderr, and the timeout was the only way out.

One detail in the logs pinned the diagnosis, by its absence. The stdout reader never reported a scanner error of its own. If the full response had arrived and fit, the call would have completed; if it had arrived and blown the stdout reader's 512 KB buffer, that reader would have logged its own error. Neither happened, so the response never fully left the child. It was blocked before it could deliver. The stream with the bug stayed quiet, and the stream that was merely downstream of it produced the three-minute hang, the cancellation, and the teardown. If I'd only looked where the symptom was, I'd have been debugging JSON-RPC framing for a week.

The fix: a reader that cannot stop

The point of the stderr loop isn't to log every byte faithfully. It's to keep the pipe drained, no matter what the child writes into it. So the fix inverts the priorities: draining is mandatory, logging is best-effort.

The rewritten loop uses a bufio.Reader with ReadSlice('\n') instead of a scanner that aborts. An oversized line is kept up to 64 KB, logged with a ...[line truncated] marker, and the rest of the line is read and thrown away. Every path through the loop keeps reading. The child can write a gigabyte on one line and the pipe never fills.

The second half of the fix removes the trigger: the server framework now caps request/response JSON in its stderr logs at 10 KB, so image payloads don't land in the logs at all. But that's hygiene, not the fix. The transport has to survive a child that logs badly, because MCP servers are, by design, other people's programs.

The regression test is my favorite part. It wires processStderr to a synchronous io.Pipe and writes a 200 KB line followed by a normal one. A synchronous pipe blocks the writer until the reader reads: the kernel's behavior with the pipe buffer turned down to zero. Stricter than the real thing, which only makes the test harder to pass. If the drain loop ever stops reading again, the test's writer goroutine hangs and a five-second timeout fails the test with the words "processStderr stopped draining after oversized line". The test doesn't check for the deadlock. It has the deadlock, in miniature, unless the code is correct.

"This is Unix 101"

If you've been doing systems work for a while, you've been waiting to say it, so let me say it for you: this is pipe backpressure, it's in every Unix textbook, it's the deadlock Python's subprocess docs warn about in so many words ("the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data"). And while we're at it, why is a server logging hundreds of kilobytes of base64 to stderr in the first place? Log a length, not a payload.

All of it lands. The failure mode is old, the rule is known, and logging binary payloads was a mistake on our side of the fence too (the framework in question is ours). I'd add one honest detail: the code even shows we knew the rule once, because the stdout scanner's buffer had been carefully raised to 512 KB. We'd thought about oversized lines, on the stream that mattered. The stderr reader didn't get that thought, because it was "just logs," and just logs is exactly the stream nobody re-audits.

The transferable part isn't the rule, it's the shape of the failure. The reader didn't crash. It exited its loop cleanly, logged one error line, and left both processes in a state where every component was healthy and no progress was possible. A reader that can stop reading is a deadlock with a delay on it.

What's still not fixed

The tool that triggered all this, preview_slide, is still not advertised to the model. The transport no longer deadlocks, so it could come back, but returning a full-resolution PNG inline is heavy for the 512 KB stdout path and costs image tokens on every call. The right version returns a downscaled image or writes the file to disk and returns a reference. Until that's done, the tool stays off. The deadlock is fixed; the payload question that exposed it is merely postponed.

And there's a fair question about the other stream. The stdout reader still has that 512 KB line limit, and a legitimate tool result past it still stops that reader, which sounds like the sin this whole post is about. The difference is what happens next. The stdout reader fails loudly: its error propagates to the pending call, the transport is torn down, and the server is killed and restarted. A hard limit signaled in seconds, not a silent hang measured in minutes. It's still a limit (a streaming JSON reader would remove it, and hasn't been written), but the rule isn't "never stop reading." It's "never stop reading silently while everyone else waits."

What I took away

1. If you spawn a child process, you own its streams. All of them, to EOF, on every error path. The pipe doesn't care what the bytes mean, only that somebody keeps taking them.
2. Backpressure turns a reader bug into a writer hang, one process away from the cause. When a child goes silent on stdout, check who stopped listening on stderr.
3. The best regression test reproduces the physics, not the symptom. The synchronous io.Pipe gives the test the same blocking behavior the kernel gives the real process, so the deadlock can't come back without the test hanging.

I keep coming back to that truncated log line, the JSON that stops at "type": "image",. The error was written where nobody was reading, by the reader that had just quit, and everything after it was silence on both streams. If you've debugged a hang that turned out to live one pipe over from where you were looking, in Savant or in your own stack, write to me at support@usesavant.com. The comments on a post like this are usually sharper than the post.

Savant is a desktop AI workspace: one app for Claude, GPT, Gemini, Grok, and local models, with built-in tools for research, documents, images, email, and more. You can download it here.