“Some people are saying the audio is broken and the video looks degraded. But it works absolutely fine everywhere I’ve checked.”

If you’ve ever run a live service — streaming or otherwise — you know this exact message. It arrives during your most important hour. You check. It works. You check from your phone. It works. And yet the complaints keep coming.

In my over 30 years of operational support, this is the most dangerous kind of problem because the instinct is to dismiss it. I have seen it numerous times that the term “It works for me” is the graveyard of unresolved incidents. As one of the YouTubers I follow says, the truth lies somewhere in between. Both parties are right: the stream was working, and it was broken — for different people, at the same time, for a reason that only made sense once I looked at the whole system.

I run the streaming for our live Satsang (a community worship gathering focused on Arti and prayers) transmissions. This is broadcast to thousands of listeners twice a day. The audience is unusually sensitive to interruptions; a thirty-second silence during a transmission is not a minor bug; it’s a broken experience for people in a moment of devotion.

The architecture is common and unremarkable: an encoder at the venue sends the feed to an origin server, which transcodes it into multiple qualities, and a set of distribution servers deliver small chunks of audio and video — called segments — to every listener’s phone.

This blog post is the story of this simple word, “segments”.

Reading the graphs like a detective

When the complaints came in, I pulled up three dashboards: the load balancer, the distribution servers, and the transcoding origin. For the non-technical reader, think of it as checking three witnesses to the same event.

The load balancer said: traffic is climbing steadily, capacity is fine, but a small percentage of requests are failing — and that percentage is growing with the audience. Nothing was overloaded. That ruled out the easy answer (“too many viewers/listeners”).

The distribution servers said something far more interesting. At exactly 14:46, all five of them — simultaneously — logged a sharp spike of “file not found” errors. Not for the audio chunks. For the playlists: the small index files that tell every player which chunks to fetch next. Then the errors vanished. The whole event lasted about one minute.

Five independent servers erroring at the same instant means the problem wasn’t on any of them. They were all faithfully reporting the same upstream truth: for one minute, the playlists disappeared at the source.

I confirmed it: the venue’s internet connection had blipped. The encoder disconnected, reconnected, and streaming resumed. Sixty seconds, start to finish. So here is the puzzle that makes this worth writing down: why did a one-minute network blip cause a much longer complaint of “broken audio”  from some listeners, while everyone else — including me — saw nothing wrong?

The optimization that turned against me

Months earlier, I had made a genuinely good change. Streaming works by having every player download a new chunk every few seconds. Each chunk, once created, never changes — so I told every player and every cache in between: this file is immutable; cache it for a year, never ask again. Request volume dropped, the experience got smoother, everything stabilized. And I was happy and proud.

That optimization had one hidden assumption, and the one-minute blip violated it. When the transcoder restarted after the reconnect, it began numbering its output chunks from zero again: index_0.ts, index_1.ts, index_2.ts — the same filenames it had used hours earlier, now containing completely different audio. But listeners’ phones, and caches along the way, still held the old files under those names, with my own instruction stamped on them: immutable, trust this for a year.

So a listener’s player would fetch index_3.ts, be handed a stale chunk from hours ago straight out of cache, and produce garbled audio. A fresh listener — or me, checking whether “it works” — had no cached history, fetched everything anew, and heard a perfect stream. Both witnesses were telling the truth.

There was a second, quieter failure stacked on top. Live playlists carry a running counter so players can track their position. When the transcoder restarted from zero, that counter jumped backward. Players interpret time moving backward as corruption and stall or bail out. Fresh players never saw the jump.

The lesson, for anyone leading a team: an optimization is a bet on an assumption. “Cache forever” bet that filenames are never reused — it paid off daily for months, then an unrelated event silently broke the assumption. When a stable system misbehaves after an incident, don’t ask “what broke?” Ask which assumption did the incident violate?

The fix: make names impossible to reuse

The elegant repair wasn’t to remove caching. Caching was doing real good. It was to make the assumption true forever. I changed the transcoder to start its chunk numbering from the current clock time when it launches. The arithmetic makes reuse impossible: chunk numbers advance by one every twelve seconds, but the clock advances by twelve in the same span. Any restart begins at a number far ahead of anything the previous run could reach. No two chunks, across any sequence of crashes, can ever share a name. As a bonus, the position counter now jumps forward on restart instead of backward. When this happens, the players handle a forward jump gracefully, snapping to the live edge instead of breaking.

I also reduced the cache duration from “one year” to 60 seconds. Why did I choose this 60-second number? The reason is that a chunk is only referenced by the playlist for about four minutes before it scrolls out and is deleted. Caching beyond that window buys nobody anything; it only extends how long a stale file can haunt you.

The lesson for engineers: set cache lifetimes based on the data’s lifecycle, not on optimism. “How long is this file legitimately needed?” has an actual answer. Use it.

Dissecting the sixty seconds

With the corruption fixed, I turned to the outage itself. Why did a network blip cost a full minute? I decomposed it, and every second had an owner.

The origin took thirty seconds just to notice the encoder was gone. A silently dropped connection doesn’t announce itself; the server waits until the configured silence threshold is reached before declaring the publisher dead. Until then, it holds the old “zombie” session. And this is where it is cruel: it rejects the encoder’s own reconnection attempts, because as far as it knows, that stream name is still occupied. Then a couple of seconds to reconnect once the zombie cleared, a moment to spin up, twelve seconds to produce the first new chunk, and then a second or two to sync out.

I cut the detection threshold from 30 to 10 seconds, bringing the worst case to roughly half a minute. And then I deliberately stopped.

Why did I not go lower?

The tempting next move was obvious: if ten seconds is better than thirty, isn’t six better than ten? The answer to this obvious question is “No”. And the reason is one of the most useful ideas in this whole episode.

Streaming rides on a transport layer that is designed to survive short losses: when packets drop, it silently retransmits with growing pauses — one second, two, four. A five-to-seven-second stall on a lossy link, or a brief CPU choke on the encoder machine, frequently recovers on its own. Every listener’s buffer absorbs it. Silence, then continuity. This is a non-event.

Set the death-detection timer inside that self-healing window, and you convert those non-events into executions.

At second six, the server kills a connection that would have recovered at second seven. This will trigger a full restart cycle for a blip that needed nothing. An aggressive timeout doesn’t make the system more responsive; it makes it trigger-happy, and every false kill costs vastly more than the seconds of detection it saves. I think ten seconds is just the right amount of patience and is beyond ordinary loss recovery. And this is well short of human perception of a real outage.

The lesson: in any system with self-healing layers underneath, impatience at the top destroys the healing below. Give the lower layers room to do their job before declaring anything dead.

How many times to knock before walking away

The encoder’s retry setting was next. The engineering reflex says: retry forever, retries are free. But the right answer came from operational experience, not technical experience. My runbook says any outage beyond a couple of minutes triggers a human response — backup encoder, secondary internet line. If a failed primary machine kept retrying for an hour, it might seize the stream back just as someone brings the backup online — two encoders fighting over one stream name, mid-transmission.

So I chose sixty retries — about two minutes. Long enough to outlast any genuine blip; short enough that when it expires, the machine takes itself out of contention exactly when a human takes over. The retry limit isn’t a technical parameter at all; instead, it is a handoff protocol between automation and people, and it must match the playbook, not be maximized in isolation.

The cleanup script that almost knifed me

One last trap, and it’s a beautiful illustration of how fixes breed new bugs. I have a cleanup script that runs whenever a publisher disconnects. It would delete the stream’s files, so late joiners to a genuinely ended stream got a clean “not found” rather than a frozen playlist.
To me it looked sensible when I wrote the script. But the disconnect hook fires on every disconnect, including transient ones. The script waited fifteen seconds, checked whether the stream was back, and if not, deleted everything.

Now, when I traced that sixty-second blip through that logic, I realized that the encoder is still inside its two-minute retry budget, mid-recovery — while the cleanup script has already declared the stream dead and deleted the playlists out from under it.

That distinction matters enormously to the software on people’s phones. A playlist that’s present but unchanging is met with near-infinite patience. The players will poll it and resume seamlessly when it updates. A missing playlist makes players give up fast. The cleanup script was transforming the most forgiving failure mode into the least forgiving one, at the worst possible moment.

The fix: teach the script the same patience as the encoder. It now waits out the full retry window (exiting early the moment the publisher returns), holds a lock so overlapping runs from a flapping connection can’t interleave, and only deletes once the encoder has demonstrably given up. By this time, the stream is already on air using a backup encoder and/or backup ISP. Every timeout in the recovery chain now agrees with every other.

The lesson — and the thesis of this whole post: recovery machinery must share one coherent definition of “dead.” Mine had three, and the most impatient component sabotaged the rest. Audit your timeouts as a set, not one by one.

The quiet hero: twelve-second chunks

A closing observation for anyone tempted by ultra-low-latency streaming. My chunks are twelve seconds long, which means standard players start about thirty-six seconds behind live — three chunks of buffer. That cushion is why, after all these fixes, most of my audience experiences a network drop at the venue as if there were literally nothing. The entire recovery completes while their player calmly drains its buffer.

Latency and resilience trade against each other, and the right balance is dictated by the content, not by fashion.

The Takeaway

If you lead a team: when told “it’s broken” and “it works fine” at once, believe both — the difference is usually hidden state (caches, buffers, session history) that makes identical requests receive different answers. And after an incident, the question isn’t “what failed?” but “which of our optimizations assumed this could never happen?”

If you’re an engineer: make identifiers globally unique so caching assumptions can’t be violated (a clock is a free counter). Derive cache lifetimes from data lifecycle. Respect the self-healing windows below you before declaring anything dead. Set retry budgets to hand off cleanly to human procedures. Align all timeouts to a single shared definition of failure.

None of the individual fixes was clever — clock-based numbering, a shorter cache header, a tuned timeout, a retry count, a patient cleanup script; each is a few lines. The value was in the diagnosis: reading five servers erroring in unison as an arrow pointing upstream, decomposing sixty seconds into its owners, and noticing that the system’s worst behavior emerged not from any component failing, but from correct components holding contradictory beliefs about time.

That, in the end, is what most production troubleshooting is: not finding the broken part, but finding the disagreement.