Last week my live-streaming transcoder started dying mid-session. The publisher would get disconnected, reconnects would fail, and the only cure was restarting nginx entirely. While poking around during one of these incidents, I ran ps aux and saw this:
root 814231 2.1 1.8 ... SL 12:04 3:11 /usr/bin/ffmpeg -i rtmp://127.0.0.1:1935/...
root 814232 2.3 1.9 ... SL 12:04 3:19 /usr/bin/ffmpeg -i rtmp://127.0.0.1:1935/...
root 815544 1.9 1.8 ... SLl 12:31 1:02 /usr/bin/ffmpeg -i rtmp://127.0.0.1:1935/...
Multiple ffmpeg processes, all in sleeping state, from sessions that should have been long dead. nginx had “killed” the transcoder when the publisher disconnected. So why were these still here?
Because I had forgotten one of the oldest facts about Unix, and it’s worth writing down so I don’t forget it again:
When a parent process dies, the kernel does not kill its children.
What actually happens
My setup used nginx-rtmp’s exec_push to launch a shell script, and the script launched ffmpeg:
nginx worker
└── bash ffmpeg_transcode.sh live
└── ffmpeg -i rtmp://... (the actual work)
When the publisher disconnects, nginx-rtmp sends a kill signal to the process it spawned — the bash script. Bash dies. And ffmpeg?
The kernel does exactly one thing with an orphaned child: it reparents it — traditionally to PID 1, or to the nearest “subreaper” (on modern systems, often systemd). That’s it. Nobody sends it a signal. ffmpeg keeps running, holding its RTMP publish sessions open, blocked on an input stream that will never send another byte. That’s the SL state, sleeping, waiting forever.
The cascade from there was my whole outage: the encoder auto-reconnects, nginx fires the script again, the new ffmpeg tries to publish to stream names the ghost ffmpeg still owns, nginx-rtmp rejects the duplicate publish, and the new session produces nothing. Every retry fails until you restart nginx, which finally tears down the ghosts.
Why the kernel behaves this way
My first instinct was “this is an OS bug waiting to happen, surely the parent’s death should cascade.” It’s not a bug. It’s the design, and half of Unix depends on it.
In Unix, the parent–child relationship is bookkeeping, not ownership. A child is a fully independent process with its own PID, file descriptors, and lifetime. The parent’s only special privilege is the right (and duty) to collect the child’s exit status with wait(). Reparenting exists purely so that someone is always around to do that collection, otherwise every orphan would become a permanent zombie.
Killing children on parent death would break, at minimum:
- Every daemon ever written. The classic daemonization recipe is literally “fork, then let the parent exit” — the child surviving its parent is the entire point.
nohupand long-running jobs. Start a build over SSH, connection drops, build keeps going.- Your desktop. Launch a browser from a terminal, close the terminal, browser stays up.
“But shouldn’t a well-behaved parent clean up?”
Yes, and this is the part that matters in practice. A parent that wants to manage its children should catch the termination signal, forward it to its children (or its process group), wait for them, and then exit. Bash can do this with trap.
Here’s the catch that sank me: nginx-rtmp kills its exec children with SIGKILL, and SIGKILL cannot be caught. No trap ever runs. Bash is vaporized instantly, mid-thought, with no opportunity to pass anything on to ffmpeg.
The mechanisms that do provide cascade-kill behavior all exist precisely because the default doesn’t, and every one of them is opt-in:
- Process groups — send a signal to
-PGIDand everyone in the group gets it. But the killer has to choose to do this; nginx-rtmp doesn’t. - cgroups — systemd kills every process in a unit’s cgroup on stop. Great, if your process tree lives in a unit.
prctl(PR_SET_PDEATHSIG, ...)— Linux-specific; a child can request “send me SIGTERM when my parent dies.” The child has to ask.
The one-word fix
Since I couldn’t make nginx kill smarter, I removed the process it was killing wrong. In the shell script, this:
/usr/bin/ffmpeg -i "rtmp://127.0.0.1:1935/esatsang/$1" ...
became this:
exec /usr/bin/ffmpeg -nostdin -i "rtmp://127.0.0.1:1935/esatsang/$1" ...
The bash builtin exec doesn’t fork a child, it replaces the shell process with ffmpeg, same PID. The script stays readable, multi-line, full of comments. But now the process nginx tracks is ffmpeg, and its SIGKILL lands exactly where it should.
The verification was satisfying. Kill the stream from the encoder side and watch:
watch -n1 'ps aux | grep [f]fmpeg'
ffmpeg vanishes within a second of the disconnect. The reconnect spawns a fresh one. No ghosts, no duplicate-publish rejections, no nginx restarts at 11pm.
The takeaway
If you ever see orphaned workers surviving their supervisor, don’t blame the kernel — it’s doing exactly what forty years of Unix software expects it to do. Ask instead: who is actually receiving the kill signal? If the answer is “a wrapper script,” the fix is probably one word long.
Next time a process “refuses to die,” check ps -o pid,ppid,stat,cmd — if PPID is 1 and it shouldn’t be, you’ve got an orphan, and somewhere upstream a wrapper ate a signal meant for someone else.
