EP07: How to strengthen resilience on weak networks¶

Open on YouTube

Recap	EP06 "Publish/play security: signing, tokens and anti-hotlinking"
Next	EP08 "Video instrumentation: measuring real end-to-end latency"

0. Goals of this episode¶

By the end, the viewer should be able to do four things:

Explain what SRT / RTP native statistics each represent, and how a product combines multiple signals into degradation conditions — different protocols' counters can't be forced into one "unrecoverable loss rate".
Understand that "weak-network resilience" is layered engineering, where detection, degradation, latency tuning and reconnect are only some of the layers.
Understand what actually happens once "unrecoverable loss rate exceeds a threshold": bitrate is cut first, a simulcast layer is only cut when that isn't enough, and whether cutting a layer also bumps the receive latency — and that this is implemented to a different degree on SRT versus WHIP.
Understand two easily-missed mechanisms that strongly affect real weak-network experience: preemption protection between P2P and the main publish, and how "reconnect" itself must be designed so it doesn't crush the system the instant the network recovers.

1. Opening hook (script notes)¶

"Degrade when there's loss" sounds simple, but in practice you first answer: does this counter count sequence-number gaps, retransmission requests, retransmitted packets, or media loss after the recovery deadline? SRT and RTP native stats differ in caliber; a product must interpret them separately, then combine loss, RTT, send-queue and available bandwidth into configurable action conditions.

2. Layer 1: interpret protocol counters first, then define product metrics¶

2.1 SRT and RTP native stats cannot be trivially unified into a ULR¶

The SRT library usually exposes loss, retransmission, dropped, NAK and send-buffer stats, but whether each field is sender- or receiver-side, a packet event or a unique packet, and whether it includes duplicate retransmissions all depend on the SRT version's API. You cannot derive the unrecovered packet count with "cumulative loss minus cumulative retransmit": one loss can trigger several retransmissions, and the counters may not even share a statistical domain.

RTP/RTCP's cumulative packets lost and fraction lost are computed mainly from expected vs received RTP sequence numbers. They reflect a receive gap but don't inherently tell the app whether the gap was later recovered by RTX/FEC in time for the decode deadline. NACK count, RTX received packets, pre-decode drops and stutter are all signals at different layers.

So the product layer should keep the native metric names and units and define its own business metric separately — e.g. "proportion of media units still missing after the recovery/playout deadline". Only when the receiver can correlate original-packet identity, RTX/FEC recovery outcome and the playout deadline can it compute such a derived metric; don't just rename an arbitrary raw counter as ULR.

2.2 Thresholds and sampling are product parameters, not protocol constants¶

Sampling window, degrade threshold, recover threshold, consecutive-hit count and cooldown should all be product parameters configured per protocol, direction, content layer and terminal capability, not constants from SRT or RTP standards, and you shouldn't assume the two protocols share one set. Production values need calibration from load tests and real network data, recorded per version.

Design usually uses a rolling window, consecutive hits and hysteresis: the degrade threshold is higher than the recover threshold, so degrading can be fast and recovering more cautious. This filters single spikes and avoids oscillating up/down near a network's critical point. Training diagrams should show parameter names only, not turn one experiment's values into universal thresholds.

2.3 Separate upstream publish degradation from downstream playback ABR¶

Upstream publish degradation happens on the encoder→Origin link. The sender can lower target bitrate, frame rate or resolution, pause upper layers, and renegotiate if needed; whether a given action is seamless, needs a keyframe, or rebuilds the session depends on the encoder and protocol implementation — you can't promise "zero cost" or a fixed interruption length uniformly.
Downstream playback ABR happens on the Origin/Edge/P2P→viewer link. The player picks an existing layer based on that viewer's goodput, RTT, loss, receive/playout buffer and decode state, and must not change global publish quality because one viewer is on a weak network.

The two control loops can share the observation system but differ in target, threshold and blast radius. Upstream anomalies affect all viewers, so actions should be more conservative; downstream ABR decides per session and usually prefers switching to a lower existing layer.

3. Layer 2: Simulcast multiple layers — which layer the detection result acts on¶

EP05 covered Simulcast's layering. Here we add the control domain: the publisher can adjust the layers it sends based on the overall uplink budget; the player selects already-published layers per session. Switching should pick a point the receiver can independently decode, coordinating keyframe requests, dependencies and buffer state. The detection module outputs network state with confidence, and the policy module decides which link, which layer, and when to switch.

RTMP/ERTMP is a capability for "staying compatible with legacy publishing devices and giving them ABR", and it makes no low-latency promise; for low latency plus ABR, use SRT/WHIP. This does not affect ERTMP's value on the cost side: ppobs can simulcast HEVC + H264 multi-track over WHIP, SRT and ERTMP alike, and saving cloud transcoding fees is the same for all three (see EP20); the difference is only in ingest latency and weak-network behavior.

4. Layer 3: once loss crosses the threshold, raise latency first, then reconnect (the unified publish-degrade protocol)¶

Simulcast answers "which layer to drop to", but not "what signal triggers a drop, and does the drop itself also need to touch some other parameter". This layer covers the unified publish-degrade protocol that actually runs on the Origin side in ppmmx (docs/design/publish-degrade-protocol.zh-CN.md): SRT and WHIP publish share one "unrecoverable loss rate" signal driving the same degrade ladder; when the ladder reaches its last rung and needs to restart the publish, it also bumps the receive latency by one step at the same time — that "bump" has to ride along on the reconnect because of a protocol-level limitation, not a design oversight.

4.1 The unified signal: unrecoverable loss rate (ULR), not raw loss¶

Following on from §2.1 — a raw counter can't just be renamed ULR; only a receiver that can correlate original-packet identity, retransmit outcome and playout deadline can compute it. ppmmx builds that correlation separately for each protocol:

SRT: ULR = (cumulative loss − cumulative successful retransmits) / expected packets, i.e. PktRecvLoss − PktRecvRetrans (unrecoverableAccumulator in internal/servers/srt/conn.go).
WHIP (RTP): mmx's WebRTC receiver absorbs reordering and NACK retransmits inside a fixed-size reorder/retransmit buffer (WebRTCInboundRTPBufferSize, default 512 packets); a late retransmit that still lands inside the window is recovered and not counted as loss — only a packet that truly falls outside the window and can't be recovered counts toward RTPPacketsLost. So RTPPacketsLost / (RTPPacketsLost + RTPPacketsReceived) is already the WHIP-side ULR (runDegradeSampling in internal/servers/webrtc/session.go), semantically equivalent to SRT's formula.

Both signals are sampled every degradeSampleSec (default 6 seconds) and fed into the same per-path degrade state machine.

4.2 A two-stage degrade ladder: cut bitrate first, cut a layer — and reconnect — only when that isn't enough¶

Degrading isn't "one threshold crossing → reconnect"; it's two stages, escalating:

Bitrate stage (zero interruption): the moment ULR exceeds degradeRaisePct (default 0.8%), the encoder bitrate drops one notch (100%→80%→60%), applied live via obs_encoder_update with no interruption to the publish.
Resolution stage (~1s interruption): once bitrate has already hit its floor (60%), if ULR is still above a separate, higher threshold degradeRestartPct (default 1.6%), the highest-resolution simulcast layer is cut — this requires restarting the publish for the new layer count to take effect, costing roughly a 1-second interruption.

In other words, the threshold behind "reconnect only once a threshold is crossed" isn't the 0.8% that triggers degrading — it's the separate, higher 1.6%. Moderate loss only cuts bitrate, no disconnect; only loss severe enough that cutting bitrate alone isn't enough is worth paying the reconnect cost for. Every state transition is followed by a degradeObservationSec (default 60s) cooldown during which no new evaluation happens; recovery (ULR falling to/below degradeLowerPct, default 0.3%, sustained for the whole cooldown) climbs back up the same way in reverse — bitrate first, then layers, adding layers back starting from the lowest resolution.

4.3 The latency/buffer parameter only moves at the moment of an actual reconnect¶

The same event that triggers the resolution stage (an imminent publish restart) also bumps that path's receive window by one step:

SRT: receive latency (TSBPD latency) is negotiated at handshake time and is immutable for the life of the connection (the gosrt Conn interface exposes no runtime setter), so the new value can only take effect via "close the connection → the publisher re-handshakes". The reconnect therefore does double duty: it lets the publisher pick up the new layer count, and it carries the raised latency along for the ride — one interruption fixes both. Latency moves by degradeRaiseLatencyStep (default 200ms) up / degradeLowerLatencyStep (default 100ms) down, clamped to [degradeLatencyMin, degradeLatencyMax] (default 300ms–3000ms); recovery does not force a disconnect, it's left to take effect at that path's next natural handshake.
WHIP: mmx's WebRTC receiver's reorder/retransmit buffer (gortsplib's rtpreceiver.Receiver) is the same kind of fixed-at-creation, immutable-for-the-connection's-life constraint as SRT's receive latency. But WHIP has no fixed bitrate, so there's no reliable way to convert SRT's millisecond step into a packet count — instead it maintains its own per-path tuned value in packets: hitting the resolution stage doubles that path's buffer size, a recovery event halves it, bounded by webrtcBufferMin/webrtcBufferMax (default 64–4096 packets). The seed and both bounds are forced to be powers of two — gortsplib's ring buffer indexes with & (length-1), so a non-power-of-two length would misaddress silently rather than error. Like SRT, WHIP needs no extra forced disconnect either: the resolution stage already makes OBS restart the publish, and the new buffer size rides along on that restart for free.

An unexpected find: webrtcInboundRTPBufferSize (default 512) was never actually wired up

Digging into this adaptive mechanism turned up a more fundamental, separate bug: ppmmx's configured webrtcInboundRTPBufferSize default (512 packets) never actually reached the rtpreceiver.Receiver a real WHIP publish creates — the code constructing PeerConnection simply left the field unset, so every WHIP publish was silently running on gortsplib's own built-in default (64 packets), one-eighth of the configured value, independent of the adaptive-buffer feature this section is about. Both are now fixed together: the buffer size is resolved per path right before PeerConnection is constructed (runPublish in internal/servers/webrtc/session.go, the same shape as SRT's LatencyFor(path) called just before Accept()), so the configured value actually takes effect, and degrade/recover events now drive it to double/halve.

4.4 Deployment status: both chains run in production now¶

The whole mechanism is gated by one master switch, degradeEnable, which replaced the earlier per-protocol switches (srtDegradeEnable/webrtcDegradeEnable/srtLatencyAutoTune, and similar). That switch is turned on on Origin nodes — the "sample ULR → two-stage degrade → bump the receive window at a threshold-triggered reconnect" chain has always been real logic running in production on the SRT side. The WHIP buffer doubling/halving mechanism from §4.3 (including the just-uncovered dead-default bug), after finishing code implementation and unit-test verification, has since been built and shipped to all four mmx nodes (origin/record/edge-1/edge-2, 2026-10-13 — see docs/deployment/ppcdn-production-deployment-summary.md, entry 续3) — this is no longer "correct code that isn't live yet"; both chains are now production fact.

5. Layer 4: main publish first, P2P is the first thing sacrificed¶

A publisher's uplink bandwidth is finite; once it carries both "the main publish to Origin" (HEVC + H264 multi-track simulcast is itself a sizeable share of uplink, see EP20) and "P2P direct to multiple viewers", the two contend on a weak network. The principle is direct: the main publish quality is always the top priority, and P2P is an add-on that can be sacrificed at any time. This lands as three hard constraints:

Independent send queues and bandwidth budgets: the main publish (Origin WHIP) and every P2P output keep their own send queue, congestion-control state and pacing, physically isolated; P2P's send logic never steals the main publish's scheduling slot.
Absolute priority, not "fair sharing": the main publish's queue and bandwidth budget always come before P2P — not "split the bandwidth", but "the main publish eats first, P2P uses what's left".
Active monitoring + active removal: the publisher continuously monitors local uplink conditions; once a congestion threshold is hit, it immediately terminates one or more P2P connections without waiting for any central command — and after recovery it does not automatically reclaim P2P; it must request a quota from the scheduler anew.

Why "not waiting for a central command" matters: if it had to report to the control plane and wait for a "disconnect P2P" command, that round-trip is itself an indeterminate delay on a weak network. P2P removal is designed so the publisher can decide and execute locally and immediately, with no network round-trip — this is also verifiable in the code: the actual removal is a direct local call to signaling (SendClose), not a report-and-wait-for-reply round trip.

Current implementation: only one signal — RTT — is actually wired up

The architecture design doc envisioned a monitoring-signal set that includes RTT, loss rate, send-queue length and available bitrate (docs/design/ppcdn-architecture-design.zh-CN.md §6.4), but the decision logic ppobs actually ships (plugins/obs-webrtc/uplink-qos-policy.cpp) only wires up one of them: uplink RTT exceeding a threshold is treated as congestion, with a default threshold of maxRttMs=300ms, checked every 2 seconds by WHIPOutput::CheckUplinkQos. By default it removes the 1 most-recently-established P2P connection (maxPeersToDropAtOnce=1, configurable to remove all of them), then enters a 10-second cooldown (cooldownMs=10000) before re-evaluating. Loss rate, send-queue length and available bitrate are not yet wired into the decision condition — that's a known gap between design intent and the current implementation, and it shouldn't be dressed up as "all four signals are already combined into the decision."

6. Layer 5: reconnect must be designed too, or the recovery moment becomes the second outage¶

After a network blip the client tries to reconnect — but if "how to reconnect" is poorly designed, it creates new problems the instant the network recovers.

6.1 Backoff strategy: full backoff+jitter on the control connection, pplayer's media reconnects now have jitter too¶

The general principle behind "should we add backoff + jitter" is: exponential backoff solves "don't keep blindly retrying the instant something fails" — avoiding hammering an already-unstable link before the network has really recovered; jitter solves a different problem — if thousands of clients that dropped at the same moment all retry at exactly the same points on the same curve, then the instant the network recovers, every one of those requests slams in together. That's the "thundering herd" effect: the recovery moment itself creates a traffic spike. A little random jitter staggers those retries in time.

These two principles land differently depending on which link you look at:

The node↔control-plane control connection (ppmmx's heartbeat/control WebSocket to ppcenter, ppmmx/internal/mmxcontrol/client.go) implements full bounded exponential backoff + jitter: an initial 1-second wait, doubling after every failure, capped at 30 seconds, with random jitter of up to half the current backoff duration layered on top.
pplayer's two media reconnects now have jitter, though not exponential backoff: the WHEP reader (MediaMTXWebRTCReader.retryPause in pplayer/ppplayer.mjs, 2-second base) and the ABR control WebSocket (MMXControlClient.reconnectDelay in the same file, 3-second base) both now call jitteredDelay(base), which returns a random value in [base, 2×base) instead of a fixed number. These two are exactly the thundering-herd scenario from before: when an Edge drops at scale, every pplayer session attached to it hits the retry branch at roughly the same moment — with jitter in place, they no longer all slam back in at that same moment.
Edge's pull from Origin is still fixed-interval, with neither exponential growth nor jitter: a fixed 5-second retry (retryPause in ppmmx/internal/staticsources/handler.go, the same constant shared by every static-source type). The thundering-herd risk on this link is still open — a known point for future optimization.

Both pplayer jitter points are live on pplayer.pp-cdn.org (2026-10-13 — see docs/deployment/ppcdn-production-deployment-summary.md, entry 续3); fetching ppplayer.mjs from production shows the jitteredDelay function right there.

The diagram below (7.4) illustrates the general "jittered exponential backoff" mechanism itself: fully in effect on the control connection today, jitter-only (no backoff growth) on pplayer's media reconnects, and neither yet on the Edge→Origin link.

6.2 Nodes carry it themselves: reconnect even when the control plane is offline¶

An Edge node caches the Origin address template and credentials for the current stream. If the Edge→Origin link blips — even if the control plane (ppcenter) itself is unavailable at that moment — Edge doesn't need to wait for it to recover: reconnecting just means locally re-resolving the cached source config (the staticsources package's resolveSource, a pure string substitution that sends no request at all), then connecting straight to Origin with the cached credential. So whether the control plane is online or offline has no bearing on this particular reconnect. Per the previous section's finding, this reconnect loop currently runs on a fixed 5-second interval, not exponential backoff. Only when reconnects keep failing, the Origin is confirmed to be genuinely down, or an upper layer decides to terminate does it actually give up and report upward.

This means "a brief control-plane failure" and "a media-path outage" are fully decoupled: as long as the cached credential hasn't expired and the Origin is alive, whether the control plane is online during that window may be entirely imperceptible to viewers.

6.3 Honest boundary: this is not "seamless failover"¶

If the Origin truly fails (not a network blip but the node going down), the publisher must republish to a new Origin before playback requests can use the new path — cross-Origin seamless failover is not promised at this stage, and the switch has a perceptible interruption. Worth stating clearly, to avoid exaggerating it into "any failure recovers invisibly".

7. Diagrams¶

7.1 Per-protocol observation and per-direction control¶

SRT native stats ──┐
                   ├──▶ interpret per protocol + rolling window / consecutive hits / hysteresis (all product params)
RTP/RTCP/RTX ──────┘                         │
                                             ▼
                              joint decision on RTT / queue / goodput / buffer
                                             │
                          ┌──────────────────┴──────────────────┐
                          ▼                                     ▼
                  upstream publish control              downstream session ABR
             adjust bitrate/fps/publish layers      pick an existing layer per viewer
             affects the whole stream, more          doesn't change others' publish quality
             conservative policy

7.2 Unrecoverable loss rate → two-stage degrade → bump latency at the threshold-triggered reconnect¶

SRT: cumulative loss - cumulative successful retransmits    WHIP: RTPPacketsLost (what the reorder
     = unrecoverable loss                                          buffer couldn't absorb)
                          │                                        │
                          └─────────────────┬──────────────────────┘
                                             ▼
                      sample every 6s: ULR = unrecoverable loss / expected packets
                                             │
                                             ▼
                              ULR > 0.8% (degradeRaisePct)?
                                   │                  │
                                   no                 yes
                                   │                  ▼
                              no action        cut bitrate one notch (100%→80%→60%, live, no interruption)
                                                       │
                                                       ▼
                                   bitrate already at the 60% floor, and ULR > 1.6% (degradeRestartPct)?
                                              │                     │
                                              no                    yes
                                              │                     ▼
                                   holds at 60% bitrate        cut the highest-resolution layer +
                                   (no reconnect)               restart the publish (~1s interruption)
                                                                     │
                                                                     ▼
                                                     SRT: receive latency += 200ms (takes effect on
                                                          this fresh handshake; live in production)
                                                     WHIP: buffer size x2 (takes effect on the next
                                                          session; live in production)

7.3 Congestion detection → active P2P removal → re-request after recovery¶

Publisher checks every 2 seconds: local uplink RTT (the only signal currently wired up;
design intent also includes loss rate / queue length / available bitrate)
        │
        ▼
   RTT ≥ threshold (default 300ms)?
        │
   ┌────┴────┐
   no         yes
   │          │
 continue     ▼
 normally   immediately terminate 1 or more P2P connections (local decision, no central command)
                │
                ▼
        main publish bandwidth budget restored
                │
                ▼
        report to center: P2P quota released; enters a 10-second cooldown
                │
                ▼
        after network recovers, no automatic reclaim — must request a new P2P quota from the scheduler

7.4 Jittered backoff: from thundering herd to staggered recovery (currently fully in effect only on the node↔control-plane connection — see §6.1)¶

[No jitter, one backoff curve]
Failure
   │
   ▼ (all clients use the same backoff times)
Retry 1: all clients fire at t=1s
Retry 2: all clients fire at t=2s
   │
   ▼
The instant the network recovers, all retry together → spike, may knock the service down again

[Exponential backoff + random jitter]
Failure
   │
   ▼ (each client adds a random offset within the window)
Retry 1: clients spread across t=0.8s ~ 1.2s
Retry 2: clients spread across t=1.6s ~ 2.4s
   │
   ▼
Retries are staggered in time → smooth recovery, no concentrated shock

7.5 Edge autonomous reconnect: what to do when the control plane is offline¶

This path is confirmed implemented (docs/ppcdn-development-progress.zh-CN.md §3.5, status "completed", which added the POST /internal/mmx/v1/origin/resolve endpoint) — it isn't just sitting in the architecture design doc's list of MUST requirements.

Edge ←→ Origin link blips
        │
        ▼
Edge dynamically resolves the Origin address + a short-lived media credential from ppcenter;
the result is cached in memory
when ppcenter is unavailable → reuse the unexpired cache to keep reconnecting
(once the credential expires, it may no longer be used)
        │
   ┌────┴────┐
  connects OK    keeps failing / Origin confirmed dead / cache expired
   │              │
   ▼              ▼
largely imperceptible   terminate the reader, report upstream
to viewers
(fixed 5-second retry
interval; the node's
control-plane credential
and the Origin media
credential are independent
of each other)

8. Wrap-up and next episode (script notes)¶

This episode drew the statistical boundaries: SRT and RTP native counters can't be unified into a ULR via a single subtraction; sampling windows and up/down thresholds are product parameters that need calibration from measurement and version management. Control must also separate whole-stream upstream publish degradation from per-viewer downstream ABR. In between sat the unified publish-degrade protocol on the Origin side — once unrecoverable loss crosses a threshold, bitrate is cut first, and only when that isn't enough does a layer get cut along with a reconnect; on SRT that reconnect also bumps the receive latency. On WHIP, digging into the same mechanism turned up a dead-wiring bug (the configured buffer default never reached gortsplib), which got fixed alongside a new doubling/halving buffer adaptation — both chains are now deployed across every mmx node. Add the main-publish preemption of P2P (currently wired up with only the RTT signal), full exponential backoff+jitter on the control connection, the jitter pplayer's two media reconnects just picked up and shipped (the thundering-herd problem itself is fixed there; exponential growth still isn't), and the Edge→Origin link that's still plain fixed-interval — and that's the weak-network resilience design of the current version — one that deliberately marks which parts are production fact and which are still known gaps.

Next episode, a different angle — not how to cope with weak networks but how to measure "how much latency there actually is": video instrumentation and end-to-end latency measurement.