EP15: The relationship between horizontal scaling and low latency¶

Open on YouTube

Recap	EP14 "Node-pool management: putting resource boundaries in the scheduler"
Next	EP16 "Handling sudden bursts of high concurrency"

0. Goals of this episode¶

By the end of this episode, the viewer should be able to do four things:

Understand the topological basis of horizontal scaling: splitting Origin (ingest) and Edge (distribution) into two separate roles is what creates room for "one publish, N-way fanout" in the first place; but the cost is symmetric — Origin's own processing latency gets amplified N times over.
Tell apart the two ways to connect Origin → Edge: the cost model of actively pushing (push fanout) versus pulling on demand (on-demand pull), and why this link is best kept on a private network within the same AZ/Region — this is "region networking."
Know how cross-region bridging works: the four node roles in ppmmx that participate in this horizontal-scaling topology (Origin/Forward/Edge/Recorder), the fact that inter-node forwarding runs over WHIP sessions, and that "one-hop latency" is currently an engineering budget, not a number measured hop-by-hop.
Get a verifiable capacity calculation: how many Edge a 64 vCPU Singapore Region quota can stand up, what "peak concurrency" comes out to under different readings, and why you can't just multiply a single machine's load-test failure point by the node count.

1. Where horizontal scaling starts: splitting Origin and Edge into two roles¶

1.1 One publish, N-way fanout¶

In PPCDN's media pipeline, Origin is the sole entry point for media data into the system, and Edge pulls from Origin and distributes to viewers; on a given Edge, multiple viewers of the same stream share one origin-pull link (EP02 §1.1). The direct benefit of this split: scaling viewer capacity only requires adding Edge, without touching the publishing side — ppobs only ever knows about one Origin address, so how many Edge get added, or how, is completely invisible to the publish side.

flowchart LR
    P["ppobs publish"] --> O["Origin<br/>sole entry point for media data"]
    O --> E1["Edge-1"]
    O --> E2["Edge-2"]
    O --> E3["Edge-3"]
    E1 --> V1["Viewer group 1"]
    E2 --> V2["Viewer group 2"]
    E3 --> V3["Viewer group 3"]

This is exactly the "Edge tier scales horizontally" point from EP02 §2.3: Edge is scheduled by node pool, steered toward the node with the most remaining capacity, and one overloaded Edge doesn't drag down the whole system (the "most remaining capacity first" ranking from EP14 §1). The first precondition for horizontal scaling is having this 1-to-N topology in place — not simply having more machines.

1.2 The symmetric cost: Origin's processing latency gets amplified N times¶

The benefit and the cost are two sides of the same coin. All N downstream branches have to pass through the same Origin before they can fan out — if Origin's own processing step (receive, forward to Recorder/Edge, no re-encoding) takes 10ms longer, every downstream branch and every viewer inherits that same 10ms, unlike a problem on a single Edge, which only affects the viewers that happen to have landed on it.

This is an engineering judgment, not an SLA figure written down in any document for this project. Nobody has specifically measured "Origin's own processing latency for this one hop" — what we do have is the end-to-end baseline recorded in EP12: direct P2P at roughly 70ms, WHIP ingest under Edge distribution at roughly 108ms, and SRT ingest at roughly 386ms. Those numbers already include the full cost of capture, encoding, network propagation, decode and render — Origin's own forwarding step is only a small slice of that. Keeping Origin's own processing overhead down in the tens-of-milliseconds range is a safety margin reserved for every layer of fanout that follows; it's a design stance, not a conclusion that's been independently measured and verified.

2. Two ways to connect Origin → Edge: push vs on-demand pull¶

2.1 push fanout: actively pushing¶

In the production config, Origin forwards outbound over forwardMmxTargets — each downstream target gets its own independent WHIP session and its own Forwarder, with no cross-interference (docs/deployment/ppcdn-production-deployment-plan.md §2). deploy/config/mmx-origin.yml's paths.all sets source: publisher + sourceOnDemand: false: as long as a target is in the config, Origin keeps pushing to it continuously, regardless of whether that Edge currently has any viewers.

The benefit is that the path stays "hot" at all times, with no link-setup wait; the cost is that every Edge added on this path requires appending a private IP to Origin's forwardMmxTargets and reloading in lockstep, and from the moment it's appended, Origin's egress immediately gains one more forwarding session — regardless of whether there are any viewers.

2.2 on-demand pull: pulling from Origin on demand¶

deploy/config/mmx-edge.yml's paths.all sets source: redirect + sourceOnDemand: true: before any viewer arrives, Edge has no pipeline at all to Origin. The flow is: a viewer issues a WHEP play request → only then does this Edge go pull that stream from Origin → multiple viewers of the same stream on the same Edge share that one origin-pull link. This is exactly the path taken by the console's self-service node creation covered in EP14 §2.3 — before each origin pull, Edge dynamically resolves the Origin address from ppcenter (POST /internal/mmx/v1/origin/resolve), so Origin doesn't need to know about it in advance.

The benefit is that an Edge with no viewers adds close to zero extra load on Origin, and adding Edge doesn't require touching Origin's config; the cost is that cold start adds link-setup and first-frame wait.

2.3 Both connection modes coexist, with different roles¶

Production isn't "pick one and apply it everywhere" — it's split by role: pre-planned, long-running core Edge use push fanout, wired into the fixed forwardMmxTargets list, in exchange for a path that's always hot and the most predictable in latency; Edge that users self-add, where scale and go-live timing are unpredictable, use on-demand pull, so Origin doesn't need to change every time one more Edge is added.

Connection mode	Config	Immediate load on Origin	Cost	Typical scenario
push fanout (actively pushing)	`source: publisher` `sourceOnDemand: false`	Takes effect the moment it's configured, occupies egress immediately	Every addition requires changing Origin's config and reloading	Pre-planned core Edge
on-demand pull (pulling from Origin on demand)	`source: redirect` `sourceOnDemand: true`	Close to 0, until viewers actually land	Cold-start link setup + first-frame wait	Self-added, dynamically scaled-out Edge

2.4 Why this link is best kept on a private network: region networking¶

docs/deployment/ppcdn-production-deployment-plan.md deliberately places Origin/Recorder/Edge in the same availability zone: within-AZ private traffic is free; same-Region, cross-AZ is billed roughly $0.01/GB both ways; cross-region costs even more. The topology diagram spells it out — forwardMmx (WHIP push) private IP · same AZ · free. Putting Origin and Edge in the same AZ/Region saves more than just the bill — it moves this link, which carries real-time media, off the public internet, avoiding the jitter that comes with public-internet congestion. This is exactly §1's "keep Origin's own hop cost down" playing out at the deployment layer.

To be precise about it: this is currently a convention at the deployment layer, not a hard boundary enforced by the scheduler. The ranking logic in ppcenter/internal/manager/node_manager.go states explicitly: "Region is deliberately not consulted: pool members compete purely on free capacity" — candidates within a pool are ranked purely by remaining capacity, with no regard for region. region / /admin/region-routes only provide a soft city-to-candidate-node priority layer (regionRank, which only affects candidate ranking and plays no part in filtering); the real isolation boundary is the PoolID covered in EP14. Which AZ or Region a node gets placed into is, today, still a manual decision ops makes when standing up the node.

3. Crossing regions: introducing the Forward node¶

A single Region's Lightsail vCPU quota has a ceiling (§5 does that math). To bridge traffic to an Origin-Edge cluster in another Region, ppmmx added a new node role: NODE_ROLE_FORWARD (ppcenter/pb3/mmx.proto).

Forward is a pure WHIP relay: it doesn't accept direct OBS publishes and doesn't serve playback to viewers; its data flow is "passively receive + actively forward" —

Region A's Origin uses the existing forwardMmx to push the stream to the Forward node (source: publisher — this is the passive-receive end);
the Forward node then uses its own forwardMmxTargets to push the entire multi-layer simulcast stream, unchanged, to Region B's Edge cluster (this is the active-forward end).

flowchart LR
    subgraph RA["Region A"]
        OA["Origin"] -- "push (forwardMmx)" --> FA["Forward node<br/>passively receives + actively forwards"]
    end
    subgraph RB["Region B"]
        EB1["Edge"]
        EB2["Edge"]
    end
    FA -- "forwardMmxTargets (push)" --> EB1
    FA -- "forwardMmxTargets (push)" --> EB2

Two details worth remembering: first, Forward reuses the existing mmx-to-mmx forwarding implementation — no new forwarding code path was written; it's essentially just push fanout with one more hop chained on. Second, Forward isn't open for self-service customers to create (it's not one of the four options in EP14 §2.3's console node-creation flow) — where and how to bridge across Regions is, for now, an internal role the platform plans centrally; it isn't handed to users to configure themselves.

Honest status: docs/ppcdn-development-progress.zh-CN.md §3.18 marks this work as "done (code and config), real cross-region integration testing still pending." In other words, the code, config and unit tests on both the ppmmx and ppcenter sides are all in place, but no real cross-region integration test has been run yet — this is a pending-verification item, not a capability already proven in production.

4. ppmmx's four horizontal-scaling roles, and the inter-node forwarding protocol¶

4.1 Four roles: Origin / Forward / Edge / Recorder¶

ppcenter/pb3/mmx.proto's NodeRole enum actually has 7 values, but only four of them actually participate in this "cross-node forwarding, horizontally scalable" topology:

Role	Responsibility	Origin-pull / fanout method
`NODE_ROLE_ORIGIN`	Accepts publishes (WHIP/SRT/ERTMP); the sole entry point for media data	Pushes downstream (`forwardMmxTargets`)
`NODE_ROLE_FORWARD`	Cross-region bridging: passively receives + actively forwards	Receives via push upstream, sends via push downstream
`NODE_ROLE_EDGE`	Pulls from Origin, serves WHEP to viewers	Push or on-demand pull (§2)
`NODE_ROLE_RECORDER`	Records per-session split-rec, uploads to object storage	Receives via push upstream (`source: publisher`)

The remaining 3 enum values are out of scope for this episode: NODE_ROLE_STANDALONE is a self-contained single-instance role that doesn't network with other nodes (already covered in EP14 §2.1) and doesn't participate in cross-node forwarding; NODE_ROLE_PUBLISHER / NODE_ROLE_PLAYER are protocol placeholders that don't correspond to any actually-deployed ppmmx server role.

4.2 Inter-node protocol: WHIP sessions, not a separate protocol¶

Whether it's Origin→Recorder, Origin→Edge, or Origin→Forward→Edge, inter-node forwarding always runs over independent WHIP sessions — "each target is an independent WHIP session with its own Forwarder, with no cross-interference" (ppcdn-production-deployment-plan.md §2). The fact that Forward reuses the existing mmx-to-mmx forwarding implementation makes the same point: no separate protocol was stood up just for "forwarding" between nodes. The WHIP session carries standard RTP/SRTP media as-is, and the entire multi-layer simulcast stream passes straight through unchanged — intermediate nodes neither decode nor transcode it.

4.3 The latency budget for one hop: tens of milliseconds in order of magnitude, but as an engineering budget, not a hop-by-hop measurement¶

Inter-node forwarding doesn't re-run the encode/decode pipeline, so in theory the overhead should be small — mainly network RTT, WHIP negotiation, and a small amount of buffering.

To be honest about it: current latency measurements have never separately covered the Origin→Edge or Origin→Forward→Edge segment. The table in docs/design/p2p-latency-measurement-plan.md §4 is explicit about this — the SEI/OBU timestamp scheme covers the full capture-to-render chain; the Edge-side WHEP currentRoundTripTime only covers the "last hop" (Edge↔viewer), it doesn't cover the Origin→Edge segment, and it can't stand in for full-chain latency. In other words, "roughly 20ms per hop" is an engineering budget worked backward from the baseline, not a number that's been verified by hop-by-hop measurement: EP12's recorded end-to-end baseline is itself only 70~108ms (386ms for SRT ingest), and if every extra forwarding hop tacked on tens or hundreds of milliseconds, a few hops stacked together would eat right through that baseline. Keeping per-hop overhead down in the tens-of-milliseconds range is what keeps "adding one Forward hop to bridge to another Region" from overdrawing the whole chain's latency budget — which is also why Forward, for now, is only used for a one-off cross-region bridge, and isn't designed to be chained into an unlimited relay chain.

5. A real capacity calculation: how many Edge can a 64 vCPU Singapore Region stand up¶

5.1 The documented part: 64 vCPU → 30 Edge¶

docs/deployment/ppcdn-production-deployment-plan.md §4.6 works this out against confirmed production figures (Origin/Recorder stay in the same Region/AZ, consuming no resources beyond their own vCPU):

Item	Calculation	Value
Media Region Lightsail vCPU quota	Requested/planned	64 vCPU
Origin usage	1 × 2 vCPU	2 vCPU
Recorder usage	1 × 2 vCPU	2 vCPU
vCPU available for Edge	64 − 2 − 2	60 vCPU
Maximum Edge count	60 ÷ 2 vCPU/Edge	30

The document's own words are explicit: "the 30 here is the computed ceiling under the 64 vCPU quota, not a business SLA."

5.2 30 Edge — which number is actually "peak concurrency"¶

Past the 30-Edge figure, "how many viewers can it hold" depends on which reading you use for each Edge's concurrency ceiling. There are four numbers here, each of a fundamentally different nature:

Reading	Per-Edge concurrency	Total across 30	Nature
Production admission (`mmxNodeCapacity`)	100	~3000	The configured capacity actually running today, used by the scheduler for admission decisions (EP16 raised this from an earlier 12 to 100 — a baseline figure set with headroom under full capacity)
Lightsail monthly-quota average	5.5	~165	Average concurrency worked back from the 3TB monthly quota, not a peak
EP14 load-test "clean pass" ceiling	250	theoretical 7500	The clean-pass point from single-machine lab load testing; EP16 has adopted this figure as Edge's full-capacity planning ceiling, but it's still a linear extrapolation that has never been verified on an actual 30-node cluster
EP14 load-test "host unresponsive" failure point	300	theoretical 9000	The failure point where a single machine runs out of memory; still not a capacity figure, and not adopted

64 vCPU Singapore Region → 30 Edge: "peak concurrency" under four different readings

 Production admission (100/Edge)   |████████████████                    3000  ← the number actually running today (EP16 raised this from 12)
 Quota monthly average (5.5/Edge)  |████                                165
 Load-test clean pass (250/Edge)   |███████████████████████████████████ 7500  ← adopted by EP16 as the full-capacity planning ceiling, still never integration-tested
 Load-test failure point (300/Edge)|████████████████████████████████████████ 9000  ← the memory-exhaustion failure point, still not a capacity figure
                                   +-----------------------------------------▶

Taking the last tier from EP14's load-test report — "host unresponsive" (300 streams, memory exhausted) — and multiplying straight by 30 does arithmetically land on roughly 9000. But that treats "a single machine's load-test failure point" as "a capacity that's safe to operate at," conflating two different readings, and it isn't adopted. EP14 §3.3 already stressed that 300 isn't the pass point, it's the failure point; the true clean pass, with zero disconnects, is 250. EP16 adopted that 250 as Edge's full capacity, while setting day-to-day configured capacity at 100 (a baseline set with headroom under full capacity, raised by EP16 from an earlier 12) — a far more generous figure than the earlier admission threshold, but still not a measured capacity ceiling, and one that's never been integration-tested at an actual 30-node cluster scale.

Conclusion: a 64 vCPU Singapore Region has a compute ceiling of 30 Edge; but all four readings above share one blind spot — every single one is a linear extrapolation of "per-Edge number × 30," and not one of them considers whether Origin itself can even forward the stream to all 30 Edge. The next section fills in that blind spot.

5.3 A later correction: none of these four readings were right — the real bottleneck is Origin's NIC¶

A real-world measurement added later in docs/deployment/ppcdn-production-deployment-plan.md §4.8 overturns the entire "per-Edge capacity × Edge count" line of thinking: Origin uses forwardMmx to fully broadcast every live stream to every single Edge, so Origin's own egress bandwidth = concurrent-stream count × Edge count × per-stream bitrate — a number that has nothing to do with viewer count at all, and that will blow out Origin's own NIC long before the Edge count climbs anywhere near 30. The numbers 7500 and 9000 weren't "a bit optimistic" — they were never measuring the right dimension to begin with: they placed the bottleneck on the Edge side (CPU/memory), but at a 30-Edge scale, Origin is what actually hits the ceiling first.

Recalculating against a 7.5Mbps full simulcast ladder (viewers actually only pull one layer of it, evaluated at 1.5Mbps/viewer; this is this episode's illustrative basis and differs from EP11's illustrative ladder), for an Origin on a 5Gbps network tier:

Edge count	Nature	Peak viewers (`$12`-tier Edge)	Peak viewers (`$24`-tier Edge)
15	5 Gbps Origin safety value	~2,700	~3,300
22	5 Gbps Origin theoretical ceiling	~4,000	~4,800

Old vs new readings (recalculated against the Origin NIC constraint)

 Old: load-test clean-pass linear extrapolation (250/Edge×30)  |███████████████████████████████████ 7500  ← blind spot: assumes Origin's forwarding capacity is unlimited
 New: 22 Edge, 5Gbps Origin theoretical ceiling                |████████████████████                 4800  ← accounts for Origin's own egress bandwidth
 New: 15 Edge, 5Gbps Origin safety value                       |█████████████                        3300  ← a planning value with headroom built in

There's one more layer in instance selection: under a 1.5Mbps viewer load, the $12 tier (2vCPU/2GB) hits a memory bottleneck on a single machine first; the $24 tier (2vCPU/4GB), at the same price, has double the memory and 33% more traffic quota, so a single machine can carry quite a few more viewers — for the same budget, the $24 tier is the better deal, which is where the gap between the table's two columns comes from. Breaking past this roughly 15-to-22-Edge ceiling has three paths: change forwardMmx from "full broadcast" to "forward on demand" (only forward to Edge that actually have viewers watching — the most effective option), fan out across multiple Origins in layers, or move to an Origin instance on a bigger network tier.

Note that this basis only counts one H264 ladder: if the publisher publishes both an H264 and an HEVC track (EP20), the Origin-to-Edge forwarding volume increases accordingly, so HEVC's downlink saving has to be accounted together with this internal forwarding increase, not looked at on the viewer side alone.

The real lesson from this correction isn't "the numbers got smaller" — it's that "the extrapolation method itself missed an entire link in the chain": what the 64 vCPU quota computes is how many Edge you can stand up, and that's a completely different question from "how many viewers can these 30 Edge actually hold together" — the hop in between (whether Origin can even forward the stream to all of them) has to be checked separately; it can't be assumed to just hold.

6. Diagrams¶

6.1 Origin/Edge split: 1-to-N fanout¶

flowchart LR
    P["ppobs publish"] --> O["Origin<br/>sole entry point"]
    O --> E1["Edge-1"]
    O --> E2["Edge-2"]
    O --> E3["Edge-3"]

6.2 push fanout vs on-demand pull¶

flowchart LR
    subgraph PUSH["push fanout: actively pushing (forwardMmxTargets)"]
        direction TB
        O1["Origin"] -- "keeps pushing continuously, regardless of viewers" --> E1["pre-provisioned core Edge"]
    end
    subgraph PULL["on-demand pull: pulling from Origin on demand (sourceOnDemand)"]
        direction TB
        V["first viewer arrives"] -- "triggers an origin pull" --> E2["dynamically scaled-out Edge"]
        E2 -. "no pipeline before this, zero extra load" .-> O2["Origin"]
    end

6.3 Crossing regions: the Forward bridge¶

flowchart LR
    subgraph RA["Region A"]
        OA["Origin"] -- "push" --> FA["Forward<br/>passively receives + actively forwards"]
    end
    subgraph RB["Region B"]
        EB1["Edge"]
        EB2["Edge"]
    end
    FA --> EB1
    FA --> EB2

6.4 64 vCPU → 30 Edge: four capacity readings, and the one that got corrected¶

flowchart TB
    Q["64 vCPU Singapore Region"] --> A["Origin 2vCPU + Recorder 2vCPU"]
    Q --> B["Edge available: 60 vCPU ÷ 2 = 30"]
    B --> C1["Production admission 100/Edge → ~3000<br/>the number actually running today (EP16 raised this from 12)"]
    B --> C2["Quota monthly average 5.5/Edge → ~165"]
    B --> C3["Old: load-test clean pass 250/Edge → theoretical 7500<br/>✗ overturned by §5.3: ignores Origin's own forwarding bandwidth"]
    B --> C4["Load-test failure point 300/Edge → theoretical 9000<br/>memory exhaustion, still not a capacity figure"]
    C3 -.->|"§4.8 real measurement"| D["New: by Origin NIC constraint<br/>15 Edge ~3,300 / 22 Edge ~4,800<br/>($24-tier Edge, 5 Gbps Origin)"]

7. Closing and next-episode preview (script notes)¶

This episode laid out the topological basis of horizontal scaling: splitting Origin and Edge into two roles is what creates room for "one publish, N-way fanout," but the cost is symmetric — Origin's own processing latency gets amplified N times over, so it has to be kept down within an engineering budget of tens of milliseconds; that's a design stance, not a number that's actually been measured hop by hop. There are two ways to connect Origin to Edge: pre-planned core paths use active pushing (forwardMmxTargets), while dynamically scaled-out edges use on-demand pull (sourceOnDemand) — the two coexist with different roles. This link is best kept on a private network within the same AZ/Region, which saves real money on cross-zone fees and avoids public-internet jitter, but that's currently a manual convention at the deployment layer, not a hard boundary enforced by the scheduler. Bridging across regions means bringing in the Forward node — passively receiving, actively forwarding, reusing the existing forwarding implementation; the code and config are all in place, but real cross-region integration testing hasn't happened yet. Inter-node forwarding runs over WHIP sessions, forwarded as-is with no decode or transcode; "roughly tens of milliseconds per hop" is an engineering budget worked backward from the baseline, not a hop-by-hop figure that's been independently measured. Finally, we worked out a real number: a 64 vCPU Singapore Region quota, after subtracting Origin/Recorder, can stand up 30 Edge — but "how many viewers can 30 Edge hold together" can't be found by multiplying per-Edge capacity straight by the node count. Origin itself fully broadcasts via forwardMmx to every Edge, and its own egress NIC is the real bottleneck: calculated against a 5Gbps-tier Origin, 15 Edge ($24 tier) gives a safety value of roughly 3,300 viewers, and 22 Edge gives a theoretical ceiling of roughly 4,800 — far smaller than the earlier linear extrapolation of "250×30=7500," because that extrapolation never considered whether Origin could even keep up with the forwarding.

Adding an Edge, done manually, is a seven-step process: provision a new machine → deploy mmx → configure the node ID and capacity → open the firewall, configure TLS → append the new node's private IP to Origin's forwardMmxTargets and reload → update the region mapping too, if one exists → confirm the node is online in ppcenter and verify with a WHEP pull. This process takes minutes at minimum, which is good enough when load is climbing slowly.

Next episode, we look at a scenario this process has no hope of keeping up with — when traffic isn't a slow climb but a sudden burst: how Edge uses configured capacity to leave elastic headroom, and how, when scaling can't keep pace, a "business degradation permit" kicks in automatically to drop quality down to 720p and ride out the window.