EP19: Load-testing methodology — from a single box to global load¶

Open on YouTube

Recap	EP18 "Recording & playback: adding VOD to live streaming"
Next	EP20 "PPCDN brings you 30%–50% savings in bandwidth & cost & latency!" (series finale)

0. Goals of this episode¶

By the end, the viewer should be able to:

Understand how single-node capacity was measured: the same method and the same standalone node spec, stress-tested on both AWS and LightNode (Manila), and why identical machine specs can produce wildly different results.
Recognize what the real constraint is: CPU and memory can be upgraded, but egress bandwidth can't simply be dialed up — true for self-hosted standalone nodes, and even more true for the platform's own Origin and Recorder nodes, whose scaling ceiling is set directly by NIC bandwidth.
Understand the real stress-test method and data: how stepped-load testing works, what to watch at each step, and what common patterns three real reports reveal when read side by side.

1. How single-node capacity was measured: same method, two datacenters¶

1.1 Method: stepped-load testing¶

What's under test is the standalone node (covered in EP17: a single instance that both ingests the publish and serves viewers directly, with no forwarding and no origin-pull). The tool is whep-loadgen — a self-built client based on pion/webrtc that opens a real PeerConnection against the node's WHEP endpoint, runs the full SDP/ICE/DTLS/SRTP handshake, then only reads RTP packets and discards them (no decoding) — so the load generator itself never becomes the bottleneck.

flowchart LR
    A["Starting step<br/>12 concurrent, 10-min soak"] --> B["Double each step<br/>25 → 50 → 75 → 100 ..."]
    B --> C{"All sessions connect<br/>this step, zero disconnects?"}
    C -->|Yes| D["Record CPU/memory/egress/loss<br/>move to next step"]
    C -->|No| E["Mark FAIL, stop<br/>previous step = max stable concurrency"]
    D --> B

Each step runs for 2 minutes (the first step is a 10-minute soak), sampling the node's CPU, memory, network egress, and the session count/packet loss/dropped frames from /metrics throughout the run; the moment a step fails to connect every session, or shows sustained reconnects/dropped frames, testing stops immediately — the previous step is that machine's maximum stable concurrency.

1.2 Same method, two datacenters, wildly different results¶

This is the most valuable finding from this round of testing: the same 2 vCPU / 2-4GB standalone node spec, tested with the same stepped method, produced a 5x difference just by switching cloud providers.

LightNode (Manila) 2vCPU/4GB
12 ████ 25 ████████ 50 ████████████████ 75 ✗ FAIL
                                        (CPU spiked to 134%/177%, connect timeouts, frames dropped)

AWS Lightsail       2vCPU/2GB
12→25→50→75→100→150→200 all PASS ████████████████████████████████████████ 250 PASS
                                                                    300 ✗ FAIL (memory exhausted)

	LightNode (Manila)	AWS Lightsail
Spec	2 vCPU / 4GB	2 vCPU / 2GB (less memory)
Nominal egress	1 Gbps	1 Gbps
Measured public egress	~50-106 Mbps	~2.4 Gbps
Max completed concurrency	50 (75 failed)	250 (300 failed on memory exhaustion)
Resource hit first	Provider network throttling	The node's own memory

The exact same machine spec collapsed at 75 concurrent on LightNode, yet on AWS Lightsail it held stable all the way to 250 — not because the machine suddenly got stronger, but because the first test was never really measuring the machine's ceiling, it was measuring the provider's hidden network throttling. This is the first lesson any single-node capacity test has to learn: a single test's "failure" conclusion, without re-testing in a different environment, can easily mistake "provider throttling" for "the machine's real ceiling."

2. The real constraint isn't CPU, it's egress bandwidth¶

2.1 The gulf between "nominal 1 Gbps" and "measured egress"¶

Before the stress test, a pure network test (no mmx running) was run against LightNode instances using iperf3, and the result was unambiguous: across three separately provisioned instances and four cities (Manila, Taipei, Singapore, Tokyo), the measured whole-machine egress consistently landed between 50-106 Mbps, never approaching the nominal 1 Gbps. Aggregate upload across 8 concurrent streams was still capped at around 103 Mbps, ruling out "a single-stream rate limit" — the throttling sits on the provider's network side, not inside the VM.

flowchart TD
    A["mmx only forwards the media stream<br/>no transcoding, no decoding"] --> B{"Is egress bandwidth enough?"}
    B -->|Enough: AWS ~2.4Gbps| C["Bottleneck is CPU/memory<br/>fixable by upgrading the machine"]
    B -->|Not enough: LightNode ~100Mbps| D["Bottleneck is the network<br/>no amount of added CPU helps"]

mmx only forwards, it never transcodes — so the bottleneck is usually egress bandwidth and the NIC, not CPU. This is the same conclusion these stress-test reports keep confirming: CPU and memory can be solved by upgrading the machine, but egress bandwidth is a hard constraint set by the cloud provider — if you plan capacity from the nominal number instead of measuring it, the resulting number is almost certainly wrong.

2.2 For Origin and Recorder, NIC bandwidth directly sets the scaling ceiling¶

This conclusion doesn't only hold for self-hosted standalone nodes — it applies even more directly to the platform's own Origin and Recorder roles, whose core job is literally "forward the stream onward," making them inherently bandwidth-sensitive:

Origin: every additional Edge it serves means forwarding one more copy of the stream. In one real deployment, 13 Edges × 5 Mbps ≈ 65 Mbps of private-IP egress, and the instance size in use at the time (t3.medium) had enough network headroom — but the deployment record specifically flagged "needs review if the Edge count grows further": double the Edge count and Origin's egress requirement doubles right along with it, while CPU and memory stay flat — the NIC is always what hits the ceiling first.
Recorder: its egress bandwidth has to carry two jobs at once — forwarding to the backup distribution line and uploading recording files to object storage (covered in EP18) — and both share the same bandwidth quota. Also from a real deployment: backup-relay forwarding runs about 1.55 TB/month, recording upload about 0.78 TB/month (estimated at 50% of inbound traffic), together accounting for nearly half the bandwidth quota — and that's just the current scale; once the number of forwarded streams or the recording duration grows, what alarms first is almost certainly the bandwidth quota, not CPU utilization.

flowchart LR
    subgraph O["Origin"]
        O1["Receives 1 publish"] --> O2["Forwards to N Edges"]
        O2 -.->|"The larger N, the more egress bandwidth grows linearly"| O3["NIC bandwidth is the scaling ceiling"]
    end
    subgraph R["Recorder"]
        R1["Receives the stream Origin forwarded"] --> R2["① Forward to the backup distribution line"]
        R1 --> R3["② Upload recording to object storage"]
        R2 & R3 -.->|"Share the same egress bandwidth quota"| R4["NIC bandwidth is the scaling ceiling"]
    end

There's a ready-made method for telling "which one hits first, CPU or bandwidth": sample the node's CPU utilization and NIC egress rate (the tx delta from /proc/net/dev) at the same time, and plot both on the same chart to see which line reaches its ceiling first — if egress hits its measured bandwidth limit while CPU still has clear headroom, you're bandwidth-bound; only when CPU climbs past 90% while egress hasn't hit its limit are you genuinely CPU-bound. Skip this chart and conclude from CPU utilization alone, and it's easy to mistake "bandwidth is choked" for "the machine isn't strong enough."

3. Test data: three real reports side by side¶

Machine tier	Max stable concurrency	Failure step	Resource hit first
LightNode 2vCPU/4GB	50 concurrent	75 concurrent	Provider network throttling (~100Mbps)
AWS Lightsail 2vCPU/2GB	250 concurrent	300 concurrent	Memory exhaustion (down to 289MB → 33MB)
LightNode 1vCPU/2GB	18 concurrent	19 concurrent	CPU (jumped from ~40% to 88-103% within 15 seconds)

3.1 Degradation is a cliff, not a gradual slope¶

In all three reports, CPU/resource usage never climbed gently to a critical point and then failed slowly — past a certain step, it jumped straight from "healthy" to "overloaded" within seconds to the low tens of seconds:

1vCPU/2GB node CPU utilization
10 concurrent ██████ 26%
15 concurrent █████████ 45%
17 concurrent ██████████ 47%
18 concurrent ██████████ 45%   ← last stable step
19 concurrent ████████████████████ 88~103%   ← jumped up within 15 seconds, frames start dropping

Between 18 and 19 concurrent, extrapolating the 10-18 slope would predict only about 48-55%, but the measured value jumped straight to 88-103% — there is no transition band. The reason is straightforward: a single-core machine has no second core to fall back on for GC, network polling, and bursts; once the main processing path can't keep up with real-time pacing, buffers start to back up, and dropped-frame/ retransmission handling in turn eats even more CPU, forming a self-amplifying congestion avalanche rather than a graceful degradation.

3.2 You can't linearly extrapolate across machine tiers¶

The slope measured on the 2 vCPU machine is roughly "CPU% ≈ 15 + 0.97% × concurrency"; halving that line to estimate the 1 vCPU machine's capacity gives an optimistic ~35 concurrent, but the real measurement was only 18. Losing one core isn't simply losing half the compute power — without a second core to fall back on, the cliff arrives much earlier and much narrower. The biggest lesson from putting all three reports side by side: changing machine tiers requires re-measuring from scratch — you cannot extrapolate from another tier's curve.

4. One diagram, tying it together¶

flowchart TD
    A["Same stepped-load method<br/>whep-loadgen, 12→25→50→75→..."] --> B["Tested once in each datacenter"]
    B --> C["LightNode: crashes at 75 concurrent"]
    B --> D["AWS Lightsail: doesn't crash until 250 concurrent"]
    C --> E["Investigation: LightNode's measured egress<br/>is only ~100Mbps, not a CPU problem"]
    D --> F["AWS's measured egress ~2.4Gbps is enough<br/>the real crash cause is memory exhaustion"]
    E --> G["Conclusion: CPU/memory can be upgraded<br/>egress bandwidth is a hard constraint set by the cloud provider"]
    F --> G
    G --> H["Same applies to Origin/Recorder:<br/>NIC bandwidth directly sets the scaling ceiling"]

5. Wrap-up and next episode (script notes)¶

This episode covered how single-node capacity was measured: the same stepped-load method, run once on LightNode's Manila datacenter and once on AWS, with the identical machine spec — LightNode collapsed at 75 concurrent, while AWS held stable all the way to 250. The difference wasn't the machine, it was the provider's real egress bandwidth (~100Mbps vs ~2.4Gbps). That's also this episode's core thread: CPU and memory can be solved by upgrading the machine, but egress bandwidth is a hard constraint set by the cloud provider — it can't simply be dialed up; for roles like Origin and Recorder, whose whole job is forwarding traffic, NIC bandwidth directly sets how far they can scale. The three real stress-test reports also jointly confirmed two more things: node degradation is a cliff, not a gradual decline; and you can't linearly extrapolate across machine tiers — changing tiers means re-measuring from scratch.

Next episode is the last one in this course. We return to the original business question: how much can PPCDN actually save you? The publisher's own uplink bandwidth replaces cloud transcoding fees, HEVC saves downlink bandwidth, and P2P saves Edge downlink and brings end-to-end latency under 110ms — we'll work out all three accounts in one go, with every number stating the conditions under which it holds, and lay out, item by item, what's measured, what's modeled, and what's still unverified.