EP08: Video instrumentation — measuring real end-to-end latency¶

Open on YouTube

Recap	EP07 "How to strengthen resilience on weak networks"
Next	EP09 "Latency stability matters more than average latency"

0. Goals of this episode¶

By the end, the viewer should be able to:

Understand why "end-to-end latency" cannot simply be "now minus timestamp", and where measuring latency in a browser is genuinely hard.
Know the three other playback-experience instrumentation metrics besides latency: video load time, connection success rate and rebuffer rate — and how each is reported and aggregated.

1. Opening hook (script notes)¶

The current measured baseline: P2P really works at around 70ms; for ingest measured on the Edge-distribution path, WHIP comes in around 108ms and SRT around 386ms (the gap comes from the ingest protocol's receive window, not the codec). How are these numbers obtained? Stamping a timestamp at the publisher isn't enough — the player must know which frame that timestamp belongs to, when that frame actually rendered, and handle the clock error between the two ends. And latency alone doesn't describe the experience; first-frame, connection success and rebuffer caliber have to fill it out.

2. The problem: this subtraction can't be done correctly in a browser¶

2.1 How timestamps get into the picture¶

ppobs stamps every video frame (not just keyframes): H.264 wraps it in a user_data_unregistered SEI, HEVC in a Suffix SEI, and AV1 in a metadata OBU. The payload is a fixed 16-byte UUID (the ASCII encoding of "OBS-ABSTS-SEI-v1", directly recognizable in a packet capture) plus an 8-byte big-endian timestamp — 24 bytes total, and it carries only a timestamp, no explicit frame number. The reason: this data is physically embedded in that frame's own bitstream and travels with it, so the instant the player decodes the SEI it inherently knows "this timestamp belongs to the frame I'm decoding right now" — no extra ID is needed to correlate them.

Worth being precise about which instant this timestamp actually marks: it isn't the moment the encoder receives the raw frame — ppobs internally has a pkt_time that's closer to the true capture instant, but the stamping code receives that parameter and discards it unused — it's the moment encoding finishes and the packet is about to be sent out over the publish connection. So encoder processing time itself gets silently folded into "end-to-end latency" rather than excluded from it. Nothing here is a bug; it's an inherent boundary of this measurement: it measures "encode output to render", not strictly "in front of the lens to on the screen" — worth stating plainly whenever this number gets cited.

Timestamps travel only along this one in-bitstream path — there is no protocol-specific side channel anymore. An earlier version additionally broadcast the same timestamp over a WHIP-only "obs-timestamp" WebRTC DataChannel, but a DataChannel is a WebRTC/WHIP capability with no equivalent for RTMP/ERTMP or SRT publishing, so that side channel has been removed. The server (whichever of RTMP, SRT or WHIP it's ingesting) scans for this same SEI/OBU directly out of its own NAL/OBU parsing pipeline — one shared mechanism across all three ingest protocols, instead of maintaining a separate side channel per protocol.

2.2 The player's dilemma: no NTP, only a local clock of unknown offset¶

A latency sample should be generated at the callback where the target frame is actually submitted for display, not when a packet arrives or decode completes. Conceptually it's render_time(frame_id) - capture_time(frame_id). A browser usually only has the OS wall clock and a monotonic clock; even if both ends did clock sync, there remain offset, drift, sync accuracy and network-asymmetry errors.

On the ppobs side, "did clock sync" is actually two layers, preferring the more accurate one and only falling back when there's no other choice:

Preferred: align to the same ppcenter anchor the player uses. While publishing, ppobs also opens a separate /ws/play WebSocket to ppcenter and runs the exact same four-timestamp probe covered in the next section — the identical wire protocol the player uses ({"type":"TIME_SYNC","t1":...} → {"type":"TIME_SYNC_ACK","t1":...,"t2":...,"t3":...}), 6 probes per round, 150ms apart, keeping the minimum-RTT sample, recalibrating every 10 minutes. As long as that offset is still fresh (under 20 minutes old), it's preferred for correcting the local clock.
Fallback: public NTP. Only when it has never connected to ppcenter, or that offset has gone stale past 20 minutes, does it fall back to public NTP: at startup it spins up a background thread that polls time.cloudflare.com, time.windows.com, pool.ntp.org, takes up to 4 samples per server and keeps the one with the lowest RTT, ending sampling early once an RTT under 30ms is seen; after that it recalibrates every 30 minutes. As long as it has synced successfully even once, it keeps using that last-calibrated offset even if every server later becomes unreachable, rather than falling back to the raw system clock — only if it has never synced successfully at all (e.g. the network blocks UDP port 123) does it fall back to an uncalibrated local wall clock.

In real live-streaming conditions, the 10-minute refresh cadence is more frequent than the 20-minute staleness window, so as long as the publisher can reach ppcenter, what's actually in effect is almost always the first layer — meaning the publisher and player aren't each separately guessing at UTC, they're independently reaching for the same anchor (ppcenter); public NTP is really just a fallback for the moment streaming starts or whenever ppcenter is briefly unreachable. A browser can't do the public-NTP layer (no access to UDP 123), but what a browser can and does do is the same "four-timestamp probe against ppcenter" ppobs runs — covered in detail next.

Why 10 minutes, not shorter: the probe doesn't open a new connection (it rides the /ws/play connection the player/publisher already has open, which ppcenter also keeps alive with its own 30-second server-side ping, independent of how often TIME_SYNC fires), so a single round is cheap. The real constraint is precision: an uncorrected consumer clock typically drifts at roughly 20–100ppm; at the worst case of 100ppm, 10 minutes of drift is about 60ms — still small next to this system's 70–386ms latency baselines. This value used to be 45 seconds; it was raised to 10 minutes once the aggregate probe volume ppcenter would see at real scale became a consideration — a point picked for the project's current size on the precision-vs-server-load tradeoff, not a uniquely correct number derived from first principles.

That offset directly breaks the computation: if the player's local clock lags ppobs's and the true latency is small enough, subtracting directly yields a negative number — and latency can't be negative, which shows the subtraction used the wrong baseline. That's the core problem this episode solves.

3. The fix: an application-layer "NTP-style" offset calibration¶

3.1 The four-timestamp algorithm¶

You don't need to implement real NTP, just borrow NTP's "four timestamps compute an offset" algorithm over an existing application-layer connection:

T1 = player's local time when it sends the request
T2 = server time when it receives the request
T3 = server time when it sends the reply
T4 = player's local time when it receives the reply

offset = ((T2 − T1) + (T3 − T4)) / 2   ≈ server clock − player clock
rtt    = (T4 − T1) − (T3 − T2)          network round-trip, excluding server processing

An easy trap: the sign of offset is "server clock minus player clock", so correcting local time means adding offset, not subtracting — the sign is easy to flip; the right approach is to verify with concrete numbers before shipping (e.g. set a known case "local clock is 100ms ahead of the server", verify the computed offset is −100, and only then does correction cancel that 100ms). After calibration, latency becomes:

corrected_now  = local reading + offset
one_way_delay  = corrected_now − embedded timestamp

3.2 Why round-trip against ppcenter, not ppobs¶

The first intuition is "the player should align to the publisher", but that assumption is structurally invalid: P2P quota is at most 3 per publisher (EP02), so the vast majority of viewers on Edge aren't connected to the publisher at all and can't do this round-trip with it.

What you actually estimate is the offset between the capture end's and player's clock domains. Using a clock-synced server as a UTC approximation is one engineering implementation, but neither the server nor the publisher is an error-free UTC: their own sync errors add into the result, and the four-timestamp algorithm also assumes roughly symmetric up/down delay. So record the minimum-RTT sample, offset age and estimation uncertainty alongside latency, rather than treating the calibration as truth. High-precision scenarios can have the capture end and player each sync to the same controlled time source and monitor drift — this isn't a hypothetical suggestion: §2.2's ppobs calibration against ppcenter is exactly that, with the publisher and player each independently aligning to the same ppcenter anchor, rather than the player guessing at UTC alone while the publisher privately carries a different one.

3.3 How to sample reliably¶

Run one round of sampling immediately after the connection is established, a fixed 6 probes per round (not a range), and take the offset of the minimum-RTT probe — a smaller RTT means that round-trip was less disturbed by queueing and jitter. This is the standard NTP/SNTP client selection method, far more reliable than a single probe.
Report no latency metric before the first calibration completes, to avoid dirty data from an untrusted offset — the same "no trusted data, no output" principle recurring throughout this course.
Re-calibrate periodically (currently every 10 minutes), because device clocks drift over time and a one-time calibration goes stale on long streams.

4. How timestamps reach the player: two connections, each handling a part¶

One easily confused point: "calibrating the offset" and "delivering timestamps" go over two fully independent connections, each handling a different job.

Players supporting Insertable Streams can parse the SEI/OBU timestamp directly from the bitstream — no extra channel needed.
Cross-browser fallback: some browsers (e.g. iOS Safari without Insertable Streams) can't read in-bitstream timestamps. In that case the receiving node the player is connected to — plainly, the node it is actually pulling from (the Edge when it plays via Edge, the Origin only when it plays straight from Origin) scans the same SEI/OBU timestamp directly out of the bitstream (the same parsing logic regardless of whether the publish used RTMP, SRT or WHIP), re-wraps it into a message, and broadcasts it over the /ws/control ABR control WebSocket the player already has open — a connection originally established for layer switching (the ABR layer selection from EP05). It now carries one more duty — forwarding timestamps — instead of opening another DataChannel. One fewer connection, reusing an existing channel, is the direct reason for the implementation choice here. One easy mistake to avoid: pplayer's control WebSocket connects to the node it is pulling from, not the Origin — the WS host is taken from the WHEP URL (MMXControlClient in pplayer/pplayer.mjs). So who broadcasts to the player depends on which node the viewer is attached to; it must not be written uniformly as "the Origin broadcasts to the player" — the Origin can only broadcast to the viewers connected directly to it.

So how do viewers attached to an Edge get the timestamp? The Origin→Edge hop uses an explicit relay: while pulling from Origin, Edge — on top of its internal WHIP pull session — dials an extra /ws/control?session_id=<pull session> connection to Origin to subscribe to the OBS_TIMESTAMP messages Origin broadcasts, then calls its own RelayObsTimestamp to forward them to the players attached to that Edge (ppmmx/internal/staticsources/webrtc/obs_timestamp_relay.go, internal/servers/webrtc/obs_timestamp_broadcast.go). In other words the cross-browser fallback message actually travels two hops: Origin → (Edge's subscribed /ws/control) → Edge → (player's /ws/control) → pplayer, not Origin pushing straight to pplayer. (Edge pulling from Origin is itself a WHIP ingest session and could in principle re-scan the SEI in the media stream on its own — that maps to the Path A fallback; the cross-browser path uses the explicit relay above.) Limited by the lack of a real multi-node environment, it's only verified that "each independent stage works as designed and degrades gracefully" — a full real multi-node end-to-end integration hasn't been run. Worth stating honestly here rather than dressing it up as "fully verified end to end".

5. Handling residual noise: invalid samples must not enter the official SLA¶

Negative latency is physically invalid, meaning at least one of clock calibration, frame matching or sampling timing is unreliable. It must not be clamped to 0 and mixed into the official SLA, nor participate as a signed raw value in official percentiles — either would artificially improve the distribution. The right approach is to keep it in a separate data-quality stream, tag the reason and trigger recalibration; the official SLA accepts only non-negative samples with successful frame matching, an unexpired offset and uncertainty within budget, while disclosing the invalid-sample rate. Persistent negatives need an alarm; occasional negatives count toward measurement quality, not business latency.

6. Complementary, not substitutes: distinguish ICE RTT from RTCP RTT¶

In WebRTC, at least two quantities are casually called RTT and must not be mixed. RTCIceCandidatePairStats.currentRoundTripTime comes from the current candidate pair's STUN connectivity check and describes the ICE transport path; RTCRemoteInboundRtpStreamStats.roundTripTime and other RTP stats are based on RTCP messages and describe the media-control feedback for the corresponding SSRC. Field availability depends on browser and session state.

Metric	Coverage
SEI/OBU timestamp one-way latency (this episode's approach)	Full path from publisher capture to player render, including encode/decode, jitter buffer, queueing
ICE candidate-pair RTT	STUN round-trip on the current ICE candidate pair; excludes encode/decode and playout buffering
RTCP RTT	Control-message round-trip for the corresponding RTP/SSRC; not equal to the ICE check RTT, and excludes the full encode/decode path

Both signals should be collected together: currentRoundTripTime is a free direct-connection quality signal, but only this episode's timestamp approach yields latency data covering the full path.

7. Beyond latency: three equally important playback instrumentation metrics¶

Knowing "what the latency is" isn't enough — a play request that takes forever to connect, connects but takes long to show a picture, or shows a picture but stutters throughout: none of those three bad experiences is captured by a single "end-to-end latency" number. The system maintains three separate instrumentation metrics, each answering one question.

7.1 First-frame time: from playback intent to the first actually-rendered frame¶

First-frame should conceptually unify its endpoints: start is when the player accepts a valid play request, end is when the first non-placeholder video frame is actually submitted for render. Merely receiving RTP packets, a complete frame, or finishing decode does not mean the user has seen a picture. Reporting should also note whether it included auth, signaling and autoplay waiting; if the business needs a separate "first frame after connect", name it as a different metric.

An easily-missed precision detail: these start/end timestamps should in principle use a monotonic clock (immune to system clock corrections), but both of pplayer's current timestamps are plain Date.now() (wall clock) — the same codebase uses performance.now() elsewhere (e.g. NAT-probe timing), just not here. If the user's device gets a system clock jump within this few-hundred-ms window, this measurement could in principle pick up a small, unpredictable skew. It's a small but real precision gap in the implementation, not a nitpick for its own sake.

This field is used in two reports: the one-shot connection-result report carries it (to measure "how was this connection"), and the play-session start report carries it too. But the server today only computes the average/min/max for this field — it doesn't yet compute P50/P95/P99 percentiles. Real percentile computation does exist in ppcenter, but it's wired to this episode's main metric (the SEI/OBU one-way latency), not to this first-frame-time field; wiring first-frame time into percentile aggregation too is still a follow-up, not something already done.

The two paths don't currently implement this at the same precision: the P2P path uses the browser's video.requestVideoFrameCallback — a callback the browser fires only when a frame is actually submitted for composited display, which maps exactly onto the "first non-placeholder frame actually submitted for render" definition above. The Edge path currently approximates "decoding has started" by polling getStats and checking for fps > 0, with a 1-second poll interval — coarser granularity than the P2P path. This isn't a doc/code mismatch; it's an engineering trade-off driven by differing browser capabilities — requestVideoFrameCallback is more precise, but today it's only used on the P2P leg.

7.2 Connection success rate: a one-shot report, not just a success/fail boolean¶

After each play decision settles (P2P or Edge chosen), the player sends a one-shot "connection result" report with these core fields:

path: whether this actually used p2p or edge.
isOK: whether the connection succeeded.
reason: on failure, the specific reason — e.g. ICE checks didn't select a usable direct candidate pair, P2P quota full, or a network/permission error; even on success it can carry the candidate type and path-selection result, for grouping by network condition. Don't judge failure by a "symmetric NAT" label alone.

Why separate "no live stream is publishing" as a failure: if a viewer reports "connection failed" but the stream is actually offline (the host just ended it that second), this failure doesn't indicate a problem with the system's connection capability — the server checks whether the stream is really online; if not, the failure record counts only toward "total play attempts" (used for stats like P2P hit rate) and not toward "connection failures", so it doesn't lower the success rate — avoiding a normal event like a host ending the stream being miscounted as a service fault.

7.3 Rebuffer rate: unify the event and the denominator first¶

Rebuffering and connection success are not the same class of problem. Define rebuffering as: after the first frame, while the user expects playback and the page is in the counted state, render progress stalls beyond the product threshold; one event lasts until rendering stably resumes. Active pause, background suspension, seek, stream switch and playback end don't count. The threshold is a product parameter recorded per platform and version — pplayer's current default is 2000ms (DEFAULT_LAG_THRESHOLD_MS in lag-tracker.mjs), meaning a <video> element's waiting event has to persist past 2 seconds before it counts as a rebuffer; shorter waits below that threshold (e.g. a brief waiting triggered by an ABR layer switch) don't count. Heartbeat and final reports carry two cumulative fields:

lagCount: how many rebuffers since playback started (the player judges "does this count as a rebuffer" against a threshold).
lagDurationMs: how many milliseconds those rebuffers totaled.

The time-based rebuffer rate aggregates as "total rebuffer duration ÷ effective watch time"; you can also report rebuffers per minute. Effective watch time excludes pre-startup, active pause and background suspension. Numerator, denominator and event-merge rules must be fixed, or different players' data aren't comparable.

An easily-missed implementation detail: the playback-end report must be sent with the browser's sendBeacon, because at the moment the page closes/refreshes, a normal async request may be terminated before it finishes; sendBeacon is the API browsers designed for "still send a request as the page unloads". But sendBeacon can't set custom request headers, so auth info can't go in the Authorization header like other APIs — it must go in the body. So here it reuses the txTime/txSecret short-lived playback token from EP06, passed directly as a body field, and the player never holds appSecret.

7.4 Why three separate metrics, not one big event¶

Video load time, connection success rate and rebuffer rate all look like parts of "playback experience", but they weren't crammed into one universal reporting API, because their timing and lifecycle are completely different: connection success is a one-shot judgment at the instant playback is established, load time is a one-shot process from request to first frame, and rebuffering is a state that accumulates throughout playback. Designing to each lifecycle as "one-shot result report" and "session heartbeat" is easier to maintain than one oversized, semantically mixed event, and lets the server aggregate independently along the three dimensions "connection quality", "startup speed" and "playback smoothness" without first deconstructing a big event to compute each dimension's numbers.

8. Diagrams¶

8.1 Timestamp (TS) transmission diagram¶

The timestamp (TS), from ppobs stamping it to the player actually receiving it

ppobs (aligned to the ppcenter anchor, public NTP as fallback)
   │  stamp an absolute timestamp into the bitstream per frame: SEI for H.264/HEVC, metadata OBU for AV1
   │  RTMP/ERTMP, SRT and WHIP all stamp the same bitstream alike
   ▼
Origin (no decode, no transcode; passes through as-is; scans this SEI/OBU
        directly out of its own NAL/OBU parsing pipeline)
   │
   ├──────────────────── Path A: player supports Insertable Streams
   │        the player parses SEI/OBU from the bitstream itself; no server forwarding needed
   │
   └──────────────────── Path B: cross-browser fallback (iOS Safari etc. can't read SEI)
             Origin wraps the timestamp it just parsed into a message
             {"type":"OBS_TIMESTAMP","data":{frame_no,timestamp,rid}}
                     │
                     ├─▶ broadcast to viewers connected directly to Origin (their /ws/control)
                     │
                     └─▶ [Origin → Edge relay]
                           while pulling, Edge dials an extra
                           /ws/control?session_id=<pull session> to subscribe to the broadcast
                                   │
                                   ▼
                           Edge calls RelayObsTimestamp on receipt
                                   │
                                   ▼
                           broadcast to the players attached to that Edge (their /ws/control)
                                   │
                                   ▼
                    pplayer receives the OBS_TIMESTAMP on the /ws/control
                    connection it already holds for layer switching

Key point: pplayer's control WebSocket connects to "the node it is pulling from"
(the Edge when playing via Edge; only the Origin when playing straight from Origin), not the
Origin — so the cross-browser fallback travels two hops, Origin → Edge → player,
not Origin pushing straight to the player.

8.2 Offset calibration: four timestamps for one trusted offset¶

Player                                   Server (ppcenter)
  │  T1: send calibration request ──────────▶ │  T2: receive request
  │                                            │
  │ ◀──────────────────────────────────────   │  T3: send reply
  │  T4: receive reply                         │

offset = ((T2 − T1) + (T3 − T4)) / 2   (server clock − player clock)
rtt    = (T4 − T1) − (T3 − T2)         (network round-trip, excluding server processing)

Repeat 6 times, take the offset of the minimum-rtt probe as this round's calibration

ppobs runs exactly this same probe on the publish side (swap "Player" for "Publisher" in the diagram), once every 10 minutes — this is the higher-priority calibration layer from §2.2: both ends independently align to the same ppcenter anchor, not just the player.

8.3 The full instrumentation lifecycle of one playback¶

Viewer clicks play
   │
   ▼
Play decision settles (P2P or Edge chosen)
   │
   ├──▶ one-shot "connection result" report: path, isOK, reason, costTime
   │      (measures: did this connection succeed, and why it failed)
   ▼
First frame received
   │
   ├──▶ session "start" report: costTime, path, qualityLevel
   │      (measures: request to first frame, how long startup took)
   ▼
Playing continuously
   │
   ├──▶ periodic "heartbeat": cumulative lagCount / lagDurationMs, p2pDelayMs
   │      (measures: how many rebuffers during playback, total duration, live P2P link latency)
   ▼
Playback end / page close
   │
   └──▶ "stop" report (sendBeacon to guarantee delivery): endReason, final lagCount / lagDurationMs
         (auth info in the body, since sendBeacon can't set custom headers)

9. Wrap-up and next episode (script notes)¶

This episode explained why the end-to-end latency subtraction is error-prone in a browser, and the fix: choose the right round-trip target (ppcenter, not the publisher), then reuse an existing connection for offset calibration — and the publisher and player each independently align to that same ppcenter anchor, rather than the player guessing alone. But latency is just one dimension of playback experience — video load time measures how fast startup is, connection success rate measures whether you can connect, and rebuffer rate measures how stable it is once connected. Together, the four metrics are the complete data behind the words "playback experience".

Next episode, why "average latency" alone isn't enough — latency stability often matters more than the average.