EP18: Recording & playback — adding VOD to live streaming¶

Recap	EP17 "Deploy from scratch: run your own CDN in 15 minutes"
Next	EP19 "Load-testing methodology: from a single box to global load"

0. Goals of this episode¶

By the end, the viewer should be able to:

Separate the two systems inside recording: ppcenter's "round events" are only audit; what really decides recording segmentation and upload is the recorder node's split-rec API.
Explain the two-phase split protocol: why recording can't "just keep recording" and must be explicitly told "start" and "end" by the business backend.
Walk the full path from live stream to VOD: fMP4 segments → the recorder node uploading directly to object storage → a playback link jointly decided by the object's ACL and the recordPublic switch.
Know the difference between implementation choices and general requirements: a 200 doesn't mean upload succeeded, and failure is silent; disk is a hard constraint; fMP4 and "record only the top layer" are implementation/cost trade-offs, not inevitable rules of a recording system.

1. Opening hook (script notes)¶

Earlier episodes were all about delivering the picture to viewers in real time. But real businesses have a reverse need: keep this session. Live video needs audit retention, auctions need to replay the bidding, online classes need catch-up for students. This episode covers how the system adds VOD to live. First an easily-missed key point: recording isn't as simple as "flip a record switch and keep recording" — because only the business system knows when "a session" starts and ends; the server doesn't know "this game round is over". So what really decides recording boundaries is a two-phase protocol actively triggered by the business backend. This episode unpacks it.

2. First, separate the two independent systems in recording¶

This is the most confusing part during integration; state it up front:

	System A: round events (audit)	System B: split-rec (actual recording)
Provided by	ppcenter	the mmx-recorder node
API	`POST /v1/records/round-events`	`POST /api/record/split`
What it does	Records the fact "a session started/ended" for console queries and audit	Actually splits the media stream and uploads the file on end
Affects recording files?	No	Yes

Both use similar field names (gameId/gameRound/roundId etc.), but the semantics are fully independent. Calling only A produces no recording file; only B truly splits and uploads. This episode focuses on B, because it's what puts "add VOD to live" into practice.

3. Recorder is a third node role¶

The media nodes in earlier episodes were only Origin and Edge. Recording introduces a third role — Recorder.

An architectural evolution worth noting: recording was initially an "embedded capability" inside the Origin process, later changed to a third node role (NODE_ROLE_RECORDER) peer to Origin/Edge. Distinguish two capabilities: registration and monitoring mean the control plane knows a recorder is online, its role and capacity; automatic scheduling requires the control plane to pick nodes by load, assign recording tasks and handle failure reassignment. The former existing doesn't mean the latter is fully implemented.

Different responsibility: Origin cares about "forwarding the stream with low latency", Recorder cares about "landing the stream completely and reliably" — one latency-sensitive, one integrity-sensitive; mixed in one process, they'd contend for resources.
Different resource profile: Recorder eats disk and upload bandwidth, Origin eats forwarding bandwidth and sessions. Separated, each scales on demand, and recording can't fill the disk and drag down the ingest entry (EP15's "see whom adding a machine pushes load to" applies here too).
The control plane can recognize the role: ppcenter can register, heartbeat and account by the recorder role; that provides a monitoring and later-scheduling basis, but whether recording requests are actually auto-assigned still depends on whether the call chain wires in selection, leasing and failure retry — you can't infer it from registration state.

In the PPCDN case, Recorder is configured as record: yes, format fmp4, and registers, heartbeats and reports capacity like other nodes. Node pool and capacity metrics can be used to observe recorder candidates, but whether recording tasks are auto-assigned must be verified separately — don't loosely call it "independent automatic scheduling".

4. Two-phase split protocol: the business backend says "start" and "end"¶

The core recording API is POST /api/record/split, which uses whether gameRound is empty to distinguish the two phases:

Phase	`gameRound`	Effect
Session start (round-start)	empty	"Cuts" each online view's stream for that table, accumulating new segments from here
Session end (round-end)	a concrete value	Verifies the caller is the owner that started the session, cuts again, renames to the final file and triggers asynchronous upload

Why two phases? Because only the business system knows the boundary of "a session". When an auction starts, when a game round ends, is business semantics the media server can't infer. So the protocol has the business backend express the fact explicitly: you say start, you say end, and I split and store the stream between the two calls.

Several design details that get stepped on in practice:

Signing uses the caller's own appId's appSecret, not some deployment-wide shared key. appId itself is part of the signature to prevent tampering into another app. Same security idea as EP06's "short-lived signature + resource binding".
Two auth modes: default simple (md5(appSecret + time + ...), time is the signing instant, tolerance ±30s); stricter advance (HMAC-SHA256 + nonce anti-replay, tolerance ±5min). Use advance for anti-replay.
Silent no-op: calling round-end on a session that never started returns 200, no error — because the end signal is emitted unconditionally and shouldn't alarm just because no start matched.
Conflict semantics are plain: one (appId, tableId) can be held by only one owner at a time; another owner grabbing it gets 500 (with specific text, not 409) — integrate by msg, not the status code alone.
Clear separation of "no publish" and "other failures": if the table has no view online, it returns code=50001, distinct from a real server error, so integrators don't guess from text.
Automatic multi-camera detection: several views under one tableId need no pre-configuration — as long as each view's stream is online, round-start records them all at once. Much less ops than "configure and call per camera".
Rate limiting: up to 500 requests/second per IP.

A practical reminder: don't call "end" immediately after "start". If the two calls are shorter than a segment, no new data has been written yet and end fails with "nothing to finalize". Real sessions are far longer; only integration/load-test scripts tend to hit this.

5. From live stream to VOD: fMP4 segmentation + the node uploads on its own¶

Splitting is only the first step; turning a recording into replayable VOD also goes through a "write to disk → remux → upload → register" pipeline — and apart from the final "register" step, the whole thing runs inside the recorder node's own process; ppcenter only receives an after-the-fact report.

① PPCDN chooses fMP4 (fragmented MP4) for continuous writing. The recorder writes fMP4 continuously, and split-rec carves out independent recording segments at business boundaries. Before upload, that fMP4 segment (moov up front, data scattered across multiple moof/mdat boxes) is first remuxed — not re-encoded — into a conventional single-moov, fast-start MP4. That's so <video src> playback, or double-clicking the downloaded file, can start without waiting for the whole download; if the remux fails, it falls back to uploading the raw fMP4 file as-is, so the segment is never lost over this. fMP4 itself interops well with modern browsers/MSE and CMAF workflows, but it isn't universally better than MPEG-TS: TS is more mature for interruption tolerance and the traditional HLS toolchain, and plain MP4 may suit single-file downloads better. The format should be chosen by player compatibility, failure recovery, container overhead and remux cost — PPCDN's fMP4 choice is an implementation trade-off.

② round-end triggers an asynchronous, best-effort upload — and the upload itself runs inside the recorder node's own process, never touching ppcenter. There's a crucial contract here: an upload failure doesn't affect the 200 already returned by /api/record/split. That is, the business backend getting a success response only means "splitting and registration completed" — not that the upload has finished, or will ever succeed. The recorder node uploads directly using the object storage's own SDK (the AWS S3 SDK — this is the path for OVH net-storage and other S3-compatible services; it falls back to the MinIO SDK when S3 isn't configured), authenticating with access keys from the node's own .env. This is not the looser presigned-URL-PUT style of coupling — the node really is bound to a specific object-storage SDK.

③ Retries happen locally on the node, and failure is silent — ppcenter may never find out. Upload failures are retried by the recorder node itself with exponential backoff (1s → 2s → 4s ... capped at 60s), up to 10 times. Only once an attempt succeeds does the node report the file info (objectKey/playbackUrl/duration/size) to ppcenter (POST /internal/mmx/v1/records/split-rec-files). If all 10 attempts fail, ppcenter is never told — there's no pending/failed status to query; the recording simply never shows up in the §6 recordings list, and the only trace is a single WARN line in the recorder node's own log. If the business backend wants to confirm whether a recording actually uploaded, the only option is to periodically poll /v1/apps/{id}/recordings and check whether the row appears — its absence could mean the upload is still in progress, or that it failed for good after 10 retries; the API can't tell the two apart.

A different recording system in the docs — don't mix them up

ppcenter separately runs a fully independent "continuous recording task" mechanism (POST /v1/records/tasks, keyed by appId + streamPath rather than this episode's gameRound/tableId). That system genuinely does have a ppcenter-side pending → uploading → uploaded / failed segment state machine, plus a "node claims and retries" protocol via /internal/mmx/v1/records/segments/retryable/claim — it serves long-running, continuous recording use cases. It's a separate product line that runs in parallel with this episode's two-phase split-rec and doesn't affect it. It's easy to conflate the two retry models when reading the API docs, so it's worth flagging explicitly here.

④ ppcenter holds no bucket credentials. This is a deliberate permission boundary: only recorder nodes' .env has the object-storage read/write credential; ppcenter doesn't. So ppcenter can't delete objects itself — when a recording expires and needs cleaning, it writes a row into a deletion queue (record_object_deletions), a recorder node claims and deletes, then reports back. Thus the credential that "can delete bucket data" exists only on the node class that most needs it; compromising the control plane can't directly delete recordings. Delete-failure retries are low (5) — repeated failure means a permission/config problem that should be surfaced to a human, not infinitely retried into a silent storage leak.

⑤ File naming and multi-environment isolation. The final object name is <tableId>-<viewName>-<roundId>.mp4 (gameId is only for reporting/audit, not in the file name). If the request explicitly passes appEnv, a <appEnv>/ prefix is added — but whatever is passed, it lands in the same bucket. That's an object-storage constraint: environments are separated only by prefix, not bucket-level permissions. During integration, pass appEnv=test so test files land under a separate prefix and don't mix with production recordings.

6. Playback: a public link the node computes itself¶

Once a recording reaches object storage, playback has its data. But here's an easy place to get this wrong: split-rec's playback link isn't a short-lived, ppcenter-issued credential with an expiry (the txTime/txSecret signing pattern from EP06) — it's a plain public URL that the recorder node itself assembles, the moment the upload succeeds, from its configured domain + bucket name + objectKey (shaped like https://<storage-domain>/<bucket>/<objectKey>, dropping the bucket segment when the domain already embeds it). It carries no signature and no expiry; after a successful upload, the node reports this string to ppcenter verbatim, and ppcenter just stores it.

Whether anyone who gets hold of this link can keep using it forever depends entirely on the object's own ACL: if the upload was configured with S3_ACL=public-read, the link is permanently public; if it relies on the bucket's default ACL and that default is private, the link might actually be unreachable by anyone. This is a completely different security model from EP06's "short-lived signature + resource binding" — no TTL means it won't expire the way a signed URL does, so once it's public there's no time window backing you up.

① Console recordings list: GET /v1/apps/{id}/recordings returns this app's list of recordings (tableId/gameId/gameRound/fileName/playbackUrl/durationSeconds/sizeBytes/creation time). One switch worth noting: "public" (recordPublic) only controls whether this API emits playbackUrl; it doesn't change the file's own access permission in object storage — that's decided by the object/bucket's own ACL, independent of this switch. Turning off recordPublic just stops ppcenter from handing out the link itself; if the object is already public-read, anyone who knows the link can still access it directly. Don't mistake "turned off public" for "the file is inaccessible".

A similarly-named endpoint that isn't part of this system

ppcenter does separately expose a POST /v1/records/rounds/manifest endpoint, which freshly issues per-segment direct links signed with txTime/txSecret and a default 5-minute validity, using the same signing function as EP06. But it serves the "continuous recording task" system mentioned in §5 (querying segments by appId/streamPath/roundId), not this episode's split-rec flow — calling it for a split-rec recording returns no segments at all. When writing integration code or reading the API docs, be careful not to treat the two as the same playback mechanism.

② Storage is billed too: recordings are billed by "stock" (GB·days), a different caliber from traffic. record_storage_daily records a stock snapshot per app per day, retention up to 90 days (MaxRecordRetentionDays). That's why "recording" isn't just a technical feature but a product capability that needs accounting (EP19/EP20 return to cost).

7. Honest boundaries¶

200 ≠ upload success, and failure is silent. Upload is asynchronous and best-effort, with retries happening entirely on the recorder node locally (up to 10 times); a success response only means splitting and registration completed. split-rec has no dedicated "upload status" query endpoint — the only way to confirm whether a recording actually uploaded is to check whether it shows up in /v1/apps/{id}/recordings, and a missing row could mean either "still uploading" or "failed for good after 10 retries" — the API can't tell the two apart.
Disk is a hard constraint. Recorder is a separate node that writes locally first and uploads only on end, so disk capacity directly decides how many streams it can record at once and for how long. Production relies on recordDeleteAfter (fixed-duration cleanup) and recordMinFreeSpace as backstops. There's a real lesson here (a pure disk-capacity estimate, not a measured test, and flagged at the time as "pending review, not verified on real hardware"): a $12-tier machine (~60GB disk, ~52GB usable) configured for mmxNodeCapacity=48 concurrent streams all recording to disk, estimated at 5Mbps/stream, can only sustain roughly 24-30 minutes; working backward, the safe ceiling for a 1/4/24-hour buffer is roughly ~20/~5-6/~1 stream(s) — single-machine recording capacity has to be calibrated by actual measurement, not by treating the admission threshold as a safe value.
PPCDN currently chooses to record only the top layer. This lowers storage, upload and indexing cost but loses multi-layer replay, low-bandwidth adaptation and redundancy when the top layer is missing. Multi- layer isn't "impossible to land at once": you can containerize separately and align timelines, at the cost of more storage, processing and manifest complexity. State "only the top layer (and preferring H.264)" as the current implementation and cost trade-off.
App credentials have a sync delay. The recorder syncs the valid app list (including appSecret) from ppcenter every 30 seconds, so a new app, an arrears recovery, or a changed appSecret takes up to 30 seconds to take effect on the recording side.
Integration tests cover only the record role's recording surface (integrationTest/record: compiling a real mmx binary, real ffmpeg RTMP publish triggering fMP4 writes, real coverage of split-rec and auth branches). And this local test only verifies the "upload doesn't block the response" contract, not real upload success — because there's no writable object storage locally. Real upload must be verified separately against a production storage environment.

8. Diagrams¶

8.1 The full path of one recording¶

  Customer backend
      │  ① round-start (cut, start accumulating)   ┌─ optional: round-events report to ppcenter (audit)
      ├───────────────────────────────────────────►│
      │                                             │
   ┌──┴──────────────────────┐                     │
   │  mmx-recorder node       │                    │
   │  · split per online view │                    │
   │  · keep writing fMP4     │                    │
   └──┬──────────────────────┘                     │
      │  ② round-end (cut again + rename + faststart remux)
      │                                             ▼
      │    file: <tableId>-<viewName>-<roundId>.mp4
      │
      │  ③ node uploads directly via S3/MinIO SDK, up to 10 retries
      ▼      (failure only logged locally — ppcenter never knows)
 ┌───────────────────┐
 │  object storage     │◀───────────────────────────────┐
 │  (S3-compatible)    │                                │ ⑤ accessed directly via playbackUrl
 │  OVH net-storage    │                                │   (plain public link, no signature/
 └─────────┬───────────┘                                │    no TTL — reachability depends on
           │ ④ reports once, only on upload success:    │    the object's ACL)
           │   objectKey + playbackUrl                  │
           ▼                                             │
   ┌───────────────┐    GET /v1/apps/{id}/recordings     │
   │   ppcenter    │────────────────────────────────────┘
   │  stores file   │   (playbackUrl included only when recordPublic=true)
   │  metadata      │
   │  (no bucket    │
   │   credential)  │
   └───────────────┘

8.2 Permission boundary: why deletion "goes around"¶

ppcenter: holds segment/playback metadata, but no object-storage credential
      │
      │ recording expires → write a deletion task (pending)
      ▼
record_object_deletions queue
      │
      │ a recorder node claims it (it has the bucket credential)
      ▼
recorder deletes the object → reports done / failed (→ manual after ≤5 tries)

9. Wrap-up and next episode (script notes)¶

This episode added VOD to live streaming: the business backend uses a two-phase protocol to explicitly mark out sessions, the Recorder node handles writing to disk, remuxing, uploading and retrying all on its own, and then reports a link to ppcenter for the console to display — a link whose public accessibility is decided by the object's ACL. Registration and monitoring aren't the same as automatic scheduling; fMP4 and recording only the top layer are PPCDN's implementation/cost trade-offs; whether the playback link is public depends on the object storage's ACL and the recordPublic switch, not a short-lived signed credential like EP06's. A 200 doesn't mean the upload is done, and upload failure is silent; disk, link visibility and node-side credential governance all need to be designed separately.

That completes Module 4's "bring the environment up" and "add the recording loop". Next episode, the most easily skipped and least skippable step in practice: load testing — how to test from a single box to global, and why passing unit tests doesn't mean surviving production traffic.