The V shape: narrative direction for Haarlem

Decided 2026-10-10. This is the leading candidate structure for the Haarlem talk (Oct 21, 40 minutes). It replaces the bottom-up order (EPMD first, Raft last). The earlier act outline in talk/talk-outline.md is left in place for reference; rewrite it once the V shape survives rehearsal.

The shape

A V. Go down the stack chasing a symptom, hit the bytes at the bottom, then climb back up giving each layer its proper name, its contract, and what it never promised. The audience meets every layer twice: first as “what is this thing?”, then as the explanation. The first pass creates the itch the second pass scratches.

The spine is one question asked at every layer: “available according to whom?” The answer at every layer is “according to this node’s local opinion”, which lands the reach-versus-agreement thesis without announcing it.

The anchor symptom

“Service not available.” NETZlive’s most common cross-cluster error, and deliberately vague: every layer down gives it a more precise name.

It is the noproc flavor, not the timeout flavor. We do not generally see timeouts on cross-cluster calls, so the busy dist buffer branch is a footnote, and the weight goes on the stale-registry chain:

1. GenServer.call to a registered name exits with noproc
2. the registry lookup returns undefined, because the name was pruned on nodedown
3. nodedown, because net_kernel stopped hearing ticks, or evicted the node to prevent an overlapping partition
4. ticks stopped, because the connection died
5. what even is the connection: the five-packet handshake
6. how was the port found: EPMD
7. the floor: Wireshark, the actual bytes. “It’s just this.”

Descent (~12 min)

Fast and a bit theatrical. One slide per layer: one question, one scary observation, no answers, no demos. Ruthless pacing is the whole trick; if the descent sprawls, the V collapses.

Ascent (~20 min)

Calm and thorough. Each layer gets its name, its contract, and what it deliberately does not do. All demos live here. Climax is the heal-and-kill: two :global coordinators after a split, one untrappable exit(pid, :kill), survivor chosen by pid comparison, nothing merged. “Not available” quietly becomes “available twice, then one of them gets shot.”

Open demo-time question: six candidate demos in twenty minutes is tight. Pick two or three to run live; the rest become pcaps and screenshots.

The ending: the NETZlive erpc wrapper

The ascent ends above where the descent started, at the abstraction the team built so nobody has to think about any of this:

• A macro inside a service that looks like a function delegate. It wraps module creation; the module’s shape (functions, attributes, versioning) is posted to a shared registry when the service starts.
• The registry is backed by Phoenix.Tracker (confirmed), the CRDT under Phoenix.Presence.
• A client calls the registry, gets the shape and the node address, and the client function wraps the :erpc call.
• Exceptions come back as typed error structs (service down, no node) instead of raw exits.

For the HTTP audience this is their world rebuilt on the BEAM: a service contract, a discovery registry, versioned interfaces, gRPC-shaped, except the transport is a SPAWN_REQUEST control message and the proto file is a macro.

And it closes the V honestly. Tracker is eventually consistent gossip, so during a split or a deploy each node’s view of who offers which service drifts for a few seconds; “service down” is literally one node’s stale opinion of the registry, wrapped in a struct. The closing contrast: :global resolves a split by killing a process; Tracker resolves it by merging two CRDTs. The wrapper did not buy agreement. It bought a vocabulary for the lack of it.

Alternatives considered and parked

1. Post-mortem. One real NETZlive incident walked as an incident review, each layer the next “why”. Strong narrative pull; wobbles if incident details are fuzzy.
2. Live IEx debugging session. Same descent, driven by tools: whereis_name, Node.list, monitor_nodes with reason, epmd -names, Wireshark. Teaches a reusable method; most fragile on stage, needs pcap and transcript fallbacks. Fold its tooling into the ascent.
3. The unveiling. Start from the one-env-var magic, ask “what actually happened” downward, failure afterwards. Wonder before fear; thesis arrives too late, around minute 25.