Cluster-Bildung

Copied from the NETZlive docs

Copy of systeme/cluster/cluster-bildung.md from the NETZlive docs site (~/Work/NETZlive/docs), taken 2026-10-09 while that directory was still uncommitted. The NETZlive site is authoritative; update there first. Code paths are prefixed with their repo so the links on this site resolve.

Before any service can be called, the nodes have to be connected. netzlive_cluster does this with libcluster and its own Kubernetes strategy, NL.Cluster.Strategy.Kubernetes (netzlive.cluster/lib/nl/cluster/strategies/kubernetes.ex). It is a fork of libcluster’s Cluster.Strategy.Kubernetes, cut down to the one mode NETZlive uses and extended so that each pod can say what its node name is.

Default topology

No consumer has to configure anything. The defaults in netzlive.cluster/lib/nl/cluster/config.ex:65-78:

kubernetes: [
  strategy: NL.Cluster.Strategy.Kubernetes,
  config: [
    mode: :auto,
    kubernetes_ip_lookup_mode: :pods,
    kubernetes_namespace: "netzlive",
    kubernetes_selector: "netzlive/pod-type=cluster-service",
    polling_interval: 10_000
  ]
]

So: every 10 seconds, ask the API server for all pods in namespace netzlive labelled netzlive/pod-type=cluster-service, and connect to them.

The libcluster supervisor is only started when topologies is non-empty (netzlive.cluster/lib/nl/cluster.ex:41-42). Setting topologies: [] — which the library’s own test config does — turns discovery off; nodes then have to be connected by hand with Node.connect/1.

One polling round

load/1 (netzlive.cluster/lib/nl/cluster/strategies/kubernetes.ex:41-82) runs at start and then every polling_interval:

1. Ask Kubernetes. get_nodes/1 (:132-190) calls GET https://kubernetes.default.svc.cluster.local/api/v1/namespaces/netzlive/pods?labelSelector=… with the pod’s service-account token from /var/run/secrets/kubernetes.io/serviceaccount/token. The CA there is used to verify TLS if present; otherwise verification is off (:98-112). The master host gets CLUSTER_DOMAIN (env, default cluster.local) appended unless it already ends with it or with . (:192-204).
2. Turn pods into node names. parse_response/2 (:208-242) builds nodename@hostname for each pod — see below.
3. Disconnect nodes that were in the previous list and are not in this one.
4. Connect to every node in the new list. Nodes that fail to connect are dropped from the remembered list, so they are retried next round.

How a failed API call is handled depends on the status:

Response	Result	Line
200	parsed pod list	:160-162
403	warning, empty list — every previously connected node is disconnected	:164-167
any other status	warning, previous list kept	:169-171
transport error	error log, previous list kept	:173-175

A 403 disconnects the cluster

An RBAC change that takes list pods away from the service account does not merely stop discovery — the next round disconnects the node from everyone. Other failures keep the last known list.

Node names

Erlang only connects nodes whose name the other side agrees with, so the name each pod computes for its peers must match the name each peer started with. The pod declares how to compute it through annotations; parse_response/2 evaluates them as JSONPath against the pod’s own JSON from the API.

Part	Source	Fallback	Line
host (after @)	annotation netzlive/hostname-naming-jsonpath	status.podIP	:215-222
name (before @)	annotation netzlive/nodename-naming-jsonpath	metadata.name (the pod name)	:224-231

Result: :"#{nodename}@#{hostname}" (:254-256). Only mode: :auto exists.

The pod’s own node name is set separately, in the release’s rel/env.sh.eex, and has to agree with what its annotations produce. The README describes two set-ups:

Single replica, stable DNS name.

metadata:
  annotations:
    netzlive/hostname-naming-jsonpath: "$.spec.containers[?(@.name >= '0')].env[?(@.name == 'EXTERNAL_DNS')].value"
  labels:
    netzlive/pod-type: cluster-service
spec:
  containers:
    - env:
        - name: EXTERNAL_DNS
          value: myname-web.netzlive.svc.cluster.local

export RELEASE_NAME=${HOSTNAME}@${EXTERNAL_DNS}

The JSONPath picks the EXTERNAL_DNS env var out of the pod spec — so peers learn the host name from the same value the pod used for itself.

Several replicas. A shared DNS name would give every replica the same node name. Leave the hostname annotation off — the IP fallback applies — and use the pod IP on both sides:

env:
  - name: POD_IP
    valueFrom:
      fieldRef:
        fieldPath: status.podIP

export RELEASE_NAME=${HOSTNAME}@${POD_IP}

In both, the part before @ is HOSTNAME, which Kubernetes sets to the pod name — the same as the metadata.name fallback. The optional netzlive/nodename-naming-jsonpath annotation exists for deployments that want something else there; whatever it selects, env.sh.eex has to produce the same string.

How each NETZlive application actually sets this up is in the Dienstkarte.

Pods without annotations are skipped silently

The pattern in parse_response/2 requires metadata.annotations to exist and status.podIP to be set (:210-213). A pod with no annotations at all, or one that has no IP yet, falls into the _ -> nil clause (:238) and is dropped without a log line. The second case resolves itself on the next round; the first never does.

Checklist for a new application

1. Depend on {:netzlive_cluster, "~> 2.10", repo: "netzlive"} — the form redispatch uses (redispatch/mix.exs:141, with ~> 2.9).
2. Label the pod netzlive/pod-type: cluster-service.
3. Run it in namespace netzlive, with a service account allowed to list pods there.
4. Make rel/env.sh.eex and the annotations agree on the node name.
5. Share the Erlang cookie with the other applications. The library does not set it; where it comes from in each deployment could not be verified from these repos.
6. Optionally add import_deps: [:netzlive_cluster] to .formatter.exs so defapi and defdelegate_api format without parentheses (shipped since 2.10.1).

Configuration

All options live under config :netzlive_cluster and are validated with NimbleOptions on start (netzlive.cluster/lib/nl/cluster/config/utility.ex:26-37) — an unknown key raises.

Key	Default	When read	Meaning
rpc_timeout	60_000	compile time	Default :erpc timeout per attempt
rpc_retry_after	10_000	compile time	Pause between retry attempts
rpc_retry_limit	210_000 (3 × 70 s)	compile time	Total retry budget
error_reporter	none	compile time	Module with send_error/3, e.g. Appsignal
topologies	Kubernetes, above	runtime	libcluster topologies
registry	[]	runtime	pool_opts for the Phoenix.Tracker, and its name
pubsub_server	NL.Cluster.V3.PubSub	runtime	Must be the same on every node

Defaults: netzlive.cluster/lib/nl/cluster/config.ex:17-83.

“Compile time” means the dependency’s compile time

The three rpc_* defaults are baked into NL.Cluster.RPC’s option schema when netzlive_cluster itself is compiled (netzlive.cluster/lib/nl/cluster/rpc.ex:15-38, which calls Config.get/1 in module attributes). error_reporter is read with Application.compile_env (netzlive.cluster/lib/nl/cluster/error_reporter.ex:22). Changing any of them in your config.exs needs mix deps.compile netzlive_cluster --force to take effect — a plain recompile of your app does not rebuild the dependency.

Two legacy keys are still accepted and translated, with a red deprecation warning printed at compile time: timeout → rpc_timeout, and config: [topologies: …] → topologies (netzlive.cluster/lib/nl/cluster/config/utility.ex:7-24).