docs: one page owns the boot recovery net

The net is built, so its description belongs in `design/` rather than in a dated
`project/` record: `design/` is how it works and why, and one page owns a mechanism
while the others link to it.

`boot-recovery-net.md` owns the deadline and its predicate, the membership rule, the
rescue, the `golden` symlink and the loop guard — including the reasoning that is
invisible in the code it describes:

  - why a deadline and not `OnFailure=`. All four members use `Restart=always` for
    reasons argued in their own unit files, and none overrides the default start
    limit, so a `btd` restarting every 5s crash-loops forever without ever entering
    `failed`. `OnFailure=` would only fire if those units were retuned to give up.
  - why the predicate is two conditions of positive evidence rather than "is it
    active". A false negative costs a diagnosis, a false positive costs a good
    release, so a stopped unit, an unloaded one and a slow one are all left alone.
  - why the rescue parses nothing, why `golden` is a symlink `updaterd` republishes
    on every start, and why a configured golden that is not installed publishes no
    link at all.
  - why clearing the breadcrumb precedes the boot counter: the trial would revert to
    `previous` while the rescue already went to golden, so an armed trial moves
    `current` back off golden a boot later.
  - what is not tested, which is systemd itself.

Two owners of overlapping mechanisms now point here instead of describing it.
`updater-design.md` carried a paragraph specifying the `OnFailure=` design that was
never built, and its on-disk layout gains the `golden` symlink. `restart-order.md`
owns the startup sequence, so `record_rescue` and `refresh_golden_links` are listed
there as the steps they are, with the reasons kept on this page — and §4 now says
that everything it describes presupposes an `updaterd` that starts, which is the gap
the net exists for.

Assisted-by: Claude:claude-opus-5

btd/src/bluez.rs

@@ -101,7 +101,7 @@ const ADAPTER_RETRY: Duration = Duration::from_secs(5);
 /// So the whole bring-up retries in place, on the same 5s cadence as the wait it already did:
 ///
 /// - **radio faults never leave this function.** A `failed` `btd` therefore means a broken binary,
-///   which is what admits it to the boot recovery net — see `docs/project/boot-recovery-net.md`;
+///   which is what admits it to the boot recovery net — see `docs/design/boot-recovery-net.md`;
 /// - **and it self-heals.** Exiting non-zero got the same retry from `Restart=always`, but only by
 ///   spending a process death on it, and only until the day the unit gains a start limit.
 ///

configd/src/main.rs

@@ -186,7 +186,7 @@ async fn main() -> ExitCode {
     //
     // It is also what admits `configd` to the boot recovery net: a unit may join the set only if it
     // waits for its dependency rather than exiting, so that a `failed` unit means a broken release
-    // and not a broken board (`docs/project/boot-recovery-net.md`).
+    // and not a broken board (`docs/design/boot-recovery-net.md`).
     let net: Arc<dyn Net> = match backend(args.fake_net).await {
         Ok(net) => net,
         Err(e) => {

configd/src/net.rs

@@ -34,7 +34,7 @@ pub trait Net: Send + Sync {
 ///
 /// It is also what makes `configd` eligible for the boot recovery net: a unit may only join the set
 /// if it waits for its dependency rather than exiting, so that a `failed` unit means a broken
-/// release rather than a broken board (`docs/project/boot-recovery-net.md`).
+/// release rather than a broken board (`docs/design/boot-recovery-net.md`).
 ///
 /// `reason` is the bus error, carried into every reply rather than logged once at startup, because
 /// the person who needs it is holding a phone and cannot see the journal.

docs/README.md

@@ -31,6 +31,7 @@ own the mechanism is the bug.
 | [`updater-design.md`](design/updater-design.md) | The update engine: verification, atomic swap, health gate, rollback, release format. |
 | [`restart-order.md`](design/restart-order.md) | Which unit restarts, at which step, on every path that moves `current` — and at boot. |
 | [`app-path-design.md`](design/app-path-design.md) | `btd` and `configd` — how a phone configures a robot over BLE. |
+| [`boot-recovery-net.md`](design/boot-recovery-net.md) | Falling back to golden when the release that booted cannot start its daemons. |
 
 ## `project/` — you are running the project
 

docs/design/boot-recovery-net.md

@@ -0,0 +1,226 @@
+# The boot recovery net
+
+What puts a robot back on a known-good release when the one it booted cannot start its daemons.
+
+This page owns the boot deadline and its predicate, the membership rule for the units it watches,
+`robot-rescue`, the `golden` symlink, and the loop guard. [`restart-order.md`](restart-order.md) owns
+the sequence this sits inside — including where `updaterd` reads the breadcrumb at startup — and
+[`updater-design.md`](updater-design.md) owns the store, the health gate and the boot counter.
+
+## The gap it closes
+
+`updaterd` never restarts itself during an update, because it is the process performing it. Three
+layers narrow what that costs, and all three run on a board that is fully up:
+
+- `self_test_updaterd` runs the release's own `updaterd --self-test` before the commit, so a
+  wrong-architecture binary, a missing library or a rejected `updater.toml` fails the update while
+  the old release is still live;
+- the deferred restarts put `updaterd` and `btd` on the new binaries five seconds after the update
+  replies;
+- the startup reconciliation checks, from the successor's side, that those restarts happened.
+
+A binary that starts on a running board can still fail at **boot**, where the network is not up, a
+peripheral is not enumerated, and unit ordering differs. And the reconciliation runs *at `updaterd`
+startup*, so it presupposes an `updaterd` that starts.
+
+That last point is the whole reason this exists. `BootCounter::record_boot` runs from
+`Engine::recover_on_start`, inside `updaterd`, and counts `updaterd` **starts** rather than literal
+boots — so an `updaterd` that does not start does not count, and the never-brick guarantee closes
+through the one process that might be the casualty. Left there, a board has a good release on disk, a
+trial armed to revert to it, and nothing running that can act on either.
+
+The boot counter handles a *different* failure: a release that boots and runs but never reports
+healthy. This one handles a release whose daemons do not come up at all.
+
+## The parts
+
+| | |
+|---|---|
+| `robot-boot-check.timer` | fires 180 s into every boot |
+| `robot-boot-check.service` | the oneshot: asks whether the release came up, hands over if not |
+| `scripts/robot-boot-check` | the predicate. Installed at `/usr/local/sbin/robot-boot-check` |
+| `scripts/robot-rescue` | swaps `current` to golden, records it, reboots. `/usr/local/sbin/robot-rescue` |
+| `<install_dir>/golden` | symlink to the release with a standing guarantee, published by `updaterd` |
+| `<state_dir>/rescued` | breadcrumb: the last attempt, and the loop guard |
+
+Both scripts are `/bin/sh` and both are **copied** into `/usr/local/sbin` by `scripts/install.sh` and
+`hooks/postinstall` — the same decision, for the same reason, as the unit files those two copy rather
+than reading through `current`. A rescue that shipped as a binary in the release payload would be
+broken by exactly the releases it exists to survive: the wrong architecture, a missing shared
+library, a panic on startup. A rescue read through `current` would route the recovery through the
+thing being recovered.
+
+## The trigger: a deadline, not `OnFailure=`
+
+`OnFailure=` looks like the natural fit — systemd fires it exactly when a unit exhausts its restarts
+and enters `failed`, with no polling and no resident watchdog. It does not fit *these* units, which
+are configured not to reach that state:
+
+| unit | policy | `RestartSec` |
+|---|---|---|
+| `btd` | `Restart=always` | 5 s |
+| `padd` | `Restart=always` | 5 s |
+| `configd` | `Restart=always` | 2 s |
+| `robotd` | `Restart=always` | 2 s |
+| `updaterd` | `Restart=on-failure` | — |
+
+`Restart=always` is deliberate in each case and argued in the unit file. No unit overrides
+`StartLimitIntervalSec`/`StartLimitBurst`, so systemd's default of five starts in ten seconds applies
+— and a `btd` restarting every 5 s never comes near it. It crash-loops indefinitely without ever
+entering `failed`, which is the state `OnFailure=` waits for. Making `OnFailure=` fire would mean
+retuning those units so the daemons *give up*: degrading the normal case to enable the rescue case.
+
+A deadline inverts the question, needs no unit changes, and catches the forever-restarter.
+`OnBootSec=180` because the board is slower than it looks — `hci0` does not exist until roughly 73
+seconds after power-on, of which `bluetooth.service` spends 26 blocked behind `dbus` — and the daemons
+wait for their hardware rather than exiting, so a slow boot is not a failed one.
+
+### The predicate
+
+Two conditions, both positive evidence that systemd tried to run a daemon and could not keep it
+running. Either one, on any member:
+
+- `ActiveState` is `failed`;
+- `NRestarts` is 3 or more.
+
+Everything else is left alone, and the asymmetry is the point. A unit an operator stopped is
+`inactive` with no restarts, and must not cost the board its release — the same call the startup
+reconciliation makes when it leaves a stopped unit stopped. A unit the release predates is not loaded
+and answers nothing, which reads the same way. A daemon still waiting for hardware is `active`. And
+one crash that recovered leaves one or two restarts, not three. **A false negative costs an operator a
+diagnosis; a false positive costs a good release.**
+
+Three restarts rather than one because `Restart=always` with a 2–5 s `RestartSec` means a genuine loop
+passes three well inside the deadline, while a daemon that died once on a transient does not.
+
+### Three ways it refuses to fire
+
+- **`Conflicts=shutdown.target`** on the oneshot. A daemon killed on the way down can be recorded as
+  `failed`; without this, a `systemctl poweroff` at the wrong moment eventually reboots a robot into
+  golden.
+- **An uptime guard.** Past ten minutes, something other than the timer started it, and the answer is
+  that the question is not this script's business.
+- **No `[Install]` section** on the oneshot, which both installers read as "do not enable this".
+  Without it, `hooks/postinstall` running `enable --now` over the units it just wrote would run a
+  rollback check in the middle of the update that installed it — with daemons legitimately
+  mid-restart, which is exactly what a rollback check reads as a release that cannot start. Timers are
+  enabled but never started for the same reason: an `OnBootSec=` timer started past its deadline fires
+  immediately.
+
+## The membership rule
+
+The objection to "any daemon fails → roll back" is that a daemon may fail for reasons a rollback
+cannot fix — a missing radio, an unpowered motor bus — and reverting a good release over a hardware
+fault is worse than doing nothing.
+
+That resolves into a rule for admitting a unit to the set: **a unit may join only if it waits for its
+hardware, absent or misbehaving, rather than exiting.** Then a member that is down means the *binary*
+is broken — wrong architecture, missing library, panic on startup, a config it rejects — which is
+what a rollback fixes.
+
+| daemon | in the set | when its dependency is absent |
+|---|---|---|
+| `robotd` | yes | waits for the motor bus, "waiting, not commanding anything" |
+| `updaterd` | yes | serves regardless; it is the recovery path and must be reachable when things are broken |
+| `btd` | yes | retries for an adapter every 5 s, and re-runs the whole bring-up if one appears and then stops answering |
+| `configd` | yes | serves `system.*` and `pad.*`, and reports `net.state=unavailable` |
+| `padd` | no | exits cleanly with no robotd socket, by design |
+
+`btd` earns its place on merit rather than symmetry: with no network, BLE is the only way in, so a
+robot that boots with no `btd` and no known wifi cannot be reached at all. `configd` earns its place
+the same way — `system.pin` is where `btd` gets the PIN a phone authenticates with, so a `configd`
+that is down takes the phone path with it.
+
+If a future daemon exits on missing hardware, either fix the daemon or leave it out of the set. Do not
+weaken the rule.
+
+## The rescue
+
+`robot-rescue` reads two symlinks and parses nothing.
+
+That is not economy. The likeliest thing this rescues is a release whose `updaterd` rejects the
+board's `updater.toml` — the operator's file, preserved across installs, while a release is free to
+change what it expects. A rescue that parses that same file dies of the disease it is treating.
+
+Golden is a version *in* that file (`ComponentConfig::golden`), so `Engine::refresh_golden_links`
+publishes it as a `golden` symlink beside `current` on every `updaterd` start: one `readlink`, nothing
+to reject. A cache rather than a second source of truth, and it fails safe — if `updaterd` stops
+starting, the link still names the release that was golden when it was last written, which is a
+release that was known good. A configured golden that is *not installed* publishes no link at all,
+because a dangling one would tell the rescue it has a target when it does not.
+
+**Golden, not previous:** when what broke is the recovery path itself, previous may be broken too.
+Golden is the one release with a standing guarantee, and `prune` never removes it.
+
+It declines, saying which it is, when there is no golden link, when golden names a release that is not
+installed, and when `current` is already golden. That last one matters most: the daemons are then
+failing on the release carrying the standing guarantee, so this is not a release fault, and swapping
+would change nothing except adding a reboot — which is how a hardware fault becomes a reboot loop.
+
+The swap renames a temporary symlink over `current`, matching `Store::link_to`, because `ln -sfn`
+unlinks and recreates and leaves a window where a daemon cannot exec. `rename(2)` does not follow
+symlinks but `mv`'s path resolution does, so a plain `mv tmp current` moves the link *into*
+`releases/<version>` and leaves `current` untouched; GNU spells the fix `-T` and BSD spells it `-h`.
+
+`--reboot` is opt-in. Every unit execs through `current`, so the swap does nothing until they restart,
+and a robot that is standing when its daemons die is a robot that falls — whoever is holding it
+decides. The boot check passes the flag; a human at a console gets the command printed instead.
+
+## The loop guard, and what opens it
+
+Two independent things stop "swap and reboot" happening twice on its own:
+
+1. the refusal when `current` is already golden, which covers the ordinary case, because a successful
+   rescue leaves exactly that state;
+2. the breadcrumb, which covers the case the first misses — an update moved `current` away from golden
+   and failed again.
+
+`Engine::record_rescue` clears the breadcrumb on the next `updaterd` start, after copying it into the
+update log. So the guard opens exactly when the board proves it can run its update daemon again, and
+stays shut when it cannot: a robot sitting still with a clear journal beats one rebooting forever.
+`robot-rescue --force` is the way past it by hand.
+
+That clearing runs **before** the boot counter, and the ordering is load-bearing. An armed trial would
+revert to `previous`; the rescue already went further, to golden. Left armed, `record_boot` would
+advance the trial and eventually move `current` *off* the release the rescue chose — the recovery net
+and the never-brick guarantee undoing each other, one boot apart.
+
+The breadcrumb is `key=value` rather than JSON, though `updaterd` parses it:
+
+```text
+at=1786453421
+install_dir=/opt/robot/daemon
+from=1.2.0
+to=1.0.0
+because=boot check: robotd.service (failed, 7 restarts)
+```
+
+The writer is a shell script running on a board where things are already going wrong, and quoting
+JSON correctly in `sh` is a way to produce a record nothing can read. `journal::Breadcrumb` is lenient
+for the same reason — every field optional — because a breadcrumb that cannot be parsed is a guard
+that never opens. `install_dir` is how the entry is attributed to a component without a second place
+that has to agree with `updater.toml` about the name `daemon`.
+
+## Evidence
+
+A silent rollback is how a day goes to "it works on my board", so the swap leaves three traces:
+
+- **the journal**, from both scripts, via `logger` when it is present and stderr always;
+- **the update log**, as `Outcome::RolledBack` with the failing unit named in its reason — which is
+  what `robotctl update log` shows, and it is permanent, unlike the breadcrumb that is cleared;
+- **`robotctl version`**, indirectly but usefully: a board running an older release than the one
+  installed is exactly what it reports on.
+
+## What is not tested
+
+The decisions are: `xtask/tests/rescue.rs` asks both scripts every question against a temporary tree,
+with a stub `systemctl` supplying unit states and recording whether a reboot was asked for, and
+`scripts/board-test.sh` covers the installer half on real ARM64 Linux.
+
+**systemd is not.** Whether the timer fires at `OnBootSec=180`, whether `Conflicts=shutdown.target`
+keeps the oneshot off the way down, and whether `NRestarts` reads the way the predicate assumes on a
+crash-looping unit are all unverified. That needs a privileged container with systemd as pid 1 —
+[`install-path-gap.md`](../project/install-path-gap.md) option C — and this is the second argument for
+building it: the recovery net is the code that must work when everything else does not.
+
+None of it has run on hardware, where the interesting path needs a board whose release cannot start.

docs/design/restart-order.md

@@ -219,6 +219,13 @@ adapter rather than failing.
 5. `Engine::recover_on_start`, **before the socket is served**, so a robot that booted into a bad
    release has already begun reverting by the time anything can ask it to do something else:
    - `clean_staging` for every component: delete `releases/.staging-*`.
+   - `record_rescue`: if `<state_dir>/rescued` exists, the boot recovery net swapped `current` to
+     golden while nothing was running. Copy it into the update log as a rollback, **clear the trial
+     for that component**, and delete the file. Before `record_boot` on purpose: the rescue already
+     went further than the trial would have, and a trial left armed would move `current` back off
+     golden a boot or two later. Deleting the file is also what releases the rescue's loop guard.
+   - `refresh_golden_links`: publish each component's configured golden as a `golden` symlink beside
+     `current`, so the rescue can find it with one `readlink` and no parser.
    - `record_boot`: increment `boots` on every armed trial in `pending.json`.
    - For each trial with `boots >= 2` (`MAX_BOOT_ATTEMPTS`): revert to `previous`, escalating to
      `golden` when `previous` is absent, missing from disk, or itself recorded as rolled back. That
@@ -243,6 +250,13 @@ A trial only exists if an apply was interrupted between step 9 and step 15 of §
 `exhausted` is `boots >= 2` while `arm` writes `boots = 0`, the *first* `updaterd` start after the
 interruption logs "update still on trial" and the *second* reverts.
 
+**Every step above presupposes an `updaterd` that starts.** One more thing happens at boot for the
+case where none of it runs: `robot-boot-check.timer` fires 180 s in, asks whether `robotd`, `configd`,
+`btd` and `updaterd` came up, and hands over to a `/bin/sh` rescue that swaps `current` to golden and
+reboots if they did not. It is deliberately outside every process on this page —
+[`boot-recovery-net.md`](boot-recovery-net.md) owns it, including why it is a deadline rather than
+`OnFailure=` on these units.
+
 ## 5. The startup reconciliation — did the restarts happen?
 
 `systemd-run` returning success means a transient timer was created. It does not mean the restart ran,

docs/design/updater-design.md

@@ -163,16 +163,14 @@ its own PR.
    engines in the same process, which surfaced as unrelated operations failing with `Busy` in the
    test suite.
 
-2. **A boot-time net outside `updaterd`**, for what slips through. `OnFailure=` on a curated set of
-   units fires exactly when one exhausts its restarts, and a tiny oneshot swaps `current` to golden and
-   reboots. Four constraints make it a net rather than a footgun: the set is curated (`btd`
-   legitimately fails on a board whose radio has not appeared, and reverting a good release over that
-   is a poor trade); it reads a `golden` **symlink** rather than parsing config, because a release that
-   breaks the config parser must not also break its own rescue; it needs a loop guard and must not fire
-   when `current` is already golden; and it cannot fix hardware — a `robotd` that fails for want of
-   servo power fails identically on golden, the same distinction the health gate draws between
-   unhealthy and degraded. Golden rather than previous, deliberately: when the recovery path itself is
-   what broke, previous may be broken too.
+2. **A boot-time net outside `updaterd`**, for what slips through: a timer three minutes into each
+   boot asks whether the release brought its daemons up, and a `/bin/sh` rescue in the installed base
+   swaps `current` to golden if it did not. It has to live outside this process and outside the
+   release, because the failure it exists for is an `updaterd` that does not start — at which point
+   `recover_on_start` and the boot counter below are both unreachable. It cannot fix hardware: a
+   `robotd` that fails for want of servo power fails identically on golden, the same distinction the
+   health gate draws between unhealthy and degraded. [`boot-recovery-net.md`](boot-recovery-net.md)
+   owns it.
 
 This is also why the update logic cannot live in `btd`: as a client of the
 update, `btd` cannot be the thing performing it — it would kill itself partway
@@ -548,12 +546,17 @@ Any non-zero hook exit, failed health probe, or timeout is treated identically:
 /opt/robot/daemon/
 ├── releases/1.4.1/     ← previous (kept for rollback)
 ├── releases/1.4.2/     ← new
-└── current → releases/1.4.2     ← systemd units point at current/
+├── current → releases/1.4.2     ← systemd units point at current/
+└── golden  → releases/1.4.1     ← never pruned; what the rescue reads
 ```
 
 Atomicity is a single `rename(2)` of the symlink on the same filesystem. No
 half-written state is ever live.
 
+`golden` is written by `Engine::refresh_golden_links` on every start, from
+`ComponentConfig::golden`. It exists so that recovery outside this process needs no
+parser — see [`boot-recovery-net.md`](boot-recovery-net.md).
+
 ### 7.2 Preflight preconditions
 
 Checked before any download or change; any failure aborts cleanly with no

scripts/robot-boot-check

@@ -2,7 +2,7 @@
 # Did the release that just booted actually come up? If not, hand over to `robot-rescue`.
 #
 # Run once per boot by `robot-boot-check.timer`, three minutes in. See
-# `docs/project/boot-recovery-net.md` for why this exists at all and why it is a deadline rather
+# `docs/design/boot-recovery-net.md` for why this exists at all and why it is a deadline rather
 # than `OnFailure=` on each daemon.
 #
 # WHAT COUNTS AS "DID NOT COME UP". Two conditions, both positive evidence that systemd tried to run
@@ -55,7 +55,7 @@ while [ $# -gt 0 ]; do
 usage: robot-boot-check [--dry-run]
 
 Decide whether the booted release brought its daemons up, and hand over to robot-rescue if not.
-Run once per boot by robot-boot-check.timer. See docs/project/boot-recovery-net.md.
+Run once per boot by robot-boot-check.timer. See docs/design/boot-recovery-net.md.
 
   --dry-run   report the verdict, and do not invoke robot-rescue
 

scripts/robot-rescue

@@ -65,7 +65,7 @@ while [ $# -gt 0 ]; do
 usage: robot-rescue [--dry-run] [--reboot] [--force] [--because <text>]
 
 Point /opt/robot/daemon/current at the golden release, without going through updaterd.
-For a board whose daemons will not start: see docs/project/boot-recovery-net.md.
+For a board whose daemons will not start: see docs/design/boot-recovery-net.md.
 
   --dry-run        report the decision and change nothing
   --reboot         reboot after the swap, instead of printing the command

updater/systemd/robot-boot-check.service

@@ -3,7 +3,7 @@
 # The last uncovered failure in the update path. `updaterd` cannot watch its own replacement start,
 # and its boot counter only advances when `updaterd` itself starts — so a release whose daemons do
 # not come up at boot leaves a board with a good release on disk, a trial armed to revert to it, and
-# nothing running that can act on either. See `docs/project/boot-recovery-net.md`.
+# nothing running that can act on either. See `docs/design/boot-recovery-net.md`.
 #
 # Started by `robot-boot-check.timer` and by nothing else. **There is deliberately no `[Install]`
 # section**: `scripts/install.sh` and `hooks/postinstall` `enable --now` every unit with one, and