The night every green light stayed on

I built an AI fleet that is supposed to run 24x7. Last night it slept for 3h50m while every check stayed green.

The autopsy is more useful than the uptime claim.

At 21:37 my laptop crashed and rebooted. At 21:51 I restarted the orchestrator session. The conversation returned. Its pulse did not.

That pulse is a three-minute heartbeat held inside the session. When the AI finishes a turn and has nothing pending, the next heartbeat wakes it. On August 10, one session died 70 times mid-stream. Of the 22 recoveries, every one came from a heartbeat reaching the idle session. Three minutes is the worst-case lost time when it is armed.

The reboot erased the heartbeat schedule. Resuming the session restored the words and left the schedule empty. Nobody checked.

The evening still looked healthy because I was talking to the system. It produced work at 01:55 and 02:12. At 02:51 the last turn ended. No message was waiting. No heartbeat was armed. Nothing remained that could fire.

From 02:51 until 06:40, the session was dark. Ten-minute API pings were green all night. The laptop stayed awake. The executor lanes sat idle with 93% of their quota free. Every measured component was available. The component that decided what to do next had stopped.

I typed at 06:40. Recovery began. The work was re-verified and shipped by about 07:15.

Three guards, one outcome

The inner guard was the heartbeat. It died with the process and came back empty.

The outer guard was a watchdog that checks whether the session transcript is moving. It had detected stale sessions in a live test on August 9. Its alarm went through a web push subscription that had been dead since August 14. The channel returned HTTP 410. Every alarm went to nobody. The watchdog retained only its latest run, so I cannot reconstruct whether it detected this particular stall.

The final guard was me. I was asleep, as designed.

The only part that recovered cleanly was the executor fleet. Those workers run under a proper operating-system supervisor, so they started after the reboot without help. The brain had no equivalent. It waited for me to type the resume command.

The human is the missing supervisor job.

What I actually bought with “24x7” is 24x7 minus the hours where the last defense was me asleep.

Five things I am keeping

Memory needs an external pulse. A session can retain its conversation while losing the mechanism that advances it. Durable context and live execution are separate checks.

A rule in memory is a wish. “Re-arm after every restart” failed silently. The heartbeat now needs to leave a disk trace, and the watchdog needs to treat a live session without that trace as a fault.

Alarm delivery is part of the alarm. A watchdog with a dead channel is a private opinion. The fallback channel has to be exercised, and channel health needs the same success checks as scheduled jobs.

Keep history. A last-run state file cannot answer what happened overnight. Each guard run needs an appended record, so an autopsy reads evidence instead of reconstructing intent.

Name the substrate. One laptop is the fleet. A crash, sleep, or power cut can stop the operation. That is the current stage.

The heartbeats are armed again. The resume rule now says exactly what failed: resuming a session does not restore its schedules, so the schedule must be checked every time. The executor lanes already have the stronger design. The orchestrator does not yet restart after a machine crash, alarm delivery still needs a live fallback, and guard history still needs to become durable.

The technical record, including the generated diagrams, is at senku.im/alive.

The fleet did not run 24x7. It gave me a clean account of why it could not. That is a better starting point than another green uptime badge.