01 Real-time platform · 2026
Turning a disaster-recovery drill into a race
A live leaderboard, team pit lanes, and surprise chaos events. Turning the lessons of a real disaster-recovery exercise into software.
Built from a real failover rehearsalExplore the build
The problem
SAP system teams are required to rehearse failover. In practice the rehearsal is a conference bridge, a spreadsheet, and four hours of somebody reading steps aloud. When I co-led a real failover-and-failback exercise for a utility client, the structured lessons-learned I wrote afterwards had five themes — and one of them was simply that nobody could see live status.
How it works
An organiser spins up a session in about a minute: name it, set a passphrase they keep, define up to fifteen teams. Out comes a shareable admin link, one read-only observer code, and a six-character join code per team. Participants work their failover issues from their own Pit Lane — resolve, bypass, or log new ones — while a live race view shows every team's position updating over websockets. The organiser broadcasts announcements, fires surprise chaos events, and at the end downloads a full post-mortem workbook.
The constraint that shaped everything: no shared credentials. Any team should be able to run their own drill without an administrator provisioning anything. Each session issues its own scoped token, so an organiser can manage their drill and nothing else, and several drills can run at once without colliding — which is why observers get a per-session code rather than one global password.
That decision cascaded. Sessions need a three-state lifecycle so a drill can sit in a lobby before it starts and freeze cleanly when it ends. Scoring has to be computed server-side, because a client that can score itself can cheat. And the post-mortem has to be generated from the event log rather than the final state, or you lose the timing data that made the exercise worth running.
