It was a Tuesday, the kind of unremarkable day that precedes most production fires. The team, sharp and capable, had built a new distributed job scheduling service. To handle failover, they implemented what seemed like a clever, simple leader electio...