We have a week per year when internal systems get to practice live failure and recovery scenarios (within reason)

• We have a week per year when internal systems get to practice live failure and recovery scenarios (within reason)
• This includes having the main office swallowed by a major earthquake, and recovery having to be done by remote offices only
• These exercises often find documentation shortfalls and dependencies/compound failure scenarios that we didn’t know about.
• They need to be done carefully though, you don’t want to cause more damage than the potential for learning. This includes an abort/revert plan.
• That’s an occasion to define and practice your revert scenario (any rollout needs to have a revert plan. Run forward only should avoided).

Practice Emergencies when everyone is awake and ready