First page Back Continue Last page Overview Image
- https://landing.google.com/sre/sre-book/chapters/automation-at-google/#xref_automation_diskerase-sidebar
- We have a few servers in some colos we don’t own for various reasons
- To protect user and our data, we have a diskerase process.
- After trying to re-erase a rack that didn’t seem fully erased, automation determined the new set of machines to erase was empty
- Due to a bug, that empty set was treated as “everything”
- The nice automation complied and did so quicker than humans were able to stop it.
- Moral of the story: double check any destructive code and make sure it gets throttled.
Overachieving engineers and computers