Google uses percent rollouts within datacenters and within the global network, they’ve always worked fine

• Google uses percent rollouts within datacenters and within the global network, they’ve always worked fine
• Until…
• One system rolled out a new config file that specified a rollout
• But it ended up having a problem that crashed every rollout server that received the file
• The percent rollout didn’t save us and our efficiency in syncing within our network, cost us dearly
• Fix is to also gate rollout of config file changes

Percent rollouts are not the ultimate safety