Ok, so how do you upgrade base packages?

• We effectively had a filesystem image that got synced to a master machine, new rpms were installed and the new image was then snapshotted.

• The new golden image can then be pushed to test machines, pass regression tests, and then pushed to a test colo, eventually with some live traffic.

• When the new image has seen enough testing, it is pushed slowly to the entire fleet.

• Normally, 2 images: the current/old one and the new one.