File level syncing, really?

• As crude as it is, file level syncing recovers from any state and bypasses package managers and their unexpected errors.

• It makes all your servers the same though, so custom packages and configs need to be outside of the synced area.

• Each server then has a list of custom files (network config, resolv.conf, syslog files, etc...) that are excluded from the sync.

• Rsync for entire machines off a master image doesn't scale well on the server side, and can bog the IO on your clients, causing them to be too slow to serve requests with acceptable latency.

• You also need triggers that restart programs if certain files change.

• So, we wrote custom rsync-like software that basically does file level syncs of all our servers from a master image and allows for shell triggers to be run appropriately.

• IO is throttled so that it does not negatively impact machines serving requests.