Lessons learned and tips #1

• If you have the expertise and many machines, maintaining your own sub Linux distribution in house gives you much more control.

• At large scales, forcing server users to use an API you provide, and not to write on the root FS definitely helps with maintenance.

• File level syncing recovers from any state and is more reliable than most other methods.

• You'll even be able to do crazy things like distribution switches.

• Don't blindingly trust and install upstream updates. They are not all good. They could conflict with your config, or even be trojanned.