Maintaining lots of apt-get capable linux machines with apt and getupdates

Marc MERLIN

marc_soft@merlins.org

OLS 2004

http://marc.merlins.org/linux/talks/getupdates/

What you don't want

Each sysadmin installed a server from some install CD. (If you're lucky, it might be the same CD, but if not, it might be different releases, or even worse, different distributions)

Each server has local, unknown, customizations from each sysadmin

You have 300 workstations with an old version of some distro, and you have to upgrade sshd, or convert them to autofs with ldap

Install server? What install server?

Challenges

You mean I have to maintain workstations where users have root and can install and remove random software?

Why maintain servers and workstations separately?

But how do I keep all this in sync when all the machines don't even have the same list of software installed?

How do I even update software? How about conf files?

Do I need to change the install each time I make changes to the installed base?

How about machines that were loaded before I changed the install but after I synced the change to the installed base?

Non working solutions I've seen

All workstations rsynced from central image

very heavy on the rsync server(s)

nightmare to maintain rsync includes/excludes

unsafe to upgrade libc

hard to maintain local software

chroot live image with ssh/scp pushes

much better (install & clients in sync)

support for local software

disaster if some clients can be down since they miss updates

nice install server with no updates

No comments...

nice install server with push updates

what about clients that are down during updates?

install out of sync with pushed updates

Requirement for a good solution

Should work for several class of clients (servers, light servers, workstations...)

clients should always end up being up to date (replay updates in order if you were down)

if an update fails, it will retry later

an update failure should not put a client into an unknown state it may not be able to recover from

The install should be updated at the same time than the installed base, and there should be no lapse of time for new installs

Some downgrade/rollback capability

Getupdates bonuses

Initial install can be on CD/DVD, nfs/ftp/http, or made with an out of date disk copy

clients are up to date as soon as they reboot after the install

lead and gold (i.e. unstable vs stable)

you know about all your clients:

cpu/memory/disk/hardware information

linux kernel version

list of packages installed

all users logged in the machine present and past / machine “owner”

convert between hostname, IP, and MAC address (useful when the client is down)

Implementation: Basics

apt-get takes care of installing packages and their dependencies (computed automatically) via nfs/http/ftp

changesets are a comprized of a shell script and a compressed tar archive containing optional files used by the update (just two files to retreive, which is easier for http/ftp)

getupdates is the core of the install and update mechanism and pulls changesets via nfs/http/ftp.

getupdates runs at install time and from cron, checks the update server for a list of changesets for the client's $TARGET, and retrieves the update if it's new, or if it's been modified since the last time it was run.

Implementation: Variables

/etc/sysconfig/getupdates:

TARGET=wkslead

NFSHOST=software.nfs

NFSSWPATH=/software -> $NFSHOST/$NFSSWPATH (boot media) or $NFSSWPATH (autofs)

PROTOCOL=nfs

WORKDIR=/var/lib/grhat

STATBASE=/auto/clientinfo -> runtime (autofs)

NFSSTATMNT=clientinfo.nfs:/clientinfo -> load time from boot media

$TARGET determines lead vs gold and wks vs srv

clientinfo is NFS only and for stats and logs

$PROTOCOL is currently http or nfs, and can be ftp easily

Implementation: Layout

File layout in the install tree:

Implementation: Postinstall

You need to bootstrap getupdates via HTTP/FTP/NFS

We read config options from /proc/cmdline: options are passed to install kernel (via syslinux / other boot loader)

Some getupdates logic in postinstall to get load options, use them to retreive getupdates and its tools, install them on disk, and save boot options in /etc/sysconfig/getupdates

Distribution specific, but needs to be run in a chroot after the distro's install has completed (%post in a Red Hat kickstart file)

Install getupdates ASAP so that it can continue the install if machine crashes or gets unplugged during postinstall

Implementation: Postinstall Code

Implementation: getupdates funcs

All changesets have to include funcs to get wrappers and install functions:

reads $WORKDIR / $LEADGOLD / $MACHTYPE (srv vs wks)

installlink: move a file out and make a symlink (like resolv.conf)

installfile: complex function that provides 8 ways to install a file

chattr: only run if argument is file, and skip symlinks

removefiles: delete file only if it's there, and after chattr -i

rotatefile: move a file out before a new one is installed (used by changesets and installfile, before replacing a stock file)

rpm/dpkg wrapper that removes expected stderr output

apt-get wrapper that defaults to -q –trivial-only for safety

apt-get-force when you really need it (apt-get -q -y)

Implementation: installfile

We need to take care of many cases:

if file is mutable, rotate out, install new file as immutable, and record operation in WORKDIR/state/changedfiles

optionally, install without rotation (overwrite), like in /etc/cron.daily

if file is mutable but in changedfiles, it's been user modified, skip

if file is immutable, upgrade getupdates maintained file in place

or force rotation anyway

or install a file in place and leave mutable by specifying the md5sum of the intended target (which leaves it alone if the user modified it)

Implementation: Changeset Rules

There are many rules for writing changesets:

changesets are numbered in the order you need them to run

changesets can be run multiple times and have to be written accordingly

do not generate any stderr output unless it's a warning or error

changesets run under bash's errexit. All lines of code have to return true

you cannot change the current directory, but you can use pushd/popd

you must not use apt-get install -y foo to get its dependencies: list all the dependencies and remove conflicting packages first. This safeguards you against some package removing other ones you didn't expect

remember, this all runs unattended, expect the unexpected and be very careful

Implementation: Changeset Examples

Implementation: Changeset Push

copy changeset code to targets/wkslead/xxxxx/runme

note that new installs could break at that time, test on lead, not on gold

copy new files used by changeset in xxxx/files/....

use makefile to rebuild workfiles.tar.bz2

go on a client and remove $WORKDIR/state/xxxx

run getupdates to check changeset on a client

change updatedate on the server for other clients to pick up the new changeset

Implementation: apt trees

apt tree setup obviously distro specific

apt-rpm setup example available with getupdates distro

recommended setup:

aptroot/targets/lead

aptroot/targets/gold

aptroot/targets/links

makefile to sync from main apt tree to lead and then gold

optional extras shadowed out with pinning

Stats and logs: clientinfo

Keep track of how many machines you have, what they run and whether updates are failing

/auto/clientinfo: hostname -> MAC and IP -> MAC

/auto/clientinfo/MAC (mutiple versions in ontap snapshots)

hardware config info: cpuinfo, df, lspci

system info: ps-auxww, free, uname-a

machine info in symlinks: hostname, ip, uuid (RHN)

getupdates run info: log/log.last/log.update

machine load info: target (wkslead vs srvgold...), rpm-qa

user info: owner, who

pushing: runcmd

Uses ssh key root logins to send commands to all machines

Imperfect: any client can be up or down at any time

Uses:

Mostly recovering from complete getupdates failure

Dirty fix on “most” machines

trigger a pull on most machines (runcmd getupdates)

quick info poll without writing a changeset

Thank god for symlinks

/etc/resolv.conf -> per office subdomains for service names

/etc/crontab -> crontab.srv | wks

/etc/auto.net

/etc/syslog.conf

/etc/nsswitch conf (with/without ldap/nis)

/etc/pam.d/system-auth (with/without kerberos/ldap)

/etc/X11/XF86Config

etc..

What's missing / TODO

Automated, fully failsafe, rollbacks

quite hard to implement without snapshots in the filesystem (lvm2 ?)

Add checksums/gpg signatures

not too hard to do, but mostly useless if you don't make sure all your contrib rpms/debs are also signed -> lots of work

crond watcher and restarter

Need to get around to that one day :)

clientinfo reporting over HTTP and not just NFS

would be a little work and require a specialized CGI with http upload

backup getupdates when you sync a bad getupdates

critical missing piece if you update getupdates and make a fatal mistake

watch /var/log/getupdates and retrieve and run shell script from a central server if the last update is too old (try 4 bytes of the IP, and then 3, 2, 1, and just failsafe)

Switch to YUM instead of apt-get on Red Hat?

Are you a good sysadmin,
or programmer?
Need a job?
(in the Silicon Valley, Zürich, or New York)
Email: marc_jo@merlins.org

Questions

Getupdates

How to maintain lots of machines in sync with getupdates

2000-01-21T00:22:29