Marc's Public Blog

Day 1 was a good powder day:

doggy was happy running down the wall

Day #2 was quite good too:

Day #3 was sunny at Heavenly:

the middle rock field was pretty bad due to very low snow though

No one came here due to the many many rocks just above

Pinnacles was closed of course

And then, I followed Johannes to Killebrew. The run down wasn't bad, but the bottom was pittyful:

this looks nice, must be fun during the summer

We took saturday off, and we were right, many lifts were closed for wind but by the end of the day, we started getting real snow, and more into the night. As predicted, we got 18 inches at kirkwood, and it was a pretty epic day, even if the crowds got everything tracked way too quicky.

Dude, some snow fell on your car :)

The line at Cornice was a bit ridiculous

A big thanks to Johannes for driving there and back, and Jim for having us at his condo for the first 2 nights.

How to use Btrfs raid5/6

Since I didn't find good documentation of where Btrfs raid5/raid6 was at, I did some tests, and with some help from list members, can write this page now.

This is as of kernel 3.14 with btrfs-tools 3.12. If your are using a kernel and especially tools older than that, there are good chances things will work less well.

Btrfs raid5/6 in a nutshell

It is important to know that raid5/raid6 is more experimental than btrfs itself is. Do not use this for production systems, or if you do and things break, you were warned :)

If you're coming from the mdadm raid5 world, here's what you need to know:

polgara:~# btrfs fi show
Label: backupcopy  uuid: eed9b55c-1d5a-40bf-a032-1be6980648e1
        Total devices 11 FS bytes used 564.54GiB
        devid    1 size 465.76GiB used 63.14GiB path /dev/dm-0
        devid    2 size 465.76GiB used 63.14GiB path /dev/dm-1
        devid    3 size 465.75GiB used 30.00GiB path   <- this device is missing
        devid    4 size 465.76GiB used 63.14GiB path /dev/dm-2
        devid    5 size 465.76GiB used 63.14GiB path /dev/dm-3
        devid    6 size 465.76GiB used 63.14GiB path /dev/dm-4
        devid    7 size 465.76GiB used 63.14GiB path /dev/mapper/crypt_sdi1
        devid    8 size 465.76GiB used 63.14GiB path /dev/mapper/crypt_sdj1
        devid    9 size 465.76GiB used 63.14GiB path /dev/dm-7
        devid    10 size 465.76GiB used 63.14GiB path /dev/dm-8
        devid    11 size 465.76GiB used 33.14GiB path /dev/mapper/crypt_sde1 <- this device was added

Create a raid5 array

polgara:/dev/disk/by-id# mkfs.btrfs -f -d raid5 -m raid5 -L backupcopy /dev/mapper/crypt_sd[bdfghijkl]1

WARNING! - Btrfs v3.12 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using

Turning ON incompat feature 'extref': increased hardlink limit per file to 65536
Turning ON incompat feature 'raid56': raid56 extended format
adding device /dev/mapper/crypt_sdd1 id 2
adding device /dev/mapper/crypt_sdf1 id 3
adding device /dev/mapper/crypt_sdg1 id 4
adding device /dev/mapper/crypt_sdh1 id 5
adding device /dev/mapper/crypt_sdi1 id 6
adding device /dev/mapper/crypt_sdj1 id 7
adding device /dev/mapper/crypt_sdk1 id 8
adding device /dev/mapper/crypt_sdl1 id 9
fs created label backupcopy on /dev/mapper/crypt_sdb1
        nodesize 16384 leafsize 16384 sectorsize 4096 size 4.09TiB
polgara:/dev/disk/by-id# mount -L backupcopy /mnt/btrfs_backupcopy

polgara:/mnt/btrfs_backupcopy# df -h .
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/crypt_sdb1  4.1T  3.0M  4.1T   1% /mnt/btrfs_backupcopy

As another example, you could use -d raid5 -m raid1 to have metadata be raid1 while data being raid5. This specific example isn't actually that useful, but just giving it as an example.

Replacing a drive that hasn't failed yet on a running raid5 array

btrfs replace does not work:

polgara:/mnt/btrfs_backupcopy# btrfs replace start -r /dev/mapper/crypt_sem1 /dev/mapper/crypt_sdm1  .
Mar 23 14:56:06 polgara kernel: [53501.511493] BTRFS warning (device dm-9): dev_replace cannot yet handle RAID5/RAID6

No big deal, this can be done in 2 steps:

polgara:/mnt/btrfs_backupcopy# btrfs device add -f /dev/mapper/crypt_sdm1 .
polgara:/mnt/btrfs_backupcopy# btrfs fi show
Label: backupcopy  uuid: eed9b55c-1d5a-40bf-a032-1be6980648e1
        Total devices 11 FS bytes used 114.35GiB
        devid    1 size 465.76GiB used 32.14GiB path /dev/dm-0
        devid    2 size 465.76GiB used 32.14GiB path /dev/mapper/crypt_sdd1
        devid    4 size 465.76GiB used 32.14GiB path /dev/dm-2
        devid    5 size 465.76GiB used 32.14GiB path /dev/dm-3
        devid    6 size 465.76GiB used 32.14GiB path /dev/dm-4
        devid    7 size 465.76GiB used 32.14GiB path /dev/mapper/crypt_sdi1
        devid    8 size 465.76GiB used 32.14GiB path /dev/dm-6
        devid    9 size 465.76GiB used 32.14GiB path /dev/mapper/crypt_sdk1
        devid    10 size 465.76GiB used 32.14GiB path /dev/mapper/crypt_sdl1
        devid    11 size 465.76GiB used 32.14GiB path /dev/mapper/crypt_sde1
        devid    12 size 465.75GiB used 0.00 path /dev/mapper/crypt_sdm1

polgara:/mnt/btrfs_backupcopy# btrfs device delete /dev/mapper/crypt_sde1 .
Mar 23 11:13:31 polgara kernel: [40145.908207] BTRFS info (device dm-9): relocating block group 945203314688 flags 129
Mar 23 14:51:51 polgara kernel: [53245.955444] BTRFS info (device dm-9): found 5576 extents
Mar 23 14:51:57 polgara kernel: [53251.874925] BTRFS info (device dm-9): found 5576 extents
polgara:/mnt/btrfs_backupcopy# 

Note that this is slow, 3.5h for just 115GB of data. It could take days for a terabyte array.

polgara:/mnt/btrfs_backupcopy# btrfs fi show
Label: backupcopy  uuid: eed9b55c-1d5a-40bf-a032-1be6980648e1
        Total devices 10 FS bytes used 114.35GiB
        devid    1 size 465.76GiB used 13.14GiB path /dev/dm-0
        devid    2 size 465.76GiB used 13.14GiB path /dev/mapper/crypt_sdd1
        devid    4 size 465.76GiB used 13.14GiB path /dev/dm-2
        devid    5 size 465.76GiB used 13.14GiB path /dev/dm-3
        devid    6 size 465.76GiB used 13.14GiB path /dev/dm-4
        devid    7 size 465.76GiB used 13.14GiB path /dev/mapper/crypt_sdi1
        devid    8 size 465.76GiB used 13.14GiB path /dev/dm-6
        devid    9 size 465.76GiB used 13.14GiB path /dev/mapper/crypt_sdk1
        devid    10 size 465.76GiB used 13.14GiB path /dev/mapper/crypt_sdl1
        devid    12 size 465.75GiB used 13.14GiB path /dev/mapper/crypt_sdm1

There we go, I'm back on 10 devices, almost as good as a btrfs replace, it simply took 2 steps

Replacing a missing drive on a running raid5 array

Normal mount will not work:

polgara:~# mount -v -t btrfs -o compress=zlib,space_cache,noatime LABEL=backupcopy /mnt/btrfs_backupcopy
mount: wrong fs type, bad option, bad superblock on /dev/mapper/crypt_sdj1,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so
Mar 21 22:29:45 polgara kernel: [ 2288.285068] BTRFS info (device dm-8): disk space caching is enabled
Mar 21 22:29:45 polgara kernel: [ 2288.285369] BTRFS: failed to read the system array on dm-8
Mar 21 22:29:45 polgara kernel: [ 2288.316067] BTRFS: open_ctree failed

So we do a mount with -o degraded
polgara:~# mount -v -t btrfs -o compress=zlib,space_cache,noatime,degraded LABEL=backupcopy /mnt/btrfs_backupcopy
/dev/mapper/crypt_sdj1 on /mnt/btrfs_backupcopy type btrfs (rw,noatime,compress=zlib,space_cache,degraded)
Mar 21 22:29:51 polgara kernel: [ 2295.042421] BTRFS: device label backupcopy devid 8 transid 3446 /dev/mapper/crypt_sdj1
Mar 21 22:29:51 polgara kernel: [ 2295.065951] BTRFS info (device dm-8): allowing degraded mounts
Mar 21 22:29:51 polgara kernel: [ 2295.065955] BTRFS info (device dm-8): disk space caching is enabled
Mar 21 22:30:32 polgara kernel: [ 2336.189000] BTRFS: device label backupcopy devid 3 transid 8 /dev/dm-9
Mar 21 22:30:32 polgara kernel: [ 2336.203175] BTRFS: device label backupcopy devid 3 transid 8 /dev/dm-9

Then we add the new drive:

polgara:/mnt/btrfs_backupcopy# btrfs device add -f /dev/mapper/crypt_sde1 .
polgara:/mnt/btrfs_backupcopy# df .
/dev/dm-0       5.1T  565G  4.0T  13% /mnt/btrfs_backupcopy   < bad, it should be 4.5T, but I get space for 11 drives

https://btrfs.wiki.kernel.org/index.php/FAQ#What_does_.22balance.22_do.3F says:
"On a filesystem with damaged replication (e.g. a RAID-1 FS with a dead and removed disk), it will force the FS to rebuild the missing copy of the data on one of the currently active devices, restoring the RAID-1 capability of the filesystem."

See also: https://btrfs.wiki.kernel.org/index.php/Balance_Filters

If we have written data since the drive was removed, or if we are recovering from a unfinished balance, doing a filter on devid=3 tells balance to only rewrite data and metadata that has a chunk on missing device #3 (this is a good way to finish the balance in multiple passes if you have to reboot in between, or the filesystem deadlocks during a balance, which unfortunately is still common as of kernel 3.14.

polgara:/mnt/btrfs_backupcopy# btrfs balance start -ddevid=3 -mdevid=3 -v .
Mar 22 13:15:55 polgara kernel: [20275.690827] BTRFS info (device dm-9): relocating block group 941277446144 flags 130
Mar 22 13:15:56 polgara kernel: [20276.604760] BTRFS info (device dm-9): relocating block group 940069486592 flags 132
Mar 22 13:19:27 polgara kernel: [20487.196844] BTRFS info (device dm-9): found 52417 extents
Mar 22 13:19:28 polgara kernel: [20488.056749] BTRFS info (device dm-9): relocating block group 938861527040 flags 132
Mar 22 13:22:41 polgara kernel: [20681.588762] BTRFS info (device dm-9): found 70146 extents
Mar 22 13:22:42 polgara kernel: [20682.380957] BTRFS info (device dm-9): relocating block group 937653567488 flags 132
Mar 22 13:26:12 polgara kernel: [20892.816204] BTRFS info (device dm-9): found 71497 extents
Mar 22 13:26:14 polgara kernel: [20894.819258] BTRFS info (device dm-9): relocating block group 927989891072 flags 129

As balancing happens, data is taken out of devid3, the one missing, and added to devid11 (the one added):

polgara:~# btrfs fi show
Label: backupcopy  uuid: eed9b55c-1d5a-40bf-a032-1be6980648e1
        Total devices 11 FS bytes used 564.54GiB
        devid    1 size 465.76GiB used 63.14GiB path /dev/dm-0
        devid    2 size 465.76GiB used 63.14GiB path /dev/dm-1
        devid    3 size 465.75GiB used 30.00GiB path   <- this device is missing
        devid    4 size 465.76GiB used 63.14GiB path /dev/dm-2
        devid    5 size 465.76GiB used 63.14GiB path /dev/dm-3
        devid    6 size 465.76GiB used 63.14GiB path /dev/dm-4
        devid    7 size 465.76GiB used 63.14GiB path /dev/mapper/crypt_sdi1
        devid    8 size 465.76GiB used 63.14GiB path /dev/mapper/crypt_sdj1
        devid    9 size 465.76GiB used 63.14GiB path /dev/dm-7
        devid    10 size 465.76GiB used 63.14GiB path /dev/dm-8
        devid    11 size 465.76GiB used 33.14GiB path /dev/mapper/crypt_sde1 <- this device was added

You can see status with:

polgara:/mnt/btrfs_backupcopy# while :
> do
> btrfs balance status .
> sleep 60
1 out of about 72 chunks balanced (2 considered),  99% left
2 out of about 72 chunks balanced (3 considered),  97% left
3 out of about 72 chunks balanced (4 considered),  96% left

At the end (and this can take hours to days), you get:

polgara:/mnt/btrfs_backupcopy# btrfs fi show
Label: backupcopy  uuid: eed9b55c-1d5a-40bf-a032-1be6980648e1
        Total devices 11 FS bytes used 114.35GiB
        devid    1 size 465.76GiB used 32.14GiB path /dev/dm-0
        devid    2 size 465.76GiB used 32.14GiB path /dev/mapper/crypt_sdd1
        devid    3 size 465.75GiB used 0.00 path  <----  drive is freed up now.
        devid    4 size 465.76GiB used 32.14GiB path /dev/dm-2
        devid    5 size 465.76GiB used 32.14GiB path /dev/dm-3
        devid    6 size 465.76GiB used 32.14GiB path /dev/dm-4
        devid    7 size 465.76GiB used 32.14GiB path /dev/mapper/crypt_sdi1
        devid    8 size 465.76GiB used 32.14GiB path /dev/dm-6
        devid    9 size 465.76GiB used 32.14GiB path /dev/mapper/crypt_sdk1
        devid    10 size 465.76GiB used 32.14GiB path /dev/mapper/crypt_sdl1
        devid    11 size 465.76GiB used 32.14GiB path /dev/mapper/crypt_sde1
Btrfs v3.12

But the array still shows 11 drives with one missing and will not mount without -o degraded.
You do this with:

polgara:/mnt/btrfs_backupcopy# btrfs device delete missing .
polgara:/mnt/btrfs_backupcopy# btrfs fi show
Label: backupcopy  uuid: eed9b55c-1d5a-40bf-a032-1be6980648e1
        Total devices 10 FS bytes used 114.35GiB
        devid    1 size 465.76GiB used 32.14GiB path /dev/dm-0
        devid    2 size 465.76GiB used 32.14GiB path /dev/mapper/crypt_sdd1
        devid    4 size 465.76GiB used 32.14GiB path /dev/dm-2
        devid    5 size 465.76GiB used 32.14GiB path /dev/dm-3
        devid    6 size 465.76GiB used 32.14GiB path /dev/dm-4
        devid    7 size 465.76GiB used 32.14GiB path /dev/mapper/crypt_sdi1
        devid    8 size 465.76GiB used 32.14GiB path /dev/dm-6
        devid    9 size 465.76GiB used 32.14GiB path /dev/mapper/crypt_sdk1
        devid    10 size 465.76GiB used 32.14GiB path /dev/mapper/crypt_sdl1
        devid    11 size 465.76GiB used 32.14GiB path /dev/mapper/crypt_sde1

And there we go, we're back in business!

From the above, you've also learned how to grow a raid5 array (add a drive, run balance), or remove a drive (just run btrfs device delete and the auto balance will restripe your entire array for n-1 drives).

A few interesting things I learned:
Gravity bends spacetime which in turn bends light (we can see 2 pictures of the same galaxy due to this effect).
Newton's laws of gravity are a good approximation. but they're not fully correct. By measuring the distance of the moon from the earth, we know it is not quite in the location newton's gravity laws predict.
This is separately from the fact that the moon is actually spinning away from the earth as a counter effect for the energy it puts into our tides, energy that is lost by the moon spinning away from us.

Turns out we've also known for around a century that Mercury doesn't follow the orbit it should either and we now know that it's because of the effects of the sun bending spacetime and in turn Mercury's orbit. I also didn't know that before Einstein gave us the theory of relativity and spacetime, scientists made up a small planet not visible to us and that they called Vulcan to make up for the real orbit of Mercury (see encyclopedia britanica and Shapiro time delay).

By the way, the moon spinning away from the earth and the friction from the tides it causes on it, also causes the earth to slow down, making our days longer. Brian Cox says in one of his shows that the earth day used to be 22 hours 600 million years ago (the earth is 5 billion years old as a reminder). In another Nova/PBS show, I remember they found fosil evidence that the earth year was some 420 days (based on layer deposits), which in turn showed that the earth day was only 18 hours back then. And if you know a few things about the moon, you also know that the moon is always facing us from the same direction, so since the earth day is slowing down, in turn we are slowing down the rotation of the moon on its own axis.

Cool stuff...

If you're interested, you can watch two old Horizon (BBC NOVA equivalent) shows with Brian Cox:

Doing much faster incremental backups than rsync with btrfs send and btrfs receive

Just like ZFS, btrfs can compute a list of block changes between 2 snapshots and only send those blocks to the other side making the backups much much faster.
At the time I'm writing this, it does work, but there still a few bugs that could cause it to abort (no data loss, but it will stop to sync further unless you start over from scratch). Most of those bugs have been fixed in kernel 3.14, so it is recommended you use this unless you're just trying it out for testing.

How does it work?

If you'd like many more details, you can find some here:

In real life, this is tedious to do by hand, and even the script to write is not super obvious, so I wrote one that I'm sharing here. I actually do a fair amount of backups on the same machine (like I backup the SSD on my laptop to a hard drive on the same laptop every hour, because SSDs fail, and they could fail while I away from home without my regular off laptop backups), but the script does allow sending the backup to another machine (--dest).

This backup script does a bit more in the following ways:

As another option, Ruedi Steinmann wrote a more fancy btrbck. It's more complicated since it's much bigger and in java, but it's more featureful, so you may prefer that.

Here is a link to the latest version of my btrfs-subvolume-backup script and a paste of a potentially outdated version for you to look at:

#!/bin/bash

# By Marc MERLIN <marc_soft@merlins.org>
# License: Apache-2.0

# Source: http://marc.merlins.org/linux/scripts

# $Id: btrfs-subvolume-backup 1012 2014-06-25 21:56:54Z svnuser $
#
# Documentation and details at
# http://marc.merlins.org/perso/btrfs/2014-03.html#Btrfs-Tips_-Doing-Fast-Incremental-Backups-With-Btrfs-Send-and-Receive

# cron jobs might not have /sbin in their path.
export PATH="$PATH:/sbin"

set -o nounset
set -o errexit
set -o pipefail

# From https://btrfs.wiki.kernel.org/index.php/Incremental_Backup

# bash shortcut for `basename $0`
PROG=${0##*/}
lock=/var/run/$PROG

usage() {
    cat <<EOF
Usage: 
cd /mnt/source_btrfs_pool
$PROG [--init] [--keep|-k num] [--dest hostname] volume_name /mnt/backup_btrfs_pool

Options:
    --init:          Print this help message and exit.
    --keep num:      Keep the last snapshots for local backups (5 by default)
    --dest hostname: If present, ssh to that machine to make the copy.
    --diff:	     show an approximate diff between the snapshots

This will snapshot volume_name in a btrfs pool, and send the diff
between it and the previous snapshot (volume_name.last) to another btrfs
pool (on other drives)

If your backup destination is another machine, you'll need to add a few
ssh commands this script

The num sanpshots to keep is to give snapshots you can recover data from 
and they get deleted after num runs. Set to 0 to disable (one snapshot will
be kept since it's required for the next diff to be computed).
EOF
    exit 0
}

die () {
    msg=${1:-}
    # don't loop on ERR
    trap ' ERR

    rm $lock

    echo "$msg" >&2
    echo >&2

    # This is a fancy shell core dumper
    if echo $msg | grep -q 'Error line .* with status'; then
	line=`echo $msg | sed 's/.*Error line \(.*\) with status.*/\1/'`
	echo " DIE: Code dump:" >&2
	nl -ba $0 | grep -3 "\b$line\b" >&2
    fi

    exit 1
}

# Trap errors for logging before we die (so that they can be picked up
# by the log checker)
trap 'die "Error line $LINENO with status $?"' ERR

init=""
# Keep the last 5 snapshots by default
keep=5
TEMP=$(getopt --longoptions help,usage,init,keep:,dest:,prefix:,diff -o h,k:,d:,p: -- "$@") || usage
dest=localhost
ssh=""
pf=""
diff=""

# getopt quotes arguments with ' We use eval to get rid of that
eval set -- $TEMP

while :
do
    case "$1" in
        -h|--help|--usage)
            usage
            shift
            ;;

	--prefix|-p)
	    shift
	    pf=_$1
	    lock="$lock.$pf"
	    shift
	    ;;

	--keep|-k)
	    shift
	    keep=$1
	    shift
	    ;;

	--dest|-d)
	    shift
	    dest=$1
	    ssh="ssh $dest"
	    shift
	    ;;

	--init)
	    init=1
	    shift
	    ;;

	--diff)
	    diff=1
	    shift
	    ;;

	--)
	    shift
	    break
	    ;;

        *) 
	    echo "Internal error from getopt!"
	    exit 1
	    ;;
    esac
done
[ $keep < 1 ]] && die "Must keep at least one snapshot for things to work ($keep given)"

DATE="$(date '+%Y%m%d_%H:%M:%S')"

[ $# != 2 ]] && usage
vol="$1"
dest_pool="$2"

# shlock (from inn) does the right thing and grabs a lock for a dead process
# (it checks the PID in the lock file and if it's not there, it
# updates the PID with the value given to -p)
if ! shlock -p $$ -f $lock; then
    echo "$lock held for $PROG, quitting" >&2
    exit
fi

if [ -z "$init" ]]; then
    test -e "${vol}${pf}_last" 	|| die "Cannot sync $vol, ${vol}${pf}_last missing. Try --init?"
    src_snap="$(readlink -e ${vol}${pf}_last)"
fi
src_newsnap="${vol}${pf}_ro.$DATE"
src_newsnaprw="${vol}${pf}_rw.$DATE"

$ssh test -d "$dest_pool/" || die "ABORT: $dest_pool not a directory (on $dest)"

btrfs subvolume snapshot -r "$vol" "$src_newsnap"

if [ -n "$diff" ]]; then
    echo diff between "$src_snap" "$src_newsnap"
    btrfs-diff "$src_snap" "$src_newsnap"
fi

# There is currently an issue that the snapshots to be used with "btrfs send"
# must be physically on the disk, or you may receive a "stale NFS file handle"
# error. This is accomplished by "sync" after the snapshot
sync

if [ -n "$init" ]]; then
    btrfs send "$src_newsnap" | $ssh btrfs receive "$dest_pool/"
else
    btrfs send -p "$src_snap" "$src_newsnap" | $ssh btrfs receive "$dest_pool/"
fi

# We make a read-write snapshot in case you want to use it for a chroot
# and some testing with a writeable filesystem or want to boot from a
# last good known snapshot.
btrfs subvolume snapshot "$src_newsnap" "$src_newsnaprw"
$ssh btrfs subvolume snapshot "$dest_pool/$src_newsnap" "$dest_pool/$src_newsnaprw"

# Keep track of the last snapshot to send a diff against.
ln -snf $src_newsnap ${vol}${pf}_last
# The rw version can be used for mounting with subvol=vol_last_rw
ln -snf $src_newsnaprw ${vol}${pf}_last_rw
$ssh ln -snf $src_newsnaprw $dest_pool/${vol}${pf}_last_rw

# How many snapshots to keep on the source btrfs pool (both read
# only and read-write).
ls -rd ${vol}${pf}_ro* | tail -n +$(( $keep + 1 ))| while read snap
do
    btrfs subvolume delete "$snap" | grep -v 'Transaction commit:'
done
ls -rd ${vol}${pf}_rw* | tail -n +$(( $keep + 1 ))| while read snap
do
    btrfs subvolume delete "$snap" | grep -v 'Transaction commit:'
done

# Same thing for destination (assume the same number of snapshots to keep,
# you can change this if you really want).
$ssh ls -rd $dest_pool/${vol}${pf}_ro* | tail -n +$(( $keep + 1 ))| while read snap
do
    $ssh btrfs subvolume delete "$snap" | grep -v 'Transaction commit:'
done
$ssh ls -rd $dest_pool/${vol}${pf}_rw* | tail -n +$(( $keep + 1 ))| while read snap
do
    $ssh btrfs subvolume delete "$snap" | grep -v 'Transaction commit:'
done

rm $lock

How to get Netapp[tm]-like snapshots with BTRFS

Linux did have crappy snapshots if you used LVM, but LVM and LVM2 snapshots were both so bad performance-wise (they were not meant to be long lived, and even a single snapshot would slow your filesysem down significantly, never mind multiple levels).

If you can't use btrfs, but you still want historical snapshots, you should look into LVM thin provisioning which are newer as of kernel 3.4. They are suppposed to be faster for multiple levels of snapshots. Considering how bad LVM2 is, I'm sure they are faster no matter what, but I didn't have a use for them now that I'm using btrfs, so I can't speak of their performance. You can read up more here:

Back to btrfs, I use the recommended layout of putting all filesystems in a subvolume. I this:

After running my script, I get multiple levels of snapshots, I'll show only root here for brevity. With this you can restore files from older versions 3 hours ago, 3 days ago, or 3 weeks ago. Here is the partial output of /mnt/btrfs_pool1:

drwxr-xr-x 1 root root  370 Feb 24 10:38 root

drwxr-xr-x 1 root root  370 Feb 24 10:38 root_daily_20140316_00:05:01

drwxr-xr-x 1 root root  370 Feb 24 10:38 root_daily_20140318_00:05:01

drwxr-xr-x 1 root root  370 Feb 24 10:38 root_daily_20140319_00:05:01

drwxr-xr-x 1 root root  370 Feb 24 10:38 root_daily_20140320_00:05:00

drwxr-xr-x 1 root root  370 Feb 24 10:38 root_hourly_20140316_22:33:00

drwxr-xr-x 1 root root  370 Feb 24 10:38 root_hourly_20140318_00:05:01

drwxr-xr-x 1 root root  370 Feb 24 10:38 root_hourly_20140319_00:05:01

drwxr-xr-x 1 root root  370 Feb 24 10:38 root_hourly_20140320_00:05:00

drwxr-xr-x 1 root root  336 Feb 19 21:40 root_weekly_20140223_00:06:01

drwxr-xr-x 1 root root  370 Feb 24 10:38 root_weekly_20140302_00:06:01

drwxr-xr-x 1 root root  370 Feb 24 10:38 root_weekly_20140309_00:06:01

drwxr-xr-x 1 root root  370 Feb 24 10:38 root_weekly_20140316_00:06:01

Note that snapshots are not backups, they give you a view into the past if your filesystem hasn't been corrupted and the disk you were using, didn't die.

I then have a cronjob that runs this:

0 * * * * root btrfs-snaps hourly 3 | egrep -v '(Create a snapshot of|Will delete the oldest|Delete subvolume|Making snapshot of )'
2 0 * * * root btrfs-snaps daily  4 | egrep -v '(Create a snapshot of|Will delete the oldest|Delete subvolume|Making snapshot of )'
3 0 * * 0 root btrfs-snaps weekly 4 | egrep -v '(Create a snapshot of|Will delete the oldest|Delete subvolume|Making snapshot of )'

This is using the script, btrfs-snaps, for which I'll paste a most likely outdated copy here:

#!/bin/bash

# By Marc MERLIN <marc_soft@merlins.org>
# License Apache 2.0.

# This lets you create sets of snapshots at any interval (I use hourly,
# daily, and weekly) and delete the older ones automatically.

# Usage:
# This is called from /etc/cron.d like so:
# 0 * * * * root btrfs-snaps hourly 3 | egrep -v '(Create a snapshot of|Will delete the oldest|Delete subvolume|Making snapshot of )'
# 1 0 * * * root btrfs-snaps daily  4 | egrep -v '(Create a snapshot of|Will delete the oldest|Delete subvolume|Making snapshot of )'
# 2 0 * * 0 root btrfs-snaps weekly 4 | egrep -v '(Create a snapshot of|Will delete the oldest|Delete subvolume|Making snapshot of )'

: ${BTRFSROOT:=/mnt/btrfs_pool1}
DATE="$(date '+%Y%m%d_%H:%M:%S')"

type=${1:-hourly}
keep=${2:-3}

cd "$BTRFSROOT"

for i in $(btrfs subvolume list -q . | grep "parent_uuid -" | awk '{print $11}')
do
    # Skip duplicate dirs once a year on DST 1h rewind.
    test -d "$BTRFSROOT/${i}_${type}_$DATE" && continue
    echo "Making snapshot of $type"
    btrfs subvolume snapshot "$BTRFSROOT"/$i "$BTRFSROOT/${i}_${type}_$DATE"
    count="$(ls -d ${i}_${type}_* | wc -l)"
    clip=$(( $count - $keep ))
    if [ $clip -gt 0 ]; then
	echo "Will delete the oldest $clip snapshots for $type"
	for sub in $(ls -d ${i}_${type}_* | head -n $clip)
	do
	    #echo "Will delete $sub"
	    btrfs subvolume delete "$sub"
	done
    fi
done

Btrfs and S3 Sleep (Suspend)

This is discussed in more details in this thread: http://comments.gmane.org/gmane.comp.file-systems.btrfs/33106

For now, I am using this crude workaround. I added this in /etc/acpi/sleep.sh:
awk '/btrfs/ { print $1 }' /proc/mounts | sort -u | while read fs; do btrfs scrub cancel $fs; done

This could easily be improved by running scrub status, pausing scrubs that are running instead of cancelling them, and resuming them after coming back from sleep.

You mostly don't need fsck on btrfs

In real life, due to how btrfs is layed out, it's not succeptible to corruption issues or things that are out of sync between locations like ext2-3-4 are, so you'll find that in normal operation you would need fsck for ext2/3/4 to clean up a few things from time to time, but this is just not needed with btrfs. Now, for very unexpected things and bad corruption, obviously you have backups right? but if you'd like to know what was corrupted and whether a restore from backup recovery is needed, or recover new data since your last backup, read on.

Run online fsck nightly or weekly, and look for errors that btrfs is noticing and reporting in syslog:

Mounting filesystems with problems, try in order:

root@polgara:~# mount -o recovery /dev/mapper/crypt /mnt/mnt8
mount: /dev/mapper/crypt already mounted or /mnt/mnt8 busy
root@polgara:~# mount -o recovery,ro /dev/mapper/crypt /mnt/mnt8
root@polgara:~#

Getting data off a filesystem you can't mount:

How to configure sec, event correlator to report btrfs filesystem errors or warnings

After installing sec.pl (apt-get install sec on debian or http://simple-evcorr.sourceforge.net/ ), install the 2 config files below.

This is not foolproof, it relies on a regex of known messages that are ok, and reports all unknown ones. You can extend the forward looking negative regex as needed.

polgara:~# cat /etc/default/sec 
#Defaults for sec
RUN_DAEMON="yes"
DAEMON_ARGS="-conf=/etc/sec.conf -input=/var/log/syslog -pid=/var/run/sec.pid -detach -log=/var/log/sec.log"
polgara:~# cat /etc/sec.conf
# http://simple-evcorr.sourceforge.net/man.html
# http://sixshooter.v6.thrupoint.net/SEC-examples/article.html
# http://sixshooter.v6.thrupoint.net/SEC-examples/article-part2.html

type=SingleWithSuppress
ptype=RegExp
pattern=(?i)kernel.*btrfs: (?!disk space caching is enabled|use ssd allocation|use .* compression|unlinked .* orphans|turning on discard|device label .* devid .* transid|detected SSD devices, enabling SSD mode|has skinny extents|device label|creating UUID tree|checking UUID tree|setting .* feature flag|bdev.* flush 0, corrupt 0, gen 0)
window=60
desc=Btrfs unexpected log
action=pipe '%t: $0' /usr/bin/mail -s "sec: %s" root

Daily or weekly btrfs scrub

This is a must have with btrfs, btrfs scrub. Note that it is disk intensive since it checks everything for consistency (including data blocks), so on a server it does add load and it could take over a day ot run if you have terabytes of data.

If you don't have shlock, install inn, copy shlock out of it, and then delete it :) (or you can remove shlock from the script, it's not vital).

The up to date version of this script is at http://marc.merlins.org/linux/scripts/btrfs-scrub

#! /bin/bash

# By Marc MERLIN <marc_soft@merlins.org> 2014/03/20
# License: Apache-2.0

which btrfs >/dev/null || exit 0

export PATH=/usr/local/bin:/sbin:$PATH

# bash shortcut for `basename $0`
PROG=${0##*/}
lock=/var/run/$PROG

# shlock (from inn) does the right thing and grabs a lock for a dead process
# (it checks the PID in the lock file and if it's not there, it
# updates the PID with the value given to -p)
# You can replace this with another lock program if you prefer or even remove
# the lock.
if ! shlock -p $$ -f $lock; then
    echo "$lock held, quitting" >&2
    exit
fi

if which on_ac_power >/dev/null 2>&1; then
    ON_BATTERY=0
    on_ac_power >/dev/null 2>&1 || ON_BATTERY=$?
    if [ "$ON_BATTERY" -eq 1 ]; then
	exit 0
    fi
fi

FILTER='(^Dumping|balancing, usage)'
test -n "$DEVS" || DEVS=$(grep '\<btrfs\>' /proc/mounts | awk '{ print $1 }' | sort -u)
for btrfs in $DEVS
do
    tail -n 0 -f /var/log/syslog | grep "BTRFS: " | grep -Ev '(disk space caching is enabled|unlinked .* orphans|turning on discard|device label .* devid .* transid|enabling SSD mode|BTRFS: has skinny extents|BTRFS: device label)' &
    mountpoint="$(grep "$btrfs" /proc/mounts | awk '{ print $2 }' | sort | head -1)"
    logger -s "Quick Metadata and Data Balance of $mountpoint ($btrfs)" >&2
    # Even in 4.3 kernels, you can still get in places where balance
    # won't work (no place left, until you run a -m0 one first)
    btrfs balance start -musage=0 -v $mountpoint 2>&1 | grep -Ev "$FILTER"
    btrfs balance start -musage=20 -v $mountpoint 2>&1 | grep -Ev "$FILTER"
    # After metadata, let's do data:
    btrfs balance start -dusage=0 -v $mountpoint 2>&1 | grep -Ev "$FILTER"
    btrfs balance start -dusage=20 -v $mountpoint 2>&1 | grep -Ev "$FILTER"
    # And now we do scrub. Note that scrub can fail with "no space left
    # on device" if you're very out of balance.
    logger -s "Starting scrub of $mountpoint" >&2
    echo btrfs scrub start -Bd $mountpoint
    ionice -c 3 nice -10 btrfs scrub start -Bd $mountpoint
    pkill -f 'tail -n 0 -f /var/log/syslog'
    logger "Ended scrub of $mountpoint" >&2
done

rm $lock

Here's the spreadsheet showing all the numbers I've gathered so far: https://docs.google.com/spreadsheet/ccc?key=0AloDTsnSvSs7dHdwS1NzcHJlRF80cVg1NUxzUlFEOFE

Year #	       PV Kwh  PG&E Kwh
2010 - #1	8503	1602
2011 - #2	8813	1209
2012 - #3	8402	 674
2013 - #4	8366	 486
2014 - #5	8840	 -30

If I count 20c/Kwh with the different tiers, the solar panels saved us $8500 so far, or one third of the price we had to pay ($26k).

This made it trivial for me to rotate the camera just a bit, and have a birdcam we were able to enjoy for 11 days (after 11 days, they had gone from egg to hatched to gone). They have however come back to our bird feeder several times, and they were not much afraid of us, maybe because I caressed them a bit when they were baby, and held a couple of them.

The parents actually attacked the squirrels on our patio and chased them away (they would fly and land on their back to spook them out):

The parents were doing a great job getting food for them. I just one hour, we saw how many grubs and insects they found, it was amazing:

Eventually they got bigger, and I grabbed one gently for a few minutes before putting it back:

And overnight, they flew off and were gone the next morning. My camera didn't get a good shot of them flying off due to lack of light. But later, we saw all 4 come back and enjoy our bird feeder (oh noes, actually those were other baby birds, Dark Eyed Junco.