Marc's Public Blog - Linux Hacking


All | Aquariums | Arduino | Btrfs | Cars | Cats | Clubbing | Dining | Dinners | Diving | Electronics | Exercising | Flying | Hiking | Linux | Linuxha | Public | Rc | Sciencemuseums | Snow | Solar | Trips

This page has a few of my blog entries about linux, but my main linux page is here
Picture of Linus


>>> Back to post index <<<

2014/03/21 Btrfs Tips: How To Setup Netapp Style Snapshots
π 2014-03-21 00:00 in Btrfs, Linux

How to get Netapp[tm]-like snapshots with BTRFS

Filesystem snapshots are something you'll never want to live without once you've had them. I learned about them in 1997 when I was working for Network Appliance, but unfortunately due to software patents, they managed to pevent most others from enjoying them until more recently.

Linux did have crappy snapshots if you used LVM, but LVM and LVM2 snapshots were both so bad performance-wise (they were not meant to be long lived, and even a single snapshot would slow your filesysem down significantly, never mind multiple levels).

If you can't use btrfs, but you still want historical snapshots, you should look into LVM thin provisioning which are newer as of kernel 3.4. They are suppposed to be faster for multiple levels of snapshots. Considering how bad LVM2 is, I'm sure they are faster no matter what, but I didn't have a use for them now that I'm using btrfs, so I can't speak of their performance. You can read up more here:

  • https://www.kernel.org/doc/Documentation/device-mapper/thin-provisioning.txt
  • https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Logical_Volume_Manager_Administration/thinly-provisioned_snapshot_volumes.html
  • Back to btrfs, I use the recommended layout of putting all filesystems in a subvolume. I this:

  • /mnt/btrfs_pool1 -> actual btrfs filesystem
  • /mnt/btrfs_pool1/root -> gets mounted to / with -o subvol=root
  • /mnt/btrfs_pool1/usr -> gets mounted to /usr with -o subvol=usr
  • /mnt/btrfs_pool1/var -> gets mounted to /var with -o subvol=var
  • After running my script, I get multiple levels of snapshots, I'll show only root here for brevity. With this you can restore files from older versions 3 hours ago, 3 days ago, or 3 weeks ago. Here is the partial output of /mnt/btrfs_pool1:

    drwxr-xr-x 1 root root  370 Feb 24 10:38 root
    drwxr-xr-x 1 root root 370 Feb 24 10:38 root_daily_20140316_00:05:01
    drwxr-xr-x 1 root root 370 Feb 24 10:38 root_daily_20140318_00:05:01
    drwxr-xr-x 1 root root 370 Feb 24 10:38 root_daily_20140319_00:05:01
    drwxr-xr-x 1 root root 370 Feb 24 10:38 root_daily_20140320_00:05:00
    drwxr-xr-x 1 root root 370 Feb 24 10:38 root_hourly_20140316_22:33:00
    drwxr-xr-x 1 root root 370 Feb 24 10:38 root_hourly_20140318_00:05:01
    drwxr-xr-x 1 root root 370 Feb 24 10:38 root_hourly_20140319_00:05:01
    drwxr-xr-x 1 root root 370 Feb 24 10:38 root_hourly_20140320_00:05:00
    drwxr-xr-x 1 root root 336 Feb 19 21:40 root_weekly_20140223_00:06:01
    drwxr-xr-x 1 root root 370 Feb 24 10:38 root_weekly_20140302_00:06:01
    drwxr-xr-x 1 root root 370 Feb 24 10:38 root_weekly_20140309_00:06:01
    drwxr-xr-x 1 root root 370 Feb 24 10:38 root_weekly_20140316_00:06:01

    Note that snapshots are not backups, they give you a view into the past if your filesystem hasn't been corrupted and the disk you were using, didn't die.

    I then have a cronjob that runs this:

    0 * * * * root btrfs-snaps hourly 3 | egrep -v '(Create a snapshot of|Will delete the oldest|Delete subvolume|Making snapshot of )'
    2 0 * * * root btrfs-snaps daily  4 | egrep -v '(Create a snapshot of|Will delete the oldest|Delete subvolume|Making snapshot of )'
    3 0 * * 0 root btrfs-snaps weekly 4 | egrep -v '(Create a snapshot of|Will delete the oldest|Delete subvolume|Making snapshot of )'

    This is using the script, btrfs-snaps, for which I'll paste a most likely outdated copy here:

    #!/bin/bash
    

    # By Marc MERLIN <marc_soft@merlins.org> # License Apache 2.0.

    # This lets you create sets of snapshots at any interval (I use hourly, # daily, and weekly) and delete the older ones automatically.

    # Usage: # This is called from /etc/cron.d like so: # 0 * * * * root btrfs-snaps hourly 3 | egrep -v '(Create a snapshot of|Will delete the oldest|Delete subvolume|Making snapshot of )' # 1 0 * * * root btrfs-snaps daily 4 | egrep -v '(Create a snapshot of|Will delete the oldest|Delete subvolume|Making snapshot of )' # 2 0 * * 0 root btrfs-snaps weekly 4 | egrep -v '(Create a snapshot of|Will delete the oldest|Delete subvolume|Making snapshot of )'

    : ${BTRFSROOT:=/mnt/btrfs_pool1} DATE="$(date '+%Y%m%d_%H:%M:%S')"

    type=${1:-hourly} keep=${2:-3}

    cd "$BTRFSROOT"

    for i in $(btrfs subvolume list -q . | grep "parent_uuid -" | awk '{print $11}') do # Skip duplicate dirs once a year on DST 1h rewind. test -d "$BTRFSROOT/${i}_${type}_$DATE" && continue echo "Making snapshot of $type" btrfs subvolume snapshot "$BTRFSROOT"/$i "$BTRFSROOT/${i}_${type}_$DATE" count="$(ls -d ${i}_${type}_* | wc -l)" clip=$(( $count - $keep )) if [ $clip -gt 0 ]; then echo "Will delete the oldest $clip snapshots for $type" for sub in $(ls -d ${i}_${type}_* | head -n $clip) do #echo "Will delete $sub" btrfs subvolume delete "$sub" done fi done


    More pages: October 2016 August 2016 July 2016 June 2016 February 2016 January 2016 May 2015 March 2015 January 2015 October 2014 May 2014 April 2014 March 2014 January 2014 November 2013 September 2013 May 2013 March 2013 January 2013 December 2012 August 2012 May 2012 March 2012 January 2012 December 2011 August 2011 July 2011 January 2011 October 2010 August 2010 June 2010 April 2010 March 2010 January 2010 December 2009 November 2009 September 2009 August 2009 July 2009 May 2009 January 2009 December 2008 November 2008 October 2008 January 2008 November 2007 August 2007 July 2006 January 2006 August 2005 April 2005 November 2004 March 2004 February 2004

    >>> Back to post index <<<