Marc's Public Blog - Linux Hacking


All | Aquariums | Arduino | Btrfs | Cars | Cats | Clubbing | Dining | Diving | Electronics | Exercising | Flying | Hiking | Linux | Linuxha | Public | Rc | Sciencemuseums | Snow | Solar | Trips

This page has a few of my blog entries about linux, but my main linux page is here
Picture of Linus


>>> Back to post index <<<

2014/03/19 Btrfs Tips: Btrfs Scrub and Btrfs Filesystem Repair
π 2014-03-19 00:00 in Btrfs, Linux

You mostly don't need fsck on btrfs

Btrfs does not have a really useful fsck, which many people think is horrible and make btrfs unsafe. In real life, there are many ways to mount btrfs filesystems with problem, or extra data out of them if they are too corrupted to mount.

In real life, due to how btrfs is layed out, it's not succeptible to corruption issues or things that are out of sync between locations like ext2-3-4 are, so you'll find that in normal operation you would need fsck for ext2/3/4 to clean up a few things from time to time, but this is just not needed with btrfs. Now, for very unexpected things and bad corruption, obviously you have backups right? but if you'd like to know what was corrupted and whether a restore from backup recovery is needed, or recover new data since your last backup, read on.

Run online fsck nightly or weekly, and look for errors that btrfs is noticing and reporting in syslog:

  • btrfs scrub is actually an online fsck everyone should run. It cannot fix anything unless you have raid1 (raid5/6 support for scrubbing and fixing hasn't been added yet as of kernel 3.14). Note that because data is checksummed, it will find data corruption, but as a result it takes longer to run since it needs to check all your data blocks too. This does put load on your filesystem and machine. With my scrubbing script below, you'll even get a list of files that are corrupted, if any.
  • use sec.pl (apt-get install sec on debian or http://simple-evcorr.sourceforge.net/ ) to report warnings or errors reported in syslog. See the example config file below.
  • Mounting filesystems with problems, try in order:

  • mount -o recovery,ro (for when regular mount isn't working). Note, you have to use ro or it will give you a misleading error:
  • root@polgara:~# mount -o recovery /dev/mapper/crypt /mnt/mnt8
    mount: /dev/mapper/crypt already mounted or /mnt/mnt8 busy
    root@polgara:~# mount -o recovery,ro /dev/mapper/crypt /mnt/mnt8
    root@polgara:~#

  • mount -o degraded (for raid5/6 with a missing drive)
  • btrfs-zero-log can help mount a filesystem if the last blocks btrfs wrote before a crash were corrupted or out of order due to bad hardware or other bugs. See https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#I_can.27t_mount_my_filesystem.2C_and_I_get_a_kernel_oops.21 In debian, I made sure this was part of the initramfs so that you can fix this problem and mount your root filesystem if you had an unclean shutdown and bad hardware that messed up the last blocks that were written.
  • Getting data off a filesystem you can't mount:

  • btrfs restore. See https://btrfs.wiki.kernel.org/index.php/Restore
  • btrfs check --repair, aka btrfsck. See http://www.phoronix.com/scan.php?page=news_item&px=MTA2MDI and https://btrfs.wiki.kernel.org/index.php/Btrfsck . btrfsck is known not to do great work, but it could be useful nonetheless. Apparently --init-csum-tree allows to then mount a filesystem with corrupted blocks with -nodatasum.
  • How to configure sec, event correlator to report btrfs filesystem errors or warnings

    After installing sec.pl (apt-get install sec on debian or http://simple-evcorr.sourceforge.net/ ), install the 2 config files below.

    This is not foolproof, it relies on a regex of known messages that are ok, and reports all unknown ones. You can extend the forward looking negative regex as needed.

    polgara:~# cat /etc/default/sec #Defaults for sec RUN_DAEMON="yes" DAEMON_ARGS="-conf=/etc/sec.conf -input=/var/log/syslog -pid=/var/run/sec.pid -detach -log=/var/log/sec.log" polgara:~# cat /etc/sec.conf # http://simple-evcorr.sourceforge.net/man.html # http://sixshooter.v6.thrupoint.net/SEC-examples/article.html # http://sixshooter.v6.thrupoint.net/SEC-examples/article-part2.html

    type=SingleWithSuppress ptype=RegExp pattern=(?i)kernel.*btrfs: (?!disk space caching is enabled|use ssd allocation|use .* compression|unlinked .* orphans|turning on discard|device label .* devid .* transid|detected SSD devices, enabling SSD mode|has skinny extents|device label|creating UUID tree|checking UUID tree|setting .* feature flag|bdev.* flush 0, corrupt 0, gen 0) window=60 desc=Btrfs unexpected log action=pipe '%t: $0' /usr/bin/mail -s "sec: %s" root

    Daily or weekly btrfs scrub

    This is a must have with btrfs, btrfs scrub. Note that it is disk intensive since it checks everything for consistency (including data blocks), so on a server it does add load and it could take over a day ot run if you have terabytes of data.

    If you don't have shlock, install inn, copy shlock out of it, and then delete it :) (or you can remove shlock from the script, it's not vital).

    The up to date version of this script is at http://marc.merlins.org/linux/scripts/btrfs-scrub

    #! /bin/bash

    # By Marc MERLIN <marc_soft@merlins.org> 2014/03/20 # License: Apache-2.0

    which btrfs >/dev/null || exit 0

    export PATH=/usr/local/bin:/sbin:$PATH

    # bash shortcut for `basename $0` PROG=${0##*/} lock=/var/run/$PROG

    # shlock (from inn) does the right thing and grabs a lock for a dead process # (it checks the PID in the lock file and if it's not there, it # updates the PID with the value given to -p) # You can replace this with another lock program if you prefer or even remove # the lock. if ! shlock -p $$ -f $lock; then echo "$lock held, quitting" >&2 exit fi

    if which on_ac_power >/dev/null 2>&1; then ON_BATTERY=0 on_ac_power >/dev/null 2>&1 || ON_BATTERY=$? if [ "$ON_BATTERY" -eq 1 ]; then exit 0 fi fi

    FILTER='(^Dumping|balancing, usage)' test -n "$DEVS" || DEVS=$(grep '\<btrfs\>' /proc/mounts | awk '{ print $1 }' | sort -u) for btrfs in $DEVS do tail -n 0 -f /var/log/syslog | grep "BTRFS: " | grep -Ev '(disk space caching is enabled|unlinked .* orphans|turning on discard|device label .* devid .* transid|enabling SSD mode|BTRFS: has skinny extents|BTRFS: device label)' & mountpoint="$(grep "$btrfs" /proc/mounts | awk '{ print $2 }' | sort | head -1)" logger -s "Quick Metadata and Data Balance of $mountpoint ($btrfs)" >&2 # Even in 4.3 kernels, you can still get in places where balance # won't work (no place left, until you run a -m0 one first) btrfs balance start -musage=0 -v $mountpoint 2>&1 | grep -Ev "$FILTER" btrfs balance start -musage=20 -v $mountpoint 2>&1 | grep -Ev "$FILTER" # After metadata, let's do data: btrfs balance start -dusage=0 -v $mountpoint 2>&1 | grep -Ev "$FILTER" btrfs balance start -dusage=20 -v $mountpoint 2>&1 | grep -Ev "$FILTER" # And now we do scrub. Note that scrub can fail with "no space left # on device" if you're very out of balance. logger -s "Starting scrub of $mountpoint" >&2 echo btrfs scrub start -Bd $mountpoint ionice -c 3 nice -10 btrfs scrub start -Bd $mountpoint pkill -f 'tail -n 0 -f /var/log/syslog' logger "Ended scrub of $mountpoint" >&2 done

    rm $lock


    More pages: January 2017 October 2016 August 2016 July 2016 June 2016 February 2016 January 2016 May 2015 March 2015 January 2015 October 2014 May 2014 April 2014 March 2014 January 2014 November 2013 September 2013 May 2013 March 2013 January 2013 December 2012 August 2012 May 2012 March 2012 January 2012 December 2011 August 2011 July 2011 January 2011 October 2010 August 2010 June 2010 April 2010 March 2010 January 2010 December 2009 November 2009 September 2009 August 2009 July 2009 May 2009 January 2009 December 2008 November 2008 October 2008 January 2008 November 2007 August 2007 July 2006 January 2006 August 2005 April 2005 November 2004 March 2004 February 2004

    >>> Back to post index <<<