Marc's Public Blog

This year, I had a bit more advance notice, and had more time to practise Japanese with language tapes, which came in quite handy in remote places like Yonaguni. Starting with somewhere more remote like Okinawa was interesting, and except for Osaka which we didn't see last year, all other cities we had seen the previous year, but not been able to see everything in, so this second visit was great for filling in the blanks we left last year :)

Here are a few random pictures taken over those 3 weeks.

Awesome liquid testing machine in the airport, with this they test your bottle and don't steal your water or juice. TSA, WTF aren't you doing this?

Store attendants training and a pep talk before opening time:

We loved the logo for that company the cat carrying a cat. We gave them our diving luggage in Osaka so that they sent it to our last hotel and we didn't have to lug it around.

How nice for cabs to actually have 2 charging plugs for cell phones :)

All the digital money cards they have, each region had its own and they were mostly incompatible. Now they finally work mostly everywhere, although this bus was an exception and it showed the only 2 cards its old reader could deal with:

I need to revise my charging station system, too many USB devices to charge nightly, that was a bit of a mess :)

French cookies, yummy :)

That one left me speechless. This temple had loaner wheelchairs so that you didn't ride your dirty wheelchair with filthy wheels on their perfect inside floor. To be honest, I find their obsession of no outside dirt at all no matter what, a bit over the top and definitely annoying when you have shoes like mine that take a long time to take in and out.

Japan has hilarious signs, even if you can't read Japanese:

Do not feed the deer, or they'll be very sad :)

Japan still rules for vending machines, available almost everywhere (although that's a lot of trash and there were very few or no places to recycle containers :( ).

fewer of them, but you can still find cigarettes

you can actually get decent coffee, a great selection even

Just like last year, trains were a lifesaver for getting around, where well organized, and on time:

Trains kind of drive themselves nowadays, but they still make it look like serious business, with white gloves :)

I wrote a lot of notes and thoughts last year, you can go refer to them, I think the only extra thing I'd add this year is that I loved many things about Japan, but I was dismayed by the fact that most Japanese people still work way too many hours because of peer pressure, and in the end, just stay at the office until they've hit 12H or so, and it's ok to go home.
For those who try to work that many hours, many studies show it's unsustainable and that you end up actually being less productive trying to work that much than having a normal work week. Somehow, I'd hope that Japanese people wisen up to that and switch to a normal work day and get to enjoy their personal lives a bit more with the extra time.

That said, I'll end with a link to this article Japan, and How I Failed to Figure it Out which actually does quite a good job explaining Japan and how different and fascinating it can be.

And finally we got to our last day in Japan. Jennifer thought she would enjoy the amenities of our hotel :)

I chose to go back to Tokyo Tower since I had gone in 1997, and figured I would see Zojo-ji, a big quiet temple just next to it, wihch happened to also be featured in X-Men's The Wolverine, much of which happens in Japan. It's a great temple, both in size and appearance:

Next, I went to Tokyo tower, just because. It's not a first attraction (obviously, the Tokyo Skytree has stole the thunder), but if you're by Zojo-ji, why not? :)

Odaiba

At the base of Tokyo Tower, there is a small aquarium. It's nothing to write home about, but I figured I'd check it out anyway. It actually had some interesting fish I hadn't seen before, as well as weird nosed turtles :)

nice snapping turtle

After this, I went back to our room and we walked to the Google Office for lunch with friends and a quick tour.

And after that it was time for a bus back to Narita to fly home. A few shots from that bus:

After a good night at the Hyatt Tokyo in Roppongi (5mn from the Google office), I had a quick look at the view from our room, but unfortunately we only got the 9th floor, so that view didn't go very far. Our room in Nagoya had been much better :)

After breakfast, we took a train to Yokohama. We started with Chinatown so that Jennifer could go bond with "her people" :) There was a fantastic Tao temple with exquisite wood carvings, paintings, and gold leaf covering. Turns out it was only 25 years old after the previous temple burned in a fire caused during Chinese New Years...

After walking by the waterfront and witnessing a boat race.

This got us to the Cup Noodle Museum which was a lot of fun. We designed our own cup noodle container, and made a custom flavour by mixing the ingredients of our choice. After that, we learned about its inventor and his life, which was actually quite interesting.

Next was the amusement park where we took the ferris wheel for nice higher up views,

landmark tower

Just after, we went to the landmark tower next gave much much better high views :)

Last but not least, we went back to Tokyo to meet to Japanese people who invited us to their home for dinner and conversation, and getting to talk with them (in English) was enlightening.

A few pictures:

nice views from the cafe

love the keyboard :)

Very Yummy :)

They had a demo on superconductivity:

Their very deep sea submarine

Asimo was there of course:

Which other science museums teach you about quantum particles and neutrinos?

nice displays on radiation

neutrino detectors from super Kamiokande

the big bang and the expansion of the universe

After a night in our smallish room in Shinbashi (but well located), and watching trains go by through the window, Jennifer got to enjoy the yummy Japanese breakfast included with the room before we went to Odaiba. I had heard that Odaiba had a lot of cool things, so we took a train there, and started exploring on foot:

There is an exact copy of the French statue of liberty in Paris as part of a cultural exchange between France and Japan:

July 4th = Bastille Day

Next we went up the Fuji TV station building and its interesting sphere that doubles as an observation point:

they have cheerleaders in Japan now?

Next, we walked to Miraikan -- National Museum of Emerging Science and Innovation. It's a stellar science museum, totally worth going to. Here is a page on it.

By then, we had little time left for the Toyota Megaweb showcase. We only spent 40mn there and it was worth more, but we had run out of time, and that time had been better spent at Miraikan.

fuel cell vehicle

isn't there a piece missing on this one? :)

Thanks to Erik Kline, we then went to a small Owl Cafe in Kiba (yes, like a Neko Cafe, but with Owls, only in Japan :)

This left us just enough time to go eat at Hamada-Ya, a fancy Japanese style restaurant with customized service

Another good day. We could have spent a bit more time in Odaiba, but I guess that'll be an excuse to go back another day :)

After Linuxcon Japan was over, we moved to Ginza and spent most of the day by Ueno Park's museums:

The Tokyo National Museum at Ueno Park is quite good, lots to see, and for a change it was a place where you were allowed to take pictures. I had planned to go to the Natural Science Museum next door again, but we ran out of time, so it'll be for another time:

the 3 missing statues from Sanjusangendo in Kyoto

After the museum, I walked through the rest of Ueno Park, passing by Toshogu Shrine, which is quite beautiful:

I then finished the other side of the park, got to see some birdies :)

Next, I went to Akihabara on my own (not really Jennifer's cup of tea):

weird mix of things available :)

Cool, I needed an MSX! :)

OMG, I remember those retro games :)

got to play the pee against the target to lift the lady's skirt game again :)

big is also beautiful

We finished in Ginza for a yummy super expensive sushi dinner at Sushi Yoshitake:

hooters ginza

manta shrimp

We took the elevator to the top, sunshine 60, for an awesome clear view after a storm. Sunshine 60 was the tallest building in 1985 and had the fastest elevator in the world .We got to see Mt Fuji for the first time (which is rare in summer):

Tokyo Skytree

We had some nice shots of Mt Fuji (thanks to my 30X zoom):

Mt Fuji in the background

Tokyo Tower

Then, we went to another Neko cafe, because why not? :)

After I had learned a fair amount about Btrfs already I felt others would benefit from a introduction to it, as well as best practises and what features it offers that would make you want to switch, I ended up submitting a talk to Linuxcon. In the process of writing the talk, I learned even more about Btrfs, and wrote some of them on my Btrfs blog while linking or putting the other relevant ones on the Main Btrfs Wiki.

The fancy talk description is here:

The presentation will give you everything you know to get up to speed with btfrs, why you should want to trust your data to btrfs, how it offers a lot of what ZFS offers without the licensing problems, as well as best practices for using it.

I will go into:

the basics of administration of a btrfs filesystem

How btrfs, swraid, dmcrypt, and lvm fit or don't fit together

how to work with a single storage pool and create all your partitions from it without having to ever resize them, or require LVM as a slow and somewhat unreliable block layer.

how to have virtually as many snapshots as you want and why you really want this

how to do very efficient block level backups of changes only and much faster than rsync ever will

how those block backups can be used to deploy OS upgrades at the file level like I explained in my talk on how Google maintains its many servers last year.

The video:

The quality isn't great due to poor lighting, sorry about this, but you get the talk slides here or view them inline below.
For easier to click on links, you may prefer the pdf version or the libreoffice version.

> To click on URLs in the presentation, click on 'with contents' in the upper left, and use those links (open to a new tab)

How to manage historical snapshots of backups with Btrfs

I have a setup where I backup a certain number of machines to a central server. There are multiple ways to do hierarchical backups with btrfs.

snapshots and rsync on top

cp -a --link + rsync

cp -a --reflink + rsync

A few pictures from the event:

Keynotes:

A few talk slides:

A very interesting talk for Japanese people on how they can better interact with the linux community and what pitfalls they should avoid:

And gatherings/people:

it had been ages since I had last seen Daniel Veillard from RH, a containerization expert. Made for good talks.

my buddy Horms

the French connection :)

Unfortunately we were not allowed to take pictures in the factory itself, but it was interesting to see how their assembly line worked, namely how all cars went through the same line, and the robots and humans adjusted to the custom build of each car (from cars to minivans) to perform the precise build specifications for each vehicle.

When I asked our tour guide why Toyoda-San would have written his name in Katakana instead of Hiragana, she had no answer :)

But at least I was able to take some pictures in the small museum in the factory:

Their personal mobility device for people needing help to get around:

Their trumpet playing robot was quite cool too:

Of course, they had cars on display:

pink, my favourite :)

The main hall shows the evolution of Japanese trains over time.

While the shinkansen (bullet train) between Tokyo and Osaka is actually not that fast (a mere 280kph) due to limitations from the train tracks built for much older and slower trains (the French TGV peaks at 350kph on newer commercial lines), the technology is nonetheless impressive, especially the part where it tries to counter lean in turns to go faster on curves that are too sharp for higher speeds.

the 3 Japanese trains that broke world records

What the new maglev will look like

The main hall is of course where all the trains and history are located:

but waiting for 2027, that's a long time to wait...

in the meantime, you can ride the short test line for fun :)

They had plenty of simulators for people to play with and demos. There is a long line for the shinkansen simulator, but to be honest I don't think it's that fun or impressive to drive a train :)

The diaroma showing the shinkansen line going through Nagoya was quite good:

The displays showing how often the shinkansens run in a single day, and then how they are maintained was quite interesting. Watch the movie if you can:

Then plenty of time to look inside all the trains:

Older trains too:

It was a great visit, really worth the time.

After waking up in our room with great view on top of the Nagoya train station, we enjoyed the view outside while having breakfast:

We took the longish journey to the Toyota Factory (about as long as taking a fast train to Tokyo :)), for our toyota factory tour (on its on page)

Next, due to how long it would have taken to get around with public transport, we took an expensive but much faster cab directly to SCMAGLEV and Railway Park which we had seen quickly last year (1h15), and I thought was definitely worth a longer 3H+ visit.

Lots of pictures from museums on this Nagoya's SCMAGLEV and Railway Park page

After a good long visit of the train museum, we went back to Nagoya to pick up our luggage and take a train to Tokyo:

Differences between two btrfs snapshots

When you have historical snapshots, it may be useful to know what changed between 2 snapshots.

The best way to do this long term is to modify "btrfs send" to compute changes between the snapshots and just output the filelist instead of a stream with data.

However, until then, there is a hack that shows you files that got added and removed between two snapshots. It's not bulletproof like btrfs send, but it can give you a quick mostly working diff between two snapshots (*it will not show renames or deletes*). See more caveats on this original serverfault post.

legolas:/mnt/btrfs_pool1# btrfs-diff usr_ro.20140513_05:00:01/ usr_ro.20140514_06:00:02/ 
share/doc/linux-image-3.15.0-rc5-amd64-i915-preempt-20140216s1/buildinfo.gz
share/doc/linux-image-3.15.0-rc5-amd64-i915-preempt-20140216s1/Buildinfo.gz
share/doc/linux-image-3.15.0-rc5-amd64-i915-preempt-20140216s1/changelog.Debian.gz
share/doc/linux-image-3.15.0-rc5-amd64-i915-preempt-20140216s1/Changes.gz
(...)

You can download my latest snapshot of btrfs-diff. Note that I am not the author, it was copied from this serverfault post.

#!/bin/bash

# Author: http://serverfault.com/users/96883/artfulrobot
# License: Unknown
#
# This script will show most files that got modified or added.
# Renames and deletions will not be shown.
# Read limitations on:
# http://serverfault.com/questions/399894/does-btrfs-have-an-efficient-way-to-compare-snapshots
# 
# btrfs send is the best way to do this long term, but as of kernel
# 3.14, btrfs send cannot just send a list of changed files without
# scanning and sending all the changed data blocks along.

usage() { echo $@ >&2; echo "Usage: $0 <older-snapshot> <newer-snapshot>" >&2; exit 1; }

[ $# -eq 2 ] || usage "Incorrect invocation";
SNAPSHOT_OLD=$1;
SNAPSHOT_NEW=$2;

[ -d $SNAPSHOT_OLD ] || usage "$SNAPSHOT_OLD does not exist";
[ -d $SNAPSHOT_NEW ] || usage "$SNAPSHOT_NEW does not exist";

OLD_TRANSID=`btrfs subvolume find-new "$SNAPSHOT_OLD" 9999999`
OLD_TRANSID=${OLD_TRANSID#transid marker was }
[ -n "$OLD_TRANSID" -a "$OLD_TRANSID" -gt 0 ] || usage "Failed to find generation for $SNAPSHOT_NEW"

btrfs subvolume find-new "$SNAPSHOT_NEW" $OLD_TRANSID | sed '$d' | cut -f17- -d' ' | sort | uniq

The poster collection was awesome, the French ones ruled ;)

For most people, Nagoya is a day stopover between Osaka/Kyoto and Tokyo. If you are very limited on time, fair enough, but otherwise there is more to see (3 days worth actually). If you go through Nagoya and do not stop to see the Maglev Train Museum, you are definitely doing it wrong :)

We got up from in Kyoto and directly went to Nagoya, where we were going to spend one night. I chose a top floor room with view at the Marriott in the main train station, which turned out to be a great call:

We then started with the Nagoya Castle. It's not the best castle to visit in Japan due to its renovation going on, but still worth a visit if you're there:

Next, we went to Tokugawa Garden and Museum:

Next, we went to the Toyota Automobile Museum, which was a bit out of the way:

our nozomi train

San Francisco could learn a thing or two about this and rebuild 101

Jennifer then found a very nice restaurant for dinner:

And we retired to our room, with the fantastic view:

I had originally planned most of a day in Kyoto to do the east side more leisurely, but since we didn't get to see Kurama the previous day, we went around Kurama in the morning (cable car and hiking to a remote temple high up in the woods). Next to the temple, there was also a treasure museum/nature center worth visiting:

this is where we went hiking to see the temple high up in the mountains

I love those trees

There was a small museum showing what can be found in the mountains around:

can't argue with a map like this :)

lying is bad for you :)

After our walk and collecting our luggage, we took a train back to Kyoto:

Kurama was nice, but we only got back to Kyoto electric bike rental shop around 13:00, which barely give us 4H to see eastern Kyoto. Anyone who knows Kyoto knows it's totally impossible in a day, never mind 4H, so we pressed on :) We started with Sanjusangendo with its 1001 statues (minus 3 that are on loan to Tokyo, which we got to see later, impressive to say the least):

many many statues :)

this is where they are housed

Then we hiked to Higashiyama street to see Tainai Meguri next to Kiomizu Dera which is now a big mess of renovation. Turns out Tainai Meguri was a bit gimmicky and not worth the trip, but oh well. The shopping street on the way there was fun seeing again though:

Next, we went to Ryozen Kannon which was closed by the time we got to it last year, but for which we saw the buddha sticking out. It's a very nice peaceful temple that has also become a war memorial:

And from there we worked back up and stopped at temples on the way. Chion-in which was mostly closed for renovation too, so we went to Shoren-in instead. After that we went back to Nanzen-ji because we were not able to climb its wooden gate last year, and that was worth it:

This time we managed to get in Eikan-do Zenrin-ji just before it closed. Well, the entrance closed 1h before actual closing time, so we kind of snuck in :) It was well worth it, it's not top on any lists, but it actually has plenty of nice things to see, I'm glad we got to see it:

Next, we went back to Honen-in which doesn't really close, and is nice and peaceful, except for 3 male ducks fighting over a single poor female :)

By then it was past 17:00 so we didn't try to go to the Silver Pavillion which we had seen last year anyway, but biked up there, so that we could bike down the Philosopher's Walk (Tetsugaku no michi) back to where I remembered the outdoors tea room with cats was, and sure enough the cats were there :)

We then went back to Maruyama Park and finished by going through Yasaka Shrine which helpfully does not close, and finished with dinner by Issian Pontocho:

This was a good day in Kurama and Kyoto thanks to the electric bikes again, not bad for a half day in Kyoto :)

After getting breakfast, we got our electric bikes at http://www.kyoto-option.com/ , bar none the best place to get electric bikes in Kyoto, with service in Englsh, and extended hours (very early in the morning to late at night).

We started with Toji, wihch has the highest Pagoda in Japan, but it's actually a huge complex with much to visit:

high tech or backup fire suppression system :)

beautiful!

Next were Higashi and Nishi Honganji, which were huge and well worth the visit despite the renovation work that's going on:

We then stoped at Shoseien, a nice garden:

which then led us to Nishiki Market, wihch was just what Jennifer was looking for :)

After returning our bikes, we took a couple of trains to Kurama, a village up in the hills north of Kyoto for our Ryokan there:

After a yummy dinner, time for a confy bed :)

I'm not sure Jennifer needed to go anywhere that morning :)

Then again, when I mentioned breakfast, she was up :)

After a good breakfast, we went for an early visit of Nara before the crowds showed up. So nice early in the morning.
We started with Kasuga Taisha, which we had seen before, but it didn't hurt to see it again :)

do not feed the deer bad food, or they get sick and sad :)

Next was Shin-Yakushi-ji:

We then cut through the park to get to Kofuku-ji, which we didn't have the time to enter last year:

We then went to Yoshikien again because it's nice and it was on the way :)

And we figured, what the hell, we had of course visited Todai-ji the previous year, but since we were walking in front of it to get back to our room, we went inside for a peek again:

Next, we went to Toshodai-ji and Yakushi-ji on our way to Kyoto. My plan was to store our luggage in the lockers by the Kintetsu train station Nishinokyo, but the lockers were too small for my big luggage :( In the end I had to lug it to both temples, but since it was raining anyway, we just took cabs:

That was a good run for Nara, so we ended there, and took the kintetsu to Kyoto (just 50mn), checked in our hotel across the train station, had a quick loook at Kyoto tower but didn't go up since we were warned it was overrun with kids by the people selling tickets (also the weather was poor), so we went to Kyoto station instead for good views, dinner, a visit of the inside (including the awesome dot matrix display built into the stairs, and back to the room).:

And this concluded another nicely filled day.

Nice murals in the train station:

In the morning, I went to Shitenoji to see an old temple with a pagoda you could actually climb to the the top of (pretty much not possible for most of them):

I then went back to Osaka station to pick up our luggage and take a train to Nara:

train driving is serious business :)

We took the loop line to arrive in Nara by the south so that we could stop at Horyu-ji, the oldest Japanese temple located in Nara. We got super lucky and got a retired engineer who spoke well English and gave us a detailled personalized tour for free:

Overrun by kids again :)

After the temple, we went to Nara proper, but got there a bit late to see anything, so after plyaing with the deer, we backtracked to our fancy Ryokan to enjoy the onsen and a yummy dinner

Except for trying to see a bit of Nara after arriving there instead of just calling it day when we arrived a bit late, we did well :) We just spent more time to finish up Nara the next day

argh more kids, at least they're nicely arranged by hat color :)

They also have a single whale shark in their smaller tank. We were lucky to see it feed:

Of course, they had more in the aquarium:

the seals were playful and would look at you and interact through the glass

so much food in them :)

The touch pool was fun :)

Osaka Aquarium is not as nice as the Okinawa one, but it's still quite nice, so we went to see it.
Like last year, it was school kid week, there were busloads of kids, and oh my god were they obniously loud in the aquarium:

argh more kids, at least they're nicely arranged by hat color :)

They also have a single whale shark in their smaller tank. We were lucky to see it feed:

Of course, they had more in the aquarium:

the seals were playful and would look at you and interact through the glass

so much food in them :)

The touch pool was fun :)

After the aquarium, we went through the food court nearby and the ferris wheel:

After that, we went to Osaka Castle:

Next, we went to downtown Osaka and its arcade towards Namba

We then headed towards the Uema Sky building to get up before sunset:

a freeway overpass going through a building, brilliant.

Then was time to go back to Namba for dinner:

The aquarium is 3rd biggest one in the world (#2 is Dubai and #1 is Atlanta), and the only one with 3 whale sharks, and the Expo Park it's in has plenty of other things to see.

Since we had left early, we entered the aquarium right after 08:30 on a monday morning when it opened, so thankfully we avoided the crazy crowds that can be there.

The outside flower statues were fantastic:

There are many displays on fish obviously, but the main attraction is the huge whale shark tank (below):

giant lobsters, too much to eat for a single person on these :)

The Aquarium is of course known for its 3 whale sharks (which are actually sharks, not whales). It's the only place in the world where you can see 3 of them, especially of that size. They may actually have the biggest habitat of any aquarium:

The rest of the aquarium was also good:

very deep sea

The rest of the Ocean Expo Park was good, although the dolphin show was maybe not Sea World level :)

flying dolphins :)

Manatees

Prostetic tail for an injured dolphin

After the aquarium (which is actually worth 3-4H if you include some of the movies they show), we went to see the rest of the expo park, which had an okinawa village recreation and a very nice botanical garden:

And after over 6H in the park, it was time to head back. We could have spent a bit more, but we got a good overview. Maybe we'll be back one day.

This isn't the rental car we picked up, but that was tempting :)

We first went to the Navy Underground Headquarters where the Japanese Navy generals hid and fought to the death when the US Marines came to take the island. It was both interesting and depressing:

some committed suicide by killing themselves with grenades

Next, we went to the Okinawa Caves and fake village to amuse tourists:

Next was the village:

Another day, we went to visit Shuri-jo, the main castle at Okinawa:

And we went to the Okinawa Museum (natural history and art):

snake skin banjo like instruments are common in Okinawa

Our hotel was actually a huge furnished apartment by the sea wall. One night, we had sushi at home, and in the morning, we could see F18s take off towards the ocean:

Obviously most of our time in Okinawa was actually for diving, and you can find those pictures on the Okinawa Diving Page to see the dive pictures from Okinawa.

We had a pretty big boat for just the two of us:

The highlight for Yonaguni was the underwater monument, which looks like an underwater atlantis:

And here are some other diving pictures we got in Yonaguni:

Manta Shrimp

Then, we flew back to Okinawa and did 3 days of diving there:

We had plenty of sea snakes

There is of course more to see in Okinawa as explained on this page

monorail in Okinawa

finally arrived in Yonaguni in 'great' weather

The weather sucked in Yonaguni, we were not able to dive on the first day, we stayed in the villa from the diving shop. To be honest, Yonaguni really doesn't have much to visit, if you're not there for diving or get away from everything else, you're likely going to be bored:

You can go to the Yonaguni Diving Page to see the dive pictures from Yonaguni.

Fixing Btrfs Filesystem Full Problems

Clear space now

Two things to note:

Note that even in the cases described below, you may have to clear one snapshot or more to make space before btrfs balance can run. As a corollary, btrfs can get in states where it's hard to get it out of the 'no space' state it's in. As a result, even if you don't need snapshot, keeping at least one around to free up space should you hit that mis-feature/bug, can be handy

Is your filesystem really full? Mis-balanced metadata and/or data chunks

Below, you'll see how to rebalance data blocks and metadata, and you are unlucky enough to get a filesystem full error before you balance, try running this first:

legolas:~# btrfs balance start -musage=0 /mnt/btrfs_pool1 &
legolas:~# btrfs balance start -dusage=0 /mnt/btrfs_pool1 &

Also, if you are really unlucky, you might get in a no more space error that requires adding a temporary block device to your filesystem to allow balance to run. See below for details.

Pre-emptively rebalancing your filesystem

Is your filesystem really full? Mis-balanced data chunks

legolas:~# btrfs fi show
Label: btrfs_pool1  uuid: 4850ee22-bf32-4131-a841-02abdb4a5ba6
	Total devices 1 FS bytes used 441.69GiB
	devid    1 size 865.01GiB used 751.04GiB path /dev/mapper/cryptroot

In the case above, because the filesystem is only 55% full, I can ask balance to rewrite all chunks that have less than 55% space used. Rebalancing those blocks actually means taking the data in those blocks, and putting it in fuller blocks so that you end up being able to free the less used blocks.
This means the bigger the -dusage value, the more work balance will have to do (i.e. taking fuller and fuller blocks and trying to free them up by putting their data elsewhere). Also, if your FS is 55% full, using -dusage=55 is ok, but there isn't a 1 to 1 correlation and you'll likely be ok with a smaller dusage number, so start small and ramp up as needed.

legolas:~# btrfs balance start -dusage=55 /mnt/btrfs_pool1 &

# Follow the progress along with:
legolas:~# while :; do btrfs balance status -v /mnt/btrfs_pool1; sleep 60; done
Balance on '/mnt/btrfs_pool1' is running
10 out of about 315 chunks balanced (22 considered),  97% left
Dumping filters: flags 0x1, state 0x1, force is off
  DATA (flags 0x2): balancing, usage=55
Balance on '/mnt/btrfs_pool1' is running
16 out of about 315 chunks balanced (28 considered),  95% left
Dumping filters: flags 0x1, state 0x1, force is off
  DATA (flags 0x2): balancing, usage=55
(...)

When it's over, the filesystem now looks like this (note devid used is now 513GB instead of 751GB):

legolas:~# btrfs fi show
Label: btrfs_pool1  uuid: 4850ee22-bf32-4131-a841-02abdb4a5ba6
	Total devices 1 FS bytes used 441.64GiB
	devid    1 size 865.01GiB used 513.04GiB path /dev/mapper/cryptroot

Before you ask, yes, btrfs should do this for you on its own, but currently doesn't as of 3.14.

Is your filesystem really full? Misbalanced metadata

In the example below, you can see Metadata DUP 9.5GB out of 10GB. Btrfs keeps 0.5GB for itself, so in the case above, metadata is full and prevents new writes.

One suggested way is to force a full rebalance, and in the example below you can see metadata goes back down to 7.39GB after it's done. Yes, there again, it would be nice if btrfs did this on its own. It will one day (some if it is now in 3.18).

Sometimes, just using -dusage=0 is enough to rebalance metadata (this is now done automatically in 3.18 and above), but if it's not enough, you'll have to increase the number.

legolas:/mnt/btrfs_pool2# btrfs fi df .
Data, single: total=800.42GiB, used=636.91GiB
System, DUP: total=8.00MiB, used=92.00KiB
System, single: total=4.00MiB, used=0.00
Metadata, DUP: total=10.00GiB, used=9.50GiB
Metadata, single: total=8.00MiB, used=0.00

legolas:/mnt/btrfs_pool2# btrfs balance start -v -dusage=0 /mnt/btrfs_pool2
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=0
  Done, had to relocate 91 out of 823 chunks

legolas:/mnt/btrfs_pool2# btrfs fi df .
Data, single: total=709.01GiB, used=603.85GiB
System, DUP: total=8.00MiB, used=88.00KiB
System, single: total=4.00MiB, used=0.00
Metadata, DUP: total=10.00GiB, used=7.39GiB
Metadata, single: total=8.00MiB, used=0.00

Balance cannot run because the filesystem is full

Note, it's even possible for a filesystem to be full enough in a way that you cannot even delete snapshots to free space. This shows how you would work around it:

root@polgara:/mnt/btrfs_pool2# btrfs fi df .
Data, single: total=159.67GiB, used=80.33GiB
System, single: total=4.00MiB, used=24.00KiB
Metadata, single: total=8.01GiB, used=7.51GiB <<<< BAD
root@polgara:/mnt/btrfs_pool2# btrfs balance start -v -dusage=0 /mnt/btrfs_pool2
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=0
Done, had to relocate 0 out of 170 chunks
root@polgara:/mnt/btrfs_pool2# btrfs balance start -v -dusage=1 /mnt/btrfs_pool2
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=1
ERROR: error during balancing '/mnt/btrfs_pool2' - No space left on device
There may be more info in syslog - try dmesg | tail
root@polgara:/mnt/btrfs_pool2# dd if=/dev/zero of=/var/tmp/btrfs bs=1G count=5
5+0 records in
5+0 records out
5368709120 bytes (5.4 GB) copied, 7.68099 s, 699 MB/s
root@polgara:/mnt/btrfs_pool2# losetup -v -f /var/tmp/btrfs
Loop device is /dev/loop0
root@polgara:/mnt/btrfs_pool2# btrfs device add /dev/loop0 .
Performing full device TRIM (5.00GiB) ...

# optional step if you have snapshots to delete, if not try the balance below
root@polgara:/mnt/btrfs_pool2# btrfs subvolume delete space2_daily_20140603_00:05:01
Delete subvolume '/mnt/btrfs_pool2/space2_daily_20140603_00:05:01'
root@polgara:/mnt/btrfs_pool2# for i in *daily*; do btrfs subvolume delete $i; done
Delete subvolume '/mnt/btrfs_pool2/space2_daily_20140604_00:05:01'
Delete subvolume '/mnt/btrfs_pool2/space2_daily_20140605_00:05:01'
Delete subvolume '/mnt/btrfs_pool2/space2_daily_20140606_00:05:01'
Delete subvolume '/mnt/btrfs_pool2/space2_daily_20140607_00:05:01'
Delete subvolume '/mnt/btrfs_pool2/space2_daily_20140608_00:05:01'
Delete subvolume '/mnt/btrfs_pool2/space2_daily_20140609_00:05:01'

root@polgara:/mnt/btrfs_pool2# btrfs balance start -v -dusage=1 /mnt/btrfs_pool2
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=1
Done, had to relocate 5 out of 169 chunks
root@polgara:/mnt/btrfs_pool2# btrfs device delete /dev/loop0 .

root@polgara:/mnt/btrfs_pool2# btrfs fi df .
Data, single: total=154.01GiB, used=80.06GiB
System, single: total=4.00MiB, used=28.00KiB
Metadata, single: total=8.01GiB, used=4.88GiB <<< GOOD

Misc Balance Resources

This Armin only was only 5h instead of the crazy 9h30 he did in 2006, but honestly it was better because it was 5h of great tunes one after another, no filler songs.
Once you add the live artists and performances, it was pretty fantastic. You can see a summary here:

Next, a few auto-awesome pictures from G+:

And then the real pictures, many courtesy of the 30X zoom on my point and shoot camera:

And before I ever had the time to get bored, it was over. A great evening, really:

It starts with some movie bits from a Sci-Fi movie I didn't know, and that felt wrong :) I wrote down some of the lyrics so that I could look them up later:

Offset 2mn:

Offset 10mn:

Offset 19mn:

Offset 28mn:

Offset 2h27:

I took note of the script bits, plugged them into google and of course it was a known Sci-Fi movie that someone went through the trouble of entirely hand transcribing, so my search was easy.
The movie was Event Horizon, http://www.imdb.com/title/tt0119081/ , which was an enjoyable watch.

But that wasn't the end of it, some lyrics in that ASOT were different:

Last but not least, 1h24mn in,there was a nice quote which I could tell was from Morgan Friedman, and I traced it to Shawshank Redemption, which I had but apparently had never watched, so I watched that too. Great movie.
"I look back at the way I was then... I young stupid kid that committed that terrible crime. I want to talk to him, I want to try to talk some sense to him, tell him the way things are...". Oh, and at the very end of the mix:

Then, while watching Shawshank Redemption, I heard this other great piano tune I already knew, Thomas Newman - Brooks was here: http://www.youtube.com/watch?v=YmjqO7DD14I
Of course, I had heard it in other Trance mixes before, it's an awesome piano track. Interestingly, the track is called "Compass and Guns" in the movie soundtrack: https://www.youtube.com/watch?v=v04F8_sHLJk

Here is the "original" trance track I had heard, which turns out was from solarstone: https://www.youtube.com/watch?v=l8hmA5ZIXDI
And I found another very nice mix done by someone else: https://www.youtube.com/watch?v=B5Rfv4b1IQA

So I went from trance to movie, which then brought me back to other trance, full circle. I like how it all pieces together, thanks to a few google searches :)