Marc's Public Blog - Linux Hacking

What follows is unix centric, and won't tell you much if you're not a programmer/unix user, but you should enjoy it otherwise (please note that I do own a legal copy of bru backup for personal use, which I got with an old boxed set of Red Hat)


I'm trying to get my backup tape library to work as I've never been able to backup the 1TB+ of media files (don't ask) I have (I've been copying them on my other drives, but now I have too many for that to fit)

So, I decided to move my tape library to another machine, just for testing (to see if the library was bad, or my SCSI connection or host PC was flaky)
But, problem, it wouldn't back up a single file over NFS.

After a while, I realized:
gargamel:~# bru --version
bru: invalid option -- -
usage: bru -cdeghitx [-#AabBCEfFGjlLmnNoOpPQRSsuUvVwXYZ] file(s)...

Copyright (c) 1994-97, Enhanced Software Technologies, Inc.
All Rights Reserved

BRU is a backup software product licensed by EST.
It is NOT public domain or shareware. Versions are
available for almost any type of Unix system.

This is the Personal Edition version of BRU 2000. It does not offer
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
support for backup or restore of remote/network filesystems.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
To order a copy of the full version of BRU 2000 or for further
information or technical support, please contact Red Hat Software

Ah, great.
There had to be a way to trick it to backup over NFS anyway (this software is more than 7 years old, I'm probably not able to buy a license today, and I'm not violating the spirit of the license, I'm just doing a backup of one system, but without being able to have the library connected to it)

So, I did a couple of straces, one of a local file and one of an NFS file and diffed them.
What did I see?

open("/bru/.serial_number", O_RDONLY) = -1 ENOENT (No such file or directory)
uname({sys="Linux", node="host.svh.domain.tld", ...}) = 0
-lstat64(0x806e5a4, 0xbffff298) = 0
-lstat64(0x8070978, 0xbffff280) = 0
-statfs("dshelf3_X10:D3_MD5_.bz2", {f_type="NFS_SUPER_MAGIC", f_bsize=8192,
f_blocks=14836773, f_bfree=1564919, f_files=15073280, f_ffree=14353562,
f_namelen=255}) = 0
^^^^^^^^^^^^^^^^^^^^^^^
-lstat64(0x8070978, 0xbfffee1c) = 0
-access("dshelf3_X10:D3_MD5_.bz2", R_OK) = 0
+lstat64(0x806e5a4, 0xbffff2a8) = 0
+lstat64(0x8070978, 0xbffff290) = 0
+lstat64(0x8070978, 0xbfffee2c) = 0
+access("profile", R_OK) = 0
+lstat64(0x8070978, 0xbfffe6f0) = 0
+access("profile", R_OK) = 0
+open("profile", O_RDONLY|O_NONBLOCK) = 4
+write(1, "c 4k of 6k [1] profile\n", 27) = 27


Aaahahh. man statfs said that NFS_SUPER_MAGIC was 0x6969, which is "ii" in ascii.
So, I just opened the bru binary with vi, found one instance of "ii" (I was lucky) and it was around the text strings, so it was definitely in the variable sections of the binary.
I changed the string to "ij", and voila, nfs backups.

Yeepee! :)

The joys of a simple geek may be hard to convey sometimes :)

Current Music: none while my disk array is being rebooted every 5mn :)
Current Mood: Hardware, you will submit :)

So, I can't believe that after two years of having that tape autoloader I finally got the thing to work. It took more sweat and tears than I care to write about, but basically, in the end, one of the two busses on one of the 4 SCSI cards in that machine was flaky (some of the cables were too though).
I can't believe that I ended up fixing the problem by putting no less than 6 SCSI busses in that machine (I had to add yet another card). That just feels wrong...
Anyway, it works now and it's happily backing up data. With a little luck, the backup will be over in a week's time.

Cool, now I only need to fix my Replay TV with its broken hard drive and my older archos jukebox player so that I can ebay it. So much to do, so little time :)
I sometimes wonder how "normal" people manage :) (but seriously, it must suck to have to throw those things away, or pay god knows how much to get them fixed by someone else)

Current Music: DJ Tiësto - Live At Innercity - Norefjell
Current Mood: Yeah another long time project finally completed

So, those who've visited my house know that I have a sick system where I use
X10 wireless remotes, that get picked up by a receiver in the wall, which
transforms the signals from RF to electrical signals, which then get picked up
by a receiver in my linen^H^H^H^H^Hcomputer closet, which transforms this into a command for my serial port, which is read by a linux daemon, which in turn runs commands of my choosing, like changing the music or the volume (it's especially awesome under the shower, because I can use a throwaway wireless remote, and still control the music. I'm sure you'll agree that it is very important to be able to do so ;-)

Anyway, I also have another PC that is my video server, and plays files from another server (my disk server), and sends the video in a S-Video cable that goes under the house and reaches my TV.
So far, so good, except that I had to use my laptop on my couch to control the video and pause it or whatever (this turned out not to be a great problem since my laptop is always on my couch, but I digress).

So, I have some TV capture card with an IR receiver, and an IR remote. With a little work, I found and setup some linux software to receive the IR codes from the remote, and pass them on as commands to mplayer, a video player on linux.
This wasn't bad, except that I have to point the IR remote in my back, towards the PC, which is nowhere near the TV, and it wouldn't even work at all if I can see the TV but I'm a place in the room where my IR remote can't reach the IR receiver.
So, I had that working a couple of years ago, but I never really used it because it was a clearly inferior and imperfect solution that no self respecting geek would bring himself down to using :)

That's where my mouseremote comes in. This handy little sucker looks like a 5 way learning remote, also supports sending X10 events (i.e. to control my lights, and also my music from the closet), and then it also sends two more sets of RF signals that get picked up by another computer. The first set allows me to move the mouse on my TV screen for the PC I use to play videos, and the second set allows me to send random keys from the remote and have them used by the PC.
This is where the interesting part comes in: using those keys to do interesting stuff, like controlling the video so that I can just use my mouseremote like a regular remote, except that I'm talking to a PC that's not even within line of sight.
I wouldn't want to further bore you with technical details, and you can find them here should you really be curious (considering that it took me a little while to get it working, I documented it so that the next guy who tries this doesn't have to spend as much time than me). I'll just leave you with the (admittedly somewhat insane) list of things that happens in real life when you press play:

1. the mouseremote sends an RF signal with the key I pressed
   
2. multimoused on linux reads from the serial port and receives a translation of the RF signal
   
3. it generates mouse events into the /dev/mumse fifo
   
4. it separates and sends keypresses (like DVD+1) to /dev/x10fifo
   
5. MouseRemote reads from the fifo and executes commands as directed in MouseRemote.conf
   
6. MouseRemote.conf calls irsend, like so ' irsend SEND_RELAY "0000000000001005 00 5" Hauppauge ', which tricks the patches lircd to think I used my IR remote to send this code
   
7. I'm running a patched lircd that understand this special trick, and relays it to its fifo as if it were coming from the infrared remote
   
8. mplayer, running in lirc mode, reads from the lircd fifo, parses ~/.lircrc, receives the fake IR code, and pauses or resumes the video
   

See, it's really simple :)
I only got that working last night once I got the special evil trick to generate fake IR events in lircd, as if I were using my IR remote. We'll put aside the fact that I've owned the remote for mmmh, 4 years now :-)
Eh, at least I get points for documenting all this , so as to make it easier for the next guy...

Anyway, that's when you get to witness someone next to me in my couch, me pressing play or pause on my remote, the video stopping or starting as appropriate, me displaying a huge grin on my face, and the person next to me wondering "what's up with him, he pressed pause, the video paused, what's so special about that?".
If they only knew ;-)))

This is where I also get to say a few words about linux vs windows.
With windows, I would definitely have spent less time on all this: either I would have found some complex software that someone wrote and that happened to do what I needed, or I would have been entirely out of luck.
With linux, it took me a fair number of hours to make this all work (10 or so), but once I found that the main pieces existed (IR and RF readers) I knew that I could write the missing glue to make it all work together. In its own way, if you are into this, it's really rewarding to not only know it all works, but to have been able to make it work and piece it up together...

Eh, everyone needs a hobby, or two, or three, or... Oh, whatever, just let it be :)

Actually, if I get really bored, I'll have to setup freevo and MythTV on one, or several of my PCs, to turn them into more tivos (although that would probably also mean that I'd have to shell out some bucks to buy a small and quiet computer, and I have that stupid bet with myself that I have so much hardware already that I don't need to buy more computers (and I haven't bought a computer in more than 5 years, everything I have is from excess spare parts that I put together).
Anyway, I already have a Tivo and ReplayTV, and I don't *really* need to be watching more TV, and I don't really have the time to get bored, but you never know :)

Current Music: Paul Oakenfold - Essential Mix - Live in Havana, Cuba - 1999-02-21
Current Mood: well, I'm pissed off, screw you guys!

It took me more than 3H to upgrade firefox from 0.8 to 1.0, ultimately due to some trace of the old package preventing the new one from installing.
After many hours of tinkering, web searches, filesystem inspections, strace logs and so forth, the fix ended up being to manually run dpkg --purge mozilla-firefox, as apt-get remove wasn't good enough.
Sometimes, linux doesn't rule, not even debian (see, I don't just badmouth windows :)
At least, I still maintain my track record of never having had to re-install linux for any reason (which isn't true of windows)

The conference is going well, it's good to see my international linux hacker buddies again (some actually live close to me, but I only get to see them in Australia and Ottawa, funny that)

Yesterday was the good old Geeks vs Nerds game show. I had ended up on the judge pannel at linuxworld in the past, and was one of the contestants this time, along with ext2 hacker and former coworker Ted T'so. On the other side were 4 contestants form the Ubuntu team I had just had lunch with the previous day to talk shop.
It was a fun game, Jemery Allison as game show host was just great, and among the questions I answered, I made the mistake of giving the correct answer to what the origin of spam was (Monty Python, the meaning of life), and got a nice viking hat to wear while singing the spam song.









As for my talk on fighting spam with Exim and SA-Exim, it went well.

Linuxworld isn't exactly an exciting show anymore, just a sad, shrinking, commercial show (nothing to do with what it used to be in its first seasons like summer 1999, winter 1999, summer 2000, and summer 2001)

Anyway, the main reason I went was that I was invited by my friends Jeremy and Chris to the Golden Penguin Bowl, I really wanted a glass penguin, and it was on my way back from the French consulate in San Francisco where I went to get my passport renewed :) (it also didn't hurt to run into a lot of former coworkers and fellow linux geeks)

We were playing against the microsoft linux team, and while I came with the appropriate geek attire (I was on the geeks team), they came disguised as storm troopers with Darth Vader, it was quite funny. They actually did pretty well; we won a few questions on microsoft, and they won a few questions on linux. In the end, we were slightly ahead, and we ended up winning hands down on the last question which was listing as many Unix and Posix operating systems as we knew. The judges compared it against a list of official answers, and we were about 5 ahead of the other team. I was amazed that they actually accepted my AUX entry, an ancient and obsolete Unix for Apple servers that no one knows about :)

Anyway, that was good time spent. The rest of the pictures are here . Jeremy gets kudos for his great two sides of the force costume :)








The Geeks won by being able to list the most Unix/Posix systems


I've been needing one of those glass penguins for a while :)

Current Mood: Good week
Current Music: Paulway - Telekinesis 1 (from my good Oz friend Paul who just gave his latest mixes to me)


The LCA conference went well. It was nice to see my see once or twice a year buddies from around the world.
The organizers did a fantastic job considering that they weren't many and that their were hosting us in a rather small New Zealand city as opposed to the Capital or Autralia, or some place like Sydney.
Like every year, it was geek heaven, with the usual suspects :)



I got to try my buddy's ever improving augmented reality project (running on linux on a custom built system that fits around the waist)



My talk went all right. I was too busy to really rehearse it, so I just delivered it on the fly, and that wasn't a problem (although I spent a little too long looking at the slides on my laptop since I hadn't memorized it)



Pretty much each evening, we had an organized dinner. Here I am with some buddies from Australia, the US, and Japan



The last evening, there was a big auction for a book (money going to charity), and to up the ante, various programmers offered to do several things if the bid reached a certain level.
In the end, the auction went for NZ$10,000, two programmers lost their facial hair, and one lost his hair since people exceeded the bids by which they promised to get those cut or shaved off.





This ended up being the attraction for that year (last year, they got dunked into a small pool)
Tomorrow morning, the organizers are very nicely offering us (the speakers) a helicopter ride above the city, and I'm flying back to San Francisco after that.

Anyway, good fun was had, and I had a great two weeks away from work :) and you can view the rest of the linux.conf.au 2006 pictures if you wish

OLS has always been a time where I end up updating my laptop's software for some reason. It's probably both because some of the talks aren't that interesting to me, so it gives me something to do :) and if something breaks, I have lots of experts around me to ask for help :)

Anyway, I used the opportunity this year to upgrade:

• from 2.6.14 to 2.6.17, along with all the fun that it implies (suspend to ram and to disk, new ipw2200 wireless driver and underlying softmac support, vmware, with new patches to make it compile)
  
• Did a major upgrade of X.org from 6.8 to 7.0 with all its new pathnames
  
• installed the ATI 3D driver, which wasn't a piece of cake, so as to get google earth for linux working with accelerated 3D (it was unusable otherwise). While researching/debugging the problem, I updated a lot online documentation so that other folks don't have to go through the same hassle has me, including this one
  
• fought the failing hard drive firmware in my laptop to get it limping around during the conference
  

By friday, as the conference is winding down, I have everything apparently working, including suspend to RAM and suspend to disk.

After many years with an old desktop machine (AMD K7 900Mhz engineering sample), I had to admit that my workstation needed an upgrade.
Similarly, my old trusty T42p laptop has not liked travelling too much and the last time I got it fixed at work, I was told it was the last time :)
Since I had just upgraded my ancient original Pentium2 Dual Xeon 450Mhz server (gargamel) last year to a more acceptable Dual P4 Xeon 2.4GHz Hyperthreading, it was time to do my workstation and laptop this time.

For the workstation, I pieced up and built a brand new up to date machine: Dual Core Duo E6850 3GHz with Asus board, and giant tower with 11U, and 12-14 drives.

For the laptop, I got a Thinkpad Z61p: Dual Core Duo T7200 2Ghz with 1920x1200 resolution. The LCD is unfortunately a bit smaller than my T42p, but the pixels are still quite readable for me.

As for the OS, I'm currently switching from an almost 10 year old hand configured/upgraded debian distribution (with a fair amount of upgrade cruft and bugglets) to Ubuntu Fiesty. I'm currently forcing myself to use the default gnome, at least long enough to see what I've been missing, if anything, and whether I can put up with it (the lack of edge scrolling, and pitiful iconbox compared to enlightenment is already annoying me). But eh, it's a learning experience and lets me start from a clean system. Once every 10 years isn't too bad compared to windows (which I had to re-install after being unable to migrate it from my older desktop)

For comparison purposes, here are performance comparison numbers between my old and new new server, laptop, and workstation (doing a kernel build with multiple levels of parallelization (-j))
The new server is almost 20 times faster than my 10 year old server:

Saroumane: Old Server/AMD K6 350Mhz
make -j2
real    488m54.485s
user    431m33.470s
sys     47m49.770s
gargamel: New Server/Dual P4 Xeon 2.4GHz Hyperthreading
make -j4
real    32m31.021s
user    87m58.847s
sys     9m22.060s
make -j2
real    39m36.286s
user    71m30.346s
sys     7m59.697s
no make -j
real    55m19.415s
user    51m24.697s
sys     6m23.695s

The new laptop is 2.5 times faster than the old one, although it'll be a lot more responsive than the old one with the second CPU core:

gandalfthewhite: Old Laptop/Thinkpad T42p/Pentium M 1.80GHz
make -j4
real    38m28.071s
user    33m58.057s
sys     2m34.672s
no make -j
real    38m24.379s
user    33m52.404s
sys     2m32.313s
gandalf: New Laptop/Thinkpad Z61p/Dual Core Duo T7200 2Ghz
make -j4
real    15m54.407s
user    27m49.496s
sys     3m2.067s
make -j2
real    16m11.416s
user    27m26.711s
sys     2m39.974s
no make -j
real    29m9.874s
user    27m16.194s
sys     2m36.370s

That said, the winner is my new workstation, top of the line of current CPU and memory speed (800Mhz DDR2, ok, DDR3 was 3 times the price, which is a bit much). What's nice is that it actually beats gargamel, my 2.4Ghz dual CPU server by a factor of 3. Impressive!
Of course, the best part is that least 15 times faster than my old workstation. I guess the old was, was old :)

moremagic: New Workstation/Dual Core Duo E6850 3GHz
make -j4
real    10m20.318s
user    18m23.385s
sys     1m35.437s
make -j2
real    10m17.382s
user    18m4.710s
sys     1m31.256s
poltron: Old Workstation/AMD K7 900Mhz
make -j2
real    155m13.288s
user    134m36.377s
sys     18m47.572s

The rest is a benchmark of two SATA boards (Silicon Image 3124 vs 3132) through a port multiplier on the faster workstation (moremagic) vs the server (gargamel). Reading speed is 150MB/s and writing speed 120MB/s in raid5 configuration in the fastest setup. Not to shabby :) (it would likely be faster without the port multipliers, but eh, who has 14 sata ports in his workstation? :) )

Setup: 10 drives on 2 5 port PMPs
3.0 Gbps to PMP, 1.5 Gbps to drives
Theory: 375MB/s per PMP/5 drives, or 750MB/s for 2 PMPs/10 drives
Sil 3132:
Out:
moremagic:/mnt/mnt# time dd if=/dev/zero of=zero bs=1M count=10240
10737418240 bytes (11 GB) copied, 89.1388 seconds, 120 MB/s
In:
moremagic:/mnt/mnt# time dd if=zero of=/dev/null bs=1M count=10240
10737418240 bytes (11 GB) copied, 71.5164 seconds, 150 MB/s
Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
moremagic.svh.me 4G 68699  95 116140  28 65026  14 72305  91 145511  17 503.0   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  5704  94 +++++ +++ +++++ +++  6342  99 +++++ +++ 21178 100
Sil 3124:
Out:
moremagic:/mnt/mnt# time dd of=zero if=/dev/zero bs=1M count=10240
10737418240 bytes (11 GB) copied, 147.033 seconds, 73.0 MB/s
In:
moremagic:/mnt/mnt# time dd if=zero of=/dev/null bs=1M count=10240
10737418240 bytes (11 GB) copied, 102.07 seconds, 105 MB/s
Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
moremagic.svh.me 4G 65831  93 72588  18 42041   9 74446  93 102407  12 485.8   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  5508  90 +++++ +++ +++++ +++  6319  99 +++++ +++ 21339 100
Sil 3124 on a Dual P4 Xeon 2.4GHz w/Hyperthreading and just 5 drives on one PMP
gargamel:/mnt/dshelf2# time dd of=zero if=/dev/zero bs=1M count=10240
10737418240 bytes (11 GB) copied, 121.61 seconds, 88.3 MB/s
gargamel:/mnt/dshelf2# time dd if=zero of=/dev/null bs=1M count=10240
10737418240 bytes (11 GB) copied, 126.053 seconds, 85.2 MB/s
Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
gargamel.svh.mer 2G 28435  98 90672  78 31061  23 29877  85 94750  31 365.5   2
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 29058  90 +++++ +++ 32252  99 32538 100 +++++ +++ 27919  87

In the old days, we had ifconfig, dhclient, APM, and things were simple.

First, came ACPI. This is not linux's fault, but boy did it make something simple as putting your laptop to sleep a real pain in the ass sometimes. I'm not sure how many hours I spent learning the acpi system, and getting it work on my thinkpad back before distros made it mostly work in most cases (but seriously APM, just worked, and ACPI was a pain in the ass)
I recently upgraded to a new laptop (thinkpad Z61p), on which I figured I'd put a brand new ubuntu feisty (now upgraded to gutsy), and I'm still running a recent kernel.org kernel instead of the vendor provided one. Maybe I'm getting punished for refusing to run gnome/KDE (I really tried, but gnome still sucks, and KDE still didn't quite do it, so I'm back to enlightenment), but simple things don't work:

• For some obscure reason, Fn+F4 calls acpi_fakekey, which then does nothing (apparently, it might still be talking to the wrong /dev/input/event0), instead of just simply calling the sleep script. Why so complicated? I mean this crap:
  

  cat /etc/acpi/sleepbtn.sh
  #!/bin/bash
  . /usr/share/acpi-support/key-constants
  #acpi_fakekey $KEY_SLEEP
  /etc/acpi/sleep.sh
  

  Seriously, WTF is acpi_fakekey, and why is there no documentation for it?
  
• tpb (thinkpad display) just worked, but was replaced by some complicated hotkey-setup package that does autodetection and still did the wrong thing for my laptop, and still doesn't do anything useful on my laptop with enlightenment (I had to hand re-install tpb, which ubuntu nicely made incompatible with hotkey-setup and ubuntu-desktop)
  
• pulseaudio just did not work due to a misbuild (/tmp/.esd vs /tmp/esd-uid), yielding broken sound laptop-wide
  
• but the best one is by far avahi, dhcdbd, and other network autoconfiguration stuff. Long are the days of simple ifplugd autoconfigure and /etc/network/interfaces is simply empty. Keeping up with all this stuff is starting to be really a mess, especially as documentation there is pretty light too.

I suppose that by the time all this is working, I'll still end up with a better config than what I can do on windows, but damn, it seems like it's getting unnecessarly hard...