Marc's Public Blog

First, the machine never rebooted after I put in the new power supply, nor would it power up with the old one (well, the fans started, but no POST). I eventually gave up and brought the machine home for further diagnostics. I found out in the end that one of the CPU slots on the motherboard donated by benley went bad, and the machine would not boot with any CPU in it (the CPUs themselves still seemed ok).
Luckily, I got an old machine called 'ins1' a while ago, as a spare should something like this happen, so it was just a matter of switching motherboards and CPUs. Good thing I had planned for that.

The part where I screwed up is that I had to replace sda with a new drive that I had prepared. I had 6 drives in the machine and no way to know which one was which outside of a label I had made on the front of the box, for a case just like this. So, I pulled the drive, and put a new one in and rebooted the machine with one CPU. I had meant to boot single user mode, but I messed up the boot command line, and when I tried to sysrq to stop multiuser, it wasn't working and the machine eventually booted in multi user mode and started to write on the degraded raid set. (turns out I had a mini keyboard that didn't support sending sysrq)
It's only a bit later that I logged in and realized that I had pulled the wrong drive and since I had written on the raidset I couldn't just shut down and put the good drive back in without some amount of filesystem corruption (I did have to do this once because I had no choice, but it's not something you do first).
(oh, and it was the wrong drive because during the install, I replaced that sata board for another one, and the other board had its port in reverse order, so my labels were also in reverse order...

By then, I only had once choice left, rebuild on a drive that was already good by using the failing drive, and sure enough the failing drive had bad sectors that prevented the rebuild to complete. I still could have forced the raid to discard the bad drive and rebuild the raidset by forcing options to use the drive I was rebuilding on, as a good drive. It works perfectly if you didn't write on raidset in between, but since I had, I figured I'd try to just clone the bad drive since it only had about 5 bad blocks.

First, I went with dd conv=noerror,sync bs=512, but then googled during the long copy that there was a better way: Gnu ddrescue (don't get confused between that in the older dd_rescue and ddr_help). ddrescue is really mostly the same, except that it copies bigger blocks until it finds and error, had a logfile with recovery, and will retry bad blocks a few times before giving up on them (dd just skips them and replaces them with zeros, which you won't find with with rsync, unless you call rsync with -c and you even know which file(s) have 0s in side, which is very non trivial with a filesystem over lvm over raid5).

The magic command is therefore: ddrescue -v -r 10 -d /dev/sda4 /dev/sdd4 log which takes about 3H on a 250GB drive at 25MB/s average speed.

If ddrescue isn't able to rescue the bad blocks, in theory I should be able to compute the parity for just those blocks from the other drives (including the one I was rebuilding on), hoping/assuming that those blocs weren't ones that got changed in the short amount of time the good drive was removed from the raid. Unfortunately, doing so is pretty non trivial, and there are no tools that I could find to hand pick sectors to rebuild in one direction vs another direction (not counting that it would be super error prone).
The good news is that ddrescue -r 10 was about right: it tried to re-read my bad block 3 times and was able to get the data off the 3rd time, so I got a perfect mirror copy of my drive with issues and won't have to wonder later which portion of which filesystem got a bunch of 0s in the middle of it. Yeah! :)
(the actual data wasn't that important, I had backups of most of it, but it would have been a bit of a pain to recreate, and I always use such an opportunity to learn about the different recovery techniques and tools so that I know what to do the day I come across something very important to restore, hopefully not my data :) )

Windmill from Golden Gate Park

So, I'll give you the answer right away: my 600W power supply wasn't delivering enough power to the drives through the disk array. It's unclear how or why, the said disk array had multiple power connectors, but everything was working fine when I first set it up for power and load, back when I had 250GB drives.
It's only later as I upgraded the drives that the new ones were just a bit too power hungry, and that the disk array had poor power routing, causing some occasional unreliability (i.e. it worked well enough and long enough that I didn't suspect that a power problem had come back). The fix was pretty simple, power each disk array from a different power source (one now uses a molex power strand while the other uses a sata power strand). Just for fun, I'll add that the entire system actually only uses 200W out of its 600W power supply, so it didn't seem obvious at the time (and still isn't), that I was simply overloading one of the power branches, or that the disk arrays really needed more than one connector to be plugged in.

This was really the problem where you can cook a frog by slowly warming up the water it is in. I never noticed that I got into a situation where the power was marginal, because it happened slowly, and I got unclear symptoms: errors on PMP, but I started using PMP back from when it was unstable and errors were common, and I was getting drive failures on my 500GB drives while the 1TB ones were rock solid (on the same power bus, go figure). The worst part is that the seagate drives would develop real bad sectors as a result, so it just looked like PMP wasn't very stable still and that the seagate drives I had were crap (for the record, those drives are still iffy as they do not reallocate bad blocks by themselves, which is not supposed to happen, marginal power or not).
The haha moment was finally when I was testing my 3rd brand "new remanufactured" drive from seagate, that drive was having issues too, even though it only had 2 hours of runtime. Then I noticed with smartctl -HAi /dev/device that the drive had 168 power on events... in 2 hours! Yes, from there I could tell it had been losing power. The rest is history...

I'm happy I finally found the problem, but I must have put 40 hours down the drain over the last 2 years as a result of this power issue :(

Networking was easy to bring up temporarily: I had to bring up the interface by hand and networkmanager would no more see loss of link and bring down eth0, which in turn triggered one of my scripts to bring up wireless on eth1.

The upgrade to Xorg 1.5.3 was supposed to be a good thing, but it made X crash every 10 minutes or so with fglrx (which was nicely upgraded for me), or with the radeonhd driver. I first had to upgrade my kernel to 2.6.27.7 (from 2.6.24) to stop the crashes, and after a fair amount of work, got the radeon and radeonhd drivers working with my mobility firegl V5200 (3d almost works, it just crashes with radeonhd and is very slow with radeon, but when I have time, I'll do some svn pull to get even later drivers and it should work I'm told).
At least the good news is that I'm now running an OSS radeon driver, no more fglrx binary blob. I also get 3D and for the very first time: AIGLX and compositing in enlightenment.

Networkmanager might be supposed to be cool, but totally fucks up your life if you're not using it exactly the way it was intended. I had auto plugging working, that stopped after an upgrade. I had auto switching from wired to wireless (through dhcp scripts) and that stopped too (after networkmanager took over the function of ifplugd), and that stopped working too. After that, I even got networkmanager to just SEGV in protest.

Then, I tried to make networkmanager work, but I soon found out that it's been riddled with bugs and not been playing nice with the rest of the system if you have non standard configs or need to admin some interfaces by hand. Sure, it has an exclude mode, where it will now not even bring the interface down on loss of link (it used to), forcing me to go back to ifplugd or the newer wicd.
The old network manager had null asserts that I reported and were never fixed.
The new one is even worse, it segvs if I manually bring up eth1:

NetworkManager: <info>  Unmanaged Device found; state CONNECTED forced. (see http://bugs.launchpad.net/bugs/191889)
NetworkManager: <WARN>  nm_supplicant_interface_add_cb(): Unexpected supplicant error getting interface: wpa_supplicant couldn't grab this interface.
[1]+  Segmentation fault      NetworkManager --no-daemon

Then I tried starting clean by removing all my interfaces from =/etc/network/interfaces=, and networkmanager refused to manage my interface anyway. It looks like it's one of the many problems that people have been seeing.
I like the fix, which says:

As a workaround removed network-manager
sudo apt-get remove network-manager
And i started my network device with:
sudo ifup eth0
Hope this helps

And then, as I read the pretty light docs with no info on real troubleshooting, or WTF won't it even manage my eth0, I see gems like these:

you may want to restart the system-settings daemon using the command:
"sudo killall nm-system-settings" to apply those changes.

NetworkManager, you're not managing any of my networks anymore. It looks like wicd will do the job, and if not I'll just go back to ifplugd and custom scripts.

Ubuntu folks: you put out a good distro, but your love affair with gnome and utter shite like networkmanager is not making you look good.

The next morning, I wasn't feeling great, but well enough to give driving a shot. I was a little bit apprehensive about being out of it and doing something stupid while driving, but I did well enough. It sure wasn't my most precise or consistent driving. I dropped a couple of wheels off the track once, and the car corrected some imperfect driving more than once, but it went ok in the end.
Unfortunately, I did lose a session due to my new brakes being much harder to change than the old ones (the entire caliper has to be taken apart, which you can't do without special tools I didn't have, and before it's cool to the touch). After I put the new race pads in, I was able to do two more sessions and score some times of 2:08, which isn't bad, but I know the car can still do better.

JBQ, trusty driver and photographer

Can't slide out the pads anymore :(

The big bolts had to be taken out and the caliper split in two

Another prey :)

All in all, while I sure wasn't feeling my top, it was still a good day. More thanks go to JBQ to driving me home after the last session: I was totally dead after my last shot of energy drink was wearing off...

Here is one video for my 3rd session.

So, the first part, the new PC went out ok because I used a bit of brains and threw money at the problem: I'm just too old to fuck around with PC hardware and build my own HTPC case: there are too many things that can not work together, requiring multiple trips to the store to exchange part, take stuff out and back in...
I sent a bid to microcenter, and they actually did a good job building the HTPC. I got a good enough case, was able to get drivers to talk to the front panel LCD, and effectively everything worked except the built in IR port that was hardwired to only talk to a microsoft remote (no thank you). After adding a PVR-350 and wiring its IR receiver, everything worked hardware-wise (Asus P5E-VM HDMI G35, dual core duo 3Ghz, and got the built in intel video chip to work with Xorg. The case is Antec Fusion Black 430 HTPC, which is not small but fairly nice).

Mmmh, and then:

Moving my mythtv setup to work on the new box cost me a lot of lost hair and sleep. This is where the DB in mythtv is a pain in the ass. I had to hand edit the DB to change the IP of my main mythtv server (I didn't even try anything as foolish as renaming the hostname, especially as I called my main myth server, 'myth', making a search/replace in the DB a guaranteed failure).
What happened is that I set a NULL value to my hostname and later fixed it back to be a 'NULL', except that phpmyadmin was nice enough to put the 'NULL' ASCII string instead of NULL, making debug output perplexing since it was effectively looking for NULL and not finding NULL in the DB.
Thanks to Mikal for steering me in the right direction for debugging this after about a week of pulling my hair... After that, everything was working.

This is where a smart person would have quit while he was ahead, but no, I had that 3rd task which was to upgrade the CPU in my old myth box (an AMD Semptron 3100+). I ordered an AMD 4000+ (2.6Ghz instead of 1.8Ghz), the fastest socket 754 upgrade available for that motherboard. I hoped that it would be fast enough to decode H264/1080p, but it turned out not to be that easy to find out.

My old HTPC


The CPU of Doom


One of my many attempts at making it working: a beefier PS from my desktop PC

So the full story took over a month, but basically the new CPU has an integrated memory controller that is very subtly incompatible with my motherboard (I probably have and older bad revision of the hardware).
End result: the CPU works fine if I limit the memory in linux to 252MB. Anything beyond that and it'll crash. Lovely!
(yes, yes, I really tried everything: other memory, other slots, better power supply, memtest, standing on one foot, etc...).
And the best part, kinda? After about a month of trying I did get linux to boot and work with 252MB, and was able to verify that even overclocked at 2.8Ghz, the new CPU can't decode H264 at 1080p anyway (including with the enhanced windows software decoder you pay for).

Boy, I want my 20 hours back!

I specifically enjoyed the talk from John & Martha King about flight safety, giving their own experience of stupid things they did and how it could have cost their life, so that we don't make the same mistakes.
"experience is a tough teacher: she gives the test first, and if you survive, then you get the lesson"

nice little plane to get around

always a nice picture to have :)

Anyway, I'm not sure this was a show to come from around the world to see, but it was still quite a good show, and something not attend if you are somewhat local.

Kudos to South Park for having an animated episode less than 24H after the results, BTW

At least the good news for 2008 is that while Bush #2 is a congenital idiot and a drunk, I at least respect McCain as being a reasonable and sensible man. I might not agree with his republican stance on the economy, or the war in Irak (which is responsible for a big hole in the said economy, even when you put aside all the money that was stolen by Dick Cheney and Halliburton), I really really wish that McCain had been the republican candidate for the last 8 years if we had to have a republican for the last 8 years. I'm somehow pretty sure that he would have done a much better job. That said, because of his political party and campaign finance money that he had to accept, even if he is a respectable straight up person, I'm worried of how much he would be able to do to fix the economy compared to Barrack without angering people and corporations who financed his campaign, but it sure can't be worse than Bush.
In a not so funny way, it's actually worrying how many Americans don't know or care that their president is seen as who he is, a dangerous blistering idiot by the rest of the world, and for as long as I've known US presidents, including when I didn't live in the US, this had never happened for prior presidents. Some were more liked than others, but none was so universally despised and disliked as Bush #2 (people in other countries actually occasionally give me crap for my living in the US under such a terrible president).
Another thing that too many people don't seem to realize here is that this two party system actually forces political candidates to pick one of two buckets, and that bucket is likely not to fit all their beliefs (like some republicans are actually fairly liberal, but republican financially, just like some democrats can have republicans financial tendencies), so in the end the party they picked is just something they had to affiliate themselves to, but it means that it's more important to look at the candidate him/herself than his/her party affiliation.
To illustrate my point, California, which is supposed to be a democratic state, actually impeached its democratic governor, and replaced him with a republican: Arnold Schwarzenegger. Arnold is not much of a liberal, he's not exactly a fan of gay rights, and even took steps to prevent gay marriages, but on the other side of the fence, he's been doing a better job running the state than the previous democrat guy (and gay marriages were fought and won in the CA supreme court). This is how things are really supposed to work. Hopefully it'll work on the federal scale too...

So, I'll be celebrating either way tomorrow because it'll be the end of Bush either way. I'm just secretly hoping for Obama because if McCain wins and then dies half way through his mandate, we'll be severely screwed with Palin: she looks even more stupid and ignorant than Bush, which I didn't think was possible (that said, you could make the same point about Obama if he gets shot by a racist, as his VP doesn't seem like a great guy either).

BTW, for those who thought by now that I was a democrat loving liberal socialist hippie, I'm more of a right wing liberal, which is something between democrat and republican and that doesn't exist in the US.

Doh! I guess I won't have my picture with the president afterall :)

Democrats would definitely raise my taxes if they win and I will personally lose out. I'm really for the more reasonable guy, whoever that happens to be at the time and would have likely picked McCain over a seemingly useless democrat candidate. Either way, it doesn't matter, both because the electoral system is broken with the electoral college and individual californian votes don't count since it's a democrat state in the end either way, and because I'm not a citizen so I don't get to vote either way :)

From: Marc MERLIN 
To: david_cohen =at= comcast.com, ralph_roberts =at= comcast.com,
        marlene_dooner =at= comcast.com, payne_brown =at= comcast.com,
        kerry_knott =at= comcast.com, esl_corp =at= cable.comcast.com,
        john_schanz =at= cable.comcast.com, john_ridall =at= cable.comcast.com,
        brian_roberts =at= comcast.com, joe_waz =at= comcast.com,
        audit_committee_chairman =at= comcast.com, darcy_rudnay =at= comcast.com,
        corporate_communications =at= comcast.com, steve_burke =at= cable.comcast.com

Dear Madam, Dear Sir,

I realize that at this point you have likely already made the decision of
appealing to the lowest common denominator customer and that angering most
of the techie population in the Silicon Valley may just be an acceptable
loss to you.
That said, if I am mistaken, I'll tell you what you are doing blatantly
wrong and what you can do about it. I am also going to warn you about the
problems with your HDTV service, and how you are very likely to get a class
action lawsuit if you continue the deployment of 5c encryption on your
channels, preventing recording through firewire (more details on that later)

Executive Summary:
- I'm very close to dumping your TV service because $67/month to watch
  basic broadcast channels plus 4 cable channels is both a ripoff and you're
  making it increasingly harder for me to record digital channels. But just
  the fact that I'm paying $12-ish for broadcast channels and $55 for a mere
  2 channels I want on top of that is very bad pricing on your part and is
  causing people like me to jump ship. I don't care if you add channels, I
  care about how much I pay for the ones I want.

  Also, as soon as you remove your channels tunable without a cable box
  (i.e. basic analog cable) and continue to prevent recording digital
  channels tuned through your unnecessary but yet required cable box, you
  will become entirely unattractive and more expensive compared to satellite
  or IPTV solutions.

- Your cable internet service, while reasonably priced while still on the
  package pricing, has been somewhat unreliable and plagued with the most
  infuriating clueless tech support I've ever worked with (not counting
  when you mistakenly shut me off and treated me like a criminal by putting
  me in the copyright infringer pile due to a temporary overusage due to a
  bug.


Phone Service
-------------
Honestly, considering the experience I had with your internet service
sometimes going down for a day or more and being told that I didn't have
business service and that I had no SLA, and that I'd probably get 1 or 2
dollars back a the end of the month to make up for the fact that I had no
internet connectivity for over 2 days (it was a city wide problem), is why I
would never consider phone service from you (and quite frankly no tech savy
person I know in the Silicon Valley has your phone service either).

TV Service
----------
1) you make TV hard to get to without your equipment
----------------------------------------------------
To your credit, that's the one thing that has mostly worked in the 12 years
I've been your customer for analog service.
Why Analog? Because with multiple TVs and 4 VCRs back then and now multiple
DVRs (not yours, mine, self built), cable had the big advantage of allowing
multiple recording streams without extra decoding hardware.

Then came digital cable, which I ignored since it required a cable box per
channel to view and was a major pain the butt to link to my VCRs and DVRs.
But now, with HDTV, I have a bigger problem. I can thankfully tune the HD
broadcast channels that you are required to carry unencrypted by law, but
I am quite antagonized by the fact that you require me to lease a cable box
I neither need nor want to tune HD channels like SCIFI-HD or Discovery-HD.

I had to beg to get an HDTV DCH-3200 cable box, you didn't even have them at
your local branch and they had no phone number for me to call before driving
there to see if they received new ones (which they didn't for several
weeks). Through much work, I was finally able to tune and record just one
channel at a time through the firewire port to my DVR in HD.
However, some of my fellow engineers who managed to do that in other markets
are increasingly getting booted off as you turn on 5c encryption on all your
channels and prevent people from recording channels that they paid for.

You are probably aware that The FCC has passed a regulation that if I am are
in the United States, and you have a HD subscription and a HD cable box,
you have to honor my request replace or upgrade your cable box with working
FireWire.
http://hraunfoss.fcc.gov/edocs_public/attachmatch/FCC-03-225A1.pdf
Page 50, section 4
    (4) Cable operators shall:
        (i) Effective April 1, 2004, upon request of a customer, replace any
        leased high definition set-top box, which does not include a
        functional IEEE 1394 interface, with one that includes a functional
        IEEE 1394 interface or upgrade the customer's set-top box by
        download or other means to ensure that the IEEE 1394 interface is
        functional.

I'm pretty sure you will lose in a class action suit when the people go
after you for effectively disabling the firewire connection by encrypting
all signals on it so that they are useless.
If you were going to tell me I should be using your PVR, don't bother. First
your PVR is slow and bad, second it doesn't share programs around the house
between TVs, and 3rd I shouldn't have to buy or lease your PVR when I have
my own that's both better and that I can program to do what I want.


2) you charge a fortune for basic cable
---------------------------------------
More specifically, you do not sell basic cable or anything a la carte. You
force me to pay $67/month for the most basic HDTV package I can get.
If I get an HD antenna and put it on my roof, I'll only be missing 2
channels: Discovery and Comedy Central.
I can now watch my Comedy Central programs streamed from their web site,
and Discovery is also making their programs downloadable from their web
site.
Pretty much everything else is on http://www.hulu.com/ or is being added
soon. My home built DVR is of course internet enabled and can stream
directly to my TV, replacing your cable box and your service as a whole.

So, please tell me why I'm currently paying you $55/month for 2 channels
that I can pretty much watch on the internet anyway?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^


Cable Internet
--------------
I'm not going to mention the shameful pretense of unlimited internet when
you had secret caps for so long, as well as traffic shaping (I can't even
blame you for them, but I do blame you for hiding them and lying about
them), or how you treated me like a criminal in error and that I had to use
my fiancee's name at my old address before I could get service back, since
both of those are past (it's another long story that I don't feel like
telling anymore).

However, I will mention that you have the worst tech support I've ever
worked with. I'm a unix sysadmin and network engineer; I know when your
service is down probably faster than you do and I do not take kindly of
having a support monkey telling me to reboot my windows machine (I do not
have windows) when I point out that I do get a DHCP address from you, that I
released it and asked for another lease, but that you only give me a
192.168 IP because at least your local service is down.
I only used your service as a backup for my faster DSL connection (although
it's indeed not cheaper than you), and now that my dual line DSL has been
working stably for long enough without any weird caps or traffic shaping,
I'm happy that I'll be able to put behind me the stress and bad experience
that having your internet service, was.
As general feedback, no one technical that I know in the silicon valley
has comcast internet, unless they are in an area where it's impossible to
get DSL.
And by far, the worst part is when you have to call support and you are
forced to listen to those forced down your throat, infuriating adds for PPV
boxing or whatever that you couldn't care less about because you're calling
about your cable internet being down (that's just after having a clueless
tech who may just hang up on you when you tell them that you aren't using
windows).


In a nutshell
-------------
As you can tell by now, while I've been your customer for as long as I've
lived here, I've been nothing but unhappy with my interactions with you,
your on the fly pricing that goes away, fake charges that are put on your
bill for equipment you don't have or didn't ask for, and the hours wasted
arguing on the phone.
Your internet service has now become irrelevant, and while you had a
technical lead on the TV side by having almost unlimited concurrent streams
of channels on a single cable, you've managed to remove that advantage by
forcing needless cable boxes and encrypting channels to make sure that they
are no more watchable without one STB per recorder or TV, just like
satellite (except that satellite is both cheaper, and does have more HD
channels and in higher quality than you).
Before long, or maybe even as of now, IPTV and internet watchable shows are
going to make people a lot less likely to pay you a ridiculous $50-$60 just
to watch the few channels they care about and lease a cable box they neither
need, nor want (recent TVs and PVRs can tune QAM256 and 8VSB digital
channels without the help of an external STB/cable box. Mine can, and I'm
pissed that you're encrypting your channels for cripple the usability of my
devices).

If you have a reasonable price you can offer me for basic HD cable, you may
keep me as a customer assuming you never turn on 5C encryption in the
silicon valley (the day you do that is likely the day you'll have that class
action suit on your hands), but otherwise this hour I spent to give you this
feedback in the hopes that someone who still cares, is reading this, is the
last hour I will spend with you.

Marc Merlin.