Marc's Public Blog - Linux Hacking

1. it can't be removed and it can't be turned off.

2. when booting with --verbose INIT_VERBOSE=true, plymouth still starts in text mode and writes useless progress dots on my screen that prevent me from seeing the debugging I need. Not only I can't remove it, but I can't even seem to stop it from spamming my screen :(

3. plymouth is not finished, it SEGVs and requires a very recent kernel with devtmpfs or the system won't boot at all (!). Many people can't just upgrade to the latest kernel, especially on servers (in this case the forced kernel upgrade forced me to upgrade lirc which is in turn broken in lucid. Some servers come with drivers that don't yet work on recent kernels; sad but still a fact of life).

4. like network-manager or upstart when they were added to ubuntu, there is virtually no useful debugging info on how to deal with this new system and how to remove/bypass it if it fails badly (which it does). By comparison, when RH introduced /etc/rc.d back in 1995 (yes, a while ago), they had a document explaining why it was better than rc.local, how it worked and how to debug problems.

In a nutshell, plymouth adds absolutely no functionality I need whatsoever and makes me want to stab ubuntu with something long and sharp because of how much of a pain in the ass it is and how much in the way it gets. It cost me many hours of painful and otherwise unnecessary debugging where I could hardly debug because of plymouth itself.
(upstart can be a pain for debugging too, but at least faster parallel booting is a good improvement, so I'll suck it up).

Of all things, if canonical only had 2 release requirements:

bonus:

Unfortunately, canonical failed 3 times on those points with at least network-manager, upstart and now plymouth.
I'm just much more angry here because plymouth is a useless piece of crap that only prevents me from booting/debugging my systems right now :(

So, like all failed upgrades, this started with a reasonably working system. A bit like getting a newer dd-wrt version: because it felt like a good idea at the time, I thought I'd jump from mythtv 0.21 to 0.23RC2 all at once.
I never went to 0.22, because quite frankly my mythtv box was working and I didn't see the point of mucking with it for features I didn't seem to need. But I was having some playback studdering problems with mythfrontend, they were getting annoying and they were apparently fixed in recent mythtv versions.

The problem was that 0.23 was only available for lucid and maybe karmic while I was still running jaunty on my mythtv (in hindsight, I should have compiled it myself for jaunty, but I figured upgrading the base OS from time to time couldn't be that bad). That said, I still remember the upgrade to jaunty where ubuntu pulled off some custom kernel patch where the base system didn't boot with a standard kernel.org kernel when you had the evms package (some obscure problem with mounting lvm devices more than once).

My plan was to upgrade as little of the system as necessary to get the mythtv packages working. Of course, I knew it might not be that simple due to the switch to upstart but I figured it'd be better than upgrading everything and risking to break a lot of things (including X, lirc, and sound, which can be interesting to tweak on a mythtv machine). In the end, that likely made no difference since the problem was mostly with plymouth.

Well, of course, it went about as badly as it could: machine didn't reboot for multiple reasons, one being that plymouth would SEGV in mountall (something that the average user would already be hard pressed to find).
Quite frankly, I think it sucks that ubuntu felt it was ok to require an experimental and somewhat controversial devtmpfs in a very recent kernel with no warning at upgrade time that the said kernel is required, and no dpkg dependency. If plymouth must absolutely depend on 2.6.32, it really should put out a warning at upgrade time that 2.6.32 and devtmpfs are required. And come on, it was non trivial for me to even find the problem and that it required devtmpfs to start with.

About upstart: I know ubuntu means well with upstart, but man does this thing not work well, at least on upgrades. My fully dist-upgraded karmic laptop still does not reliably boot due to a (race condition in upstart, bug 499361), and lucid turned out to be worse on my mythtv: it somehow managed to start init scripts and things that mounted all my filesystems before fscking them in mountall, which then caused mountall to silently fail without being able to give me the usual sulogin shell, likely because of plymouth.

As others have already echoed, plymouth looks like half baked still, and when it fails, it gives you a broken system that it super hard to debug :-/

For debugging, I knew about --debug and nosplash, but they didn't help. Andrew told me about additional --verbose INIT_VERBOSE=true which I can't find documented anywhere and they didn't seem to do anything useful with the ubuntu kernel. Strangely enough when I switched to my kernel, I started getting stuff written on the screen although several processes were overlaying one another, it wasn't pretty...

I'm afraid to state my experience with Ubuntu lucid, upstart, and now especially plymouth:
Debugging a boot has become almost as hard as windows. Ubuntu, you're done: you've made the linux desktop as invasive as you could and almost matched windows on the "suckiness of debugging" scale.
As for dependencies ending up so broken that mountall in rcS.d runs _after_ my partitions have been mounted and therefore fails and hangs the boot with no prompt I can see due to plymouth brokenness, that's just plain bad.

The best part is that you can't even fix/debug a non booting system with init=/bin/bash so much because anything in /etc/init won't start if upstart wasn't launched as pid 1 (which is bash in this case). This "progress" really does not make debugging systems easier :(
I also hate the fact where fsck is hidden and where the boot is just looking like it's not doing anything spinning the ubuntu dots while fsck is actually doing something or has silently failed or hung somewhere else that is not necessarily visible.

Is this really an improvement from the old known good text boot?
(maybe it's supposed to work better than that, but considering how badly it's capable of failing then, I don't really care).

About the switch to fbconsole: killing X gives me an unusable flashing text console. It might be fixable, but is that really an improvement as a default: something I can't debug when it goes wrong?

For now, after rebuilding a customer kernel in the hopes that I could boot and that lirc would work (I can only boot unfortunately, no lirc yet despite new versions and module rebuilds), the system almost boots but gdm won't auto start. Upstart of course makes debugging and fixing this non fun :-
Of course, the problem here is that it's easy to just criticize other people's work without contributing. That's always been true: when you complain a piece of open source software doing something questionable or plain wrong, that's always an issue. At least, I pay my "dues" by making plenty of contributions myself (albeit not to ubuntu per se outside of the occasional detailed bug report) and I'm not making a big fuss about how my alsa sound got muted to 0 (master volume) after the upgrade: it's not hard for me to fix, even if the less technical end user all this extra polish is supposed to be for, might not fix this quickly.

Ubuntu/Canonical, I switched from Debian to you because you were just plain better. You made me reconsider this with the network-manager fiasco that just broke my networking for a whole 2 years before it got finally fixed, and after this now pretty nightmarish upgrade that burned almost a weekend on getting my mythtv server working again, I'm really starting to reconsider my decision. Please remember that less can be more, and that more wiz bang polish is just a bad idea when it's a the expense of not having a simple system to understand and debug anymore when things go wrong (and will go wrong when you give it to end users).

I was also looking at upgrading bunch of servers at work to lucid (from something much older), but now that I've seen plymouth and upstart dependencies that still don't work and are a pain in the ass to debug (hell, you never fixed/figured out why they still don't work on my laptop 6 months later, aka bug 499361), I am really reconsidering trusting servers with ubuntu karmic or newer now (aka pre/post-upstart).
(quick update, upstart has gotten some debugging docs put up now. They weren't there when I really needed them, just like NetworkManager,

Ubuntu/Canonical, I switched from Red Hat to Debian because Red Hat was focussing so much on the desktop that it started sucking on servers (at the time). You are now doing the same from where I stand: I don't really care about all that time wasted on which corner of the window the X will be, I don't even use gnome anyway, but don't lose focus on the server: this is what Linux is good at and if I can't trust Ubuntu for servers anymore (after my experience on my mythtv and broken/hard to debug upstart network dependencies that just hang boot on my laptop, I can't see myself putting lucid on servers), I am reconsidering Debian now: last I checked they still considered that for the server simpler is better.
Also, if you introduce bold new systems which quite frankly aren't going to work quite right for a while, make sure easy to find documentation on how to debug them/work around them is posted in the release notes and easy to find places on the ubuntu site (being in search indexes for "foo debugging" is of course pretty much a must too). And on the mountall/plymouth front, if it's as bad as still getting SEGVs that just stop the boot very early (as of lucid beta2), wouldn't it be wiser to make plymouth a removable non required component? Yes, it's more work but it's also the safer thing to do and better for the users for whom plymouth either doesn't work or is something they could really do without.

Now back to trying to fix lirc on my mythtv as it's been broken for a week after the upgrade (after 3 kernel recompiles and 2 upgrades of lirc userland stuff), but that's likely not ubuntu's fault but more likely the random suckiness that comes with upgrades and the fact that lirc still hasn't been deemed kernel merging worthy yet.
*update* lirc was actually broken as shipped in lucid, I filed this bug. Man, lucid was a painful upgrade for my mythtv...