Dear google, please index this:
Some old python-central package and compat breakage I wasn't interested in debugging, caused upgrades and uninstalls to fail. Causing a complex upgrade to fail in the middle due to a failed package removal on a pre-uninstall failing on pycentral, broke the system quite badly.
I restored it from a brrfs snapshot and then disabled pycentral (/bin/true) to allow its removal and apt-get -f install to suceed. After that, upgrades were possible again.
Removing gnome-codec-install (0.4.7+nmu1) ...
File "/usr/bin/pycentral", line 87
raise ValueError, 'unknown version info %s' % vinfo
Removing python-gst0.10 (0.10.22-2) ...
File "/usr/bin/pycentral", line 87
raise ValueError, 'unknown version info %s' % vinfo
^
SyntaxError: invalid syntax
dpkg: error processing package python-gst0.10 (--remove):
installed python-gst0.10 package pre-removal script subprocess returned error exit status 1
dpkg: too many errors, stopping
File "/usr/bin/pycentral", line 87
raise ValueError, 'unknown version info %s' % vinfo
saruman:~# mv /usr/bin/pycentral /usr/bin/pycentral.orig
saruman:~# ln -s /bin/true /usr/bin/pycentral
saruman:~# ln dpkg --remove python-central python-gst0.10 python-bluez gnome-codec-install bzrtools
(Reading database ... 503974 files and directories currently installed.)
Removing python-bluez (0.18-1+b2) ...
dpkg: warning: while removing python-bluez, directory '/usr/lib/python2.6/dist-packages/bluetooth' not empty so not removed
dpkg: warning: while removing python-bluez, directory '/usr/lib/python2.7/dist-packages/bluetooth' not empty so not removed
Removing gnome-codec-install (0.4.7+nmu1) ...
Removing bzrtools (2.3.1-1) ...
Removing python-gst0.10 (0.10.22-2) ...
dpkg: warning: while removing python-gst0.10, directory '/usr/lib/python2.6/dist-packages/gst-0.10/gst/extend' not empty so not removed
dpkg: warning: while removing python-gst0.10, directory '/usr/lib/python2.7/dist-packages/gst-0.10/gst/extend' not empty so not removed
Removing python-central (0.6.14+nmu2) ...
Processing triggers for hicolor-icon-theme (0.15-1) ...
Processing triggers for man-db (2.7.0.2-4) ...
My own "I'm just going to spend 5mn to fix this" story, that lasted over 6H.
Power outage rebooted one of my energy monitors and it's now counting one of my solar arrays backwards because of an apparent bit flip in the flash (it's an old device)
10 days later I'm finally home long enough to look at this, I "just" need to connect to it via some windows only app, connect via the serial port and reprogram the lost/corrupted settings
Ah, but my only desktop/windows machine left in the house blew a capacitor on its motherboard recently, and I haven't been able to justify the time to fix it (that would be a rabbit hole on its own and I know this), so I can't use that
unfortunately that one device only works with a real serial port, not any USB-serial adapters I've been able to try (and I have 4 different brands/chips), so that prevents use of any of my laptops
oh, wait, I do have a much older thinkpad with a docking station that has a _real_ serial port, let's try this.
ok, it boots, but the windows XP partition gives some "NTLDR missing". I haven't used it in some 15 years, so I wouldn't know
ok, let's boot linux, ubuntu oneiric, oh my. Virtualbox won't work, modules are broken, I try a few of the kernels, a mix of 32bit kernels with himem for the 8GB of RAM, and some x64 kernels with 32bit userspace. None of them want to build working virtualbox modules.
apt-get install is of course broken with a weird dependency chain, not going to go to _that_ rabbithole either (see, I've learned from previous times)
maybe I can make that WXP partition boot again? Ok, I'll skip details on how that ends up in cleaning half my bedroom while looking for old boot/recovery CDs (BartPE, UBCD, etc...), another 1.5h rabbithole
eventually get the boot CDs, they also BSOD during boot, look in bios, change the SATA support from AHCI to compatible, that fixes it
fast forward through some "details", boot WXP recovery console, apply "fixboot" and "fixmbr" which of course removes the install-mbr bootloader I had installed to dual boot WXP and llinux, and was not present on my linux recovery CDs
eventually boot some linux recovery that allows me to decrypt the linux partition (luksopen), chroot into it because it's not the same binary format, and install-mbr -t 54 -v -e 14FA -p 2 -is /dev/sda
eventually get WXP to boot, for some reason the wireless support does not work anymore and can't connect. Ok, use ethernet instead and everything is so old that none fhe web browsers work, because everything is https and the ciphers are all incompatible with older browsers now
retrieve the device windows software from my normal laptop, copy it over, install it, it runs, uses the real serial port on the ultrabay, and connects to my power monitor. I revert the broken settings on the device over serial, and save them.
SUCCESS!
That only took about 6H, why it it dark outside? Oh, it's past midnight. What was I doing before that?
Well, this was interesting: a hybrid remote/in person conference. About half the speakers were presenting remotely and the other half (or was it third?) went in person to the Seattle Hyatt where the conference was held. I was one of them.
There were not that many attendees, I think 4/5th were remote, so most speakers (including myself) gave talks to a handful of in person attendees, it was a bit weird, but hopefully it worked. The conference required everyone to be vaccinated or have a negative test every day (and they checked). Given that, I found that requiring masking the entire time was a bit over the top, and annoying for speaking to others, but so be it:
Jim Zemlin opened the conference as always:
The various keynotes were good:
Linus and Dirk came in person for their usual firechat
Most speakers, including myself, gave talks in breakout rooms during the 3 days:
My talk went well, despite the low in person attendance:
video of the talk:
And of course we had a vendor floor and per person wrapped snacks (per covid rules):
While I wrote this for my Lenovo Thinkpad P73, this is likely equivally relevant to P53, P72, and P52.
Thinkpad P73 vs P70, not a win all around: only one 2.5" drive instead of 2, and a badly designed power system
So, when lenovo came out with the Thinkpad P70, I wasn't very happy because if you had a 90W power supply, it refused to charge from it, at any rate whatsoever. I was not impressed, but eh, at least it would still power the laptop so that its batteries didn't go flat while plugged in.
Well, leave it to lenovo engineers to make things worse the for the P73. The minimum power supply was raised from 135W (170W recommended) to 170W (230W recommended) which is understandable, but lenovo decided to ensure that the laptop will not take any power from any power supply that does not identify itself as 170W or more. This means that even ifit only needs 40W to sustain itself without digging into the batteries, it will completely refuse to use a 90W or even 135W power supply for anything at all, and kill the battery instead. Lenovo, you just plain suck, there is no excuse for this.
P70 vs P73, they look pretty similar
*Update*: it seems if that if you get a cheaper nvidia chip with the P73, it is then configured to accept 135W power supplies as the minimum required. That said, it will still refuse to work with any regular 90W power supply or external battery back, unless you force it with the center pin resistor swap.
While I have no plans to use windows on that machine, I thought I'd just try it out to see how it does on power. This is where I was impressed, windows can idle at less than 10W for more than 11H runtime, while I'm lucky if I can get linux at 15W. This is definitely a place where linux should do better, of course, it's not as if Lenovo put any work into making linux more efficient on their hardware either:
*Update* : with tlp and using the nouveau driver just enough to turn off the nvidia chip, I'm now able to tune the laptop down to 10W, almost matching windows.
See tlp issue 494 for details on how to setup tlp to run in low power mode when power is plugged in.
Another disappointment is that the P73 is mostly the same size and weight than the P70, but it has less room for storage. It has a bunch of empty spaces that aren't used for anything, and it can't use two 2.5" SATA drives anymore, like the P70 could. Worse, the now single 2.5" slot uses a lenovo only ribbon cable that does not ship with the laptop and basically means you cannot even add a 2.5" drive without that special ribbon cable, which isn't in stock yet. Well done... Ah yes, the battery is also not hot swappable, even if it is replaceable (unlike a Mac laptop where everything is sealed shut).
this shows the unobtanium lenovo cable for the now only single drive that fits, along the unused space
Ok, stop complaining, just buy a bunch of 170w or 230W power supplies and move on with your life
Well, yes and no:
I literally have 10 power supplies between home and work, not really looking at replacing all 10. Lenovo wants $137 per power supply by the way, even if they are $85 from other sellers
those 170W/230W power supplies are huge. They also weigh as much as some small notebooks (!)
I have 12V to 20V car adapters, those won't work anymore
I have external battery packs for extended runtime, and I haven't found a single one that can deliver the amps a P73 tries to needlessly require
Tricking the P70 and P73 into accepting a power supply it wouldn't otherwise use
Lenovo uses the a center pin resistor to know how much power they can draw from the power supply, see: http://www.thinkwiki.org/wiki/Power_Connector .
For the P70, I built this power supply adapter with a resistor bridge to tell the laptop how big it should think the power supply, is:
It's basically a configurable version of this. Yes, lenovo, I thank you for the hours I wasted opening up power supply plugs and replacing the center pin resistors:
Here's how the P73 responds:
- 230W works fine 4.6k => I have seen power supplies work with 170W but fail at 230W, so the laptop does draw more
- 170W works ok 1.9k (1.8k also ok)
- 130W rejected 1k
- 90W rejected 550
The rejected power supplies will be used to charge the laptop if it is shutdown, but they will not be used in any way otherwise. On the P70 the laptop would at least use the power supply to keep the laptop alive, and use half battery half external power supply. Not so with the P73, it just ignores it entirely.
This is utter bullshit as I have plenty of 90W power supplies, including 12V car converters, or a 90W external 20V battery pack I can't use anymore.
You can go read my Hacking a thinkpad slim tip adapter to output more than 90W (required to charge a Thinkpad P70) page for details, including this nice battery pack I couldn't use anymore:
*Update* : so, actually with some serious tlp hacking (basically I told it to force battery mode even if a power supply is plugged in), I've managed to throttle the laptop enough, even when plugged in, so that it only uses 20W. At that point, I'm actually able to use my old external battery pack, as well as a 90W power supply, as long as I lie to the laptop and pretend they are 230W power supplies with the resistor trick. In my tests with windows, it was not possible to throttle the laptop enough when plugged in, not to have it overwhelm a smaller power supply (not that 90W is small for a laptop that normally uses 20-30W when it's not charging batteries).
If you scroll to the bottom of the page, you'll also see a terrible buffer lipo hardware hack I did that allows to use the battery pack with higher amp draws, but it's a bit ridiculous (and bulky).
See tlp issue 494 for details on how to setup tlp to run in low power mode when power is plugged in.
Without the tlp hack or the buffer lipo hack, when I lie to the laptop and tell it is connected to a a bigger power supply, manage power draw with what I run, and disable battery charging in software, but the laptop will still draw the power supply for over 100W when you plug it in for a fraction of a second, and refuse it if the voltage drops.
Obviously this would not be a problem if the laptop simply had a 90W power supply mode where it throttle things down and turned off battery charging. This is mostly what the P70 does.
In the meantme, on top of hacking my power supplies, I also made this for my laptop, it looks silly and makes the thinkpad not look like a professional laptop, but well, that's lenovo's fault:
this takes any power supply and replaces the center tip resistor with a 1.9k one to emulate a 170W power supply
From talking to Lenovo, they don't think that this is really a problem, so since I'm an engineer, I made my own external battery pack, but I otherwise recommend to road warriors to avoid thinkpads from now on, given the backward power design in this one.
Making a Thinkpad P73 compatible external battery pack
I did some testing and confirmed that the laptop is very picky about power supplies. It even rejects a 19.7V 20A power supply I had, because it's 19.8V and not 20V. Same thing for amps, it needs to be able to draw maybe around 5A for a short time to accept the power supply (they sure are putting a lot of effort into making sure the power supply is not under-spec'ed).
Prototype with 150W step up converter which takes my 16V lipo to 20V while delivering enough amps to make the laptop happy:
it works, and the laptop thinks it's connected to a 230W power supply thanks to the center pin resistor.
Here's a quick demo:
Version 2 was to have a way to recharge the battery pack while it's being used. I've used this to use the battery pack as a buffer to absorb peaks from the laptop without tripping an external power supply, including in a car limited to 100W or a plane power supply limited to even less:
this works in theory, but the lipo charger is quite slow and wouldn't keep up for long, but I made a better version shown lower down
Lenovo's P73 airplane mode simply stops using the external power supply and reverts to batteries. Sigh...
Oh yes, let's talk about airplane mode. The lenovo engineers thought of everything: if they detect that the power supply drops a few times in a row, they offer a nice setting which is supposed to make the plane more airplane friendly. How friendly you ask? Well, you could throttle down the CPUs, disable battery charging, do smart stuff like that. Or, if you're lenovo, you can have airplane mode simply refuse to use the power supply altogether. Thank you lenovo, you wrote a feature that saves me the trouble of unplugging an otherwise perfectly good power supply that you refuse to use (to be super clear, my 230W power supply is plugged in and airplane mode just disabled it):
Making a battery pack to act both as buffer for a smaller power supply, and as emergency external power (even power the laptop from 12v)
Anyway, back to the battery pack, I found the ISDT H605 Air lipo charger which is small enough and can charge the lipo at 5A, which should be enough to keep up with the laptop when not doing CPU crazy stuff. This also allows using a 12V power supply or a lower wattage lenovo power supply to recharge the pack while it's in use, or not:
version 1 was a bit bigger than I wanted, 90W power supply shown for scale
here, the 90W power supply is recharging the battery at 2.5A while it's being discharged at 3A on the output side, using the battery as buffer
I made version 3 a bit smaller, with a built in 12V lipo to act as buffer for a smaller power supply. Yes, it's a beautiful piece of art, I know :)
Pushed to the extreme, I can now use my original external battery pack again by having it recharge my lipo+150W step up that can output more amps than the ravpower pack can. Of couse, it's inefficient, the ravpower pack outputs 20V that gets down converted to 12V by the H605 Air lipo charger, which charges the built in 3S lipo in the box, and then gets up converted back to 20V without the amp limitation (the phone used to control the lipo charger also inside the box):
The really cool thing is that by using tlp, it's actually possible to tune the laptop down to very low power use, even when plugged in, something that windows probably can't do:
7.3W with the screen off (and around 10W with wifi off and the screen on low dim) is not bad for a laptop that big
4S Lipo vs 4x 18650 or 26650 batteries
I do have a few lipos laying around, so that's free energy for my laptop if I'm willing to carry them. I have however found that for higher draws, the step up converter doesn't quite keep up at 20V/5A+ with just 12V input (3S), but is fine with 16V input (4S):
That said, as I recently found out that 16650 (or better 26650) batteries are both lighter and smaller. The only thing the lipos do, is offer a better discharge rate, but while that's useful for a high power RC plane motor that can empty the batteries in 10mn, it's not needed for a laptop:
Outside of 26650 batteries, there are other ones like https://www.18650batterystore.com/21700-p/samsung-50e.htm .
26650s are 195 Wh/kg while the lipo I gave was 184Wh/kg. The 3rd battery listed is supposed to be 260Wh/kg which is much nicer, that said, it looks like those samsung batteries are actually smaller than 26650s, lighter, and yet offer the same 5Ah at 3.6V. If so, that's very impressive.
As of this writing, I have however not found 26650 battery holders that hold 26650 protected cells that are a bit longer. This seems to the be only one available, and it's too short to hold the batteries: https://www.amazon.com/gp/product/B074GVPWSH
A few months later, I ended up cutting them and taping them to make them longer. It doesn't work great, but it was enough for a few tests. In the end, I found that somehow I wasn't getting enough amps out of them, which surprised me, so they didn't work that well compared to a lipo:
Conclusion
Lenovo, please make the P73 work like the P70, and fix this airplane mode thing that turns off the external power supply. That's embarrassing...
As part of Linux.conf.au 2020, I gave my main talk Planning for and handling failures from open hardware, aviation, to production at Google.
The talk focussed on failures I've encountered in multiple fields, and what I learned from reading from other people's failures, a common practise in aviation that has saved countless lives in not re-creating failures and accidents out of ignorance.
As they say in aviation "experience is a cruel teacher: she gives you the test first, and if you survive, then you get to learn the lesson".
After looking at examples of failures we experienced at Google, I give a fair amount of examples from aviation, from AF447, QF32, and the Boeing 737 Max disaster which is so many failures in so many ways that it takes a while to describe in details. My hope is that engineers in a similar situation where they know they are getting overruled, can use other escallation steps to avoid disaster for others.
You can get the talk PDF or openoffice source from here. Otherwise, you can read the slides below and watch the video recording:
I've been going to linux.conf.au for 18 years now (since 2001), and presented a fair amount of linux talks related there, but the big change for me was the open hardware miniconf that started in 2010. Thanks to its projects every year, I got to learn a lot about microcontrollers and some about electronics.
This talk was my first non linux talk which detailled everything I learned from those miniconfs and projects I worked that stemmed from them. I presented it at LCA 2019 Christchurch.
you can find the talk pdf here: http://marc.merlins.org/linux/talks/Using_Open_Hardware/Using_Open_Hardware.pdf (you'll want this one to get all the clickable links in the slides)
you can view the talk slides in html here or below:
Talk video below:
I arrived the sunday before the conference and helped out the open hardware organizers with a bit of last minute setup. I also got to do some last minute testing and tuning of my panels:
hacked up ESP32 with level converters on breadboard to run 3x 64x32 SmartMatrix panels with SmartMatrix::GFX
64x64 P3.8 SmartMatrix::GFX panel vs 3x 64x32 SmartMatrix::GFX P4 flexible panels vs 4x 16x16 FastLED::NeoMatrix P10 panels
After finishing the code tuning and demos just in time, gave a 20mn miniconf talk on the history of linux.conf.au hardware miniconf. I went through how much I learned from those confs and what I was able to achieve as a result. I sure got to learn a lot about microcontroller and driver programming:
I wasn't able to bring my burning man 4096 neopixel matrix, it doesn't even fit in my car, but the irony is that my small 64x64 rgbpanel has the same resolution and fits easily in my backpack
The 64x64 compact display is showing the hand X-ray here
A few days later, I gave the longer version of my talk at the main conference. By then it had grown to over 160 slides in a 45mn slot, or 16 seconds per slide. Ooops...
The full talk went into details on what I learned in the hardware hacking field, a lot of it was simply electricity, U=RI, wires, pre-made components (small inline volt/amp meters, DC-DC converters, and so forth).
and for good measure the talk ends with how I was able to apply some of that knowledge to design simple solutions for hardware testing racks for Google's Fuchsia on arm platforms.
Hopefully the talk and/or slides are useful to you. Links:
This year, the Open Hardware Miniconf team designed a donkeycar for us at LCA 2019 Christchurch. It's a car that navigates by itself using its onboard camera connected to a Raspberry Pi using training video data gathered and analysed offline by tensorflow. That sure was an ambitious project!
I arrived the day before to help finish up the kits for the next morning:
the cars were eager to perform :)
Andy and Jon who ended up working all night to make sure the kits would work the next morning
The next morning, we showed up to build the kit:
rPi with custom last minute hat for the donkey car
done!
Jon gave a talk about the car design
Nice way to support 5V neopixels on 3.3V microcontrollers