Marc's Public Blog - Linux Hacking

Some old python-central package and compat breakage I wasn't interested in debugging, caused upgrades and uninstalls to fail. Causing a complex upgrade to fail in the middle due to a failed package removal on a pre-uninstall failing on pycentral, broke the system quite badly.

I restored it from a brrfs snapshot and then disabled pycentral (/bin/true) to allow its removal and apt-get -f install to suceed. After that, upgrades were possible again.

Removing gnome-codec-install (0.4.7+nmu1) ...
  File "/usr/bin/pycentral", line 87
    raise ValueError, 'unknown version info %s' % vinfo
Removing python-gst0.10 (0.10.22-2) ...
  File "/usr/bin/pycentral", line 87
    raise ValueError, 'unknown version info %s' % vinfo
                    ^
SyntaxError: invalid syntax
dpkg: error processing package python-gst0.10 (--remove):
 installed python-gst0.10 package pre-removal script subprocess returned error exit status 1
dpkg: too many errors, stopping
  File "/usr/bin/pycentral", line 87
    raise ValueError, 'unknown version info %s' % vinfo

saruman:~# mv /usr/bin/pycentral /usr/bin/pycentral.orig
saruman:~# ln -s /bin/true /usr/bin/pycentral
saruman:~# ln dpkg --remove python-central python-gst0.10 python-bluez gnome-codec-install bzrtools
(Reading database ... 503974 files and directories currently installed.)
Removing python-bluez (0.18-1+b2) ...
dpkg: warning: while removing python-bluez, directory '/usr/lib/python2.6/dist-packages/bluetooth' not empty so not removed
dpkg: warning: while removing python-bluez, directory '/usr/lib/python2.7/dist-packages/bluetooth' not empty so not removed
Removing gnome-codec-install (0.4.7+nmu1) ...
Removing bzrtools (2.3.1-1) ...
Removing python-gst0.10 (0.10.22-2) ...
dpkg: warning: while removing python-gst0.10, directory '/usr/lib/python2.6/dist-packages/gst-0.10/gst/extend' not empty so not removed
dpkg: warning: while removing python-gst0.10, directory '/usr/lib/python2.7/dist-packages/gst-0.10/gst/extend' not empty so not removed
Removing python-central (0.6.14+nmu2) ...
Processing triggers for hicolor-icon-theme (0.15-1) ...
Processing triggers for man-db (2.7.0.2-4) ...

SUCCESS!

That only took about 6H, why it it dark outside? Oh, it's past midnight. What was I doing before that?

Jim Zemlin opened the conference as always:

The various keynotes were good:

Linus and Dirk came in person for their usual firechat

Most speakers, including myself, gave talks in breakout rooms during the 3 days:

My talk went well, despite the low in person attendance:

video of the talk:

And of course we had a vendor floor and per person wrapped snacks (per covid rules):

We finished the conference with a reception at the Seattle Aquarium, which I had been to before, but was still fun to see again:

Thinkpad P73 vs P70, not a win all around: only one 2.5" drive instead of 2, and a badly designed power system

P70 vs P73, they look pretty similar

*Update*: it seems if that if you get a cheaper nvidia chip with the P73, it is then configured to accept 135W power supplies as the minimum required. That said, it will still refuse to work with any regular 90W power supply or external battery back, unless you force it with the center pin resistor swap.

While I have no plans to use windows on that machine, I thought I'd just try it out to see how it does on power. This is where I was impressed, windows can idle at less than 10W for more than 11H runtime, while I'm lucky if I can get linux at 15W. This is definitely a place where linux should do better, of course, it's not as if Lenovo put any work into making linux more efficient on their hardware either:

*Update* : with tlp and using the nouveau driver just enough to turn off the nvidia chip, I'm now able to tune the laptop down to 10W, almost matching windows.
See tlp issue 494 for details on how to setup tlp to run in low power mode when power is plugged in.

Another disappointment is that the P73 is mostly the same size and weight than the P70, but it has less room for storage. It has a bunch of empty spaces that aren't used for anything, and it can't use two 2.5" SATA drives anymore, like the P70 could. Worse, the now single 2.5" slot uses a lenovo only ribbon cable that does not ship with the laptop and basically means you cannot even add a 2.5" drive without that special ribbon cable, which isn't in stock yet. Well done... Ah yes, the battery is also not hot swappable, even if it is replaceable (unlike a Mac laptop where everything is sealed shut).

this shows the unobtanium lenovo cable for the now only single drive that fits, along the unused space

Ok, stop complaining, just buy a bunch of 170w or 230W power supplies and move on with your life

Tricking the P70 and P73 into accepting a power supply it wouldn't otherwise use

For the P70, I built this power supply adapter with a resistor bridge to tell the laptop how big it should think the power supply, is:

It's basically a configurable version of this. Yes, lenovo, I thank you for the hours I wasted opening up power supply plugs and replacing the center pin resistors:

Here's how the P73 responds:

- 230W works fine   4.6k => I have seen power supplies work with 170W but fail at 230W, so the laptop does draw more
- 170W works ok     1.9k (1.8k also ok)
- 130W rejected     1k
- 90W  rejected     550 

The rejected power supplies will be used to charge the laptop if it is shutdown, but they will not be used in any way otherwise. On the P70 the laptop would at least use the power supply to keep the laptop alive, and use half battery half external power supply. Not so with the P73, it just ignores it entirely.
This is utter bullshit as I have plenty of 90W power supplies, including 12V car converters, or a 90W external 20V battery pack I can't use anymore.

You can go read my Hacking a thinkpad slim tip adapter to output more than 90W (required to charge a Thinkpad P70) page for details, including this nice battery pack I couldn't use anymore:

*Update* : so, actually with some serious tlp hacking (basically I told it to force battery mode even if a power supply is plugged in), I've managed to throttle the laptop enough, even when plugged in, so that it only uses 20W. At that point, I'm actually able to use my old external battery pack, as well as a 90W power supply, as long as I lie to the laptop and pretend they are 230W power supplies with the resistor trick. In my tests with windows, it was not possible to throttle the laptop enough when plugged in, not to have it overwhelm a smaller power supply (not that 90W is small for a laptop that normally uses 20-30W when it's not charging batteries).
If you scroll to the bottom of the page, you'll also see a terrible buffer lipo hardware hack I did that allows to use the battery pack with higher amp draws, but it's a bit ridiculous (and bulky).
See tlp issue 494 for details on how to setup tlp to run in low power mode when power is plugged in.

Without the tlp hack or the buffer lipo hack, when I lie to the laptop and tell it is connected to a a bigger power supply, manage power draw with what I run, and disable battery charging in software, but the laptop will still draw the power supply for over 100W when you plug it in for a fraction of a second, and refuse it if the voltage drops.
Obviously this would not be a problem if the laptop simply had a 90W power supply mode where it throttle things down and turned off battery charging. This is mostly what the P70 does.

In the meantme, on top of hacking my power supplies, I also made this for my laptop, it looks silly and makes the thinkpad not look like a professional laptop, but well, that's lenovo's fault:

this takes any power supply and replaces the center tip resistor with a 1.9k one to emulate a 170W power supply

From talking to Lenovo, they don't think that this is really a problem, so since I'm an engineer, I made my own external battery pack, but I otherwise recommend to road warriors to avoid thinkpads from now on, given the backward power design in this one.

Making a Thinkpad P73 compatible external battery pack

Prototype with 150W step up converter which takes my 16V lipo to 20V while delivering enough amps to make the laptop happy:

it works, and the laptop thinks it's connected to a 230W power supply thanks to the center pin resistor.

Here's a quick demo:

Version 2 was to have a way to recharge the battery pack while it's being used. I've used this to use the battery pack as a buffer to absorb peaks from the laptop without tripping an external power supply, including in a car limited to 100W or a plane power supply limited to even less:

this works in theory, but the lipo charger is quite slow and wouldn't keep up for long, but I made a better version shown lower down

Lenovo's P73 airplane mode simply stops using the external power supply and reverts to batteries. Sigh...

Making a battery pack to act both as buffer for a smaller power supply, and as emergency external power (even power the laptop from 12v)

version 1 was a bit bigger than I wanted, 90W power supply shown for scale

here, the 90W power supply is recharging the battery at 2.5A while it's being discharged at 3A on the output side, using the battery as buffer

I made version 3 a bit smaller, with a built in 12V lipo to act as buffer for a smaller power supply. Yes, it's a beautiful piece of art, I know :)

Pushed to the extreme, I can now use my original external battery pack again by having it recharge my lipo+150W step up that can output more amps than the ravpower pack can. Of couse, it's inefficient, the ravpower pack outputs 20V that gets down converted to 12V by the H605 Air lipo charger, which charges the built in 3S lipo in the box, and then gets up converted back to 20V without the amp limitation (the phone used to control the lipo charger also inside the box):

The really cool thing is that by using tlp, it's actually possible to tune the laptop down to very low power use, even when plugged in, something that windows probably can't do:

7.3W with the screen off (and around 10W with wifi off and the screen on low dim) is not bad for a laptop that big

4S Lipo vs 4x 18650 or 26650 batteries

That said, as I recently found out that 16650 (or better 26650) batteries are both lighter and smaller. The only thing the lipos do, is offer a better discharge rate, but while that's useful for a high power RC plane motor that can empty the batteries in 10mn, it's not needed for a laptop:

26650s are 195 Wh/kg while the lipo I gave was 184Wh/kg. The 3rd battery listed is supposed to be 260Wh/kg which is much nicer, that said, it looks like those samsung batteries are actually smaller than 26650s, lighter, and yet offer the same 5Ah at 3.6V. If so, that's very impressive.

As of this writing, I have however not found 26650 battery holders that hold 26650 protected cells that are a bit longer. This seems to the be only one available, and it's too short to hold the batteries: https://www.amazon.com/gp/product/B074GVPWSH

A few months later, I ended up cutting them and taping them to make them longer. It doesn't work great, but it was enough for a few tests. In the end, I found that somehow I wasn't getting enough amps out of them, which surprised me, so they didn't work that well compared to a lipo:

Conclusion

The talk focussed on failures I've encountered in multiple fields, and what I learned from reading from other people's failures, a common practise in aviation that has saved countless lives in not re-creating failures and accidents out of ignorance.
As they say in aviation "experience is a cruel teacher: she gives you the test first, and if you survive, then you get to learn the lesson".

After looking at examples of failures we experienced at Google, I give a fair amount of examples from aviation, from AF447, QF32, and the Boeing 737 Max disaster which is so many failures in so many ways that it takes a while to describe in details. My hope is that engineers in a similar situation where they know they are getting overruled, can use other escallation steps to avoid disaster for others.

You can get the talk PDF or openoffice source from here. Otherwise, you can read the slides below and watch the video recording:

Jon, presenting the miniconf as per how many

Raspberry Pi running the car

This year's dingo car, controlled by the rPi

All done

Andy presenting the software side of the car

We then trained our cars' neural network by manually driving them arount the track:

and before long, I got my car to drive around on its own

Here is a video of my car driving on its own, including how it learned to drive:

During the miniconf, I also gave a talk on my recent LED panel/ESP32 work and brought my recent RGBPanels:

I gave a couple of talks this year:

I also had fun at the Open Hardware Miniconf 2020: Dingo Car with Tensorflow Video Analysis v2

I met Chris and Jon for dinner before the conference:

Chris had a fancy wheelchair enhanced with Jon's improvements

Then came time for the conference:

I much enjoyed a very cool talk on emulating old game consoles in an FPGA

took a lot to run those video consoles back then, lots of it was RAM

there was no framebuffer, sprites and backgrounds were rendered real time during CRT gun scans

backgrounds were rendered fro a bunch of tiles to same RAM

Random talk slides:

great to have Bunnie back

Nice to see the usual people again:

of course I 'need' an ESP32 watch :)

this one is cute too

Nice dinners as usual:

I brought my new LED outfit

And just like this, it was conference close, where we we told to guess where the next conference would be:

it'll be back in Canberra a 3rd time

Jon Oxer won the Rusty Wrench award this year, and it was well deserved

After an ice cream social, the conference was over.

you can find the talk pdf here: http://marc.merlins.org/linux/talks/Using_Open_Hardware/Using_Open_Hardware.pdf (you'll want this one to get all the clickable links in the slides)
you can view the talk slides in html here or below:

Talk video below:

I arrived the sunday before the conference and helped out the open hardware organizers with a bit of last minute setup. I also got to do some last minute testing and tuning of my panels:

hacked up ESP32 with level converters on breadboard to run 3x 64x32 SmartMatrix panels with SmartMatrix::GFX

64x64 P3.8 SmartMatrix::GFX panel vs 3x 64x32 SmartMatrix::GFX P4 flexible panels vs 4x 16x16 FastLED::NeoMatrix P10 panels

After finishing the code tuning and demos just in time, gave a 20mn miniconf talk on the history of linux.conf.au hardware miniconf. I went through how much I learned from those confs and what I was able to achieve as a result. I sure got to learn a lot about microcontroller and driver programming:

I wasn't able to bring my burning man 4096 neopixel matrix, it doesn't even fit in my car, but the irony is that my small 64x64 rgbpanel has the same resolution and fits easily in my backpack

The 64x64 compact display is showing the hand X-ray here

A few days later, I gave the longer version of my talk at the main conference. By then it had grown to over 160 slides in a 45mn slot, or 16 seconds per slide. Ooops...

http://marc.merlins.org/perso/arduino/post_2015-01-06_Driver-for-direct-driving-single-to-3-color-LED-Matrices-with-software-PWM.html

http://marc.merlins.org/perso/arduino/post_2017-04-03_Arduino-328P-Uno-Teensy3_1-ESP8266-ESP32-IR-and-Neopixels.html

http://marc.merlins.org/perso/arduino/post_2017-04-24_Adafruit-GFX-on-NeoMatrix-and-RGB-Matrix-Panel-Demo.html

http://marc.merlins.org/perso/arduino/post_2017-06-02_LED-Pants-and-Shirt-Programmed-With-Arduino-on-ESP8266.html

http://marc.merlins.org/perso/arduino/post_2018-04-23_FastLED_NeoMatrix-library_-how-to-do-Matrices-with-FastLED-and-Adafruit_GFX.html

http://marc.merlins.org/perso/arduino/post_2018-05-29_EDM-Party-Shirt-powered-with-FastLED_NeoMatrix-and-Adafruit_GFX_-plus-160Wh-_10Ah-4S_-worth-of-lipos.html

http://marc.merlins.org/perso/arduino/post_2018-07-13_AnimatedGIFs-for-SmartMatrix-or-NeoMatrix-_Neopixel-WS2812B_-from-SDcard-or-SPIFFS_-on-Teensy-3_x_-ESP8266_-or-ESP32.html

http://marc.merlins.org/perso/arduino/post_2018-07-30_Building-a-64x64-Neopixel-Neomatrix-_4096-pixels_-running-NeoMatrix-FastLED-IR.html

https://github.com/marcmerlin/SmartMatrix_GFX

Hopefully the talk and/or slides are useful to you. Links:

I arrived the day before to help finish up the kits for the next morning:

the cars were eager to perform :)

Andy and Jon who ended up working all night to make sure the kits would work the next morning

The next morning, we showed up to build the kit:

rPi with custom last minute hat for the donkey car

done!

Jon gave a talk about the car design

Nice way to support 5V neopixels on 3.3V microcontrollers

We then had a few talks:

Including mine on the history of linux.conf.au hardware miniconf

After the miniconf, we had a few tries at getting our own cars to self drive after training:

I decked out my car with neopixels, because bling! :)

nice car parked outside

The first day, I went to the Open Hardware Miniconf to build a monkey car with video analysis fed to tensorflow

LCA had lots of attendees as usual, enough ot overfill the main keynote room. The usual suspects/friends were there as usual:

Ah yes, the shirt and glasses, that's because of my talk Using Open Hardware from my shirt to OS testing for Google's Fuchsia

Lots of talks, thankfully they were recorded as there were 2-3x as many talks as I was able to see:

really nice keynote showing how to hack insulin dispensers to deliver the right amount at the right time

Commodore 65 slide presentation program

fpga based cell phone running alongside a commodore 65 fpga, awesome

Rusty's keynote for the win :)

the first LCA in 1999

Rusty gave a few talks since then :)

very interesting talk on how painful it was to take over grsecurity maintenance

Random pictures:

love this little laptop I found

As usual, nice parties in the evenings: speaker dinner, professional DNS, Penguin Dinner, and nice adhoc BBQ on friday:

And just like this, it was conference close:

next year will be the Gold Coast

thank you, come again :)