Marc's Public Blog - Linux Hacking


All | Aquariums | Arduino | Btrfs | Cars | Cats | Clubbing | Dining | Diving | Electronics | Exercising | Flying | Halloween | Hiking | Linux | Linuxha | Monuments | Museums | Public | Rc | Sciencemuseums | Snow | Solar | Trips

This page has a few of my blog entries about linux, but my main linux page is here
Picture of Linus


Table of Content for linux:

More pages: May 2020 January 2020 January 2019 December 2018 March 2018 January 2018 September 2017 January 2017 October 2016 August 2016 July 2016 June 2016 March 2016 February 2016 January 2016 May 2015 March 2015 January 2015 October 2014 May 2014 April 2014 March 2014 January 2014 November 2013 September 2013 May 2013 March 2013 January 2013 December 2012 August 2012 May 2012 March 2012 January 2012 December 2011 August 2011 July 2011 January 2011 October 2010 August 2010 June 2010 April 2010 March 2010 January 2010 December 2009 November 2009 September 2009 August 2009 July 2009 May 2009 January 2009 December 2008 November 2008 October 2008 January 2008 November 2007 August 2007 July 2006 January 2006 August 2005 April 2005 November 2004 March 2004 February 2004




2020/05/04 Hacking Power Supplies and Battery Pack To Get Around ThinkPad P73 Broken Power Supply Design
π 2020-05-04 01:01 in Electronics, Linux
While I wrote this for my Lenovo Thinkpad P73, this is likely equivally relevant to P53, P72, and P52.

Thinkpad P73 vs P70, not a win all around: only one 2.5" drive instead of 2, and a badly designed power system

So, when lenovo came out with the Thinkpad P70, I wasn't very happy because if you had a 90W power supply, it refused to charge from it, at any rate whatsoever. I was not impressed, but eh, at least it would still power the laptop so that its batteries didn't go flat while plugged in.
Well, leave it to lenovo engineers to make things worse the for the P73. The minimum power supply was raised from 135W (170W recommended) to 170W (230W recommended) which is understandable, but lenovo decided to ensure that the laptop will not take any power from any power supply that does not identify itself as 170W or more. This means that even ifit only needs 40W to sustain itself without digging into the batteries, it will completely refuse to use a 90W or even 135W power supply for anything at all, and kill the battery instead. Lenovo, you just plain suck, there is no excuse for this.

P70 vs P73, they look pretty similar
P70 vs P73, they look pretty similar

*Update*: it seems if that if you get a cheaper nvidia chip with the P73, it is then configured to accept 135W power supplies as the minimum required. That said, it will still refuse to work with any regular 90W power supply or external battery back, unless you force it with the center pin resistor swap.

While I have no plans to use windows on that machine, I thought I'd just try it out to see how it does on power. This is where I was impressed, windows can idle at less than 10W for more than 11H runtime, while I'm lucky if I can get linux at 15W. This is definitely a place where linux should do better, of course, it's not as if Lenovo put any work into making linux more efficient on their hardware either:


*Update* : with tlp and using the nouveau driver just enough to turn off the nvidia chip, I'm now able to tune the laptop down to 10W, almost matching windows.
See tlp issue 494 for details on how to setup tlp to run in low power mode when power is plugged in.

Another disappointment is that the P73 is mostly the same size and weight than the P70, but it has less room for storage. It has a bunch of empty spaces that aren't used for anything, and it can't use two 2.5" SATA drives anymore, like the P70 could. Worse, the now single 2.5" slot uses a lenovo only ribbon cable that does not ship with the laptop and basically means you cannot even add a 2.5" drive without that special ribbon cable, which isn't in stock yet. Well done... Ah yes, the battery is also not hot swappable, even if it is replaceable (unlike a Mac laptop where everything is sealed shut).


this shows the unobtanium lenovo cable for the now only single drive that fits, along the unused space
this shows the unobtanium lenovo cable for the now only single drive that fits, along the unused space

Ok, stop complaining, just buy a bunch of 170w or 230W power supplies and move on with your life

Well, yes and no:
  • I literally have 10 power supplies between home and work, not really looking at replacing all 10. Lenovo wants $137 per power supply by the way, even if they are $85 from other sellers
  • those 170W/230W power supplies are huge. They also weigh as much as some small notebooks (!)
  • I have 12V to 20V car adapters, those won't work anymore
  • I have external battery packs for extended runtime, and I haven't found a single one that can deliver the amps a P73 tries to needlessly require
  • Tricking the P70 and P73 into accepting a power supply it wouldn't otherwise use

    Lenovo uses the a center pin resistor to know how much power they can draw from the power supply, see: http://www.thinkwiki.org/wiki/Power_Connector .

    For the P70, I built this power supply adapter with a resistor bridge to tell the laptop how big it should think the power supply, is:


    It's basically a configurable version of this. Yes, lenovo, I thank you for the hours I wasted opening up power supply plugs and replacing the center pin resistors:


    Here's how the P73 responds:

    - 230W works fine   4.6k => I have seen power supplies work with 170W but fail at 230W, so the laptop does draw more
    - 170W works ok     1.9k (1.8k also ok)
    - 130W rejected     1k
    - 90W  rejected     550 

    The rejected power supplies will be used to charge the laptop if it is shutdown, but they will not be used in any way otherwise. On the P70 the laptop would at least use the power supply to keep the laptop alive, and use half battery half external power supply. Not so with the P73, it just ignores it entirely.
    This is utter bullshit as I have plenty of 90W power supplies, including 12V car converters, or a 90W external 20V battery pack I can't use anymore.

    You can go read my Hacking a thinkpad slim tip adapter to output more than 90W (required to charge a Thinkpad P70) page for details, including this nice battery pack I couldn't use anymore:


    *Update* : so, actually with some serious tlp hacking (basically I told it to force battery mode even if a power supply is plugged in), I've managed to throttle the laptop enough, even when plugged in, so that it only uses 20W. At that point, I'm actually able to use my old external battery pack, as well as a 90W power supply, as long as I lie to the laptop and pretend they are 230W power supplies with the resistor trick. In my tests with windows, it was not possible to throttle the laptop enough when plugged in, not to have it overwhelm a smaller power supply (not that 90W is small for a laptop that normally uses 20-30W when it's not charging batteries).
    If you scroll to the bottom of the page, you'll also see a terrible buffer lipo hardware hack I did that allows to use the battery pack with higher amp draws, but it's a bit ridiculous (and bulky).
    See tlp issue 494 for details on how to setup tlp to run in low power mode when power is plugged in.

    Without the tlp hack or the buffer lipo hack, when I lie to the laptop and tell it is connected to a a bigger power supply, manage power draw with what I run, and disable battery charging in software, but the laptop will still draw the power supply for over 100W when you plug it in for a fraction of a second, and refuse it if the voltage drops.
    Obviously this would not be a problem if the laptop simply had a 90W power supply mode where it throttle things down and turned off battery charging. This is mostly what the P70 does.

    In the meantme, on top of hacking my power supplies, I also made this for my laptop, it looks silly and makes the thinkpad not look like a professional laptop, but well, that's lenovo's fault:


    this takes any power supply and replaces the center tip resistor with a 1.9k one to emulate a 170W power supply
    this takes any power supply and replaces the center tip resistor with a 1.9k one to emulate a 170W power supply

    From talking to Lenovo, they don't think that this is really a problem, so since I'm an engineer, I made my own external battery pack, but I otherwise recommend to road warriors to avoid thinkpads from now on, given the backward power design in this one.

    Making a Thinkpad P73 compatible external battery pack

    I did some testing and confirmed that the laptop is very picky about power supplies. It even rejects a 19.7V 20A power supply I had, because it's 19.8V and not 20V. Same thing for amps, it needs to be able to draw maybe around 5A for a short time to accept the power supply (they sure are putting a lot of effort into making sure the power supply is not under-spec'ed).

    Prototype with 150W step up converter which takes my 16V lipo to 20V while delivering enough amps to make the laptop happy:


    it works, and the laptop thinks it's connected to a 230W power supply thanks to the center pin resistor.
    it works, and the laptop thinks it's connected to a 230W power supply thanks to the center pin resistor.

    Here's a quick demo:

    Version 2 was to have a way to recharge the battery pack while it's being used. I've used this to use the battery pack as a buffer to absorb peaks from the laptop without tripping an external power supply, including in a car limited to 100W or a plane power supply limited to even less:

    this works in theory, but the lipo charger is quite slow and wouldn't keep up for long, but I made a better version shown lower down
    this works in theory, but the lipo charger is quite slow and wouldn't keep up for long, but I made a better version shown lower down

    Lenovo's P73 airplane mode simply stops using the external power supply and reverts to batteries. Sigh...

    Oh yes, let's talk about airplane mode. The lenovo engineers thought of everything: if they detect that the power supply drops a few times in a row, they offer a nice setting which is supposed to make the plane more airplane friendly. How friendly you ask? Well, you could throttle down the CPUs, disable battery charging, do smart stuff like that. Or, if you're lenovo, you can have airplane mode simply refuse to use the power supply altogether. Thank you lenovo, you wrote a feature that saves me the trouble of unplugging an otherwise perfectly good power supply that you refuse to use (to be super clear, my 230W power supply is plugged in and airplane mode just disabled it):


    Making a battery pack to act both as buffer for a smaller power supply, and as emergency external power (even power the laptop from 12v)

    Anyway, back to the battery pack, I found the ISDT H605 Air lipo charger which is small enough and can charge the lipo at 5A, which should be enough to keep up with the laptop when not doing CPU crazy stuff. This also allows using a 12V power supply or a lower wattage lenovo power supply to recharge the pack while it's in use, or not:


    version 1 was a bit bigger than I wanted, 90W power supply shown for scale
    version 1 was a bit bigger than I wanted, 90W power supply shown for scale

    here, the 90W power supply is recharging the battery at 2.5A while it's being discharged at 3A on the output side, using the battery as buffer
    here, the 90W power supply is recharging the battery at 2.5A while it's being discharged at 3A on the output side, using the battery as buffer

    I made version 3 a bit smaller, with a built in 12V lipo to act as buffer for a smaller power supply. Yes, it's a beautiful piece of art, I know :)
    I made version 3 a bit smaller, with a built in 12V lipo to act as buffer for a smaller power supply. Yes, it's a beautiful piece of art, I know :)

    Pushed to the extreme, I can now use my original external battery pack again by having it recharge my lipo+150W step up that can output more amps than the ravpower pack can. Of couse, it's inefficient, the ravpower pack outputs 20V that gets down converted to 12V by the H605 Air lipo charger, which charges the built in 3S lipo in the box, and then gets up converted back to 20V without the amp limitation (the phone used to control the lipo charger also inside the box):


    The really cool thing is that by using tlp, it's actually possible to tune the laptop down to very low power use, even when plugged in, something that windows probably can't do:

    7.3W with the screen off (and around 10W with wifi off and the screen on low dim) is not bad for a laptop that big
    7.3W with the screen off (and around 10W with wifi off and the screen on low dim) is not bad for a laptop that big

    4S Lipo vs 4x 18650 or 26650 batteries

    I do have a few lipos laying around, so that's free energy for my laptop if I'm willing to carry them. I have however found that for higher draws, the step up converter doesn't quite keep up at 20V/5A+ with just 12V input (3S), but is fine with 16V input (4S):


    That said, as I recently found out that 16650 (or better 26650) batteries are both lighter and smaller. The only thing the lipos do, is offer a better discharge rate, but while that's useful for a high power RC plane motor that can empty the batteries in 10mn, it's not needed for a laptop:


    Outside of 26650 batteries, there are other ones like https://www.18650batterystore.com/21700-p/samsung-50e.htm .

    26650s are 195 Wh/kg while the lipo I gave was 184Wh/kg. The 3rd battery listed is supposed to be 260Wh/kg which is much nicer, that said, it looks like those samsung batteries are actually smaller than 26650s, lighter, and yet offer the same 5Ah at 3.6V. If so, that's very impressive.

    As of this writing, I have however not found 26650 battery holders that hold 26650 protected cells that are a bit longer. This seems to the be only one available, and it's too short to hold the batteries: https://www.amazon.com/gp/product/B074GVPWSH


    A few months later, I ended up cutting them and taping them to make them longer. It doesn't work great, but it was enough for a few tests. In the end, I found that somehow I wasn't getting enough amps out of them, which surprised me, so they didn't work that well compared to a lipo:


    Conclusion

    Lenovo, please make the P73 work like the P70, and fix this airplane mode thing that turns off the external power supply. That's embarrassing...
    2020/01/15 Linux.conf.au 2020 Main Talk: Planning for and Handling Failures from Open Hardware, Aviation, to Production at Google
    π 2020-01-15 01:01 in Google, Linux, Public
    As part of Linux.conf.au 2020, I gave my main talk Planning for and handling failures from open hardware, aviation, to production at Google.

    The talk focussed on failures I've encountered in multiple fields, and what I learned from reading from other people's failures, a common practise in aviation that has saved countless lives in not re-creating failures and accidents out of ignorance.
    As they say in aviation "experience is a cruel teacher: she gives you the test first, and if you survive, then you get to learn the lesson".

    After looking at examples of failures we experienced at Google, I give a fair amount of examples from aviation, from AF447, QF32, and the Boeing 737 Max disaster which is so many failures in so many ways that it takes a while to describe in details. My hope is that engineers in a similar situation where they know they are getting overruled, can use other escallation steps to avoid disaster for others.

    You can get the talk PDF or openoffice source from here. Otherwise, you can read the slides below and watch the video recording:




    2020/01/14 Linux.conf.au Open Hardware Miniconf 2020: Dingo Car with Tensorflow Video Analysis v2
    π 2020-01-14 01:01 in Arduino, Electronics, Linux
    Another LCA, another Open Hardware Miniconf. This year was an improved version of the Dingo Car from last year.

    Jon, presenting the miniconf as per how many
    Jon, presenting the miniconf as per how many





    Raspberry Pi running the car
    Raspberry Pi running the car



    This year's dingo car, controlled by the rPi
    This year's dingo car, controlled by the rPi

    All done
    All done

    Andy presenting the software side of the car
    Andy presenting the software side of the car

    We then trained our cars' neural network by manually driving them arount the track:


    and before long, I got my car to drive around on its own
    and before long, I got my car to drive around on its own

    Here is a video of my car driving on its own, including how it learned to drive:

    During the miniconf, I also gave a talk on my recent LED panel/ESP32 work and brought my recent RGBPanels:


    2020/01/13 Linux.Conf.au 2020 in the Gold Coast
    π 2020-01-13 01:01 in Linux
    LCA had been nearby twice, in Brisbane, but this was the first time in the Gold Coast. I had been there twice in the past though, so there were fewer things left to see, I went to check them

    I gave a couple of talks this year:

  • Planning for and Handling Failures from Open Hardware, Aviation, to Production at Google
  • ESP32 Memory Management, Neopixels and RGBPanels
  • I also had fun at the Open Hardware Miniconf 2020: Dingo Car with Tensorflow Video Analysis v2

    I met Chris and Jon for dinner before the conference:


    Chris had a fancy wheelchair enhanced with Jon's improvements
    Chris had a fancy wheelchair enhanced with Jon's improvements

    Then came time for the conference:





    I much enjoyed a very cool talk on emulating old game consoles in an FPGA



    took a lot to run those video consoles back then, lots of it was RAM
    took a lot to run those video consoles back then, lots of it was RAM

    there was no framebuffer, sprites and backgrounds were rendered real time during CRT gun scans
    there was no framebuffer, sprites and backgrounds were rendered real time during CRT gun scans

    backgrounds were rendered fro a bunch of tiles to same RAM
    backgrounds were rendered fro a bunch of tiles to same RAM

    Random talk slides:










    great to have Bunnie back
    great to have Bunnie back



    Nice to see the usual people again:


    of course I 'need' an ESP32 watch :)
    of course I 'need' an ESP32 watch :)

    this one is cute too
    this one is cute too



    Nice dinners as usual:






    I brought my new LED outfit
    I brought my new LED outfit


    And just like this, it was conference close, where we we told to guess where the next conference would be:



    it'll be back in Canberra a 3rd time
    it'll be back in Canberra a 3rd time

    Jon Oxer won the Rusty Wrench award this year, and it was well deserved
    Jon Oxer won the Rusty Wrench award this year, and it was well deserved


    After an ice cream social, the conference was over.

    2019/01/24 Linux.conf.au 2019: Using Open Hardware from my shirt to OS testing for Google's Fuchsia
    π 2019-01-24 01:01 in Arduino, Electronics, Linux
    I've been going to linux.conf.au for 18 years now (since 2001), and presented a fair amount of linux talks related there, but the big change for me was the open hardware miniconf that started in 2010. Thanks to its projects every year, I got to learn a lot about microcontrollers and some about electronics.
    This talk was my first non linux talk which detailled everything I learned from those miniconfs and projects I worked that stemmed from them. I presented it at LCA 2019 Christchurch.

    you can find the talk pdf here: http://marc.merlins.org/linux/talks/Using_Open_Hardware/Using_Open_Hardware.pdf (you'll want this one to get all the clickable links in the slides)
    you can view the talk slides in html here or below:

    Talk video below:

    I arrived the sunday before the conference and helped out the open hardware organizers with a bit of last minute setup. I also got to do some last minute testing and tuning of my panels:

    hacked up ESP32 with level converters on breadboard to run 3x 64x32 SmartMatrix panels with SmartMatrix::GFX
    hacked up ESP32 with level converters on breadboard to run 3x 64x32 SmartMatrix panels with SmartMatrix::GFX

    64x64 P3.8 SmartMatrix::GFX panel vs 3x 64x32 SmartMatrix::GFX P4 flexible panels vs 4x 16x16 FastLED::NeoMatrix P10 panels
    64x64 P3.8 SmartMatrix::GFX panel vs 3x 64x32 SmartMatrix::GFX P4 flexible panels vs 4x 16x16 FastLED::NeoMatrix P10 panels

    After finishing the code tuning and demos just in time, gave a 20mn miniconf talk on the history of linux.conf.au hardware miniconf. I went through how much I learned from those confs and what I was able to achieve as a result. I sure got to learn a lot about microcontroller and driver programming:




    I wasn't able to bring my burning man 4096 neopixel matrix, it doesn't even fit in my car, but the irony is that my small 64x64 rgbpanel has the same resolution and fits easily in my backpack
    I wasn't able to bring my burning man 4096 neopixel matrix, it doesn't even fit in my car, but the irony is that my small 64x64 rgbpanel has the same resolution and fits easily in my backpack

    The 64x64 compact display is showing the hand X-ray here
    The 64x64 compact display is showing the hand X-ray here

    A few days later, I gave the longer version of my talk at the main conference. By then it had grown to over 160 slides in a 45mn slot, or 16 seconds per slide. Ooops...

  • The full talk went into details on what I learned in the hardware hacking field, a lot of it was simply electricity, U=RI, wires, pre-made components (small inline volt/amp meters, DC-DC converters, and so forth).
  • I also gave a is a quick summary of my hacking an arduino board to turn it into a sleep analysis machine, a longer talk I gave elsewhere in 2012.
  • There is a section on what I learned from Tridge and RC planes with ardupilot
  • I had to mention microcontroller hacking and driver writing for IoTuz on ESP32 given how much I learned from it
  • And IoTuz got me to write a primitive Neopixel driver for ESP32, a very slippery slope that led me to many LED projects
  • http://marc.merlins.org/perso/arduino/post_2015-01-06_Driver-for-direct-driving-single-to-3-color-LED-Matrices-with-software-PWM.html
  • http://marc.merlins.org/perso/arduino/post_2017-04-03_Arduino-328P-Uno-Teensy3_1-ESP8266-ESP32-IR-and-Neopixels.html
  • http://marc.merlins.org/perso/arduino/post_2017-04-24_Adafruit-GFX-on-NeoMatrix-and-RGB-Matrix-Panel-Demo.html
  • http://marc.merlins.org/perso/arduino/post_2017-06-02_LED-Pants-and-Shirt-Programmed-With-Arduino-on-ESP8266.html
  • http://marc.merlins.org/perso/arduino/post_2018-04-23_FastLED_NeoMatrix-library_-how-to-do-Matrices-with-FastLED-and-Adafruit_GFX.html
  • http://marc.merlins.org/perso/arduino/post_2018-05-29_EDM-Party-Shirt-powered-with-FastLED_NeoMatrix-and-Adafruit_GFX_-plus-160Wh-_10Ah-4S_-worth-of-lipos.html
  • http://marc.merlins.org/perso/arduino/post_2018-07-13_AnimatedGIFs-for-SmartMatrix-or-NeoMatrix-_Neopixel-WS2812B_-from-SDcard-or-SPIFFS_-on-Teensy-3_x_-ESP8266_-or-ESP32.html
  • http://marc.merlins.org/perso/arduino/post_2018-07-30_Building-a-64x64-Neopixel-Neomatrix-_4096-pixels_-running-NeoMatrix-FastLED-IR.html
  • https://github.com/marcmerlin/SmartMatrix_GFX
  • and for good measure the talk ends with how I was able to apply some of that knowledge to design simple solutions for hardware testing racks for Google's Fuchsia on arm platforms.


  • Hopefully the talk and/or slides are useful to you. Links:

  • http://marc.merlins.org/linux/talks/Using_Open_Hardware/Using_Open_Hardware.pdf
  • http://marc.merlins.org/linux/talks/Using_Open_Hardware
  • http://marc.merlins.org/perso/arduino
  • https://www.youtube.com/watch?v=YqeBXCCabo0
  • 2019/01/21 Donkey Car with Tensorflow Video Analysis at Open Hardware Miniconf at Linux.Conf.au 2019
    π 2019-01-21 01:01 in Arduino, Linux
    This year, the Open Hardware Miniconf team designed a donkeycar for us at LCA 2019 Christchurch. It's a car that navigates by itself using its onboard camera connected to a Raspberry Pi using training video data gathered and analysed offline by tensorflow. That sure was an ambitious project!

    I arrived the day before to help finish up the kits for the next morning:



    the cars were eager to perform :)
    the cars were eager to perform :)

    Andy and Jon who ended up working all night to make sure the kits would work the next morning
    Andy and Jon who ended up working all night to make sure the kits would work the next morning


    The next morning, we showed up to build the kit:




    rPi with custom last minute hat for the donkey car
    rPi with custom last minute hat for the donkey car

    done!
    done!

    Jon gave a talk about the car design
    Jon gave a talk about the car design

    Nice way to support 5V neopixels on 3.3V microcontrollers
    Nice way to support 5V neopixels on 3.3V microcontrollers

    We then had a few talks:



    Including mine on the history of linux.conf.au hardware miniconf


    After the miniconf, we had a few tries at getting our own cars to self drive after training:


    I decked out my car with neopixels, because bling! :)
    I decked out my car with neopixels, because bling! :)


    2019/01/21 Linux.Conf.au 2019 in Christchurch
    π 2019-01-21 01:01 in Linux
    This year, LCA was back in New Zealand in the city of Christchurch.


    nice car parked outside
    nice car parked outside



    The first day, I went to the Open Hardware Miniconf to build a monkey car with video analysis fed to tensorflow

    LCA had lots of attendees as usual, enough ot overfill the main keynote room. The usual suspects/friends were there as usual:







    Ah yes, the shirt and glasses, that's because of my talk Using Open Hardware from my shirt to OS testing for Google's Fuchsia

    Lots of talks, thankfully they were recorded as there were 2-3x as many talks as I was able to see:



    really nice keynote showing how to hack insulin dispensers to deliver the right amount at the right time
    really nice keynote showing how to hack insulin dispensers to deliver the right amount at the right time



    Commodore 65 slide presentation program
    Commodore 65 slide presentation program

    fpga based cell phone running alongside a commodore 65 fpga, awesome
    fpga based cell phone running alongside a commodore 65 fpga, awesome







    Rusty's keynote for the win :)
    Rusty's keynote for the win :)



    the first LCA in 1999
    the first LCA in 1999

    Rusty gave a few talks since then :)
    Rusty gave a few talks since then :)


    very interesting talk on how painful it was to take over grsecurity maintenance
    very interesting talk on how painful it was to take over grsecurity maintenance

    Random pictures:

    love this little laptop I found
    love this little laptop I found

    As usual, nice parties in the evenings: speaker dinner, professional DNS, Penguin Dinner, and nice adhoc BBQ on friday:











    And just like this, it was conference close:




    next year will be the Gold Coast
    next year will be the Gold Coast

    thank you, come again :)
    thank you, come again :)

    2018/12/20 Accessing USB Devices In Docker (ttyUSB0, /dev/bus/usb/... for fastboot, adb) without using --privileged
    π 2018-12-20 01:01 in Linux

    Problem:You want to use devices like /dev/ttyUSB0 or need to access raw USB devices under /dev/bus/usb in a docker container

    This is not going to work easily for a variety of reasons. Let's go through them.

    Before I go there, yes, giving --privileged will "fix" all your problems but it will also give full access to everything in your container, where the container can wipe the host device, steal password from memory outside the container and so forth. Maybe that's ok for you anyway, but if not, read on:

    /dev/ttyUSB0 does not show up under /dev in the container

    This does 2 things: 1) create the device in the container, and 2) give the container write access to the block device (more or that later)

    docker run --device=/dev/ttyUSB0 -i -t --entrypoint /bin/bash --cap-add SYS_PTRACE debian:amd64

    I added SYS_PTRACE so that you can use strace -e trace=file command to debug access problems

    /dev/bus or /dev/serial do not show up under /dev in the container

    Try this:
    docker run -v /dev/bus:/dev/bus:ro -v /dev/serial:/dev/serial:ro -i -t --entrypoint /bin/bash --cap-add SYS_PTRACE debian:amd64

    This will make /dev/bus and /dev/serial show up as bind mounts in your container. Yet you will find that _fastboot devices or adb_ will not work. Strace will show you permission denied. What you want instead is this:

    docker run --device=/dev/bus -v /dev/serial:/dev/serial:ro -i -t --entrypoint /bin/bash --cap-add SYS_PTRACE debian:amd64

    See https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities

    This mounts /dev/bus in your container and gives you access to all the block/character devices in there. Now, fastboot will work. Well, it will work until...

    /dev/bus, fastboot/adb work until I disconnect/reconnect the device

    Yeah, this is where things get more hairy. what the --device command above did was give access to all the block/character devices that existed when your container was created. If new ones appear (unplugging and replugging a device or rebooting it), those don't work.
    I have not found a way to get docker to make them work after the container has started.

    Thankfully there is a clue here: https://www.kernel.org/doc/Documentation/cgroup-v1/devices.txt

    You want to go back to running

    docker run -v /dev/bus:/dev/bus:ro -v /dev/serial:/dev/serial:ro -i -t --entrypoint /bin/bash --cap-add SYS_PTRACE debian:amd64

    Then, grab the container ID in $A:

    A=$( docker ps |awk '/bin.bash/ { print $1 }' )

    Note the major number of your device:

    root@fuchsia-tests-x64-lab01-0002:/sys/fs/cgroup# l /dev/bus/usb/004/* | head -1
    crw-rw-r-- 1 root root 189, 384 Dec 18 17:51 /dev/bus/usb/004/001
    Here it's 189. What changes it the minor number (384) when you plug/unplug devices.

    Finally, tell the linux container via an opaque cgroup interface, that all character devices of major number 189, are allowed:

    root@fuchsia-tests-x64-lab01-0002:~# echo 'c 189:* rwm' > /sys/fs/cgroup/devices/docker/$A*/devices.allow 

    After that, things should work.

    My /dev/ttyUSB device keeps changing and the new one isn't allowed

    See above and run
    root@fuchsia-tests-x64-lab01-0002:~# echo 'c 188:* rwm' > /sys/fs/cgroup/devices/docker/$A*/devices.allow 

    Then inside the container, create them all:

    root@f47dbbab392b:/dev# for i in $(seq 1 255); do mknod ttyUSB$i c 188 $i; done

    Hope this helps!

    2018/12/20 Getting Around USB3 xhci 32 Device Limit "Max number of devices this xHCI host supports is 32"
    π 2018-12-20 01:01 in Linux
    Are you stuck with xhci_hcd 0000:00:14.0: Max number of devices this xHCI host supports is 32 ?

    Wait, you're saying, but USB should support 127 devices (which include hub ports), why only 32?
    Well, xhci/USB3 actually has an extra limit of 96 endpoints and each device uses 3 endpoints, which leaves you with only 32 devices. Yes, this is terrible, I agree. It's also not well documented and few people know how to get around it.
    https://forums.intel.com/s/question/0D50P00004905stSAA/hardware-limitations-on-usb-endpoints-xhci?language=en_US contains more details, although apparently some further details are only available under NDA, but it looks like a mess.

    Originally I found this page which explains how to connect USB3 hubs with USB2 cables to save a few USB IDs, but ultimately it does not contain the real solution which is:
    If you need more than 32 devices, do not use (and disable) USB3/xhci
    Yes, it is that sad, intel has actually designed a newer USB that is much much worse than the previous version as far as number of devices is concerned.

    What wasn't clear to me is that xhci supports USB2 and USB1 devices, but no matter what devices you use (USB2 or USB1), you're still limited by that artificial 32 device limit that is due to the broken design of xhci as long as the xhci controller is handling those USB2/USB1 devices for you.
    I've been told that some motherboards auto assign their USB ports to an ehci controller or xhci controller depending on what you plug into them. If so, this is the time to plug your USB3 hub with a USB2 cable to force the port to be routed to EHCI. This didn't work on the motherboard I tried on, however.

    Now, the good news is that because motherboards try to work with older operating systems that may not support xhci, they usually provide an ehci controller too, the only issue is to get those USB ports assigned to the ehci controller(s).

    How do you do this?

  • Turn off USB3 in the bios altogether (what I did). The supermicro motherboard I was using is old enough that it actually contains 2 ehci chips which then multiplied the number of devices you can plug in, gave me 8x more devices than when I had USB3 enabled (ehci can do close to 128 instead of 32 and 2 controllers instead of 1).
  • remove/blacklist the linux xhci driver modules. If the motherboard auto assigns xhci/ehci depending on what driver is loaded, this should work
  • If the motherboard only comes with an xhci controller, you are up a creek without a paddle, but you can still get an external USB2 PCI(-e) card: https://www.amazon.com/gp/product/B002RL8V7E "PCIe USB 2.0 Card - PCI-E USB 2.0 Card" actually comes with 2 USB1 controllers and one USB2 controller. It's not very clear how they are assigned (maybe dynamically depending on what is plugged into the ports). In either case the USB1 and USB2 have the high device limit you're looking for, but if somehow you're needing lots of devices (past 128), you may be able to get more out of the USB1.1 controllers (although at USB1 speeds, so that's probably not what you want). This is how that external card looks like
  • 06:04.0 USB controller: VIA Technologies, Inc. VT82xx/62xx UHCI USB 1.1 Controller (rev 62)
    06:04.1 USB controller: VIA Technologies, Inc. VT82xx/62xx UHCI USB 1.1 Controller (rev 62)
    06:04.2 USB controller: VIA Technologies, Inc. USB 2.0 (rev 65)

    Now, I found another interesting way to do this, on some motherboards you can reroute the USB ports to the USB2 chipset without changing the bios. See https://www.systutorials.com/241533/how-to-force-a-usb-3-0-port-to-work-in-usb-2-0-mode-in-linux
    In the case of my supermicro board, I didn't look at how the two EHCI (USB2) controllers were routed, but it had extra USB cables on the motherboard, so I bought USB headers installed them on the front, and moved some USB devices to them: https://www.amazon.com/gp/product/B00NMBXUXI .

    Note that you likely don't want to buy a USB-3 PCI(-e) card like https://www.amazon.com/gp/product/B00FPIMICA because it creates a new root hub but only allows a measly extra 30-ish devices to be added (vs 120+ with the USB2 card mentioned above).

    Big thanks to

  • Alan Stern who helped me figure this out in https://www.spinics.net/lists/linux-usb/msg175224.html
  • Ruchira Ravoori from fuchsia kernel team who explained things to me further and pointed me to https://forums.intel.com/s/question/0D50P00004905stSAA/hardware-limitations-on-usb-endpoints-xhci?language=en_US
  • 2018/03/09 Btrfs Tips: Rescuing A Btrfs Send Receive Relationship
    π 2018-03-09 01:01 in Btrfs, Linux
    I had a problem when migrating disks and moving read only btrfs receive snapshots. I did a
    btrfs snapshot dir1/Video_ro.20180220_21:03:41 dir2/Video_ro.20180220_21:03:41 
    which in turn broke the snapshot by making it read-write and losing the btrfs receive relationship (Received UUID)

    Later, I wanted to re-establish the btrfs send/receive relationship since it was an 8TB subvolume, and I wanted to avoid having to copy back all the data I already had there. Simply making the destination snapshot read only again did not work

    The solution was python-btrfs, and this code: https://github.com/knorrie/python-btrfs/commit/1ace623f95300ecf581b1182780fd6432a46b24d

    gargamel:python-btrfs/examples# ./set_received_uuid.py 2afc7a5e-107f-d54b-8929-197b80b70828 31337 1234.5678 /mnt/btrfs_bigbackup/DS1/Video_ro.20180220_21:03:411G Current subvolume information: subvol_id: 94887 received_uuid: 00000000-0000-0000-0000-000000000000 stime: 0.0 (1970-01-01T00:00:00) stransid: 0 rtime: 0.0 (1970-01-01T00:00:00) rtransid: 0

    Setting received subvolume...

    Resulting subvolume information: subvol_id: 94887 received_uuid: 2afc7a5e-107f-d54b-8929-197b80b70828 stime: 1234.5678 (1970-01-01T00:20:34.567800) stransid: 31337 rtime: 1520488877.415709329 (2018-03-08T06:01:17.415709) rtransid: 255755

    Then make it read-only again: gargamel:python-btrfs/examples# btrfs property set -ts /mnt/btrfs_bigbackup/DS1/Video_ro.20180220_21:03:41 ro true

    But this didn't work: ABORT: btrfs send -p /mnt/btrfs_pool1/Video_ro.20180220_21:03:41 Video_ro.20180308_07:50:06 | btrfs receive /mnt/btrfs_bigbackup/DS1//. failed At subvol Video_ro.20180308_07:50:06 At snapshot Video_ro.20180308_07:50:06 ERROR: cannot find parent subvolume

    You can see that with set_received_uuid.py, I set Received UUID to match UUID on the source:

    gargamel:/mnt/btrfs_pool1# btrfs subvolume show /mnt/btrfs_bigbackup/DS1/Video_ro.20180220_21:03:41 DS1/Video_ro.20180220_21:03:41
            Name:                   Video_ro.20180220_21:03:41
            UUID:                   cb4f343c-5e79-7f49-adf0-7ce0b29f23b3
            Parent UUID:            0e220a4f-6426-4745-8399-0da0084f8b23 
            Received UUID:          2afc7a5e-107f-d54b-8929-197b80b70828 << changed this
            Creation time:          2018-02-20 21:13:36 -0800
            Subvolume ID:           94887
            Generation:             250689
            Gen at creation:        250689
            Parent ID:              89160
            Top level ID:           89160
            Flags:                  readonly
            Snapshot(s):
    

    Name: Video_ro.20180220_21:03:41 UUID: 2afc7a5e-107f-d54b-8929-197b80b70828 Parent UUID: e5ec5c1e-6b49-084e-8820-5a8cfaa1b089 Received UUID: 0e220a4f-6426-4745-8399-0da0084f8b23 Creation time: 2018-02-20 21:03:42 -0800 Subvolume ID: 11228 Generation: 4174 Gen at creation: 4150 Parent ID: 5 Top level ID: 5 Flags: readonly

    Turns out however that because the source had a Parent UUID value too, I was actually supposed to set Received UUID on the destination to it:

    gargamel:python-btrfs/examples# ./set_received_uuid.py 0e220a4f-6426-4745-8399-0da0084f8b23 313 37 1234.5678 /mnt/btrfs_bigbackup/DS1/Video_ro.20180220_21:03:41 Current subvolume information: subvol_id: 94887 received_uuid: 2afc7a5e-107f-d54b-8929-197b80b70828 stime: 1234.5678 (1970-01-01T00:20:34.567800) stransid: 31337 rtime: 1520488877.415709329 (2018-03-08T06:01:17.415709) rtransid: 255755

    Setting received subvolume...

    Resulting subvolume information: subvol_id: 94887 received_uuid: 0e220a4f-6426-4745-8399-0da0084f8b23 stime: 1234.5678 (1970-01-01T00:20:34.567800) stransid: 31337 rtime: 1520537034.890253770 (2018-03-08T19:23:54.890254) rtransid: 256119

    gargamel:python-btrfs/examples# btrfs property set -ts /mnt/btrfs_bigbackup/DS1/Video_ro.201802 20_21:03:41 ro true

    After this, I was able to re-start my btrfs send/receive and /mnt/btrfs_bigbackup/DS1/Video_ro.201802 20_21:03:41 was properly accepted as a destination. Yeah!


    More pages: May 2020 January 2020 January 2019 December 2018 March 2018 January 2018 September 2017 January 2017 October 2016 August 2016 July 2016 June 2016 March 2016 February 2016 January 2016 May 2015 March 2015 January 2015 October 2014 May 2014 April 2014 March 2014 January 2014 November 2013 September 2013 May 2013 March 2013 January 2013 December 2012 August 2012 May 2012 March 2012 January 2012 December 2011 August 2011 July 2011 January 2011 October 2010 August 2010 June 2010 April 2010 March 2010 January 2010 December 2009 November 2009 September 2009 August 2009 July 2009 May 2009 January 2009 December 2008 November 2008 October 2008 January 2008 November 2007 August 2007 July 2006 January 2006 August 2005 April 2005 November 2004 March 2004 February 2004

    Contact Email