Marc's Public Blog - Linux Hacking


All | Aquariums | Arduino | Btrfs | Cars | Cats | Clubbing | Computers | Diving | Dreamstate | Edc | Electronics | Exercising | Festivals | Flying | Halloween | Hbot | Hiking | Linux | Linuxha | Monuments | Museums | Oshkosh | Outings | Public | Rc | Sciencemuseums | Solar | Tfsf | Trips



>>> Back to post index <<<

2014/01/06 My Live Upgrading Many Thousands of Servers ProdNG talk at Linux.conf.au 2014
π 2014-01-06 01:01 in Linux
LCA 2014 was the last event I gave my ProdNG talk at, the summary of 3.5 years of work at Google and the craziest upgrade I've done in my life. I also wrote a more detailled ProdNG LISA/Usenix Paper if you'd like more details.


The fancy talk description is here:

This talk will look at how we upgraded our ancient linux distribution on all the Google production servers to a more modern one based on debian stripped down and built from source.
We do live upgrades on running machines using an rsync-like rollout mechanism, and we slowly switched distributions on the fly without rebooting or downtime unless required by separately upgraded kernel, but doing a 10 year jump of linux userspace without breaking users was no piece of cake.
The talk will explain how something so crazy made sense, was even possible, executed, and worked out, as well as recommendations for maintaining linux big servers and dealing with upstream upgrades.

The video:

If you'd like the presentation in other formats:

  • ProdNG.odp
  • ProdNG.pdf
  • html (full screen)
  • More importantly, you can get the ProdNG LISA Paper which has many more details on how the upgrade went and the implementation.


    >


    More pages: July 2002 February 2004 March 2004 November 2004 April 2005 August 2005 January 2006 July 2006 August 2007 November 2007 January 2008 October 2008 November 2008 December 2008 January 2009 May 2009 July 2009 August 2009 September 2009 November 2009 December 2009 January 2010 March 2010 April 2010 June 2010 August 2010 October 2010 January 2011 July 2011 August 2011 December 2011 January 2012 March 2012 May 2012 August 2012 December 2012 January 2013 March 2013 May 2013 September 2013 November 2013 January 2014 March 2014 April 2014 May 2014 October 2014 January 2015 March 2015 May 2015 January 2016 February 2016 June 2016 July 2016 August 2016 October 2016 January 2017 September 2017 January 2018 March 2018 December 2018 January 2019 January 2020 May 2020 January 2021 September 2021 March 2023 April 2023 December 2023 June 2024

    >>> Back to post index <<<

    Contact Email