LinuxWorld Convention & Expo Summer 2001

• New format
• Linuxworld this year
• Day 1 Keynote: Linux Throughout the Enterprise - Ready o r Not?
• Conference: 2.4 Kernel Scalability Features
• Conference: JFS for linux
• Feature Presentation: Golden Penguin Bowl
• Video, Graphics, DVR/PVR -> Scaling linux to 1024 CPUs the easy way
• No Parties
• Day 2 Keynote: The New New Old War
• IDG Linus Torvalds Community award
• Advances in video capture and digital video under linux
• Feature Presentation: The state of Open Source
• Showfloor
• Evening Parties
• That's all folks

• I finally retired my old Ricoh 4200 digital camera (for sale, Email me for details ), and my new camera (Canon S10) takes 1600x1200 pictures and has enough memory that I use that size by default. The problem is that I used to resize my pictures so that they weren't too big and could be viewed by anyone on a normal sized screen (my desktop at home is 2048x1536, at at work it's 3600x1440 (dualhead), a bit beyond "normal"), however this doesn't work anymore: I didn't want to sacrifice the picture size for the lowest common denominator, so it became clear that a viewer with multiple sizes was necessary.
  I first started to use IDS, but it was way too slow on my machines and would kill the P120-64Megs of Ram that you're getting this page from. Thankfully Raphael Moll, gave me a pre-release of his excellent RIG, which re-implements IDS in PHP and C++, and is noticeably faster. Without this, I would not have been able to make the picture library available to more than 2 or 3 concurrent users
• After having read a few reports from the excellent folks at Linux Weekly News, I realized that the one long page format isn't that bad afterall, especially if there isn't loads of things to say for each portion of the event.
• The one page format is faster to write for me . Writing these pages isn't my full time job. Actually, it's not my job at all . I unfortunately can't spend the 30 or more hours I've spent to write some of my past reports, and half the time, when I was done, it was 7 to 10 days after the event, when it was "old and boring news" in web time.
• Having recently come back from the Ottawa Linux Showcase, I didn't have the time to spend 5 days at the show, but that said, it didn't seem much worth it either, so that was ok.

Since the first two days of Linuwrold were really a couple of days of tutorials and the conferences and showfloor really opened on tuesday, I'll refer to tuesday as the first day and wednesday the second. Thursday was the last day, and developer conferences ended on wednesday, so it didn't really seem worth going.

He explained that we had had a decade of technological advancement and it was a decade of tremendous growth for the internet

A few numbers and points he gave were:

• 7 Trillion Emails sent in the US.
• At the apogee of napster, 1.4B songs downloaded in one month, 8.5% PCs tapped in and napster was responsible for 5% net traffic in the US.
• Rebirth of open source: projects like sendmail, linux, or apache, that then became extremely widespread (rebirth is compared to the sixties and seventies when open source was quite common)
• Because of the new demands from internet users, there have been big challenges on the server side in reliability expectations, hard drive storage needs and response time.
• Most enterprises are shooting for less than 5mn of downtime per year.
• Security breaches cost $15B annually and they need to be addressed in near real time.

• Providing better support and services for big enterprise customers
• helping with integration and migration
• adopting and adhering to new standards like hotplug PCI, which is one of the things compaq works on
• scalability to 4 and more CPUs, which has been addressed to some extent by the 2.4 kernel (although, some work is still needed for good performance beyond 8 CPUs). Process limits and threads have similarly been worked on.

• broader hardware support (like the hotplug PCI work)
• applications and development software
• Middleware
• management software
• enterprise applications
• other random work like mobile IPV6 on the Ipaq

He gave an impressive demo of an Ipaq running linux with a camera and a wireless LAN card running and roaming with mobile IPv6 between two wireless networks while broadcasting video back to the server.

• CPU scalability:
  • less use of the global lock_kernel
  • more subsystem based spinlocks
  • spinlocks embedded in data structures
  • semaphores used to serialize address space access
  • more of a spinlock hierarchy established Lock granularity has a tradeoff though: the more fined grained you get, the less lock contention you have, but the more overhead you get because of all the locks you need to go through of any single operation.
    The hard part is to find the sweet spot in the number and location of locks, and obviously this depends on how many CPUs you have in your target platform. Right now, for Larry, 4 CPUs is the sweet spot in linux 2.4
  
  
• Multithreading improvements:
  • multiple threads can access address space specific data structures simultaneously
  • Single mmap_sem semaphore was replaced with multiple reader/single writer semaphore (down_read/down_write operations)
  • A reader lock is now acquired for reading per address space structures
  • exclusive write lock is acquired when altering per address space structures
  
  
• 32bit UIDs and GIDs
  
• 64bit virtual address space
  • architectural limit of the virtual address space was expanded to 64 bits on alpha and IA64
  • IA64 implements 51 bits (16 disjoins 47 bit regions)
  • alpha implements 43 bits (2 disjoint 42 bit regions)
  • future alpha is expanted do 48 bits (2 disjoint 47 bit regions)
  
  
• Unified filesystem cache
  • Single pagecache was unified from previous pagecache read/buffermem write functionality
  • eliminates copying buffers from buffermem to pagecache on file read ops
  • reduces memory consumption by eliminating double buffered copies of FS data
  • eliminates overhead of searching two levels of data cache
  
  
• Distribution of interrupts
  • more hardware interrupt service routines can be processed simulteanously on multiple CPUs (irq_attempt())
  • Software interrupts (softIRQs) can be processed simultaneously on all CPUs with hardware interrupts enabled
  • More SMP spin locks are maintained with device specific data structures
  
  
• Increased number of threads and tasks
  • maximum allowable number of tasks/address spaces as well as threads was increased
  • Configuration of maximum number of threas is runtime tunable via /proc filesystem (no more NR_TASKS)
  • Scheduler optimizations minimize overhead of context switching between sibling threads
  
  
• IA64 Port and architecture optimizations
  • ia64 specific TLB optimizations
  • large rotating register file
  • ia64 SMP specifics
  • ia64 IO specifics
  • 64 bit virtual address spac
  • NUMA support under development
  
  
• Alpha Port and architecture optimizations
  • 64 bit virtual address space (EV67 is 43 bits: half user, half kernel; EV68 supports 48 bits: also half user, half kernel)
  • 2TB (41 bits) physical address limit for EV67
  • 64TB (46 bits) physical address limit for EV68
  • accurate SMP compatible processor time optimizations
  • NUMA support under development
  
  
• Bigmem for IA32
  • 1GB physical memory limitation in the linux kernel
  • 4GB physical meory limitation for 32bit systems
  • 4GB physical memory optimizations in the linux kernel
  • 64GB physical memory using PAE on IA32
  
  
• Disks and filesystems
  • File size increase
    • File offset was increased from 31 bits to 44 bits in the VFS layer
    • Filesystem size and file size are increased to 16TB
  • buffermem and pagecache were merged in 2.4 to unify fs cache
  • Raw IO
  • Journaling filesystems (ext3, reiserfs, xfs, jfs)
  • logical volume manager included in the main kernell
  • Working NFS V3 (udp only in the stock kernel)
  
  
• Other miscellaneous
  • fast memory transferts for Pentium, AMD, Cyrix, Winchip and a few others
  • network rewrite to optimal performance and better use of SMP systems
  • kernel based http daemon (khttpd)
  • Iptables/Netfilter replacement for ipchains
  • Tunable memory parameters (/proc/sys/vm/)

JFS is proven FS technology (it's been in AIX for 10 years), and is been rebuilt from the ground up in 1995. Because of the shared code base, it is capable of mounting partitions from OS/2
The features of JFS are as follows:

• Support for file sizes up to 512 terabytes (0.5K blocks) to 4 petabytes (4K blocks)
• Max aggreagate is from 4PB to 32PB and is currently limited by linux I/O structures not being 64 bits yet.
• Journaling of metadata only (asynchronous for added speed)
• Filenames in unicode (UTF16)
• B+Tree with extents (variable-length sequence of aggregate blocks allocated as one unit)
• Built to scale: in memory and on disk data structures are designed to be scalable, and code is designed to be run on SMP systems with 8 or more CPUs
• Dynamic Inode allocation
• Support for symlinks in inodes
• Directories can store up to 8 entries directly inside the inode, and then B+trees keyed on name are used
• Support for both sparse and dense files (chosen at mkfs time)
• Ability to increase filesystem size and option to integrate with LVM

Evolution of the code and current status:

• Announced and shipped in 2000/02/02 at Linuxworld in NYC
• 41 code drops so far
• JFS patch and tar files supports 2.2.14 onward and 2.4.x from a single source tree.
• Completely independent of any kernel changes
• Beta 1 in 2000/12
• Release 1.0.0 in 2001/06 (current is 1.0.3)

Steve has good hopes that it will be included in distributions soon as it's easy to add (it doesn't require a massive kernel patch like XFS does)

• What is the number of Neo's apartement in the Matrix?
• How about the room Trinity was in?
• In the Alien series, what was Repley's first name?
• And for 10,000 points: what is CPIP in RFC 1149?

He explained how companies have in the past modified their kernels to have very granular locks were all wrong (he's worked at both Sun and SGI)
Not only is it hard to make a kernel very modular, but as you add more locks to remove the contention on each individual lock, you make the kernel more bloated and complex, and especially you slow down the common case as those locks have to be taken every single time a function is called (also, on a hardware level, on most architectures, taking a lock is expensive and causes bus contention)

Larry's solution to this problem is not to try to scale an OS to many CPUs, but to run multiple instances of the OS on the hardware and setup some ways for them to communicate (a principle somewhat similar to RT Linux where linux runs as the lowest priority task under an RTOS, except that in this case you'd have several linux tasks)

That was a definitely a novel approach, and Larry is looking for coders if you're interested in helping.

By regulation, phone companies are not allowed to discriminate on the content that travels on phone lines.
However, on cable companies, there is no such law and they are allowed to decide what travels on their cables, discriminate traffic, favor some streams or protocols over other ones.

He then obviously mentioned the DMCA and the fact that content providers and the government are now able to scare people off from writing code that doesn't fit with their agenda (the Dmitry case was an obvious example)

He explained that they have the upper hand now, and either ignoring them because we have better things to do, or in disgust, or trying to work around it by building free networks or products that can't be stopped, just isn't the right way to go. We need to fight the actual laws: it's not alright that guns can be sold whether they are used for defence or used to kill a policeman but that one can be put in jail for 25 years for writing code that is a threat to companies that are behind the DMCA
Napster had to garantee that 100% of the content being traded through their system didn't violate copyrights and had to shut down. Xerox doesn't have to garantee that their copy machines aren't used for copyright violation.

Professor Lessig concluded that we have to do something about the situation, either by talking to politicians, by marching, or at the very least, by supporting people who are doing so by supporting the EFF.

We should also thank IDG for giving money back to the community with the two money awards they gave.

What was the biggest thing in open source that happened in the last year?

Linus has been impressed with KDE, which he switched to this past year after it went past the eye candy stage to a stage where you have office applications
Brian has been impressed with how big companies, including microsoft have responded to open source
For Jeremy, the 2.4 kernel actually is the most important thing we've had in the last year, bring linux to the enterprise level.
Dirk does see the applications as the important part. The fact that LSB 1.0 was released is the most important part for him so that we can start solving the incompatibilities between distributions.

How about microsoft calling the GPL a threat to the capitalism?

Jeremy answered: "so what? and the problem is?"
Dirk thinks that the GPL being a threat is actually a good thing as it drives competitors to innovate.
Linus explained that for him Linux got where it is thanks to the GPL license as it keeps people who use it honest if they contribute and use it commercially (with a BSD style license, you can use the code and you don't have to contribute back)

How do you look at new software that needs to be created?

There were two views on the matter: you write whatever software you need as you need it (Linus and Jeremy) but Dirk thinks that we should look at the needs of other people and try to respond to it.

When does open source make sense?

Who knows who wrote the paperclip in MS office? If it where open source, you could go to his house and shoot him
Opinions differed a bit, some thought that everything should be open source within 50 years, but Dirk and Linus believed that highly specificialized applications (like a software for a medical company) but that comodity software (office) definitely benefits from being OSS.

How about single signons?

The whole panel was worried about that because it would give microsoft too much control and information that it really shouldn't have. However, they are once again going to leverage their monopoly on the desktop to take over another market and put infrastructure in MSIE 6.0 to allow web sites to make all commercial transactions through microsoft's servers
Linus however thinks that microsoft will not be able to tax every single transaction over the web because governments would step in to do the taxing themselves.

You have a clue? You need a job?


Transgaming makes your windows games run under wine


You really want to support their efforts and subscribe to their service (I will, even though I don't even play games)


Obviously, I'm biased since VA Linux designed this server (and sold it to Appro when it got out of the hardware business)
but this it's really a sweet box: Dual Athlon 1.2G, 4 hot swappable SCA drives + CD Rom all in a 1U


You can touch, but you can't take home


Too bad the trip you can win is only for two

After the movie, we discussed with the director, J.T.S. Moore, and Bruce Perens, featured in the movie

This event unfortunately conflicted with EFF's party a few blocks away where Dmitry spoke, as well as Stallman and a representative from the EFF, but I got to see most of the crowd just before the speeches were over.

Yes, we were having fun


Come on, buy the few linux10 shirts we have left




'cause the money is going to the EFF

A big thank you goes to Raphael Moll for not only giving me a prerelease of RIG (Ralf's Image Gallery), and fixing it up to the last minute so that I could release this report.

With IDS, I just couldn't have made all the pictures available, but with RIG, I can, they're all here (note that if you browse anything else than the default size pictures or the 640 pixel wide resized pictures, they're going to have to be generated on the fly, and it could be a tad slower)

And here's the now usual little footer note:
If you'd like to use some pictures or text, please Email me first.

'Till next time
Marc Merlin (marc_web@merlins.org)

2001/09/05 (04:26): Version 1.0. Enjoy!
2001/09/05 (11:11): Version 1.1. Added a few words about IBM
2001/09/05 (13:45): Version 1.2. IPCP is really CPIP. Thanks Ron