ELC: Morton and Saxena on working with the kernel community

In many ways, Andrew Morton's keynote set the tone for this year's Embedded Linux Conference (ELC) by describing the ways that embedded companies and developers can work with the kernel community in a way that will be "mutually beneficial". Morton provided reasons, from a purely economic standpoint, why it makes sense for companies to get their code into the mainline kernel. He also provided concrete suggestions on how to make that happen. The theme of the conference seemed to be "working with the community" and Morton's speech provided an excellent example of how and why to do just that.

Conference organizer Tim Bird introduced the keynote as "the main event" for ELC, noting that he often thought of Morton as "kind of like the adult in the room" on linux-kernel. Readers of that mailing list tend to get the impression that there's more than one of him around because of all that he does. He also noted that it was surprising to some that Morton has an embedded Linux background—from his work at Digeo.

Morton believes that embedded development is underrepresented in kernel.org work relative to its economic importance. This is caused by a number of factors, not least the financial constraints under which much embedded development is done. An exceptional case is the chip and board manufacturers who have a big interest in seeing Linux run well on their hardware so that they can attract more customers. But even those do not contribute as much as he would like to see to kernel development.

An effect of this underrepresentation is a risk that it will tilt kernel development more toward the server and desktop. The kernel team is already accused of being server-centric, and there is some truth to that, "but not as much as one might think". Kernel hackers do care about the desktop as well as embedded devices, but without an advocate for embedded concerns, sometimes things get missed.

Something Morton would like to see is a single full-time "embedded maintainer". That person would serve as the advocate for embedded concerns, ensuring that they didn't get overlooked in the process. An embedded maintainer could make a significant impact for embedded development.

Not all kernel contributions need to be code, he said. There is a need just to hear the problems that are being faced by the embedded community along with lists of things that are missing. "Senior, sophisticated people" are needed to help prioritize the features that are being considered as well. Morton often finds out things he didn't know at conferences, things that he should have known about much earlier: "That's bad!"

Morton is trying to incite the embedded community to interact with the kernel hackers more on linux-kernel. He said that a great way to get the attention of the team is to come onto the mailing list and make them look bad. Unfavorable comparisons to other systems or earlier kernels, for example, especially when backed up with numbers, are noticed quickly. He said that it is important to remember that the person who makes the most noise gets the most attention.

One of the areas that he is most concerned about is the practice of "patch hoarding"—holding on to kernel changes as patches without submitting them upstream to the kernel hackers. It is hopefully only due to a lack of resources, but he has heard that some are doing it to try and gain a competitive advantage. This is simply wrong, he said, companies have a "moral if not legal obligation" to submit those patches.

There are many good reasons for getting code merged upstream that Morton outlined. The code will be better because of the review done by the kernel hackers; once it is done, the maintenance cost falls to near zero as well. He also touted the competitive advantage, noting that getting your code merged means that you have won—competing proposals won't get in. Being the first to merge a feature can make it easier on yourself and harder on your competition.

There are downsides to getting your code upstream as well. Most of those stem from not getting code out there early enough for review. The kernel developers can ask for significant changes to the code especially in the area of user space interfaces. If a company already has lots of code using the new feature and/or interface, it could be very disruptive; "sorry, there's no real fix for that except getting your code out early enough".

Another downside that companies may run into is with competitors being brought into the process. Morton and other kernel hackers will try to find others who might have a stake in a new feature to get them involved so that everybody's needs are taken into account. This can blunt the "win" of getting your feature merged. Some are also concerned that competitors will get access to the code once it has been submitted; "tough luck" Morton said, everything in the kernel is GPL.

Morton had specific suggestions for choosing a kernel version to use for an embedded project. 2.6.24 is not a lot better than 2.4.18 for embedded use, but it has one important feature: the kernel team will be interested in bugs in the current kernel. He suggests starting with the current kernel, upgrading it while development proceeds, freezing it only when it is time to ship the product.

He also suggests that a company create an internal kernel team with one or two people who are the interface to linux-kernel. This will help with name recognition on the mailing list, which will in turn get patches submitted more attention. Over time, by participating and reviewing others' code, the interface people will build up "brownie points" that will allow them to call in favors to get their code reviewed, or to help smooth the path for inclusion.

The kernel.org developers appear to give free support, generally very good support, Morton said, but it is not truly free. Kernel hackers do it as a "mutually beneficial transaction"; they don't do it to make more money for your company, they do it to make the kernel better. Morton is definitely a big part of that, inviting people to email him, especially if "five minutes of my time can save months of yours".

The decision about when to merge a new feature is hard for some to understand. Many consider Linux a dictatorship, which is incorrect, it is instead "a democracy that doesn't vote". The merge decision is made on the model of the "rule of law" with kernel hackers playing the role of judges. Unfortunately, there are few written rules.

Some of the factors that go into his decision about a particular feature are its maintainability, whether there will be an ongoing maintenance team, as well as the general usefulness of the feature. Depending on the size of the feature, an ongoing maintenance team can be the deciding factor. It is not so important for a driver, but a new architecture, for example, needs ongoing maintenance that can only be done by people with knowledge of and access to the hardware.

MontaVista kernel hacker, Deepak Saxena, gave a presentation entitled "Appropriate Community Practices: Social and Technical Advice" later in the conference that mirrored many of Morton's points. He showed some examples of hardware vendors making bad decisions that got shot down by the kernel developers, mostly because they didn't "release early and release often". There is a dangerous attitude that "it's Linux, it's open source, I can do anything I want" which is true, but won't get you far with the community.

Saxena has high regard for the benefits of working with the system: if your competitor is active in the community, they are getting an advantage that you aren't. Like Morton, he believes that some members of the development team need to get involved in kernel.org activities. "The community is an extension of your team, your team is an extension of the community".

He also has specific advice for hardware vendors: avoid abstraction layers, recognize that your hardware is not unique, and think beyond the reference board implementation. Generalizing your code so that others can use it will make it much more acceptable, as will talking with the developers responsible for the subsystems you are touching. Abstraction layers may be helpful for hardware vendors trying to support multiple operating systems, but they make it difficult for the kernel hackers to understand and maintain the code. The kernel.org folks are not interested in finding and fixing bugs in an abstraction layer.

He also points out additional benefits of getting code merged. Once it is in the kernel, the company's team will no longer have to keep up with kernel releases, updating their patches to follow the latest changes. The code will still need to be maintained, but day-to-day changes will be handled by the kernel.org folks. An additional benefit is that the code will be enhanced by various efforts to automatically find bugs in mainline kernel code with tools like lockdep.

It is clear that the kernel hackers are making a big effort to not only get code from the embedded folks, but also some of their expertise. There are various outreach efforts to try and get more people involved in the Linux development process; these two talks are certainly a part of that. By making it clear that there are benefits to both parties, they hope to make an argument that will reach up from engineering to management resulting in a better kernel for all.

Sorry for this highly offtopic/political and nontechnical reply. Please don't hate me ;).

>The decision about when to merge a new feature is hard for some to understand. Many consider
Linux a dictatorship, which is incorrect, it is instead "a democracy that doesn't vote". The
merge decision is made on the model of the "rule of law" with kernel hackers playing the role
of judges. Unfortunately, there are few written rules. 

I'm not a native english speaker, so i might be reading it wrong. But the above paragraph does
seem a bit "loaded" (In lack of a better word).

Don't claim that some thing is hard to understand "for some". That's arrogant? Especially
since the author seems to understand, from the following sentence(s): 
>Many consider linux a dictatorship, which is incorrect, it is a "democracy that doesn't
vote".

That's a phrase coming from a dictator defending himself ;). At the least the argument "that's
incorrect" certainly doesn't make it easy to understand. 

>The merge decision is made on the model of the "rule of law" with kernel hackers playing the
role of judges. Unfortunately, there are few written rules. 

A democracy with no voting and judges that have no rules. Don't claim the linux model is  a
democracy then, it obviously isn't. Democracy should more or less be equivalent to "Majority
rules" - when you don't vote how can majority rule? Explain why the model works for linux
instead ;). 

Being a dictatorship does not have to be bad. As long as people are free to go their own way
and leave(fork) it counters the arbitrary power of maintainers for the specific project(with
name).

Even though the consensus is that linus is a benevolent dictator. Which is my claim, google is
my reference ;). Maybe it's more of an an oligarchy today with many more maintainers.

>A democracy with no voting and judges that have no rules. Don't claim the linux model is a
democracy then, it obviously isn't. Democracy should more or less be equivalent to "Majority
rules" - when you don't vote how can majority rule? Explain why the model works for linux
instead ;). 

No it's a democracy to the right limit. i.e. you can post a feature, even if Linus himself
said OK, and another developer found something bad in your code, Andrew or subsystem
maintainer won't merge the code till it has zero objections.

It's just hidden democracy. i.e., there's no formal process but there's informal way which is
more pragmatic and solve the 'merge/not-merge' problem quite well (in most cases).

Even Linus cannot subvert the process. His request can be turned down by the maintainer, and
if he tried to force the kernel will fork. Even if Linus, Morton and the subsystem maintainer
try together to hoodwink the whole community, it will not be easy to prevent a fork.

> Even though the consensus is that linus is a benevolent dictator. Which is my claim, google
is my reference ;). Maybe it's more of an an oligarchy today with many more maintainers.

In any project with large number of humans, there *must* be a final authority. Without that
authority, people will just go in endless debates and *stop* doing the useful work. 

There's a fine line between dictatorship and 'final authority' but I think Linus is walking on
that line very well.

Lord Hailsham would have called it an "elective dictatorship" (the form of the British
constitution) -- but lets not get too constitutional because it requires some study or hell
will break loose on convention vs written.

>The linux model is obviously democracy. You don't even need 50% of votes to fork the kernel:
... So far Linux does not have significant forks - that means that system works.

Forking is individual freedom and not a form of government. Actually a key part of democracy
is that "majority rules", which is obviously >50%. I agree it's important that the possibility
of forking exists, as i wrote
"...As long as people are free to go their own way and leave(fork), it counters the arbitrary
power of maintainers for the specific project(with name)."


>The fact that there are no known rules is advantage, not weakness. If you have written rules
you have a way to game the system, but if the only way to get anything done is rough consensus
it's much harder. Take a look on Microsoft's attack on two systems: IETF (no strict rules) and
ISO (rigid structure with a lot of bureaucracy). First attack failed: after four years
SenderID is still "an Experimental Protocol for the Internet community" and not "an Internet
standard of any kind". Second attack succeded: now we have this huge mess called ISO/IEC 29500
standard.

Yes rules is always an complication. I don't disagree with this. The linux model has obviously
worked. But that doesn't make it a democracy.

My gripe was about the use of democracy as the true solution for all problems. If that holds,
then for linux to be a solution, we must argue that it is some kind of democracy. 

I think the problem is that we have created this dichotomy between democracy (good) and
everything else (bad) so we cannot apply some of the other terms that are probably closer to
the reality of the kernel development.

To me it looks more like an aristocracy where the best and the brightest have absolute control
based on internally developed conventions and pressures. Their power comes from 'knowing' what
is best and right, which is the main reason why they don't abuse their powers. The threat of a
peasant revolt (fork?) is what keeps them inline plus the ego massage from belonging to the
elite group.

A pure democratic method would probably produce a much lower quality product in the sense that
it would tend to converge on a boring consensus kernel that tries to be something to everybody
without pleasing anyone.

>I think the problem is that we have created this dichotomy between democracy (good) and
everything else (bad) so we cannot apply some of the other terms that are probably closer to
the reality of the kernel development.

Exactly what i wanted to get across - but much clearer than what wrote. Thank you! :)

Exactly what i wanted to get across - but much clearer than what I wrote. Thank you! :)

> To me it looks more like an aristocracy where the best and the brightest
> have absolute control based on internally developed conventions and
> pressures. Their power comes from 'knowing' what is best and right, which
> is the main reason why they don't abuse their powers.

This is called a meritocracy.  An aristocracy is where rule is hereditary.

In fact, the word "aristocracy" derives from the Greek for "rule by the best". (It's also
worth noting that the coiner of the term "meritocracy" - presumably "rule by the most
deserving" - saw it as a *dystopian* ideal; and it doesn't take too much imagination to see
that what starts out as meritocracy soon ossifies into aristocracy.)

Nice post. Lets try and avoid calling things by labels we like, as opposed to labels which
actually apply.

I applaud Andrew and Deepak for their efforts here. The best Linux projects are run as
extensions to the community. The company gains by interacting with experts in the community,
resulting in better code. Linux gains by new features or hardware support being added to the
kernel.

We need to make more hardware and silicon vendors aware of the benefits in working with the
community. Too many SoC vendors, for example, don't submit their kernel patches to the
community. They prefer to distribute their own "patched" kernel to their customers directly,
as if they're adding value by doing so. In the last two years, I've worked on projects at
different companies using SoCs from three different silicon vendors and none of those silicon
vendors were actively working on submitting their code to the kernel. Why? The article gives
several reasons that I won't repeat again here. A few more are

- High perceived difficulty in getting patches approved. Companies who try to submit work to
the kernel lists can be scared off by what they perceive as aggressive or unreasonable
comments from the community. The Linux community could do better in this area. This is where
Andrew's initiative comes in - educating people about how to work with the community. Greg KH
is also active on this.

- Concern that releasing open source drivers for hardware somehow gives away competitive
advantage. This is a hard nut to crack. If companies understood more about the potential
advantages of submitting code into the kernel, more might do so. Some companies though are
just stuck in their ways.

- The quality of the vendor's code isn't up to scratch to be accepted into the kernel. The
hardware vendor knows it so doesn't bother submitting it. I even saw one comment in one
vendor's kernel distribution along the lines of "Heaven forbid we are forced to release this
code to linux-mips.org. We'll be laughed at." Unfortunately, some code written by or for
hardware vendors is done in a way that is never intended for submission into the kernel. As a
result, it is often buggy and difficult to keep up to date when new kernel features come
along.

The bottom line is that if you are starting an Embedded Linux project and are evaluating
hardware alternatives, choosing hardware that is already supported by the Linux kernel could
save months of software effort, even if the hardware vendor claims Linux support by "special"
out-of-tree code. Let's help them understand what's in it for them in working with the
community.

Shameless plug. I published a white paper on Best Practices in Embedded Linux a while ago.
http://www.katalix.com/doc/embedded_linux_best_practices.pdf

-- 
James Chapman
Katalix Systems Ltd
http://www.katalix.com
Catalysts for your Embedded Linux software development

> Abstraction layers may be helpful for hardware vendors trying to support multiple operating
systems, but they make it difficult for the kernel hackers to understand and maintain the
code. The kernel.org folks are not interested in finding and fixing bugs in an abstraction
layer.

I have always found this to be rather a shame, as the Linux kernel is the largest repository
of free device drivers in existence.  It would be nice if some effort were made to make them
benefit other projects which could make use of them as well.

We have a lot of cross-pollination between Linux and the BSDs, the people developing for each
O.S. can read the other's source code just fine.

Granted, you can't always re-use the code itself, but that *really* isn't what matters most,
especially for drivers.  In fact, it is best if things remain as they are.  We get better code
quality and a much higher degree of freedom to change interfaces that way.

The really precious information (how to talk to the hardware) is already shareable among all
since the code is open anyway.  That's the only thing that matters.

You may be interested in the videos that we took of Andrew's and Deepak's  presentations. All
the videos that we took are announced on http://free-electrons.com/news/news.2008-05-13