IT managers urged to dip toes into community-based support

For traditional businesses, navigating the world of open source software development is very different from working with a vendor. Just ask Ed Reaves, the platform product line manager for Canada-based Nortel Networks, which uses Linux to run the switches that handle mobile telephone call routing.

At the Linux Foundation's second annual Collaboration Summit, Reaves asked a panel of Linux kernel experts what he needed to do to get help from the kernel community to integrate code changes that would support Nortel's switches.

The switches run Wind River's Linux Platform for Networking Equipment Version 1.4, which uses the 2.6.14 kernel. Nortel wants to upgrade to the 2.6.21 kernel, but that takes a lot of work because Nortel has to do extensive custom coding with each new kernel to make it work with its switches. That's because the needed code isn't built into the mainline Linux kernel.

Nortel has been trying to get code changes included in the kernel, which would sharply reduce the amount of rewriting that's needed when new kernels are adopted.

To solve Nortel's problem, the panel of experts offered a suggestion: Learn to speak "community."

"His problem is fairly well-represented in the industry," said panelist James Bottomley, a board member of the Linux Foundation and chairman of the group's technical advisory board. "It stems from the change in paradigms that Linux forces on people."

If the problem involved IBM's AIX Unix operating system, Nortel could ask IBM to help them add the code to the AIX kernel. "In Linux, there's nobody actually directly working on that job" of making changes for a specific company, Bottomley said.

In Linux, a company has to get involved in the community to get the changes it needs incorporated into the mainline kernel code, Bottomley said. "It's a very complicated thing. But at the end of the day, it also gives you a lot more opportunity" to get what you need included in future kernels.

"It's not the way the industry is used to operating. The conversation in the old world would have started at the executive level. It's a much more frightening kind of thing, especially from the project planning point of view," Bottomley said.

"In Linux, you always have the opportunity to push it in whatever direction you want as long as you get involved" in the community, he said. "The problem is really one of education," he said, explaining that enterprise IT leaders need to learn to get what they need from the open source community by working within it, rather than simply picking up a phone and calling a traditional software vendor.

Going to the wrong place for help

In Nortel's case, the company was sending the needed kernel code patches to their embedded Linux vendor, Wind River, which provides the operating system used in Nortel's phone switch hardware.

In general, Bottomley said, such patch requests or bug fixes should be sent to the Linux kernel maintainer teams, rather than to the Linux operating system vendors that build their applications around the kernel.

By sending requests to the kernel teams, Bottomley said, Nortel engineers would have been directly involved in getting the code changes they wanted.

As others join in to discuss their situations on the kernel mailing lists, they can start collaborating to pursue similar changes. Such discussion will solve a majority of the problems for all the users, he said. "You need to work with others who want it," Bottomley said. "Those others who want it [ironically] are often your competitors in real life."

Page Break

In Nortel's case, the engineers want to get their needed code into the kernel in order to control high-resolution timers that send "wake-up" calls to the kernel for use in the switches that route mobile phone calls.

Nortel had its engineers create custom code to get the work done, but since they did it on their own, they have to redo their work every time a new Linux kernel comes out for their phone switches.

Such a problem is common for many companies that do their development work outside the open source community, Bottomley said. "That's where the big strain comes," he said. Going solo can cost companies thousands of dollars in costs and many hours in staff time, and then they have to do it all again manually each time a new kernel is released.

To avoid having to rewrite the code each time a new kernel comes out, Nortel wants to get its changes into the mainline kernel so they are there automatically for every future version.

"Your maintenance burden is significantly reduced," Bottomley said. "You still have to test and check the code, but you don't have the massive front-end work in the race to keep up with the kernel."

Typically, Linux kernels are updated every two to three months by the kernel maintainers. That can be a challenge for companies like Nortel that want to keep up with the latest kernel, but it's also where getting involved in the process pays off, Bottomley said.

Getting started

The way to start, Bottomley said, is to have your developers post to the kernel community mailing lists and monitor them for the issues you are experiencing. "Start posting and getting involved in the discussions," he said. "It's a community, sharing development. They're negotiating to get the best implementation that suits them all."

Some of the key kernel mailing lists are:

• {xref:http://marc.info/|marc.info]], the kernel mailing list archive.

• kernel.org, the place to start and to find the proper mailing lists for the communities and subcommunities where your developers want to get involved.

• kernelnewbies.org, for resources about the kernel, the processes and the communities.

• The Linux Foundation, for more helpful information and collaboration-building resources.

Once your developers are involved, he said, they can stay involved after your changes are included so you can watch over the kernel and keep your changes updated.

Daniel Frye, manager of IBM's Linux Technology Center and another conference participant, said he's seen plenty of business users that have similar issues to what Nortel faces with its switches. For those companies, he said, having to learn the open source community's way of doing things is "counterintuitive."

"They don't how it works," Frye said. "Some of them have cultural barriers, some of them have misconceptions."

The Linux Foundation is working on ways to teach businesses how to get what they need from the open source community and how to communicate better with open source developers, Frye said.

That starts with knowing who makes up the communities that are working on the code in the kernel parts that affect your business, he said. "If you don't know who the kernel maintainer (the lead person for a specific part of the kernel development process) is, then you're not even started," Frye said. "You've got to know who the subcommunity is" and who makes up those five to 10-person groups.

To get this information, Frye recommends that companies ask lots of questions, do research and poke around in the mailing lists to see how they work. Communities can work in different ways, making them even more intimidating, he said. "Listen and find out how decisions are made, and then go ahead and participate, then adapt to the community."

Page Break

Frye also urged companies to get their legal team on board early. Since a company's developers will be contributing code, the legal team will want to ensure there are no violations of the company's intellectual property rights.

Also key, Frye said, is that you have to appraise work done by developers differently in the open source community. "We don't care whether someone gets their own code accepted" in an open source project, he said. "We care if we get out of the community code that we need."

Let someone else do it

Another panel member, Theodore Ts'o, an IBM technical staffer and a fellow and chief platform strategist at the Linux Foundation, urged a bit of caution for businesses that are considering jumping into the kernel development community.

"There are many, many options," Ts'o said. "You don't have to do it yourself." It can be very intimidating to jump into a kernel mailing list, which can receive 800 or more posts daily. "It's very daunting," he said.

What might work better for some enterprise users is to bring in a development partner to handle much of the community work, he said. "If people want to do it, I don't want to discourage them, but at the same time, I worry that they'll jump in and then jump out."

"It's like a car," Ts'o said. "Some people like to fix a car themselves. Some will go to a dealership and some will go to an independent repair shop."