| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
LWN.net Weekly Edition for July 10, 2008SELinux and Fedora Red Hat has undoubtedly done more to make SELinux usable than any other organization, but has it actually reached the point where it can be enabled by default for all desktops? The Fedora project clearly thinks so. Not only is SELinux enabled, but the installer no longer has an option to disable it or to put it into "permissive" mode. Most of the posts in a thread on the fedora-devel mailing list see that as the right choice, but some are not so sure. Jon Masters started things off by making a request to restore the installation option, giving several reasons summing up with:
But there are numerous other justifications I could give, including my
personal belief that it's absolutely nuts to thrust SE Linux upon
unsuspecting Desktop users (who don't know what it is anyway) without
giving them the choice to turn it off.
His reasons were unconvincing to many as he was not considered to be a "normal" desktop user; the things he was doing were much more technical than the users that are being targeted by the SELinux policies distributed with Fedora 9. The problems he reported were resolved quickly, but the fact remains that there are paths through Fedora—even just using desktop applications—that will result in SELinux-caused failures. The Red Hat SELinux team is very responsive, but users will get frustrated quickly if things they are trying to do fail in mysterious (to them) ways. Alan Cox argues against providing an installation choice because he doesn't think users have enough context to make a sensible choice. He likens it to a car with multiple choices for safety features:
"This car has brakes, enable them ?"
"Would you like the seatbelts to work ?" "Shall I enable the airbag ?" When push comes to shove, Masters and a few others see the default of SELinux installed in "enforcing" mode as being too restrictive. It is likely to cause users to become annoyed with Fedora as a whole because one or more paths through the applications have not yet been tested. That, unfortunately, is the crux of the issue: SELinux policies are being developed in a reactive manner based on testing applications and adding exceptions for actions they perform. As a security tool, SELinux is a good choice, because it essentially denies everything by default. Policies are added that will allow certain actions for users and applications. Its complexity is legendary, however, which is why Red Hat (and others) have made a substantial effort to make it work semi-invisibly. They started by generating policies for network-facing services and have now moved into securing desktop applications, particularly programs like web browsers which are increasingly the target of attacks. SELinux has three modes, disabled, which turns off SELinux, permissive, which just logs attempts to do things that violate the policies, and enforcing, which disallows any access that is denied by the policies. When getting applications to work with SELinux, permissive mode is typically used. The log messages are analyzed to determine what changes should be made to the policies or to the application so that they work together. If there are features that were not tested in the application that require additional privileges, the first user that tries that feature in enforcing mode will run into trouble. When that happens, SELinux can be put into permissive mode with a simple GUI or configuration file change, followed by a reboot. One of the problems is that users may very well not know that SELinux is the source of their problem. There are tools, like SETroubleShoot, that can help alert users, but it is still a frustrating, hard to comprehend problem at times. Once the user has "fixed" the problem by disabling SELinux, they are unlikely to turn it back on. It is a difficult choice, but Fedora is firmly on the side of forcing non-technical users into using SELinux, at least until it breaks. More technical users will know about SELinux and, perhaps, be able to make more informed choices. One of Red Hat's SELinux developers, James Morris, neatly sums up the reasons it is important to continue pushing SELinux:
The only way to really make progress in improving security is to make it a
standard part of the computing landscape; for it to be ubiquitous and
generalized, which is the aim of the SELinux project.
[...] Punting the decision to the end user during installation is possibly the worst option. It's our responsibility as the developers of the OS to both get security right and make it usable. It's difficult, indeed, but not impossible. There are efforts underway to add easier ways for users to report SELinux log messages, perhaps even in an automated way, so that policy or application problems get identified and fixed more quickly. While it may not be easy for long-time Linux users to adjust to an SELinux-enabled system, it is getting to the point where average users, who never use the command line, rarely run into problems. And those are just the kind of users who need the level of security that SELinux can provide.
Questions and answers with Stormy Peters Those who have followed the GNOME project over the last few years have seen the wishlist item for a "business manager" or "executive director" for the GNOME Foundation; the subject was especially likely to come up during Foundation board elections. This position has remained unfilled for some time, seemingly a result of uncertain funding and the difficulty of finding the right person. These problems would appear to be in the past now; on July 7, the GNOME Foundation announced that this position would be filled by Stormy Peters, formerly of OpenLogic.Stormy now has the challenge of helping an energetic and independent-minded development community build on its success and achieve its ambitious goals for the future. We asked her a few questions about how she thought that might go; here's what we got back. LWN: This is a new position, in that the GNOME Foundation has never had an executive director before. So people may be wondering what you'll actually be doing. How do you expect to be spending your time in this position?
Actually, the GNOME Foundation has had an executive director before
but not for the past few years. I will spend my time strengthening
relationships with the existing sponsors, working on finding new
industry partners and helping the Board of Directors and the
community execute some of their great ideas for GNOME. The GNOME
community's goal is to provide an easy to use, intuitive interface
for Linux and Unix as well as a powerful development platform.
A year from now, what do you hope your biggest accomplishments will be?
The GNOME community has a tremendous amount of passion and a real
dedication to making a development platform and a desktop that is
easy to use. I think showing the world that, getting the word out
and showing how it is changing the way people are able use their
computers and mobile devices is key. So to answer your question,
I'd like to see a stronger Foundation (more sponsors and members),
increase the amount of great ideas that get executed, and make
GNOME a household name. :)
Next year, it seems reasonably likely that there will be a combined GNOME/KDE developers conference in Europe. What are your thoughts on the current state of cooperation with KDE, and how do you think it could be improved?
I hope we have a combined GUADEC/Akademy next year. KDE and GNOME
have been working more closely together during the past year or so
and they have accomplished some good things like with dbus. I think
anytime you get great developers together, good things happen.
One high-profile GNOME goal was 10x10 - 10% of the desktop market by 2010. In mid-2008, it seems fairly clear that this goal will not be achieved. Do you think that the desktop remains a suitable target for free software, or should GNOME deemphasize the traditional desktop in favor of other goals?
I do think that a free and open source desktop is still a great
goal. While the number of free and open source desktops out there
might be small, it is growing tremendously. Just look at the number
of laptops that ship with GNU/Linux (from Dell, Asus and other) as
well as the number of mobile devices that are based on free and
open source software.
Though the GNOME Foundation is not intended to control the technical direction of the project, it clearly cannot be without influence there. Are there technical directions you would like to see the development community take, directions which would help to convince manufacturers to incorporate GNOME technologies and contribute to GNOME development?
I'll be working closely with the community and the board of
advisors to figure out how I can best help with technical
directions. One thing we'd like to see from our sponsors - through
our board of advisors - is more information on what end-users would
like to see in GNOME.
In the past you have spoken about how introducing money into free software development can have a demotivating effect on developers. Do you fear that sort of problem as GNOME becomes more commercially successful? How would you hope to avoid that kind of difficulty?
I don't think it's an issue in the short term as growing the GNOME
Foundation doesn't directly correspond to hiring lots of
developers. But that said, I think the key is maintaining the
intrinsic motivations that make GNOME contributors such a
passionate group of developers.
Thanks to Stormy for being kind enough to answer our questions in the middle of what must have been a highly busy time at GUADEC in Istanbul.
Notes on the Viacom ruling Google's purchase of YouTube always seemed questionable to some observers: it looked as if Google were buying itself a whole new source of copyright lawsuits. One of the benefits of that purchase came through on July 2, when a U.S. District Court ordered Google to hand over its complete set of YouTube traffic logs, containing information about every video viewed on the service. See Groklaw for the full text of the order. If this order stands (and it appears that Google will not appeal it), millions of users worldwide will have their viewing data handed over to a litigious entertainment industry company. There's a couple of important implications to draw from this turn of events, so LWN will venture a little far afield and take a look.The data involved includes, for each video viewed, the time, which video was involved, which YouTube user account was used, and the IP address the request came from. Viacom claimed that the privacy of YouTube users is not threatened by this release of data, and the court agreed. But account names can be correlated across sites, and IP addresses (especially time-correlated IP addresses) can easily identify exactly who was watching a particular video. Viacom promises it would never use this data to launch enforcement actions against individuals; the fact that the company feels the need to make that promise suggests that Viacom feels it could use this data to that end. One other interesting aspect of the ruling which has been commented upon less is this: Google has also been ordered to hand over every video which has been removed from the site. Once again, that is a great deal of data. It also drives home the point that, on a site like YouTube, nothing is really removed: all of those "removed" videos are still there, waiting for some company with enough lawyers to go after it. All of this data is to be handed over regardless of what jurisdiction the users thought they were in. Nobody's privacy or data retention laws apply here. This is a worldwide compromise of personal data. So lesson number one is obvious: attending to one's personal security requires being very careful about the data tracks that one leaves on other peoples' servers. Regardless of any site's privacy policy or any country's data sharing laws, that data is there for the grabbing. The course of events which led to the compromise of vast amounts of video-viewing data can also lead to the disclosure of electronic mail, accounting data, online chat sessions, purchase histories, software downloads, or which edgy Second Life neighborhood one likes to hang out in. Indeed, records of video viewing activity are more strongly protected in the U.S. than many other types of data; other types of information may well prove easier to get. What we leave on remote machines seems to stay there indefinitely, and it's an open book for those with sufficient legal power on their side. [PULL QUOTE: If you gather together that much information on the behavior of many millions of people, somebody, somewhere, is going to try to get their hands on it. END QUOTE] The second lesson is for anybody running a publicly-available server, as many LWN readers do. The video activity database being grabbed by Viacom is said to be about 12 terabytes deep - before getting into the "removed" videos. It should not be surprising that a data stash of that size would attract this kind of action. If you gather together that much information on the behavior of many millions of people, somebody, somewhere, is going to try to get their hands on it. How could it possibly be any other way? Not enough people are asking this question: why does Google/YouTube hold that much data about its users? Why does it retain the ability to replay their actions years after the fact? And why do "removed" videos not go away? If that data did not exist in the first place, there would be no question of disclosing it to an attacking corporation. A company which keeps that amount of data around is prioritizing whatever commercial value it sees in that data over the privacy and security of its users. And, by inviting raids from corporations (which we hear about) and governments (which we might not hear about), such companies are not helping their own security either. So there are strong arguments for simply not retaining all that data in the first place. Naturally, some governments are doing their best to force that kind of retention, but that's a different battle. In the absence of legal constraints, a standard policy mandating short data retention periods makes a lot of sense. It behooves all of us to think about what kind of data we leave lying around - either through our activities or by facilitating the activities of others - and to keep it to a minimum. The most secure data is data which does not exist.
Page editor: Jonathan Corbet Security Secrecy and the DNS flaw By now, most folks will have seen reports of the design flaw discovered in DNS as it has seen fairly widespread coverage, even in the non-technical press. It is rare to see such a coordinated disclosure and security update amongst that many of the big players in the computer industry. While fixes abound, the actual problem has yet to be disclosed, which has both positives and negatives. Responsible disclosure policies dictate that vulnerabilities be kept secret until all affected vendors can create an update. Because this flaw is in the design of DNS, most implementations were affected. This still doesn't quite explain the roughly six months between the discovery of the problem and the release of the fix. Evidently it took a meeting of the minds at the Microsoft campus in March to decide upon the right course of action. Once the fixes were done, presumably they were released on the next "patch Tuesday"—Microsoft's monthly security update day. Normally, once fixes are available, information about the vulnerability is released. But, for a number of reasons, that has not happened in this case. One of the main reasons is that DNS is an essential internet service and it will take time for affected users to patch their systems. In addition, there have been no reports of this flaw being exploited "in the wild", reducing the pressure to divulge it. Security researcher Dan Kaminsky discovered the flaw and he has yet another, "blatantly selfish" reason for keeping it quiet as he would like to be able to announce it at Black Hat in Las Vegas in early August:
While I'm out there, trying to get all these bugs scrubbed — old and
new —
please, keep the speculation off the @public forums and IRC channels. We're
a curious lot, and we want to know how things break. But the public needs
at least a chance to deploy this fix, and from a blatantly selfish
perspective, I'd kind of like my thunder not to be completely stolen in
Vegas.
None of these seem like horrible reasons to keep the vulnerability quiet for a time (roughly 30 days), but they do leave some DNS implementations and worried administrators without the information they need to evaluate the situation. Administrators do not know what traffic patterns or other symptoms to look for to determine if exploits are being attempted. Smaller, less prominent DNS implementations were not included in the collaboration, thus they don't have enough information to decide whether they are vulnerable or not. A perfect example is Dnsmasq, a lightweight DNS server for smaller networks. Dnsmasq is often used in embedded Linux distributions targeted for home wireless routers. Simon Kelley, Dnsmasq developer, was asked about the vulnerability; his response speaks volumes:
I wasn't contacted in advance about this, and no patch for dnsmasq has
been released. Since the exact nature of the new vulnerability has not
(as far as I know) been announced, I don't know if dnsmasq is vulnerable.
Kelley has since released a patched version, but it is still unknown whether it is needed or, really, if it even fixes the problem. It is difficult to know for sure that a security hole has been closed if information about the hole is not available. This points to the problems that can come from withholding vulnerability information. Based on the patches and some information from Kaminsky and others, it is clear that this is a cache poisoning vulnerability. Since source port randomization is the change that was applied to alleviate, but not eliminate, the flaw, we can surmise that Kaminsky found a way to reduce the number of spoofed replies that need to be sent to something tractable. According Internet Systems Consortium, developers of the BIND DNS server, the only true solution is DNSSEC, which implies that the current fixes only make cache poisoning less likely, not impossible. Source port randomization is a technique that has been advocated by Daniel J. Bernstein (i.e. djb) for many years. He implemented it in his djbdns name server long ago. Essentially, it chooses a random source UDP port for each query that the name server makes, which has the effect of increasing the randomness that an attacker needs to be able to predict before being able to poison the cache. While the market share of Dnsmasq may be miniscule, there are certainly other DNS implementations that are also concerned. In addition, we are relying on those who are "in the know" to be on the lookout for suspicious traffic that might indicate the vulnerability being exploited. Kaminsky is certainly under no obligation to reveal anything, but one wonders if the safest course would have been for him to provide details now, even at the expense of his "thunder".
Security news Dan Kaminsky Discovers Fundamental Issue In DNS: Massive Multivendor Patch Released (Securosis.com) Dan Kaminsky has found a flaw in the design of DNS that can allow cache poisoning as an article at Securosis.com details. This has lead to a CERT advisory as well as a coordinated release of patched DNS servers from all affected vendors. Evidently source port randomization is helpful in alleviating the problem. "The issue is extremely serious, and all name servers should be patched as soon as possible. Updates are also being released for a variety of other platforms since this is a problem with the DNS protocol itself, not a specific implementation. The good news is this is a really strange situation where the fix does not immediate reveal the vulnerability and reverse engineering isn't directly possible." That last claim seems rather strong, time will tell, but it makes sense to be prepared to upgrade affected servers as soon as distributions make them available.
Mozilla Foundation developing a model for a security metric (heise online) An article at heise online describes Mozilla's new security metrics project, which is an attempt to measure the relative security of Firefox. "One of the main factors cited is how long Firefox users are exposed to a threat while a hole remains unpatched. The developers say they want to use the security metric derived from the results to identify any problematic stage in the development and patch process."
New vulnerabilities bind9: DNS cache poisoning
glib2: buffer overflow
jetty: multiple vulnerabilities
linuxdcpp: denial of service
mercurial: unauthorized access
openldap: denial of service
php: multiple vulnerabilities
phpMyAdmin: cross-site scripting
pidgin: buffer overflow
poppler: memory management bug
ruby: directory traversal vulnerability
ruby: integer overflow
sipp: buffer overflows
squid: denial of service
vsftpd: denial of service
webkit: memory corruption
Page editor: Jake Edge
Kernel development Release status Kernel release status The current 2.6 development kernel is 2.6.26-rc9, released on July 5. "Enough changes that we needed another -rc, and the regression list isn't emptying fast enough either (probably because a number of people, including reporters, are vacationing)." Along with the fixes there's a new driver for cameras which implement the standard USB video class spec. The long-format changelog has the details.A few dozen changesets have been merged since 2.6.26-rc9, as of this writing. They are mostly fixes, but there is also a printk() extension which allows for higher-level format string specifiers; see below for details. The current -mm kernel is 2.6.26-rc8-mm1. Recent changes to -mm include the MMU notifiers patch, a new alloc_pages_exact() function for more efficient large allocations, a lot of checkpatch.pl tweaks, and a patch series ending with "revert-revert-revert-revert-linux-next-revert-bootmem-add-return-value-to-reserve_bootmem_node.patch" that one probably doesn't want to know about. The current stable 2.6 release is 2.6.25.10, released on July 2. "It contains a number of assorted bugfixes all over the tree. And once again, any users of the 2.6.25 kernel series are STRONGLY encouraged to upgrade to this release."
Kernel development news Quotes of the week
The problem is that SystemTap hasn't really benefited from
community based innovation largely because it doesn't have much of
a community. The bigger picture problem Red Hat didn't see when
they accepted the cash was that this project wouldn't generate a
community just from the usual publish the code and they will come
philosophy. The result is what we see to day: a klunky and
accident prone tool that has sun engineers writing trite little
homilies on the benefits of a planned political operating system.
-- James Bottomley
Think about some of the evil perpetrated by hal and the userspace
suspend-resume scripts (and how much complexity with random XML
fragments getting parsed by various dbus plugins), and tell me with
a straight face that you would trust these modern-day desktop
application writers with this interface. Because they *will* find
some interesting way to (ab)use it.....
-- Ted Ts'o
It's readily obvious that people (ie: top-level maintainers) aren't
even compile-testing their own stuff once it's merged into
linux-next. You say (but don't provide evidence that) linux-next
is too unstable to develop against. Well guess why? Because
people are choosing to let it be that way.
-- Andrew Morton
Quotes of the week part 2: the firmware flamewar
Not 15 minutes after David posted his note, we're now up to 11
reports; and this is only from an -mm patch series. Can you
imagine the number of bug reports if this were allowed to ship in a
mainline kernel.org release? One good thing is that we can
definitely show that there people that are downloading, compiling
and trying to build the -mm kernel. :-)
-- Ted Ts'o
External firmware is by design an error prone system, even with
versioning. But by being built and linked into the driver, it is
fool proof.
-- David Miller
On a technical basis alone, we would never disconnect a crucial component such as firmware, from the driver. The only thing charging these transformations, from day one, is legal concerns.
All forms of change introduce _some_ risk of breakage, of
course. In this case, as usual, I've tried to be careful to avoid
regressions. The most important part, obviously, was having a way
to build firmware into the static kernel.
-- David Woodhouse
When it comes to _modules_, doing that would introduce a certain amount of complexity which just doesn't seem necessary -- if you can load modules, then you have userspace, and you can use hotplug for firmware too. Especially given that so many modern drivers already _require_ you to do that, so the users understand it, and the tools like mkinitrd already cope with it -- checking MODULE_FIRMWARE() for the modules they include and including the appropriate files automatically. [...] You need to stay sober for long enough to say 'Y' when it asks you if you want to build the required firmware into the kernel. And we even made that the _default_ now, for the benefit of those who can't stay sober that long. (Perhaps we'll make 'allyesconfig' the default next?)
The current development kernel is...linux-next? One of the development process advantages brought by git (and by BitKeeper before it) is the ability to see the up-to-the-second, bleeding-edge status of Linus's tree. So any developer who wants to know where the front edge of development lies can grab that tree and make patches fit into it. But the value of the mainline repository for development would appear to be less than it once was. The mainline is no longer where the action is.Consider, for example, this response from Andrew Morton after finding that a patch posted to linux-kernel would not compile for him:
I assume this patch was prepared against some ancient out-of-date
kernel such as current Linus mainline. Guys, we have a new
development tree now.
He followed up with this statement:
But what I am repeatedly seeing is people cheerfully raising 2.6.27
patches against the 2.6.26 tree when we have a nice 2.6.27 tree for
developing against. Those days are over, guys.
So the message would appear to be clear: development work should be done against the linux-next tree rather than against the mainline kernel. There are some clear advantages to having work done in this way. Patches developed against linux-next should merge cleanly during the next merge window. Developers will be testing each other's trees as they work, causing bugs to turn up earlier in the process. And, of course, Andrew won't have to complain about patches which fail to build for him - at least, not as often. Linux-next is a somewhat strange base on which to try to develop, though. It is built anew every day from over 100 subsystem trees, each of which can, itself, change from one day to the next. So linux-next is a moving target, just like the mainline is. But, unlike the mainline, linux-next has no consistent or coherent history. Every day's linux-next tree is a completely new creation with a unique - and transient - history. Consider a developer who bases some work on a mainline release - 2.6.26-rc9, say. That developer's work will be derived from a specific commit in the mainline tree, known as b7279469d66b55119784b8b9529c99c1955fe747 in this case. The history from 2.6.26-rc9 is well defined, and that series of patches can be merged into any other repository which also contains 2.6.26-rc9; the identity of that commit is consistent and immutable across all repositories. With such a development tree, it is (relatively) easy to track the mainline as it advances, and to merge one's work when the time comes. A git tree based on the mainline sits on a solid foundation. It is not possible to base a tree on linux-next in the same way. Development can begin at a specific commit, but tomorrow's linux-next tree may not contain that commit at all. The various component trees will have advanced independently of the previous day's linux-next tree, which can, in itself, complicate things. But the process of making all those trees come together can involve tasks like moving patches from one tree to another, or fixing intermediate patches which break things. That makes the end result better, but at the cost of rebasing those trees. Rebasing completely rewrites the development history, causing the old history to disappear from the tree. So a patch series based on the previous history loses its foundation. And, since linux-next is built from its components every day, a patch developed on top of linux-next may, when integrated into that tree, be merged somewhere in the middle of the sequence; in other words, the patch will be merged into a tree which differs considerably from the tree on which it was developed. As Stephen Rothwell, the maintainer of the linux-next tree, put it:
One downsides of the way linux-next works is that, because it is
recreated every day, you cannot really base anything on it that is
to be merged into it.
Another interesting aspect of linux-next development involves API changes. The longstanding rule in kernel development is that internal kernel interfaces can be changed if there is a good reason to do so, but that the person making the change is obligated to fix all in-tree code broken by that change. If an API change is introduced into linux-next, though, the developer is simply not able to fix any code which enters linux-next by way of the other subsystem trees. If the developer does get patches into those trees for the API change, they can no longer be built on top of kernels which lack that change - the mainline, for example. API changes have, in other words, become harder to do - a situation which some may see as a good thing. What all this means is that API changes must be handled through techniques like the creation of backward-compatibility layers; those layers can then be removed a development cycle or two later once the transition is complete. Or changes can be split up and added to individual subsystem trees; that, however, can lead to interesting ordering dependencies between the trees. In some cases, we are seeing 2.6.27 changes being merged into 2.6.26 in stub form as a way of making all of the pieces fit together. Then, there is the simple matter that developers like to have a stable base upon which to create their code. The linux-next tree, since it contains large amounts of relatively new code, will also contain its share of new bugs. That makes developers, who are often having enough trouble just tracking down their own bugs, somewhat grumpy. Development against the mainline tends to have a lower probability of forcing developers to look for bugs which are not of their own making. Many of these complaints have an easy answer: the pain which comes from making all the pieces fit together in linux-next must be faced at some point anyway. The real difference is that linux-next allows those problems to be dealt with at leisure, while the older "merge everything in the mainline" model compressed much of that work into the merge window. How beneficial that really is will be seen for the first time in the 2.6.27 merge window; if linux-next is serving its intended function, 2.6.27 should come together with rather less hassle than its immediate predecessors did. But, regardless of the value provided by linux-next for integration and testing purposes, the fact remains that it is a difficult platform upon which to develop patches. That process is somewhat like building a house on a sand bar; overnight the tide comes in and completely reshapes the land underneath you. That is why most (possibly all) of the subsystem trees used to assemble linux-next are, themselves, based on the mainline. The solution to that problem will have to evolve over time. The linux-next tree is a new institution which is still finding its proper place in the development process. Easier ways to develop patches against the linux-next tree will certainly be worked out; it may well turn out that quilt-like tools work better for this task than git. But, for now, linux-next is an excellent integration and testing resource, but it has not quite yet managed to become the true Linux kernel development tree.
Multiqueue networking One of the fundamental data structures in the networking subsystem is the transmit queue associated with each device. The core networking code will call a driver's hard_start_xmit() function to let the driver know that a packet is ready for transmission; it is then the driver's job to feed that packet into the hardware's transmit queue. The result is a data structure which looks vaguely like this:
"Vaguely" because the list of sk_buff structures (SKBs - the internal representation of packets) does not exist in this form within the kernel; instead, the driver maintains the queue in a way that the hardware can process it. This is a scheme which has worked well for years, but it has run into a fundamental limitation: it does not map well to devices which have multiple transmit queues. Such devices are becoming increasingly common, especially in the wireless networking area. Devices which implement the Wireless Multimedia Extensions, for example, can have four different classes of service: video, voice, best-effort, and background. Video and voice traffic may receive higher priority within the device - it is transmitted first - and the device can also take more of the available air time for such packets. On the other hand, the queues for this kind of traffic may be relatively short; if a video packet doesn't get sent on its way quickly, the receiving end will lose interest and move on. So it might be better to just drop video packets which have been delayed for too long. On the other hand, the "background" level only gets transmitted if there is nothing else to do; it is well-suited to low-priority traffic like bittorrent or email from the boss. It would make sense to have a relatively long queue for background packets, though, to be able to take full advantage of a lull in higher-priority traffic. Within these devices, each class of service has its own transmit queue. This separation of traffic makes it easy for the hardware to choose which packet to transmit next. It also allows independent limits on the size of each queue; there is no point in filling the device's queue space with background traffic which is not going to be transmitted in any case. But the networking subsystem does not have any built-in support for multiqueue devices. This hardware has been driven using a number of creative techniques which have gotten the job done, but not in an optimal way. That may be about to change, though, with the advent of David Miller's multiqueue transmit patch series. The current code treats a network device as the fundamental unit which is managed by the outgoing packet scheduler. David's patches change that idea somewhat, since each transmit queue will need to be scheduled independently. So there is a new netdev_queue structure which encapsulates all of the information about a single transmit queue, and which is protected by its own lock. Multiqueue drivers then set up an array of these structures. So the new data structure can, with sufficient imagination, be seen to look something like this:
Once again, the actual lists of outgoing packets normally exist in the form of special data structures in device-accessible memory. Once the device has these queues set up for it, the various policies associated with each class of service can be implemented. Each queue is managed independently, so more voice packets can be queued even if some other queue (background, say) is overflowing. David would appear to have worked hard to avoid creating trouble for network driver developers. Drivers for single-queue devices need not be changed at all, and the addition of multiqueue support is relatively straightforward. The first step is to replace the alloc_etherdev() call with a call to:
struct net_device *alloc_etherdev_mq(int sizeof_priv,
unsigned int queue_count);
The new queue_count parameter describes the maximum number of transmit queues that the device might support. The actual number in use should be stored in the real_num_tx_queues field of the net_device structure. Note that this value can only be changed when the device is down. A multiqueue driver will get packets destined for any queue via the usual hard_start_xmit() function. To determine which queue to use, the driver should call:
u16 skb_get_queue_mapping(struct sk_buff *skb);
The return value is an index into the array of transmit queues. One might well wonder how the networking core decides which queue to use in the first place. That is handled via a new net_device callback:
u16 (*select_queue)(struct net_device *dev, struct sk_buff *skb);
The patch set includes an implementation of select_queue() which can be used with WME-capable devices. About the only other required change is for multiqueue drivers to inform the networking core about the status of specific queues. To that end, there is a new set of functions:
struct netdev_queue *netdev_get_tx_queue(struct net_device *dev,
u16 index);
void netif_tx_start_queue(struct netdev_queue *dev_queue);
void netif_tx_wake_queue(struct netdev_queue *dev_queue);
void netif_tx_stop_queue(struct netdev_queue *dev_queue);
A call to netdev_get_tx_queue() will turn a queue index into the struct netdev_queue pointer required by the other functions, which can be used to stop and start the queue in the usual manner. Should the driver need to operate on all of the queues at once, there is a set of helper functions:
void netif_tx_start_all_queues(struct net_device *dev);
void netif_tx_wake_all_queues(struct net_device *dev);
void netif_tx_stop_all_queues(struct net_device *dev);
Naturally, there are a few other details to deal with, and the multiqueue interface is likely to evolve somewhat over time. At one point, David was hoping to have this feature ready for inclusion into 2.6.27, but that goal looks overly ambitious now. It does seem that much of the ground work will be merged in the next development cycle, though, meaning that full multiqueue support should be in good shape for merging in 2.6.28.
Enhanced printk() merged A change very late in the development cycle for 2.6.26 provides a framework for extending printk() to handle new kinds of arguments. Linus Torvalds just merged the change—after -rc9—presumably partially because he knew he could trust the author, but also because it should have no effect on the kernel. It will provide for better debugging output once code is changed to take advantage of it. The core idea is to extend printk() so that kernel data structures can be formatted in kernel-specific ways. In order to get some compile-time checking, the %p format specifier has been overloaded. For example, %pI might be used to indicate that the associated pointer is to be formatted as a struct inode, which could print the most interesting fields of that structure. GCC will be able to check for the presence of a pointer argument, but because it does not understand the I part, cannot enforce that it is a pointer of the right type. Extending printk() in this manner allowed Torvalds—who authored the patch—to add two new types to printk(): %pS for symbolic pointers and %pF for symbolic function pointers. In both cases, the code uses kallsyms to turn the pointer value into a symbol name. Instead of a kernel developer having to read long address strings and then trying to find them in the system map, the kernel will do that work for them. The %pF specifier is for architectures like ppc and ia64 that use function descriptors rather than pointers. For those architectures, a function pointer points to a structure that contains the actual function address. By using the %pF specifier, the proper dereferencing is done. As an example of how the augmented printk() could be used, Torvalds converted printk_address(). The CONFIG_KALLSYMS dependency and the kallsyms_lookup() were removed, essentially leaving a one-line function:
printk(" [<%016lx>] %s%pS\n", address, reliable ? "": "? ", (void *) address);
If kallsyms is not present, the new printk() just reverts
to printing the address in hexadecimal, which allows the special case
handling to be done there.
The clear intent is to allow additional extensions to printk() to support other kernel data structures. The change to vsprintf(), which underlies printk(), actually allows for any sequence of alphanumeric characters to appear after the %p. The new pointer() helper function currently only implements the two new specifiers, but others have been mentioned. The mostly likely additions are for things like IPv4, IPv6, and MAC addresses. Torvalds specifically mentions using %p6N as a possibility for IPv6 addresses. Some would rather have seen a different syntax be used, %p{feature} was suggested, but that would conflict with some current uses of %p in the kernel. Torvalds is happy with his choice:
I _expressly_ chose '%p[alphanumeric]*' because it's basically
totally insane to have that in a *real* printk() string: the end result
would be totally unreadable.
The patch took an interesting route to the kernel, with much of the discussion evidently going on in private between Torvalds, Andrew Morton, and others before popping up on the linuxppc-dev and linux-ia64 mailing lists. The patch itself has not been posted to linux-kernel in its complete form, but was committed on July 6. While it is a bit strange to see such a change this late in the development cycle, it is a change that should have no impact as there are no plans to actually use the new specifiers in 2.6.26.
Patches and updates Kernel trees
Core kernel code
Development tools
Device drivers
Documentation
Filesystems and block I/O
Memory management
Networking
Architecture-specific
Security-related
Virtualization and containers
Benchmarks and bugs
Miscellaneous
Page editor: Jonathan Corbet
Distributions News and Editorials Fedora takes Linux to college The idea of Linux in the classroom is nothing new. From a grassroots push for district-wide adoption in secondary schools, to a plan to offer the One Laptop Per Child program in every developing country, the FOSS community is always looking for ways to encourage schools to use Linux. Recently, however, there's a new movement afoot that's aimed at snapping up a segment of computer users before they spend their money on computers with commercial operating systems. Linux is headed to college. For the last few weeks, volunteer members of Fedora's marketing team have been kicking around ideas on ways to encourage college students to give Linux a try and draw new users into the Fedora fold. Rather than approach university IT departments running Windows to convince them to switch operating systems, the team hopes to create a groundswell of college-aged users who will march into classrooms and lecture halls with Fedora-laden laptops and eagerly dive into work-study projects that focus on Linux development. Jack Aboutboul, Red Hat's Community Engineer and the main impetus behind the tentatively-named Campus Ambassador program, says though it is similar to Fedora's existing Ambassador program, the new program will have a different governance model and slightly different goals. Students from Auburn, Texas A&M, Berkeley, and other U.S. colleges, as well as team members attending universities in other countries, have already shown an interest in assuming the role of Campus Ambassadors, and have agreed to speak at campus events about the benefits of Fedora and of Linux in general. Taking the idea a step further, many Fedora team members would like to see the development of promotional material designed with college students in mind, such as posters that encourage students in the art department to volunteer their skills creating artwork for Linux distros. As one Fedora marketing team member notes, "How many marketing majors are aware that there are real life marketing opportunities for them within the Fedora project while they are still students? Reaching these students should be one focus of any campus outreach." At least one school, Cabrillo College in Santa Cruz County, California, is already hard at work promoting Fedora on its campus. In addition to forming a GNU/Linux Users Group (LUG) and holding regular installfests, the LUG is also creating its own Fedora-based distro called Seahawk GNU/Linux, named after the school mascot. LUG President Larry Cafiero explains, "Not that the world needs yet another distro, mind you, but we're using the project as a teaching tool more than an actual distro that will take the world by storm." He says that not only do students gain hands-on familiarity with Linux, but "those who get introduced to GNU/Linux through the school-based distro get a sort of introduction to Fedora as well." Since Fedora already has a strong Ambassador Program, the question of why a separate university Ambassadorship is necessary has come up. Essentially, it boils down to a difference in how users will be mentored. In the typical Ambassador arrangement, Fedora users simply evangelize Linux and encourage people to give Fedora a try while offering assistance and tips along the way. Marketing team member Chris Tyler sees the role of Campus Ambassador as more finely-tuned and as a "a matchmaker between a student, a potential need (project), and community resources." Tyler says that there are many benefits to this arrangement, including the opportunity for students to work on projects with a larger user base which, will therefore have a bigger real-world impact than student projects that remain inside the walls of the school. Team member Jeff Spaleta says finding projects with a long shelf life is vital to keeping students interested in Linux, and good for the long-term health of the community. "If students as part of their degrees need to work on a year or semester-long project, I want Fedora to be obvious place to look for compelling things to work on, with an aim towards well scoped projects that have a good chance for long lived utility," he says. "I hate seeing good academic projects die because there was no real plan to hand them off outside of that academic group which incubated them." Team members seem to be in agreement that the Ambassador program is a winning situation for everyone. Students get hands-on experience — and, in some cases, a grade — for participating in a software development project. Computer technology departments can offer a wider learning environment with little to no investment, Fedora may garner new users, and the Linux community as a whole grows. In an effort to move the Campus Ambassador project forward, Jack Aboutboul plans to formally present the idea at a Community Architecture meeting later this month.
New Releases Gentoo Linux 2008.0 released Gentoo Linux 2008.0 is out. "Code-named 'It's got what plants crave,' this release contains numerous new features including an updated installer, improved hardware support, a complete rework of profiles, and a move to Xfce instead of GNOME on the LiveCD."
Foresight 2.0.3 Released The Foresight team has announced the latest release of Foresight. Foresight 2.0.3 GNOME Edition is a minor release, featuring the latest release of GNOME, 2.22.3, which has a new login manager for users, as well as a number of bug fixes and updates.
Ubuntu 8.04.1 LTS released The first maintenance release to the Ubuntu "Hardy" release is available. They've fixed a number of issues, but not all of them... "While we have fixed a number of audio-related issues, including a scheduler problem that caused audio stuttering under load, other audio playback problems may still exist, because so far we have been unable to verify a targeted fix that does not cause regressions for other users."
openSUSE Build Service 1.0 The openSUSE project has announced the 1.0 release of its build service. "The 1.0 release provides all the features necessary to support building openSUSE in the public build systems and allowing direct contributions to openSUSE from all contributors. Developers can now submit contributions to openSUSE directly at build.opensuse.org." The service can also be used to build packages for several other distributions.
Distribution News Fedora Rawhide users: brace for a new RPM Fedora Rawhide users are about to be treated to an "alpha snapshot" of the crucial RPM utility. This is a good thing in the long run; RPM development has been stalled for far too long. But the developers are warning that life could be interesting in the short term. "BACKUP YOUR RPMDB, NOW! We're not aware of any baby-eating bugs in rpm but I'd be shocked if there were no new bugs at all... Better safe than sorry.
Fedora Board Recap 2008-JUL-01 Click below for a look at the July 1st meeting of the Fedora Board. Topics include Board Elections, Guarding Privacy and Trademark Guidelines.
Gentoo Linux New Gentoo council elected Elections for Gentoo council are finished. The newly elected council members are Donnie Berkholz, Mark Loeser, Diego Pettenò, Petteri Räty, Luca Barbato, Markus Ullmann, and Tobias Scherbaum.
Distribution Newsletters Ubuntu Weekly Newsletter #98 The Ubuntu Weekly Newsletter for July 5, 2008 covers: Ubuntu 8.04.1 LTS released, Intrepid Alpha 2 due out Thursday, Ubuntu Brainstorm, Two new Ubuntu Teams, Kubuntu Intrepid News, Ubuntu Nicaragua TV show, Launchpad 1.2.6 released, Launchpod episode #6, New Ubuntu Forums Interviews Host, Ubuntu-UK podcast #9, and much more.
OpenSUSE Weekly News/29 This issue of the OpenSUSE Weekly News looks at openSUSE 11.1 Roadmap, Novell Client for Linux Public Beta for openSUSE 10.3, People of openSUSE: Jan-Simon Möller, John Anderson: Find duplicate files by content not name, Lukas Ocilka: Package Search and One Click Install in YaST, Miguel de Icaza: Moonlight 0.7 is now Available, and several other topics.
Fedora Weekly News Issue 133 The Fedora Weekly News for July 5, 2008 looks at Planet Fedora articles "Func and Certmaster 0.20 Released", "Transifex receives some updates", "Inkscape tutorials", "New Fedora business cards", and "Bug Triage meeting", plus Fedora reviews, Help Wanted at the Utah Open Source Conference, Development articles "SELinux Eats Babies, Confines Wives, Gives Birth", "Help Wanted: Samba4, Heimdahl, OpenChange", "Java, So Many Free Choices", "Fedora 9 Now Officially Supported On Itanium/IA64", and "Running As Root", and several other topics.
DistroWatch Weekly, Issue 260 The DistroWatch Weekly for July 7, 2008 is out. "What were the most exciting Linux events of the first half of 2008? The continued success of Linux on ultra-portable laptops? The arrival of KDE 4? The miscellaneous distribution releases? Our lead article takes a quick look at the most interesting events in the Linux world that shaped the year so far. In the news section, the first stable release of Gentoo Linux in 14 months hits the download servers, Ubuntu receives high marks from French legislators, Xandros acquires Linspire and its software assets, PC-BSD releases the first BSD with integrated KDE 4.1, and OpenBSD prepares for the forthcoming release of symbolic importance - version 4.4. Also not to be missed, news about Ikki Boot, a compilation CD with an excellent collection of rescue utilities, and the latest distro statistics from this site's web server logs. Finally, we are pleased to announce that the recipient of the DistroWatch.com's monthly donation for June 2008 is the MythDora project."
Debian Misc developer news (#9) This edition of Misc developer news has some advice on quilt usage, plus update-grub starts using UUIDs by default, wxwidgets2.8 upload to unstable, and Volunteers needed to handle update of release notes.
Distribution meetings Next Ubuntu Developer Summit The next Ubuntu Developer Summit (UDS) has been scheduled for December 8 - 12, 2008 at the Google Campus, Mountain View, California, USA. The primary topic will be version 9.04. See the UDS wiki for more information.
Interviews People of openSUSE: Joe Brockmeier Here's a People of openSUSE interview with Joe Brockmeier, openSUSE community manager. "What especially motivates you to participate in the openSUSE project? I want to see openSUSE become a more independent project, to allow external contributors to have a strong voice in its development and direction, and I want to see as many people as possible introduced to Linux through openSUSE."
Page editor: Rebecca Sobol
Development What's coming in OpenSSH 5.1 OpenSSH is an important tool for remote connectivity: "OpenSSH is a FREE version of the SSH connectivity tools that technical users of the Internet rely on. Users of telnet, rlogin, and ftp may not realize that their password is transmitted across the Internet unencrypted, but it is. OpenSSH encrypts all traffic (including passwords) to effectively eliminate eavesdropping, connection hijacking, and other attacks. Additionally, OpenSSH provides secure tunneling capabilities and several authentication methods, and supports all SSH protocol versions." On July 6, 2008 a call for testing was issued for OpenSSH version 5.1: "OpenSSH 5.1 is almost ready for release, so we would appreciate testing on as many platforms and systems as possible. This release is one of the biggest in recent years, with two hackathons' worth of improvements and fixes for some of our most recalcitrant bugs." A large number of new features are being added to the OpenSSH suite of utilities. Some of the feature highlights include:
For those who would like to experiment with the new features, a series of snapshot releases are available for download.
System Applications Database Software PostgreSQL Weekly News The July 6, 2008 edition of the PostgreSQL Weekly News is online with the latest PostgreSQL DBMS articles and resources.
Filesystem Utilities g4l: verison 0.26 released (SourceForge) Version 0.26 of g4l has been announced. "G4L is a hard disk and partition imaging and cloning tool. The created images are optionally compressed and transferred to an FTP server instead of cloning locally. Latest changes with 0.26: added mdev as hotplug program, updated ntfs-3g, kernel 25.9, busybox 1.11.0 with patches to 7/02/2008, g4l30 script that shows partition size and type, added sfdisk and fsck programs."
Interoperability Samba Team Releases Samba 3.2 Version 3.2 of Samba has been announced. "Samba 3.2 builds upon the success of Samba 3.0 by modernizing and enhancing the code whilst still retaining compatibility with all existing Samba installations."
LDAP Software python-ldap 2.3.5 released Version 2.3.5 of python-ldap has been announced, it includes several bug fixes. " python-ldap provides an object-oriented API to access LDAP directory servers from Python programs. It mainly wraps the OpenLDAP 2.x libs for that purpose. Additionally it contains modules for other LDAP-related stuff (e.g. processing LDIF, LDAPURLs and LDAPv3 schema)."
Mail Software Vacation 1.2.7.0 released Version 1.2.7.0 of vacation has been announced, Chris Samuel has stepped up as the maintainer of the SourceForge version "This is a complete rebase of the current Vacation code base from the closely related version at http://savannah.nongnu.org/cvs/?group=vacation which had been released under the two clause BSD license (no advertising clause). This means Vacation finally links legally with the GPL's GDBM (something I don't believe people previously realised)!"
Web Site Development dobrado: 0.10 released (SourceForge) Version 0.10 of dobrado has been announced. "Dobrado is an easy to use online website creator. There are no templates, just drag and drop content anywhere on the screen. This release of Dobrado allows the system to be used as a wiki, with user defined copy and edit permissions, and undo/redo from history log."
scgi 1.13 announced Version 1.13 of scgi has been announced, some new capabilities have been added. "The SCGI protocol is a replacement for the Common Gateway Interface (CGI) protocol. It is similar to FastCGI but is designed to be easier to implement. The scgi package includes implementions of the SCGI protocol for Python, Apache 1 and Apache 2."
Desktop Applications Audio Applications jack_capture V0.9.19, Snd-ls V0.9.8.17 and Rollendurcmesserzeitsammler V0.0.4 New versions of jack_capture, Snd-ls and Rollendurcmesserzeitsammler have been announced. "jack_capture is a program for recording soundfiles with jack. Snd-ls is a distribution of Bill Schottstaedt's sound editor SND. The Audio Rollendurchmesserzeitsammler is a conservative garbage collector providing general ways to efficiently allocate and use memory inside a realtime audio thread without having to manually free it later."
SLV2 0.6.0 announced Version 0.6.0 of SLV2, a library for accessing LV2 audio plugins, is out with several new capabilities and bug fixes.
Desktop Environments GNOME 2.22.3 released Version 2.22.3 of the GNOME desktop environment has been announced. "This is the third update to GNOME 2.22. This includes all the bug fixes, all the new translations, updated documentation and other improvements made by the various GNOME teams since the last stable release."
GARNOME 2.22.3 released Version 2.22.3 of GARNOME, the bleeding edge GNOME distribution, has been announced. "It includes a wealth of new application releases, updated translations and bug fixes as part of this GNOME release -- as well as updates and fixes after the GNOME freeze and a host of third-party GNOME packages."
GNOME Software Announcements The following new GNOME software has been announced this week:
KDE Commit-Digest (KDE.News) The June 1, 2008 edition of the KDE Commit-Digest has been announced. The content summary says: "Amarok 2 gets basic video playing support, and a connection to Librivox public domain audio books. Major porting to KDE 4 continues in K3b. More work on "Fuzzy Search" integration in Digikam. The start of support for sound effects in KGoldRunner, and the addition of a sound feature in KPresenter. Improvements in the msword-odf and kpr-odf import facilities in KOffice..."
KDE Software Announcements The following new KDE software has been announced this week:
Xorg Software Announcements The following new Xorg software has been announced this week:
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
![[Network transmit queue]](/images/ns/kernel/net-tx-old.png){kind=small}
![[Multiqueue tx structure]](/images/ns/kernel/net-tx-mq.png)