With the nomination period beginning for the Ubuntu Technical Board, big changes like Unity having arrived in Ubuntu recently, and the upcoming UDS for being what will likely be a new LTS release of Ubuntu, it’s as good as time as any to ask big questions about the development process, challenge assumptions, and make suggestions for big changes.

Cadence

The Ubuntu release process is well known, and its developers talk regularly about the cadence of it. A new release of Ubuntu comes out every six months, and each release follows a predictable pattern. I’ve stolen the following image from OMG! Ubuntu’s recent series about Ubuntu Development.

Each developer working on Ubuntu follows this cycle. When Ubuntu 11.10 is released on October 13th, they’ll begin again. After they recover, of course.

First there’ll be a bit of a wait for the archive to be open, this gets quicker and quicker each release but since it depends on a toolchain being built and other similarly fundamental things, it tends to be a period where most people figure out what they’re going to discuss at UDS.

UDS is a bit late for the 12.04 cycle, so the merge period will probably occupy developer time both before and after UDS. This isn’t represented on Daniel’s chart above, but this is the time when massive amounts of updates arrive from Debian; it’s a time of great instability for Ubuntu. At some point there will be an Alpha 1, but you won’t want to try and install that.

Planning for UDS is going to take up some time, and writing up the results of the plans afterwards and turning that into work items. There’s also a UDS Hangover which nobody (except Robbie Williamson, when drafting the 10.10 Release Cycle) seems to like to talk about. Nothing gets done in the week or two following UDS, everybody is too wiped out.

So realistically speaking, development of features for 12.04 is going to start around mid-November at the earliest. And by features I mean the big headline things in Ubuntu; like Unity, like the Software Center, like the Installer. These things are important to get right.

Pretending for a moment that features are developed over the winter holidays like Thanksgiving, Christmas and New Year, you’ve got clear development time until Feature Freeze. The 12.04 Release Schedule isn’t published yet, but I figure that’s going to be somewhere around February 16th after which everyone switches to bug fixing and release testing.

That’s just 13 weeks of development time!

Chaos

So you’re an Ubuntu developer working on features for the upcoming release, you don’t have anywhere near as much time as you’d expect to actually do the development work. What happens if you’re replacing something that works with something completely new? Can’t you just target a later release, and work continually until the feature freeze of that release?

It turns out that you can’t. There is an incredible emphasis on the Ubuntu planning process of targeting features for particular releases. This is the exact thing you’re not supposed to do with a time-based release schedule.

Unfortunately Canonical’s own performance-review and management is also based around this schedule. The Ubuntu developers so employed (the vast majority) have such fundamentals as their pay, bonuses, etc. dictated by how many of their assigned features and work items are into the release by feature freeze. It’s not the only requirement, but it’s the biggest one.

Your new feature is going to take twelve months of development time to fully develop before it’s truly a replacement for the existing feature in Ubuntu. What you don’t do is spend twelve months developing and land it when it’s a perfect replacement.

What you do do is develop it in 12-13 week bursts, which means it’s going to take you roughly four release cycles before it’s ready rather than two. And you land the quarter-complete feature in the first release, replacing the older stable feature.

Consequence

If this were true, you would expect to see new features repeatedly arriving in Ubuntu before they were ready. Removing the old, deprecated feature and breaking things temporarily with the promise that everything will be better in the next release, certainly the one after that, definitely by the LTS.

Maybe you don’t believe that characterizes Ubuntu, in which case you should probably just stop reading now because we’re not going to agree with my fundamental complaint.

But I will say this: I know I’m responsible for doing this on more than one occasion because I had to; and I saw the exact same pattern in others’ work, when I was a manager my reports complained that they had to follow this pattern and I still see the same pattern today with features such as Unity and the Software Center.

Follow this pattern and developers are going to complain that they need a release where they don’t have any features to work on, and can just spend the time stabilizing and bug fixing.

Worse, follow this pattern and you’re going to create a user expectation that releases are going to be largely unstable and contain sweeping changes that are going to be surprising to administrators of Enterprise desktop deployments, and discourage them from using your distribution at all.

A kludge to this would be to overlay a second release schedule onto your first one, with more of an emphasis on stability and support. It’s a target for your developers to complete their features, or at least stabilize them in those 12 weeks; and it’s a target for your users to consider deployment. So three out of four of your releases are really just unstable previews of that final fourth release.

Complacency

This second LTS release cycle solves the unstable release issue, so why is this a problem?

Because developer time is wasted; because user time is wasted; because user confidence is lost.

Because features can take longer than two years to develop; or if even if a feature takes just two years, if it’s not begun immediately after the previous LTS release, it’s not going to be ready for the next one so you might postpone and lose the lead.

Because you might expect a knock-on degeneracy effect in the LTS releases as well; with 12.04 LTS being less stable than 10.04 LTS, which was less stable than 8.04 LTS which was less stable than 6.06 LTS. And it’s far too late now to have considered the 10.10/11.04/11.10/12.04 cycle to have been a Super-Long-Term-Support release and kept back the complete replacement of the desktop environment.

Because the original reason for the six-month cycle has already been forgotten: features are targeted towards releases, rather than released when ready; because the original base for the release schedule (GNOME) is no longer a key component of the distribution; because no other key component has adopted this schedule.

Because these might be a better way.

Cataclysm

What I’m going to suggest here is a completely new development process for Ubuntu, complete with details about how it would be implemented.

I’m going to suggest a monthly release process, beginning with the 11.10 release. It so happens that this fits perfectly with Ubuntu’s version numbering system, the next release would be 11.11, followed by 11.12, followed by 12.01 and so on.

This monthly release would be simply known as release in your sources.list, updates would be published to it on the first week of the month. There would be no codenames, and due to the rapid releases, changes would be largely unsurprising and iterative on the previous releases.

In order to provide user testing, a second release known as beta would exist. It’s from this release that release would be copied from on that first week of the month. beta would be updated every two weeks, on the first week of the month after it became the new release, and then on the half-way point of the month. Users who like a little bleeding on their edge can change their sources.list to use this more exciting release, or download appropriate disk images.

Developers wouldn’t run either of these, they would run the third release branch alpha. It’s from here that beta is updated; and from here that daily disk images would be generated.

Publishing from alpha to beta, and then from beta to release is handled semi-automatically. The release manager will track Release Critical bugs, and will hold up packages from copying from one to the other if they have outstanding problems. If this sounds familiar, it’s because this is exactly how the Debian testing distribution works and I recommend using the same software (which Ubuntu already uses to check for archive issues).

So where do developers upload? It’s tempting to just say to alpha, but if we say that, alpha will end up looking very different from release because it will be filled with unstable software that’s not ready for users yet. This will make it harder for problems in the release branch to be fixed, because none of the components are left in alpha because they’ve been replaced by something that’s not ready yet.

Developers will upload to an unpublished trunk branch. Packages will be copied to alpha provided:

• there is a signed-off code review for the upload
• the upload meets policy (lintian clean)
• the upload builds on all released architectures
• unit tests pass on all released architectures
• functional and verification tests pass on all architectures for the archive as a whole

I just introduced a bunch of new checks to the developer process there; I just introduced code review, mandatory unit tests and then piled functional tests and verification tests on top.

The first four are relatively self-explanatory; fail any of these tests and your upload has marked the tree red. In which case not only will your package fail to copy to alpha, but you’re about to have a conversation with the Release Manager.

For functional and verification tests, this means doing more automated QA. A failing test could be an automated installer run, or an automated boot-and-test run, etc. They’ll run sometime after the fact and will make the entire tree red. The Release Manager or their team will have to examine the logs to figure out the culprit.

So things aren’t copying to alpha, now one of two things is going to happen.

• the Release Manager reverts your upload. Because trunk is unpublished, this is simply overwriting with the older package from alpha; nobody except the original developer is going to have known about it
• after talking with the developer, it’s decided that further uploads of other packages are required (e.g. due to dep-wait, or the bug being elsewhere) in which case the tree remains red while the developer (or another in rare cases) prepares that fix upload.

While the tree is red, nobody else is allowed to upload unless it’s a fix for the problem. All effort should go to fixing the tree.

If the archive has to always remain stable, how do you develop large features such as Upstart, Unity, Ubiquity, Software Center, etc.? You use a PPA to do development, on your own timeline.

If your feature takes twelve months to develop, you take twelve months to develop it in that PPA. You’re going to be posting regularly to mailing lists or blogging about your feature to encourage users to add your PPA to their sources.list to gain testing. Obviously you’ll be doing various uploads to the main series over time to get all your dependencies in early where they don’t conflict with what’s already there.

Conclusion

My proposal is a radical change to the Ubuntu Release Process, but surprisingly it would take very little technical effort to implement because all the pieces are already there including the work on performing automated functional and verification tests.

I believe it solves the problem of landing unstable features before they’re ready, because it almost entirely removes releases as a thing. As a developer you simply work in a PPA until you’ll pass review, and land a stable feature that can replace what was there before.

It solves the need for occasional stabilization and bug-fixing releases because the main series is always stable and can receive bug-fixes easily separate to any development work going on. A developer can chose to focus on looking after the main series for some of their time in addition to their feature development work, or devote all of their time to it.

Another problem I’ve not talked about is that of building software on an unstable foundation, also solved by this change. Since developers will run alpha, and vendor developers can just run a relatively up-to-date, yet stable, release branch, software can be built on a solid foundation. Only the new feature or software itself is unstable until ready.

Canonical can keep its review schedule, and use developer uploads and work items; except rather than landing in a release, they can now land in a PPA.

Merges from Debian unstable can be handled pretty much continually as long as they keep the tree green, alternatively one can decide that users ultimately don’t care about an updated version of cat and until a case can be made (e.g. an open bug) for a package’s update, it need not be merged.

Users can now be confident of always receiving a stable operating system, because of the multiple testing and QA passes each change continually receives. Updates come in monthly, two-weekly or dailyish batches depending where in the main series they chose to run.

Enterprise administrators can run this stable release, because it only changes gradually with well-tested updates. The big changes and features have a long gestation period in PPAs, with many advance notices and blog posts about them. They’re not a surprise and can be planned for well in advance of their landing.

Downsides will, doubtless, be found in the comments below.

For your consideration.

Graphics cards from different manufacturers are very different beasts, in fact, often different generations of graphics cards from the same manufacturer can be pretty different too. While there’s a great deal of standardisation for things such as resolutions, colour depths and talking to monitors; the software side has almost no standardisation whatsoever.

In fact, one of the most fundamental operations, setting the resolution and color depth we desire (aka. the Mode) can be different for each and every card, let alone each and every manufacturer.

Practically speaking, this means that there are only two things that know how to set the mode. The Video BIOS and the Graphics Driver. Historically, within Linux, the Graphics Driver has resided within the X Window System server.

There are lots of different places that we need to set the mode:

• during boot, the Video BIOS will set an initial mode
• this may be changed by the boot loader using the Video BIOS or VBE
• when the X server is started, it will likely set a new accelerated mode using the card’s native protocols rather than VESA
• when switching between away from the X server to another VT, the previous video mode has to be restored
• when switching back to the X server from another VT, the correct video mode has to be set
• when resuming from suspend or hibernate, the correct video mode for whatever VT we resume to has to be restored

The two biggest problems here are switching from/to the X server VT, and resuming from suspend.  In the former case, the biggest difficulty is actually switching between X servers on different VTs (e.g. user switching).  The process ends up looking like this:

• user switch initiates VT switch from X server on VT7 to X server on VT8
• X server on VT7 restores the previous video mode (likely VGA text)
• X server on VT8 sets the video mode appropriately

This leads to the black screen and flashing cursor between VT switches that everybody hates.  In the resuming from suspend case, it’s really tricky; we don’t necessarily know the right video mode to restore, calling into the Video BIOS (even with VBE) is tricky, error-prone and doesn’t work if it wasn’t a VESA mode, and the X server’s mode setting code was never designed to be called externally.  This means we used to basically rely on resuming into the X server and having that restore the mode for us.

All this changes with Kernel Mode Setting (KMS).  Throughout the development of Lucid, you’ve probably seen that term mentioned a few times.

Simply put, Kernel Mode Setting is all about taking the bulk of the Graphics Driver code out of the X server and putting it into the Kernel.  This means that the kernel has Graphics Drivers just like the kernel has Network Card Drivers, Wireless Drivers, USB Drivers, etc.

Most importantly, the kernel can set the mode whenever we need to and restore it on resume.  The three most user-visible results of this are:

• a high-color, high-resolution splash screen during boot with a seamless (no black screen) transition into the X server
• fast and seamless transition between X servers when user switching
• extraordinarily fast resume from suspend directly into graphics (no blinking text cursor)

Behind the scenes there’s even more awesome waiting to be used in future releases.

Obviously most of the effort on writing these new drivers has gone to the “big three” graphics card vendors’ hardware.  Intel themselves have contributed a large part of the KMS work, and their own drivers for it; nVidia owners are covered by the reverse-engineering effort that created the nouveau driver; and ATI owners are covered by the new radeon driver.

Those with graphics cards from other vendors are a little out in the cold here, but at least they’re no worse off than they were before.  The biggest source of complaint comes from those with cards for which there is also a binary driver available (usually nVidia).

By switching from the in-kernel graphics driver (i.e. nouveau) to one supplied as a binary loaded into your X server (i.e. nvidia-glx), you will not have Kernel Mode Setting support and are effectively regressing your own system to Ubuntu 9.04 state.

I’m sure that nVidia users will be quick to point out that in Ubuntu 9.04, they had more than 16-colors for the splash screen, and that’s true.

One of the other changes made in 10.04 is the switch from using usplash to draw the splash screen to using Plymouth.  Both had the ability to draw to frame buffers (basically kernel 2D graphics canvases), but Plymouth had much better support for multiple heads, native panel resolutions, and most importantly the smooth transition into X.

Supporting KMS properly for those using the open source drivers meant regressing slightly in prettyness for those who don’t.

usplash used to support higher colors and resolutions by using SVGAlib as a backend when a frame buffer was not available; writing a new Plymouth SVGAlib renderer was simply too much work on top of the existing integration worked that needed to happen for KMS.  (If someone wanted to do it as a personal project though, go right ahead!)

An alternative to using SVGAlib would have been to at least set a better mode through VESA or VBE.  There are two common ways to do this, firstly by using the VESA frame buffer (vesafb) or secondly by having the boot loader set the graphics mode and keep it set when running the kernel, triggering the use of the EFI frame buffer (efifb).  The problem with both of these is resuming from suspend.  As discussed in detail above, we don’t have the ability to restore this mode on resume.

So we picked the third alternative, which has the added attraction that it works on all hardware and doesn’t cause other issues.  We use good, old, reliable 16-color VGA.

The great thing about the Ubuntu distro team development sprints is that you get to sit around a table and share your knowledge about workarounds for all of the broken things in the current release:

• To get the machine to resume from suspend, boot an older kernel
• To get X to start, disable usplash
• To get a useful desktop, wait for the white screen, press Alt+F2 and type “metacity --replace“

The Ubuntu QA community have put together an awesome new resource for Ubuntu users and developers – Ubuntu Brainstorm.  This allows you to suggest ideas for improvements, and to vote on the ideas others have suggested.

We have of course been inspired by the IdeaStorm site from our good friends at Dell but modified the concept to fit our needs.

The development team can now take the pulse on the most pressing user issues and propose the ideas as topics at the Ubuntu Development Summits and ultimately as specifications. Ubuntu development is in turn driven by detailed specifications written up in the wiki and tracked as blueprints in Launchpad.

An idea on brainstorm can easily be linked to a Launchpad blueprint as well as to a bug or a forum discussion thread. In this way we expect to bridge the locations where ideas are often submitted now, as forum posts or bug reports, with the blueprint format they should be expressed in to be implemented.