New in Ubuntu 14.04 LTS: nginx 1.4 in main

Ubuntu 14.04 LTS ships with nginx 1.4, which is now in main for the first time. Packages in main are covered by the Ubuntu Security Team and generally receive particular focus and attention in Ubuntu. This brings nginx up to par with Apache as a first class citizen in Ubuntu.

This move also led us to closer collaboration with nginx upstream. This is great to see happening in Ubuntu, and can only help to improve quality in our ecosystem.

Note that it is only nginx, nginx-core and the other support packages nginx-doc and nginx-common that are in main. The other packages (extras, full, light, naxsi etc) contain third party plugins and thus remain in universe. See below for details.

Background

Thomas Ward had been looking after the nginx packages in Ubuntu for quite a while, so when I received requests to get nginx into main, I made sure to contact him. One requirement for main inclusion is a team commitment to look after the package. We concluded that Thomas would carry on looking after nginx in general in Ubuntu, but that the rest of the Ubuntu Server Team would be able to back him as necessary.

Following Jorge's blog post about nginx plans for main, Sarah Novotny from nginx upstream contacted us to see how we might be able to collaborate. We are now all in touch so that we can work together to make the nginx experience better for Ubuntu users. I made sure that we all connected with the Debian nginx team also.

Thomas also blogged about nginx in main as soon as it landed.

Packaging notes

There are a couple of notable differences in Ubuntu's nginx packaging (inherited from Debian):

  1. The default path served is not /var/www/ like it is with Apache. Instead, it is /usr/share/nginx/html/. This directory contains the index.html file that is served by default. However, /usr/share/ is not a suitable location to place your own files to serve, since this area is maintained by packaging. Instead, you should configure nginx to serve from a different path, and then use that. According to the Filesystem Hierarchy Standard, /srv is a suitable path to use for this.

    Placing your own files in /usr/share/nginx/html/ is dangerous, as they can be arbitrarily overwritten by package upgrades. This unfortunate behaviour has been reported in bug 1194074, and there there has been some discussion in Debian bug 730382. But as this is a consequence of the choice of default document root as a deliberate decision by the Debian nginx maintainers, there isn't yet any solution to stop users falling into this trap, except to know about it. So please heed this warning and make sure that you change your document root appropriately.

  2. The nginx daemon does not start by default as soon as the package is installed. You must do this by hand using service nginx start. This makes sense since you will usually need to reconfigure nginx to use a different document root first (see the previous point).

Adjustments for main inclusion

A requirement for main inclusion in Ubuntu is quality and maintainability from a security perspective. The security team reviewed nginx and passed this requirement for nginx itself, noting that "Nginx is high-quality legible code, excellent explanatory comments and platform notes, very useful utility functions, and defensive error checking and logging".

However, some third party modules shipped with nginx in Debian varied in quality, so did not pass this requirement for main inclusion.

Since nginx does not currently support dynamically loadable modules, it is not possible for binary distributions such as Debian and Ubuntu to independently build plugin modules using separate source packages. Since module selection is done at build time, this makes it impossible for users to select the precise set of modules they want to have enabled in their nginx binaries, or to add modules written by third parties afterwards, as the distribution has already built the nginx binaries as part of the distribution.

So instead, Debian supplies a selection of third party modules as part of the nginx packaging. This results in binary packages such as nginx-light, nginx-full and nginx-extras, so that users can at least pick from a list of predefined sets of modules, which include common third party modules.

Since third party modules could not be included in main in Ubuntu, a new binary package nginx-core was created which contains only modules supplied in the nginx source itself. It is nginx-core, generated from the nginx source only, and related support packages nginx, nginx-common, and nginx-doc, that were promoted to main.

nginx 1.6

nginx 1.6 was released on 24 April, which was a week after the release of 14.04 LTS. This means that it will not be available as part of Ubuntu except in future releases. If you need nginx 1.6 on 12.04 or 14.04, you can use a PPA or the upstream-provided package repository. Read on for details.

Multiple package sources

You now have a variety of sources for nginx packages. You can install nginx from the Ubuntu repository itself, use the Launchpad nginx team PPA or use the packages provided by nginx upstream.

When deciding which source to use, I suggest that you consider the differences in release management, how security updates are handled and by whom, and your deployment's external repository dependencies.

This sort of choice in package repository source seems to be becoming increasingly common for key packages as the Free Software ecosystem continues to develop. Ubuntu Server LTS releases remain the stable, solid ground that production server deployments are built on, but the faster development pace of upstreams means that there is constant demand for newer upstream releases to be made available in older LTS releases.

Here are my own personal opinions on the pros and cons of the different approaches, from an nginx perspective.

nginx from Ubuntu itself

nginx as shipped within Ubuntu follows the Ubuntu release cycle and release management. You get the version available at the time the Ubuntu release you're using entered feature freeze, with only high-impact bugfixes and security updates issued as updates, as curated by the Ubuntu Server Team and the Ubuntu Security Team.

This provides a stable platform, where by stable I mean that the package does not functionally change in the lifetime of the Ubuntu release. From a production perspective, this means that if you successfully deployed last week, you can have maximum confidence in performing an identical redeployment next week. If your workflow is to have a validated, consistently reproducable deployment, then this approach minimises the chance of your deployment regressing, by not changing it. More information on Ubuntu's stable release policy can be found on the Stable Release Updates page.

The trade-off to this release stability is that the latest and greatest is not available, except through six-monthly non-LTS releases, whose use is less common on production servers.

You can see which nginx version ships with which Ubuntu release on the Ubuntu nginx package page. If the version of nginx shipped with Ubuntu is suitable for your needs, then I recommend using this option.

nginx from the Launchpad nginx team PPAs

The Launchpad nginx team PPAs are mainly maintained by Thomas Ward nowadays, who is the same person who generally looks after the nginx packaging in Ubuntu itself. You have a choice of two PPAs "stable" and "mainline", which follow the two lines of upstream development.

The version of nginx in these PPAs move along with upstream releases. For example: if you installed nginx 1.4 from this repository on 12.04 previously, it would automatically have upgraded to 1.6 when you performed your regular system updates after the PPA was updated to 1.6. This effectively gives you a "rolling release" of the latest nginx, but based on the stable release of Ubuntu 12.04 LTS.

The advantage of this approach is that you get the latest version of nginx, assuming that this matters to you. For example, if you need a more recent feature that is not present in the version of nginx shipped with the latest LTS release of Ubuntu, then this is useful.

If you want to continue to have the latest version, then this option will work well for you.

However if you want the latest version but for it to subsequently not change, then this is dangerous, since not updating your system from this PPA having used it also means that you will not receive security updates, and bugfixes will generally not be available to you unless you also bump to the latest release version.

Assuming that you do stay up-to-date with the PPA, then instead of managing regression risk by not changing things, regression risk must now necessarily be managed by the nginx upstream team's QA process. In this case, I know that they do have a comprehensive test suite that they run before release, but clearly the regression risk is higher than the approach of not changing anything at all.

Also, note that as a PPA this does not receive the attention of the Ubuntu Security Team. Thomas is very good at keeping this PPA up to date, but clearly PPA maintanance primarily by one person does have a very low bus factor in the context of timely updates.

nginx packages from upstream

nginx upstream also publish package repositories for nginx. The trade-offs in using these are quite similar to using the Launchpad nginx team PPA from a release management perspective.

The key difference is in packaging. nginx upstream packaging is designed to appear largely the same regardless of which distribution you are using, for more consistency across distribution families. This is different from the nginx distribution packaging, which is generally designed to follow the patterns commonly used across the Debian and Ubuntu distributions. So if you move from distribution packaging to upstream packaging or vice versa, you will probably need to adapt your deployment configuration.

I assume that the bus factor for timely updates here is much higher than for the PPA, since this repository is managed by the larger upstream nginx team. Security updates generally originate from upstreams anyway, so in general all nginx repositories are to some extent reliant on the upstream nginx team for updates, of course, regardless of their direct source.

Note that if you choose this option, your deployment will additionally rely on the availability of the upstream nginx package archive. If you use many upstream repositories for many different components in your deployment, then this magnifies to many points of failure. You can mitigate this risk entirely by mirroring the packages you are using. Of course, this type of deployment dependency also applies to anything based on the Ubuntu archive, in that your deployment already has a dependency on the Ubuntu archive if you are not mirroring the Ubuntu packages you use. I think that how you consider this trade-off, or whether it is even a trade-off at all, is a matter of opinion.

Backports

I will note that the Ubuntu Backports repository exists, but it is not currently used for nginx, so I will not discuss this option further here.

Getting help

As always, see Ubuntu's main page on community support options. askubuntu.com, #ubuntu-server on IRC (Freenode) and the Ubuntu Server mailing list are appropriate venues.

Thanks

A big shout out is due to Thomas Ward, who has been looking after both nginx in Ubuntu and the Launchpad nginx team PPA for quite a while now. Thomas was pivotal in getting nginx into main, and blogged about it when it landed.

Thanks also to the Debian nginx packaging team. Ubuntu's nginx packaging is based on their hard work.

And to Sarah Novotny of nginx upstream, for reaching out and collaborating with us to help make the nginx experience of Ubuntu users better.