A Pelican Bloghttp://www.justgohome.co.uk/blog/2017-08-21T16:53:44+00:00More on the imported repositories2017-08-21T16:53:44+00:00Robie Basaktag:www.justgohome.co.uk,2017-08-21:blog/2017/08/more-on-the-imported-repositories.html<p><img alt="git ubuntu illustration" src="http://www.justgohome.co.uk/blog/2017/07/git-ubuntu.png" /></p>
<p>This is the third post in our series on our git workflow tooling in
Ubuntu. There is an <a href="http://www.justgohome.co.uk/blog/2017/07/developing-ubuntu-using-git.html#multi-post-index">index of all our planned posts in the first
post</a>.
As mentioned there, it is important to keep in mind that <a href="http://www.justgohome.co.uk/blog/2017/07/developing-ubuntu-using-git.html#experimental-status">the tooling
and implementation are still highly
experimental</a>.</p>
<p>Nish introduced our imported repositories <a href="https://naccblog.wordpress.com/2017/08/01/git-ubuntu-clone/">using the <code>git ubuntu clone</code>
wrapper</a>, as well
as using git directly.</p>
<p>I'd like to go into a little more detail about what we're actually importing.</p>
<p>Automatic imports watch for new source publications in Launchpad. When they
occur, the importer will push new commits and tags to match those publications
and move up branch pointers automatically. All branches maintained by the
importer are fast-forwarding. A temporary exception to this is during our
experimental stage, before we release a 1.0. Until then, we may re-import
packages as we work on the importer.</p>
<p>Our importer maintains one repository per source package in
<a href="https://code.launchpad.net/~usd-import-team/+git">lp:~usd-import-team</a> for
now. Eventually we expect to make these available under an alias of the form
<code>lp:ubuntu/+source/<package></code>.</p>
<p>The imported trees correctly reflect the ancestry of individual source
packages. This includes both Debian and Ubuntu history, with Ubuntu history
correctly parented from the appropriate points in Debian's history.</p>
<p>There is a branch corresponding to each Debian and Ubuntu series (eg. <code>sid</code>,
<code>stretch</code>, <code>xenial</code>, <code>artful</code>), and in the case of Ubuntu, pocket (eg.
<code>trusty-updates</code>, <code>xenial-security</code>, <code>artful-proposed</code>). We’ll cover details on
how exactly these work, and the logic behind their design, in future posts.</p>
<h2>Single source of truth</h2>
<p>A goal of the importer is to reflect the single source of truth, which
we define as Launchpad's source publication history. Therefore, the only
source of the imported tags and branches must come via the importer and
not directly from uploaders. Only the importer can push to the imported
repositories. We do have a mechanism to preserve "rich history" provided
by git committers that in any other project they'd be able to push
directly; more on this in a future post.</p>
<p>Note that this methodology means that upstream's commit graphs are typically
not made available by the importer. You won't find individual upstream commits
by looking in our imported respository against a particular package. We may
decide to make this possible in the future by relying on the "rich history"
import mechanism combined with the use of our parenting invariant. More on this
in a future post.</p>
<p>We're trying hard to make sure that there is no case where a source package
upload can cause the importer to fail. We're treating any failure to import a
source package as a bug in our importer. The importer should be able to import
anything that was ever uploaded to Debian or Ubuntu.</p>
<h2>Available branches</h2>
<p>As Nish mentioned, there are two main types of branches:</p>
<ol>
<li>Devel branches: the branches on which to start development.</li>
<li>Pocket branches: understanding the current state of the archive.</li>
</ol>
<p>There are also "applied" versions of these branches, which provide quilt
patches already applied.</p>
<h3>The "devel" branches</h3>
<p>Ubuntu developers are used to using <code>pull-lp-source</code> to grab the latest version
of a package source on which to base development. We want to make <code>git clone</code>
work instead.</p>
<p>What is the correct version on which to base development? In Ubuntu,
this can vary depending on the exact publication state. Usually you want
the highest version published in the development release, or the highest
version published in a particular series for an
<a href="https://wiki.ubuntu.com/StableReleaseUpdates">SRU</a>. For this purpose,
the git importer maintains <code>devel</code> branches. For the development
release, this is just called <code>ubuntu/devel</code>. For a stable release, it is
<code>ubuntu/<codename>-devel</code>; for example: <code>ubuntu/xenial-devel</code>.</p>
<p>This is intended to save developers the effort of figuring out which pocket to
use (between the release pocket, <code>proposed</code>, <code>updates</code> and <code>security</code>) and
saves scripts from having to figure it out themselves.</p>
<p>One exception to this is in the rare case that a version in the proposed pocket
has been discarded, such as a failed SRU. The importer only follows new version
publications and does not follow deletions. In this case you may wish to start
directly from a pocket branch (described below). Alternatively, before you
begin you could start from the devel branch and use <code>git revert</code> to add a
commit reversing the published change in line with the deletion. The
infrastructure doesn’t care which method you choose.</p>
<h3>Pocket branches</h3>
<p>A different use case is to look at the git branches to understand what the
archive looks like today, or how it looked in the past.</p>
<h4>Ubuntu’s pockets</h4>
<p>First let’s review what Ubuntu pockets actually are.</p>
<p>In Ubuntu, the archive for a particular release (“series”) is divided up into a
number of “pockets”, which correspond to the entries in your <code>sources.list</code>
file:</p>
<ol>
<li>
<p>The release pocket. This pocket has no specific name. During
development, the release pocket is where package eventually land.
Upon release, this pocket is frozen and never changes again.</p>
</li>
<li>
<p>The “proposed” pocket. This pocket is used for two different
purposes depending on whether the series is in development or has
been released. During development, this is the staging area in which
packages are built, tested and checked for installability before
they land. When all tests pass, packages are moved into the release
pocket in a process called <a href="https://wiki.ubuntu.com/ProposedMigration">proposed
migration</a>. After
release, the proposed pocket is used for a different purpose: manual
user verification of proposed stable release updates through the
<a href="https://wiki.ubuntu.com/StableReleaseUpdates#Verification">SRU verification
process</a>.
When a package passes SRU verification, the package is moved into
the updates pocket, rather than the release pocket is as the case
during development.</p>
</li>
<li>
<p>The “updates” pocket. Only used for stable releases, this pocket is
used to issue recommended updates to users during the lifetime of a
stable release.</p>
</li>
<li>
<p>The “security” pocket. Only used for stable releases, this pocket is
used to issue security updates to users during the lifetime of a
stable release. Updates to the security pocket are always also
copied into the updates pocket.</p>
</li>
</ol>
<p>The name of a pocket is in the form <code><series>-<type></code>, such as
<code>xenial-security</code> and <code>artful-updates</code>. As an exception, the release pocket is
the series with no suffix, such as <code>trusty</code>.</p>
<h4>Importing pockets into git</h4>
<p>The importer maintains a branch for each pocket, in the form <code>ubuntu/<pocket
name></code>, such as <code>ubuntu/trusty</code> (the Trusty release pocket),
<code>ubuntu/xenial-security</code>, <code>ubuntu/artful-proposed</code> and so on.</p>
<h3>Combining pocket branches: the devel branches</h3>
<p>To support the archeology use case, the importer must maintain these pocket
branches. However, for the ongoing development use case, it's far more
convenient for developers to use our "devel branches".</p>
<p>We do this by making use of our "parenting invariant", which we’ll describe in
more detail in a future post. For now, it’s sufficient to understand that the
devel branches are the points at which you should start development, and the
pocket branches can be used to examine the current state of the various pockets
in the Ubuntu archive.</p>
<h3>Applied branches</h3>
<p>For our drive-by contributor use case, it would be nice if these drive-by
contributors didn't have to understand quilt. It's reasonable to expect that,
straight after running <code>git clone</code> from Ubuntu and selecting an appropriate
branch, the tree that appears is exactly the source used to build the
corresponding Ubuntu package. For drive-by contributors unfamiliar with Debian
packaging, this means that quilt patches should appear applied. Our "applied"
branches and tags provide this function.</p>
<p>The non-applied branches are the normative imports from Debian and Ubuntu
developer uploads. They are intended to represent the source package exactly
and without any derived components. This means that quilt patches appear in
<code>debian/patches/</code> as normal, but are not applied.</p>
<p>From one of these import commits, all quilt patches (if any) are applied one by
one. Each application results in a separate git commit. A final commit with an
identical tree is added for branch fast-forwarding purposes, and these commits
form the line of the "applied" branch.</p>
<p>Consequently, you can switch to an "applied" branch, or corresponding tag, and
you'll see the state of the Ubuntu source package with all patches applied and
as if quilt doesn't exist. Drive-by contributors can then file pull requests
against such a branch (eg. <code>applied/ubuntu/devel</code>) and bots and sponsors will
be able to understand exactly what is being requested.</p>
<p>It will often be necessary to pull out requested changes into a separate
quilt patch, add <a href="http://dep.debian.net/deps/dep3/">dep3</a> headers, and
possibly squash the requested change into an existing quilt patch,
before uploading. We hope to automate some of this in the future, but
for now this is left as a task for sponsors who accept git workflow
upload requests. Contributors can either submit pull requests against
the applied branches this way, or fold up into quilt patches themselves
and submit pull requests against the non-applied branches instead.</p>
<p>In the future, it may be desirable for all Ubuntu developers to forget quilt
and live entirely inside git-based patchsets. For the time being, we want to
support both traditional and git-based workflows. So both applied and
non-applied branches are maintained by the importer.</p>
<p>It is a matter of debate as to whether the non-applied or applied branches
should appear by default. We are open to further discussion on this.</p>
<h2>Available tags</h2>
<p>The first time the importer imports a new version of a source package, it tags
it using an "import tag", which is of the form <code>import/<version></code>. The
patches-applied equivalent is derived, and the result is tagged in the form
<code>applied/<version></code>.</p>
<p>Pocket copies, such as from Debian into Ubuntu via autosync, are not tagged;
the pocket branches are moved forward with new commits (with identical trees)
as necessary.</p>
<p>Since not all valid package version strings are valid git tag names, the tag
names are escaped using the same rules as specified in
<a href="http://dep.debian.net/deps/dep14/">dep14</a>, Debian's recommendation on git
repository layouts.</p>
<p>We use tags of the form <code>upload/<version></code> to supply rich history for adoption
into the imported commit graph. This cannot be pushed directly; more on this in
a future post.</p>
<p>To support the preservation of orig tarballs using pristine tar, the importer
also tags <code>upstream/<version></code>.</p>
<h2>pristine-tar and the dsc branches</h2>
<p>So that the git repositories contain all the information needed to reconstruct
an imported source package, the importer also stores the orig tarballs using
pristine-tar and the signed dsc file in separate branches. Orig tarballs are
automatically extracted on package build as needed by the <code>git ubuntu build</code>
wrapper. This means that <code>git ubuntu clone</code> followed by <code>git ubuntu build</code>
should Just Work.</p>
<p>Orig tarballs between Debian and Ubuntu may vary in exceptional cases, so the
importer keeps these properly namespaced in <code>debian/pristine-tar</code> and
<code>ubuntu/pristine-tar</code> branches to avoid collisions, and likewise for the dsc
files <code>debian/dsc</code> and <code>ubuntu/dsc</code>. pristine-tar doesn't currently support
multiple branches; it assumes a single branch of <code>pristine-tar</code>. Our <code>git
ubuntu build</code> wrapper works around this for now. In the future I'd like for us
to drive getting parameterised branch name support into <code>pristine-tar</code> upstream
to support our use case.</p>
<h2>Conclusion</h2>
<p>In order to provide a full understanding of what we're doing to those
developers interested in all the detail, I've tried to cover all the git
reference objects presented by our importer in this post today.</p>
<p>In our next post, Nish will continue discussing the <code>git ubuntu</code> tooling by
introducing <code>git ubuntu tag</code>.</p>Developing Ubuntu using git2017-07-24T16:29:35+00:00Robie Basaktag:www.justgohome.co.uk,2017-07-24:blog/2017/07/developing-ubuntu-using-git.html<p><img alt="git ubuntu illustration" src="http://www.justgohome.co.uk/blog/2017/07/git-ubuntu.png" /></p>
<p>Back in 2014, I published some <a href="http://www.justgohome.co.uk/blog/2014/08/ubuntu-git-merge-workflow.html">information and tooling on using git for Ubuntu
development</a>,
even though most Ubuntu development wasn’t done with git at the time.</p>
<p>Three years on, this work has expanded significantly. Most of the server team
is using git for daily work when touching Ubuntu packaging. We have expanded
our tooling. With the significant interest we’ve received, we’re now interested
in developing this work to make git become <em>the</em> way of working with Ubuntu’s
source code. Our plan is to do this with no disruption to existing developer
workflows.</p>
<p><a name="multi-post-index">
This post is part of a blog series. Here’s an index of all our planned posts.
We’ll amend this index and update links as we go.
</a></p>
<ul>
<li><a href="http://www.justgohome.co.uk/blog/2017/07/developing-ubuntu-using-git.html">Developing Ubuntu using git</a>
(this post)</li>
<li><a href="https://naccblog.wordpress.com/2017/08/01/git-ubuntu-clone/"><code>git ubuntu clone</code></a></li>
<li><a href="http://www.justgohome.co.uk/blog/2017/08/more-on-the-imported-repositories.html">More on the imported
repositories</a></li>
<li>Available branches</li>
<li>History and parenting</li>
<li>Repository objects</li>
<li>Rich history</li>
<li>Wrapper subcommands</li>
</ul>
<h2>Why is this so hard?</h2>
<p>Most Free Software development projects already use git. So why has Ubuntu
taken so long?</p>
<p>Unlike most software projects, Ubuntu (like other distributions) derives its
sources from upstreams, rather than being the originator of the software. This
means that we do not “control” the git repositories. Repository elements such
as branch names, tag names, branching policies and so forth are not up to us;
and nor is the choice to use git in the first place.</p>
<p>For git use in Ubuntu development to be effective, we need these repository
elements to follow the same schemes across all our packages. But upstream
projects use different schemes for their branches, tags and merge workflows. So
our task isn’t as trivial as just adopting upstreams’ git trees.</p>
<h3>Existing packaging repositories</h3>
<p>While Ubuntu makes key changes to packages as needed, the long tail of packages
that we ship are derived from Debian with no changes. Debian package
maintainers use the VCS system of their choice, which may be nothing, git, or
something else like Mercurial or Bazaar. These repositories are arranged in the
manner of the package maintainers’ choices. They may be derived from their
upstreams’ respective repositories, or they may instead be based on wholesale
upstream code “import” commits done when the packaging is updated to a latest
upstream release.</p>
<p>Right now, 68% of source packages in Ubuntu are listed as having their
packaging maintained in git (whether in Debian or Ubuntu):</p>
<p><img src='http://www.justgohome.co.uk/blog/2017/07/vcs-chart-1.svg' style="width: 30em;"/>
<img src='http://www.justgohome.co.uk/blog/2017/07/vcs-chart-2.svg' style="width: 46em;"/></p>
<p><em>Note: the data I have used cannot tell us the difference between a
package not being maintained in a VCS and the package maintainer not
having declared in metadata that a particular VCS is in use.</em></p>
<h3>Choices for Ubuntu and git</h3>
<p>We’re not in a position to mandate that everyone uses git. Even if we did do
that for Ubuntu, we cannot expect mandate it in Debian and certainly not in
every upstream project in our repositories.</p>
<p>One of the problems we want to solve is to be able to answer the question
“where do I git clone from to get the Ubuntu source for package X?”. We don’t
want to be forced to say “ah, but for package X, it’s the same as Debian, and
they’re using Mercurial in this case, so you can’t git clone you have to use hg
clone from this other place”, and then have the answer be different for package
Y and different again for package Z. We know this will happen for 3 out of 10
packages. We want to eliminate all the edge cases so that, for <code>git clone</code>
against any Ubuntu package, the repository structures are all consistent and
all subsequent developer expectations always work.</p>
<p>To achieve this consistency, we need to find a way to use git for all Ubuntu
packages: regardless of what VCS Debian or upstreams use for each package and
project; and regardless of their different branching, tagging and merging
models.</p>
<p>We think we’ve achieved this with our design; more on this in a future post.</p>
<h2>Ubuntu, Bazaar, and UDD</h2>
<p>Some readers may be familiar with a previous effort in Ubuntu,
<a href="https://wiki.ubuntu.com/DistributedDevelopment">UDD</a>, which was largely a
similar effort but with Bazaar. Nine years later, git has largely won the “VCS
wars”, and appears to be preferred by the majority of developers. Our current
effort could be seen as “UDD, but with git”, if you wish.</p>
<h2>Project goals</h2>
<p>We'd like to avoid flag days and forced workflow changes. Ubuntu git
integration will develop over time, but we don’t expect Ubuntu
developers to be forced to switch to it. We’d prefer for developers to
choose to use our integration on its own merits, switching over if and
when they feel it appropriate. If, after further consultation with users
and Launchpad developers, we did switch to git as the primary source of
truth from Launchpad, we expect to be able to wrap for backwards
compatibility with dput.</p>
<p>Being central to all code, “moving to git” can be somewhat
all-encompassing in terms of desirable use cases. Our original goal was
very specific: to make what we call “Ubuntu package merges” easier. I
achieved this back in 2014, and the server team has since made big
improvements to this particular use case. Now we want to use git for
much more, so this necessarily encompasses a wide range of use cases. We
have accepted the following use cases as falling within the scope of our
project:</p>
<h3>For drive-by contributors and new developers</h3>
<ul>
<li>
<p>Provide a single place from which any developer can <code>git clone</code>
to see the current state of a package across all our releases, and to
provide branches against which pull requests can be received.</p>
</li>
<li>
<p>Make automatic checking of contributions possible via a linter,
for contributors to run locally but also run by a bot automatically
against pull requests, to tighten the feedback loop where automatic
advice is possible.</p>
</li>
<li>
<p>Simplify and flatten the learning curve by eliminating the need to
use some of the arcane tooling that has built up over the decades in
the case of simple contributions. Most developers either know git or
have many others readily available to teach them git, so we can take
advantage of this instead of requiring them to learn pull-lp-source,
debdiff, etc.</p>
</li>
</ul>
<h3>For routine Ubuntu development</h3>
<ul>
<li>
<p>Faster and more accurate “Ubuntu package merges” by using git
(already achieved).</p>
</li>
<li>
<p>Collaborative working for sets of complex package changes, such as
SRUs and backports, so that planned changes can be shared, reviewed
and amended before upload.</p>
</li>
</ul>
<h3>For experienced Ubuntu developers</h3>
<ul>
<li>
<p>Automatic linting of contributions to allow contributors to fix some
issues directly and immediately themselves, to relieve sponsor and
review workload.</p>
</li>
<li>
<p>All publication history available for debugging, bisection and other
general software archeology tasks.</p>
</li>
<li>
<p><code>git push</code> to upload to the Ubuntu archives.</p>
</li>
</ul>
<h2>Current status</h2>
<p>We have an importer running that automatically imports new source package
publications into git, so the entire publication history of that package
becomes available to git users. Until we’re ready to scale this up, we’re
importing a subset of packages from a whitelist, with other packages imported
on request for interested developers. You can also run the importer yourself
locally on any package.</p>
<p>Tooling is available as an extension to git providing a set of subcommands
(<code>git ubuntu ...</code>). The CLI is still experimental and subject to change, and we
have a set of high-level subcommands planned that we have yet to write.</p>
<h2><a name="experimental-status">Experimental status</a></h2>
<p>If you’re interested, please do take a look! We’d appreciate feedback. However,
note that we aren’t “production ready” yet:</p>
<ul>
<li>
<p>There are a number of developer UX issues we’d like to fix before declaring
the CLI “stable”.</p>
</li>
<li>
<p>For scaling reasons, Launchpad needs improved git shared object support
before we’re ready for developers to push cloned package repositories
en-masse.</p>
</li>
<li>
<p>We expect to re-run the importer on all packages before declaring ourselves
ready, so git commit hashes for our published branches will change until we
declare them stable.</p>
</li>
</ul>
<h2>What would make a 1.0</h2>
<ol>
<li>Launchpad shared object support.</li>
<li>Hash stability declared.</li>
<li>Developer UX issues fixed.</li>
<li>Anything else? Please let us know what you think should be essential.</li>
</ol>
<h2>The wrapper</h2>
<p>On our way, we hit a bunch of edge cases which may confuse developers. Some
examples:</p>
<ul>
<li>
<p>An upstream may have placed a <code>.gitattributes</code> file that will unexpectedly
“modify” the upstream source (<code>$Id$</code> etc) as we add packaging commits.</p>
</li>
<li>
<p>git will by default convert line endings and suchlike for you; but in
packaging work, we want to leave the upstream sources untouched except where
we have some reason to explicitly patch them.</p>
</li>
<li>
<p>The build may depend on empty directories, which git cannot currently
represent.</p>
</li>
</ul>
<p>These edge cases can be worked around, often automatically, but this won’t
happen when a new developers use <code>git clone</code> directly.</p>
<p>To avoid having to introduce too much at once, we have written a wrapper
that handles these edge cases automatically, or at least warns you about
them on the occasions that they are relevant. There are also some common
repetitive actions that are specific to our workflows; the wrapper also
composes these for convenience to save developer time.</p>
<p>We don’t want to mandate use of our wrapper. To better suit advanced
developers, we’ve designed everything to be directly accessible without the
wrapper, and we consider this method of access to be a first class citizen in
our work. We’ll talk more about the wrapper and its capabilities in a future
post.</p>
<h2>Next</h2>
<p>In the next post, we’ll cover details of where the imported repositories are
and what they look like.</p>On Ubuntu and License Compliance2015-03-24T12:54:43+00:00Robie Basaktag:www.justgohome.co.uk,2015-03-24:blog/2015/03/on-ubuntu-and-license-compliance.html<p>I found it quite frustrating to read <a href="http://mer-project.blogspot.ru/2015/03/some-doubts-about-gpl-licensing-and-bq.html">Carsten Munk's concerns about GPL
and licensing related to the kernel shipping with the bq Ubuntu
phone</a>.
Clarity is essential in these matters. That Carsten can't tell what is
going on for certain is a problem. It shouldn't have happened and I'm
pleased to see that my colleagues are working hard to clear it all up.</p>
<p>As a Canonical employee and an Ubuntu developer I work hard to make sure
that the work I'm involved with is fully compliant. Sometimes this takes
me considerable time and effort. So for me the frustrating part of
reading Carsten's investigation is that only our mistakes are evident.
When things are done right people often don't notice, and so it's all
too easy for outsiders to draw the conclusion that we are "evil". I'd
like to present an example of how I work hard to do things right, in an
effort to balance this view.</p>
<p><a href="https://jujucharms.com/">Juju</a> is a particularly challenging project to
package using the traditional distribution model. It's a cross-platform,
cross-distribution and cross-release tool, and a single deployment needs
to be able to deal with all of this simultaneously. But from a licensing
perspective, the challenge comes from it being a major Go project. It
follows standard Go practices in handling its dependencies, so by the
time an an upstream release gets to me the release tarball contains all
of Juju's dependencies embedded within it. As the person who uploads new
Juju packages to the Ubuntu archive, it's my responsibility to make sure
that everything is compliant from a licensing perspective. The embedding
means that instead of having to verify just the Juju code itself, I also
have to verify all dependencies, recursively. Many of the dependencies
are small third party projects that appear to not have been packaged for
a distribution before, with little attention paid to licensing
compliance before I looked at them. Dependencies are added and versions
bumped frequently. Every time, I have to check again. Right now, the sum
of Juju and its dependencies involves over 3000 files over 37 separate
projects.</p>
<p>Back in July I did a full review over all of this code and developed a
process to follow further changes incrementally, since the situation
here is quite radically different from a traditional distribution
package. In my initial review, I found a whole slew of clearly
unintentional errors, but sought to have them fixed anyway. I filed an
<a href="https://bugs.launchpad.net/juju-core/+bug/1341589">extensive bug
report</a> describing
the contradictions and ambiguities I found. I have also filed bugs in
upstream projects as appropriate: for example in
<a href="https://github.com/xeipuuv/gojsonschema/issues/37">gojsonschema</a>. I was
pleased to find that it wasn't just me focusing on diligence in this
area: as you can see from the first bug, my colleagues on the Juju team
all took the issues I raised seriously, addressed them and committed
fixes in just a week. Bugs I have filed more recently about licensing
errors introduced in newer releases have continued to result in a quick
response.</p>
<p>So, please do not misconstrue our intentions. Mistakes may happen but we
do care, and do seek to resolve them as quickly as we can.</p>What's in an Ubuntu package version string?2015-01-15T12:01:47+00:00Robie Basaktag:www.justgohome.co.uk,2015-01-15:blog/2015/01/ubuntu-package-versions.html<p>Here's a typical example:</p>
<div class="highlight"><pre>corosync 2.3.3-1ubuntu1
</pre></div>
<p>The part before the hyphen (<code>2.3.3</code>) is the "upstream version". This is
the version of the release tarball from upstream that the package is
based on.</p>
<p>The part after the hyphen (<code>1ubuntu1</code>) is the packaging revision. But
this splits further into the part before <code>ubuntu</code> and the part after.
The part before (<code>1</code>) is the Debian packaging revision the packaging in
Ubuntu is based on. The part after (also <code>1</code> in this case) is the Ubuntu
packaging revision.</p>
<p>So here's how we can interpret this. The <code>ubuntu</code> tells us that there
are Ubuntu specific changes that have been made in the package. The
string after it (<code>1</code>) is the packaging revision assigned by the Ubuntu
developer, and suggests that it has only been modified once. Going
backwards, we can see that this Ubuntu modified package is based on
Debian's package of corosync version <code>2.3.3-1</code>. The Debian maintainer
has assigned packaging revision <code>1</code> also, and his package is based on
corosync's upstream release version <code>2.3.3</code>.</p>
<p>Another example:</p>
<div class="highlight"><pre>apache2 2.4.10-8ubuntu2
</pre></div>
<p>This package has also been modified from Debian, since <code>ubuntu</code> is
present in the version string. It is on the second revision of packaging
modifications in Ubuntu, and these modifications are based on the eighth
Debian maintainer's packaging revision of upstream's 2.4.10 release.</p>
<h2>Being in sync</h2>
<div class="highlight"><pre>haproxy 1.5.10-1
</pre></div>
<p>The absence of an <code>ubuntu</code> string inside the version number tells us
that this package source has not been modified from Debian. We usually
describe this as being "in sync" with Debian, which is a common goal for
us in Ubuntu for most packages. This is the first Debian packaging
revision of upstream haproxy's <code>1.5.10</code> release.</p>
<h2>Not directly based on Debian</h2>
<div class="highlight"><pre>libvirt 1.2.8-0ubuntu19
</pre></div>
<p>The <code>-0ubuntu</code> tells us that this package is not based on Debian's
packaging of upstream release <code>1.2.8</code> at all. This may be for a number
of reasons, which cannot be determined solely from the version number:</p>
<ol>
<li>
<p>Ubuntu may have pushed ahead with a newer upstream release of <code>1.2.8</code>
before Debian uploaded it. This can happen when Ubuntu developers have a
tighter deadline than Debian for a particular version. For example,
since Ubuntu has a faster release cycle than Debian, it may be the case
that a Debian maintainer hasn't had time to upload to Debian yet, but an
Ubuntu developer wants to make the shorter deadline of an imminent
Ubuntu release.</p>
</li>
<li>
<p>The package doesn't exist in Debian at all. This isn't ideal, but
happens for very Ubuntu-specific packages. An example of this is
<code>nvidia-304</code>, which is a "restricted" component binary non-free driver.</p>
</li>
<li>
<p>Ubuntu developers have decided to deliberately diverge from Debian
for some reason. We try to avoid this situation as much as possible, but
the situation does exist for some packages.</p>
</li>
</ol>
<h2>SRUs</h2>
<div class="highlight"><pre>vsftpd 3.0.2-1ubuntu2.14.04.1
</pre></div>
<p>This follows a common scheme used for updates to a stable release (a
Stable Release Update, or SRU), where it is often necessary to "insert"
a version in between the version in the stable release and version in a
future release, so that upgrades to future releases still work
correctly.</p>
<p>Use of this scheme isn't mandatory, but unless an Ubuntu developer is
trying to be misleading, this version string means that the second
Ubuntu modified packaging revision (<code>ubuntu2</code>) of the first Debian
packaging revision (<code>-1</code>) of the upstream vsftpd release <code>3.0.2</code> has had
one SRU (<code>.1</code>) applied to it in the <code>14.04</code> release. See the wiki page
on <a href="https://wiki.ubuntu.com/SecurityTeam/UpdatePreparation#Update_the_packaging">security update
preparation</a>
for more details of this scheme, which is also the recommended scheme to
use for non-security updates.</p>
<p>Another (possibly more) common pattern you'll see in SRU version numbers
is:</p>
<div class="highlight"><pre>freeipmi 1.1.5-3ubuntu3.1
</pre></div>
<p>This uses the same scheme. But here, "inserting" a version didn't need
the <code>14.04</code> style prefix since the same <code>1.1.5-3ubuntu3</code> didn't appear
in multiple releases. So, a little more straightforwardly, this has had
one SRU (<code>.1</code>) applied to the third (<code>3</code>) Ubuntu (<code>ubuntu</code>) modification
of the third (<code>-3</code>) Debian packaging revision of upstream freeipmi
release <code>1.1.5</code>.</p>
<h2>Comparing version numbers</h2>
<p>Underpinning all of this is the definition of how Debian and Ubuntu
package version numbers are compared. A strict ordering is <a href="https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Version">defined in
Debian
policy</a>,
and familiarity with the scheme is essential for package maintainers.
When in doubt, <code>dpkg --compare-versions</code> can verify the ordering for
you.</p>
<h2>Autosync</h2>
<p>It is useful to know that the way that autosync mechanism prevents
Ubuntu-specific changes from being overwritten is by detecting <code>ubuntu</code> in the
version string. This is how, for example, no-change rebuilds are automatically
synced over, but regular packages with Ubuntu deltas present are not.</p>
<h2>Edge cases</h2>
<p>Unfortunately, there are many edge cases which make it difficult to
write a comprehensive guide. Here are some that come to mind that you
can expect to eventually come across:</p>
<ul>
<li>
<p>Native packages (eg. <code>dpkg</code>) have no packaging revision and so have no
hyphen.</p>
</li>
<li>
<p>Hyphens are permitted in upstream version numbers, which leads to
multiple hyphens in the Debian package version string!</p>
</li>
<li>
<p>Upstream release tarballs that contain files that cannot be
redistributed by Debian or Ubuntu for legal reasons are modified by
the maintainer to meet policy, and then grow something like <code>+dfsg</code> to
the "upstream" version string.</p>
</li>
<li>
<p>A tilde (<code>~</code>) is defined to sort before anything else, including
nothing. So it's used to "insert" a version before a standard one.
This is commonly used in PPAs and backports.</p>
</li>
<li>
<p>Version epochs prefix a number followed by a colon to "reset" version
numbers, for example when an upstream version numbering scheme changes
and a newer version number would otherwise evaluate "backwards".</p>
</li>
<li>
<p>No change rebuilds lead to versions numbers like <code>1.5.2-3build1</code>.</p>
</li>
<li>
<p>Versions "in the middle". I've said things like "second" and "eighth"
above, but without checking we don't know for example that there
wasn't an extra <code>2.4.10-6.1</code> revision in between <code>2.4.10-6</code> and
<code>2.4.10-7</code>, so this isn't strictly correct. We can't assume the number
of versions that exist between two version strings based just on the
version strings. The version numbering system is deliberately designed
so that it is always possible to create a new version string in the
middle.</p>
</li>
<li>
<p>Debian non-maintainer uploads (NMUs) use an additional suffix instead
of bumping the first part of the packaging revision number as the
maintainer would. This is to prevent any conflicts with an upload the
maintainer might be preparing, and has the benefit of making it clear
to others that an NMU took place.</p>
</li>
</ul>My git-based Ubuntu package merge workflow2014-08-04T16:17:45+00:00Robie Basaktag:www.justgohome.co.uk,2014-08-04:blog/2014/08/ubuntu-git-merge-workflow.html<p><em>Originally posted to the ubuntu-devel mailing list
(<a href="https://lists.ubuntu.com/archives/ubuntu-devel/2014-August/038418.html">archive</a>).</em></p>
<p>I thought it was about time that I shared my own merge workflow, as I
think it is quite different from most other Ubuntu developers. I'm an
advanced git user (even a fanatic, perhaps), and I make extensive use of
git's interactive rebase feature. To me, an Ubuntu package merge is just
a rebase in git's terminology, and in this case I use git as nothing
more than an advanced patchset manager.</p>
<p>I find my workflow allows me to handle arbitrarily complex package
merges - something I've not been able to do any other way. And once I've
merged a particular package with this workflow once, future merges take
me far less time because checking individual broken down diffs is even
quicker still.</p>
<p>This workflow may be useful to others, but probably only if you are
already very familiar with git's interactive rebase feature. I don't
suggest that you try to use this workflow without first being
extremely comfortable with this (for example, working with git while not
attached to a branch).</p>
<p>On the other hand, if you are very familiar with rebasing in git, then
like me you may find this workflow to be the logically obvious way of
doing package merges in Ubuntu. I wonder if anybody else feels like
this.</p>
<p>In my mind, this write-up may seem complex, but I think this complexity
is just a reflection of the reality of what's really going on when one
does an Ubuntu package merge. But by using git, the complexity gets
moved to the complexity of doing git rebases, and this is something that
only needs to be learned once.</p>
<p>I'm also interested to know how this fits in with other recent work in
using git with Debian packaging. My impression is that it doesn't fit so
well, because in Ubuntu we need to deal with all Debian packages,
including those not managed in git in Debian. Comments, feedback and
criticism are all appreciated.</p>
<h1>Considering merges</h1>
<h2>Merge essentials</h2>
<p>Let's first consider what an Ubuntu package merge really is. Existing
Ubuntu developers probably want to skip this section.</p>
<p>First, some terminology. For a given package that needs merging, Ubuntu
has applied some set of changes from the Debian version it is based on.
So we have some Debian version from which Ubuntu diverged (the base
version), the latest Debian version, and the current Ubuntu version. The
old Ubuntu delta is the diff between the base version and the current
Ubuntu version. The new Ubuntu delta will be the diff between the latest
Debian version and the newest Ubuntu version that we will upload.</p>
<p>To do a package merge, we must re-apply all of the Ubuntu delta that is
still required onto the latest Debian version. On the way, we might find
that some changes are no longer required, some changes that have to be
modified to work against the latest Debian version, and may perhaps need
to introduce new changes.</p>
<p>We expect the result to contain a changelog entry summarising what
remains in the Ubuntu delta, what was modified or dropped, and any new
changes that were made.</p>
<h2>The logical delta</h2>
<p>So when doing a package merge, it is essential to understand what
exactly logically constituted the previous Ubuntu delta, so that we can
identify what changes are no longer required, how we might need to
modify some previous changes, and what new changes may be needed.</p>
<p>When the Ubuntu delta is relatively trivial, checking all of this
by examining the diffs produced by merge-o-matic is normally fine. Even
if the delta consists of a few changes, they are easy to identify and
understand in a small diff.</p>
<p>But when the delta is larger, I find it far more difficult to follow it
all in my head at once, particularly when multiple logical changes apply
changes to similar overlapping areas across multiple files. This is, of
course, yet another good reason why we should be sending our changes to
Debian and keeping our delta small, but in some cases maintaining a
large delta is necessary, at least in the short term.</p>
<p>In following my workflow, I have come across a number of merge errors
made by multiple Ubuntu developers where the claimed delta in the
changelog for a merge did not match the delta itself. This suggests to
me that developers are not always checking and understanding the delta
as they should.</p>
<h1>Applying git</h1>
<p>git makes it easy to take a large "squashed" diff and split it into
multiple constituent logical parts. This is what I've been doing here.
Once split like this, I use <code>git rebase</code> to apply the logical parts back
on to the latest Debian version. This allows me to examine each logical
part of the delta separately, modifying or removing them as required.
When I'm done, it is easy to review each part, and even compare against
the previous version. And I can save the broken down parts for the next
merge.</p>
<p>So broadly, my workflow for packages with complex deltas is:</p>
<ol>
<li>
<p>Import the base, latest Debian and all Ubuntu revisions since the
base version into a git repository.</p>
</li>
<li>
<p>Break down the Ubuntu revisions into constituent logical parts using
<code>git rebase</code>. Or if I followed this workflow last time, then I just run
<code>git am</code> against what I saved previously. One might consider this step
to be the opposite of a "squash" operation. "Unsquash", if you like.</p>
</li>
<li>
<p>Rebase onto the latest Debian version, dropping any metadata changes
(eg. <code>debian/changelog</code> changes and <code>update-maintainer</code>) and amending
the delta on the way as required.</p>
</li>
<li>
<p>Update <code>debian/changelog</code>, apply <code>update-maintainer</code>, review, test
and upload.</p>
</li>
<li>
<p>Run <code>git format-patch</code> to save my set of logical changes for next time.</p>
</li>
</ol>
<p>To help with these tasks, I have written some tooling that I use. I've
pushed these to <code>git://github.com/basak/ubuntu-git-tools.git</code>:</p>
<ul>
<li>
<p><code>xgit</code> is a wrapper around setting <code>GIT_WORK_DIR</code> and <code>GIT_DIR</code> so
that I can operate with a <code>.git</code> directory that is outside my working
tree. This means that <code>dpkg-buildpackage</code>, <code>dpkg-source</code> etc. don't
need to know or care that I'm using git, and I can run git commands
without necessarily being in my working tree.</p>
</li>
<li>
<p><code>git-dsc-commit</code> imports a source package by just committing and
tagging a new commit (in the current branch, or detached HEAD) that is
exactly the unpacked source package.</p>
</li>
<li>
<p><code>git-merge-changelogs</code> is a wrapper around <code>dpkg-mergechangelogs</code> that
takes its input changelogs from <code>debian/changelog</code> files found in
specific git revisions.</p>
</li>
<li>
<p><code>git-reconstruct-changelog</code> extracts commit log messages from a set of
git commits and writes them to <code>debian/changelog</code>.</p>
</li>
</ul>
<p>These tools are incomplete. I didn't know where I was going when I wrote
them, and there is certainly scope for more automation. I addressed the
biggest needs first, and what is remaining costs me little time so I
have not spent time to automate any more yet.</p>
<h2>Importing revisions into a git repository</h2>
<p>I generally start with:</p>
<div class="highlight"><pre># Download relevant source packages. This could probably be automated
# with the help of grab-merge.
pull-debian-source -d <package>
pull-debian-source -d <package> <base-revision>
pull-lp-source -d <package>
pull-lp-source -d <package> <version-since-base> # 0 or more times
# Set up git repository for this merge
. /path/to/xgit.bash
xgit
mkdir git gitwd # git = moved .git directory; gitwd = working directory
# without .git
git init
</pre></div>
<p>Next I import the sources package into git, modelling the Ubuntu
divergence by having the new Debian package have a parent commit of the
base Debian package, and the Ubuntu packages on a separate branch also
rooted at the base package:</p>
<div class="highlight"><pre>git dsc-commit <base-revision .dsc>
git dsc-commit <latest-debian-version .dsc>
git checkout <base-revision tag>
git dsc-commit <Ubuntu version since base .dsc> # 0 or more times
git dsc-commit <current Ubuntu version .dsc>
</pre></div>
<p><code>git-dsc-commit</code> automatically tags revisions, but since <code>~</code> and <code>:</code> are
invalid in git tag names, <code>_</code> is substituted. So right now, I have to
correctly name the tag that <code>git-dsc-commit</code> used in the <code>git checkout</code>
call above.</p>
<p><code>git-dsc-commit</code> commits "3.0 (quilt)" source packages without patches
applied. I prefer to work with quilt patches directly if they need
refreshing or other changes made. Otherwise I just get noise in <code>.pc/</code>,
and it is difficult to rationalise any changes made back into the
separate quilt patches they belong to.</p>
<p>Note that <code>git-dsc-commit</code> commits the entire source package tree
exactly as it is. It is not like a normal commit, where logically you're
committing a change. Underneath, git commits are really snapshots, not
changesets, so <code>git-dsc-commit</code> just commits a snapshot identical to the
source package. For example, if you have made a change, then
from your point of view <code>git-dsc-commit</code> will effectively commit the
reverse of that change if necessary so that the result looks identical
to the source package you're importing.</p>
<p>When this step is done, I have a git repository with imported source
packages in commits that mirror the Ubuntu divergence.</p>
<p>I find this point very useful in itself, since I can now easily compare
things. If I want to know if two files specific are different between
specific Debian and Ubuntu source package versions, or how they are
different, or want a list of files in a particular <code>debian/</code>
subdirectory that have changed, then I just ask and git will tell me.
Querying for changes between arbitrary revisions and files is something
that git does very well.</p>
<h2>Breaking down the delta into logical parts</h2>
<p>If I have already perfomed this next step for a previous upload, then a
simple <code>git am</code> against my saved work allows me to skip this step. I can
verify the result by diffing against the imported squashed equivalent.</p>
<p>I won't go into how to use <code>git rebase</code> here; I assume you know that.
For every commit I edit, I generally <code>git reset HEAD^</code> back to the
previous version, so all changes made in this particular source package
version become unstaged. Then I go through the changelog entries one by
one, staging only those changes (often using <code>git add -p</code>) and
committing them one by one.</p>
<p>The point in this step is to reflect what was logically present in an
already-uploaded source package, errors and all. Some notes:</p>
<ul>
<li>
<p>I generally aim to end up with commits that follow the same order as
the entries in <code>debian/changelog</code>.</p>
</li>
<li>
<p><code>git log --decorate</code> is useful here, since all the imported source
packages are tagged.</p>
</li>
<li>
<p>I make the commit message for each logical change identical to its
entry in <code>debian/changelog</code> where possible, including leading
whitespace and the <code>*</code>, <code>-</code> or <code>+</code> bullet points.</p>
</li>
<li>
<p>I make the <code>debian/changelog</code> file change for the entire upload a
separate commit at the end (most recent) for each source package
version.</p>
</li>
<li>
<p>If <code>update-maintainer</code> was run and thus modified <code>debian/control</code>, or
<code>VCS-*</code> entries changed to <code>XS-Debian-VCS*</code> entries, I put this in a
separate logical commit with an "ubuntu-meta" commit message.</p>
</li>
<li>
<p>Quilt patches that exist only in Ubuntu involve logical commits that
add the file in <code>debian/patches/</code> and add a single line to
<code>debian/patches/series</code>. Patches remain unapplied. Similarly, for
other types of quilt patch modification, only changes to
<code>debian/patches/</code> end up in the commit.</p>
</li>
<li>
<p>This is the stage that I often find errors in the previously
documented changelog. Where this happens, I just figure out what
happened logically and try and commit something that matches. If
<code>debian/changelog</code> specified a change that was actually not present,
it doesn't get a logical commit, but the commit with the full
<code>debian/changelog</code> change does include the erroneous text.</p>
</li>
</ul>
<p>When done, it is trivial to run <code>git log -p</code> and check that all commits
match their description. I also run <code>git diff <tag></code> and verify that the
result is still identical to the source package import we started from
by checking that the reported diff is empty.</p>
<h2>Rebasing onto the newest Debian version</h2>
<p>Again, this should be straightforward to follow for git rebasers, and I
assume you know how to operate the details.</p>
<p>First, I drop any previous commits that changed <code>debian/changelog</code> only,
as well as any "ubuntu-meta" commits. Then something like <code>git rebase
--onto <new_debian_version> <base_version></code> does the job. If there are
conflicts, they can be handled during the rebase in the normal way.</p>
<p>While I'm doing this, I take notes of the changes I made so that I can
write up the changelog later. Where possible, I directly squash these
notes into the commit messages in the form of the future changelog
entry. If the rebase step drops commits because they have been applied
in Debian, then it's important to note these. git doesn't specifically
point these out except as they scroll past.</p>
<p>Next, I check that all quilt patches apply and are still correct, and do
any further editing required. This includes test builds, running dep8
tests, etc. As I do this, I use <code>git rebase</code> extensively again,
squashing the commits down into their original places and updating
commit messages (which will be the basis of the future changelog
message).</p>
<p>When doing test builds at this stage, I don't want to overwrite the
Debian source package in my parent directory, so I do have to insert a
temporary changelog entry or something. I haven't worked out a strong
pattern for this yet; sometimes I complete the remaining steps first to
avoid this issue, and rebase and squash any changes I needed back in.
<code>git-buildpackage</code> can probably help me here; I haven't looked into
integrating it into my workflow at this step yet.</p>
<p>It is important to note that this stage really uses git as an advanced
patchset editor. I am editing the patchset itself. I specifically do not
add new commits to the end, except temporarily before I squash them down
again.</p>
<p>When this step is complete, my commits start from the imported latest
Debian source package version, and show the logical delta (one logical
entry per commit) that will form the new Ubuntu upload. Changelog
entries exist only as commit messages; <code>debian/changelog</code> is not
modified at all yet.</p>
<h2>Updating the changelog</h2>
<p>Since an Ubuntu merge is expected to include a merged changelog, adding
to the Debian changelog will not do; we need to import all previous
Ubuntu changelog entries too.</p>
<p>Most of this could probably be automated more.</p>
<h3>Merging old changelog entries</h3>
<p>My tool <code>git-merge-changelogs</code> does this. Calling it as <code>git
merge-changelogs <base version tag> <latest Debian version tag> <current
Ubuntu version tag></code> fetches the changelog entries out of the imported
source packages, calls <code>dpkg-mergechangelogs</code> and writes out
<code>debian/changelog</code> in the working tree. Then I usually just <code>git commit
-mmerge-changelogs debian/changelog</code> to commit this step.</p>
<h3>Automatically creating new changelog entries</h3>
<p>Next, I need to add the changelog entry for the merge itself. I do this
with my tool <code>git-reconstruct-changelog</code>. Calling <code>git
reconstruct-changelog <latest Debian version tag></code> inserts the commit
messages into <code>debian/changelog</code>. Then I usually run <code>git commit
-mreconstruct-changelogs debian/changelog</code> to commit this step.</p>
<h3>Finishing the changelog</h3>
<p>Reconstructing the changelog will miss out the merge introduction, and
also will fail to mention any dropped changes since there are no commits
that correspond to these. Consulting my notes from earlier, I edit up
the changelog manually, fix any whitespace/wrapping issues, release it
with <code>dch -r ''</code> and commit it with <code>git commit -mchangelog
debian/changelog</code>.</p>
<h2>Other metadata</h2>
<p>Next I run <code>update-maintainer</code>, and then <code>git commit -mubuntu-meta
debian/control</code> to commit this step. Any <code>VCS-*</code> to <code>XS-Debian-VCS-*</code>
type translation goes into this commit, too.</p>
<h2>Uploading</h2>
<p>That's it. Since my working tree has no <code>.git</code> directory, I can just run
<code>debuild</code> as usual to create my source package ready for upload.</p>
<p>If there's a problem and I need to go round again, it's quite easy to
squash a change in where I need it, re-run <code>git reconstruct-changelog</code>
and edit the changelog, and rebuild the source package.</p>
<h2>Saving the logical delta for future use</h2>
<p>After upload, I make sure to save my logical delta by using <code>git
format-patch</code>. This allows me to reconstruct it quickly the next time I
merge the same package. There is no need for me to keep the git
repository around.</p>
<p>The patchsets I've saved this way don't always follow what I've written
here precisely, as I have taken a while to settle on it, and I still
deviate on a whim. It doesn't really matter though; by separating out
logical changes into separate commits, when I look at it the next time
it's easy to mould a patchset into whatever form I will need.</p>
<h1>Example of use</h1>
<p>This workflow allows me to handle any merge that is thrown at me,
however complex it may be. When I merged mysql-5.5 last cycle, it had
diverged considerably from Debian, but with much cherry-picking going
both ways. The sheer complexity of it, and the time necessary to figure
it all out, had put off developers before me from sorting it out.
Instead, some changes kept getting cherry-picked and other changes were
getting lost.</p>
<p>When I reconstructed the logical set of changes made in Ubuntu since we
diverged, I ended up with a branch of around 120 commits (IIRC). With
extensive rebasing, I ended up reducing this to 8 logical changes to
send to Debian, and just 4 commits remaining in the Ubuntu delta.
Importantly, I did this in a way that I could be confident about the
results, since I could easily verify my work.</p>
<p>I'm now going to do the same for mysql-5.6, and I'm much happier doing
it knowing that I can manage it this way.</p>
<h1>Future</h1>
<p>I have a local store of these logical delta patchsets. Currently this is
for apache2, facter, nginx, php5, subversion and vsftpd. If others want
to follow the same workflow, we should work out some way to share them.</p>
<p>And if many people find a git repository that follows Debian and Ubuntu
source packages useful, then perhaps we should set one of those up to
share, too, to save doing the import step.</p>
<p>I did have some code that auto-imported into git from UDD bzr, and
cached, so I could just <code>git clone</code> a UDD branch, but this is limited to
UDD's package import reliability, so I stopped using it. My
<code>git-dsc-commit</code> tool should always work. I have had to fix a number of
edge cases, but I am not aware of any that are outstanding.</p>
<h1>The End</h1>
<p>What I think I have here are the pieces needed to make merging Ubuntu
packages with git work. The workflow itself doesn't matter so much - you
can mix and match, and you should be fine.</p>Ubuntu Server Q&A Sessions2014-06-09T11:38:48+00:00Robie Basaktag:www.justgohome.co.uk,2014-06-09:blog/2014/06/ubuntu-server-qanda-sessions.html<p>As part of the Ubuntu Server team's participation in the <a href="http://fridge.ubuntu.com/2014/05/28/calling-for-ubuntu-online-summit-sessions/">Ubuntu Open
Summit</a>, we'll be running two Q&A sessions this week aimed at Ubuntu
Server users. We want to gather questions from the community both before
and during the event, so that users can get direct and authoritative
answers from those in the know.</p>
<p>This will be the first Ubuntu Open Summit, combining the previously
separate Ubuntu Developer Summit and Ubuntu Open Week events. The event
will run online, with live video streaming and participation using
Google Hangouts and IRC. See the <a href="http://summit.ubuntu.com/uos-1406/">schedule</a> for details. Sessions
will be recorded and the videos will be available afterwards.</p>
<p>I suggest that questions be asked beforehand in any place we can grab
them (eg. <a href="https://lists.ubuntu.com/mailman/listinfo/ubuntu-server">the mailing list</a>, <a href="http://www.reddit.com/r/Ubuntu">r/Ubuntu</a>, etc), and then we can
best prepare answers for you.</p>
<p>Alternatively, and for more interactivity, there will be an IRC channel
that we will monitor during the sessions themselves.</p>
<h2>Ubuntu Server Security Q&A</h2>
<p>Currently scheduled for: Tuesday (2014-06-10) 19:00 - 19:55 UTC</p>
<p>Also known as "Security team reads mean tweets", Marc Deslauriers and
Seth Arnold of the Ubuntu Security Team will be on hand to answer all
questions security.</p>
<p>If you want to know how to secure your freshly installed Ubuntu Server,
the pros and cons of various individual hardening approaches, how to
make sure that your system has received a particular security update, or
have any other question related to Ubuntu Server and security, this is
your opportunity to find out.</p>
<p>And of course, I presume everyone wants to hear the Ubuntu Security Team
read mean tweets!</p>
<h2>Ubuntu Server systemd Q&A</h2>
<p>Currently scheduled for: Wednesday (2014-06-11) 18:00 - 18:55 UTC</p>
<p>Following a long debate, Debian <a href="http://article.gmane.org/gmane.linux.debian.devel.ctte/5453">chose systemd</a> in Februrary as the
default init system for its upcoming "jessie" release, and Mark has
<a href="http://www.markshuttleworth.com/archives/1316">confirmed</a> that Ubuntu will follow Debian and also switch to systemd
by default.</p>
<p>What are the implications of this move for Ubuntu Server users? Will
16.04 switch to systemd, and if so, when? When will Ubuntu Server users
need to rewrite all of their custom upstart jobs?</p>
<p>Dimitri John Ledkov, an Ubuntu Core Developer closely involved with the
systemd switch, will be online to answer your questions.</p>
<h2>Submitting questions</h2>
<p>We'll accept questions at any time, including during the event on IRC.
Or if you know what you'd like to ask already, please post them to
Reddit or to the Ubuntu Server mailing list and we will pick them up and
prepare answers for you.</p>
<h2>Schedule changes</h2>
<p>Please keep an eye on <a href="http://summit.ubuntu.com/uos-1406/">the schedule</a> in case it changes!</p>New in Ubuntu 14.04 LTS: PHP 5.52014-05-08T14:11:51+00:00Robie Basaktag:www.justgohome.co.uk,2014-05-08:blog/2014/05/new-in-14-04-php.html<p>Ubuntu 14.04 LTS ships with PHP 5.5, which is a significant upgrade over
PHP 5.3 as found in 12.04.</p>
<p>PHP 5.5 actually first appeared in 13.10, though of course if you intend
to do an LTS to LTS upgrade, you won't notice this until now.</p>
<p>PHP upstream introduced some incompabilities in this update, and
recommend testing before upgrading production environments. For more
details, see the <a href="http://php.net/manual/en/migration55.changes.php">PHP migration
guide</a>.</p>
<h2>Getting PHP</h2>
<p>PHP 5.5 is available in the <a href="https://launchpad.net/ubuntu/+source/php5">main Ubuntu repository for
Trusty</a>, so <code>apt-get install
php5-cli</code> (or <code>libapache2-mod-php5</code>, etc) will suffice. Ondřej Surý, the
primary Debian PHP maintainer, also maintains a series of PPAs on
Launchpad if you need a different version. Taking a quick look at
<a href="https://launchpad.net/~ondrej">Ondřej's Launchpad page</a>, I see:</p>
<ul>
<li><a href="https://launchpad.net/~ondrej/+archive/php5-oldstable">PPA for PHP 5.4</a></li>
<li><a href="https://launchpad.net/~ondrej/+archive/php5">PPA for PHP 5.5</a></li>
<li><a href="https://launchpad.net/~ondrej/+archive/php5-5.6">PPA for PHP 5.6</a></li>
</ul>
<p>Thanks to Ondřej for kindly taking the time to maintain these PPAs for
the community.</p>
<h2>Help Wanted</h2>
<p>Ondřej Surý, the Debian maintainer of PHP, posted a <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=664595">request for
help</a> with PHP
Debian packaging. If you are an available Debian developer or
maintainer, please join him. Helping with Debian PHP packaging also
helps Ubuntu, since Ubuntu's PHP packages are heavily based on Debian's
work.</p>
<h2>Upstream dissonance</h2>
<p>There are a few areas in which multiple distributions (at least the
Fedora and Debian families) patch PHP, but these patches have not been
taken upstream. Perhaps I'm mistaken, but I believe that most PHP users
consume PHP through a distribution, so it does not seem ideal that
multiple distributions carry these patches instead of their changes
being adopted in upstream directly. I'd love to see better collaboration
with upstream in these areas so that these particular distribution
patches become unnecessary.</p>
<p>I mention some of these differences here since the differences appear to
have affected some Ubuntu users.</p>
<h3>JSON module</h3>
<p>PHP upstream ship a JSON module whose licence is not considered
acceptable to distributions. This was reported and accepted in
<a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=692613">Debian bug
692613</a>, and
<a href="https://fedoraproject.org/wiki/Licensing:Main?rd=Licensing#Bad_Licenses">Fedora also
concurs</a>.</p>
<p>As a workaround, Remi Collet from Fedora removed the JSON module, but
arranged to ship a PECL module that builds JSON from an external,
API compatible source that is not based on the upstream that is licensed
dubiously. Debian fixed the licensing problem by doing the same thing.</p>
<p>Ubuntu follows the Debian lead by default, so Trusty also ships without
the upstream core PHP module. Instead, since Trusty, the <code>php5-json</code>
module is automatically brought in as a dependency. Users will still see
an API-compatible JSON module available; it is just an alternative
implementation that is more acceptable to Debian (and to Fedora).</p>
<p>Note that there is a claim in <a href="https://launchpad.net/bugs/1287726">bug
1287726</a> that there exist edge cases
where behaviour has changed. This bug is waiting on a volunteer to fix
behaviour in the alternative JSON implementation. It sounds like this
other upstream would be happy to take the change. This is an unfortunate
consequence of the situation, but at least there are no fundamental
disagreements on how to fix this. It is "just a bug" with a simple
technical solution; we just need an expert in the area to take a look
and fix it.</p>
<p>Some background: the <a href="https://www.debian.org/social_contract#guidelines">Debian Free Software Guidelines
(DFSG)</a> ensures that
Debian is free to install and use by everyone, without restriction.
Ubuntu is primarily based on Debian, so inherits this status. Much of
the hard work to make this reality has been done by the <a href="https://ftp-master.debian.org/">Debian FTP
masters</a> who have painstakingly reviewed
every package in Debian (and consequently most of the packages in
Ubuntu) to ensure that this is the case. So all software shipped by
Debian must comply with the DFSG, and the PHP JSON module does not meet
this requirement.</p>
<p>The other side of this argument is that the PHP JSON module is too
important to compromise quality over a petty licensing issue. Why can't
distributions just ignore the nonsensical clause?</p>
<p>This issue has brought out many angry people on both sides of the
debate, and there have been some flames over this. I would like to
remind readers of the <a href="http://www.ubuntu.com/about/about-ubuntu/conduct">Ubuntu Code of
Conduct</a>:</p>
<blockquote>
<p>Disagreements, social and technical, are normal, but we do not allow
them to persist and fester leaving others uncertain of the agreed
direction.</p>
<p>We expect participants in the project to resolve disagreements
constructively. When they cannot, we escalate the matter to structures
with designated leaders to arbitrate and provide clarity and
direction.</p>
</blockquote>
<p>In Ubuntu, we make decisions based on consensus, or (rarely) through
leadership when necessary. Right now, Ubuntu has not specifically made
any decision on this point. We are simply following Debian by default,
which is what we do unless we have particular reason to diverge.</p>
<p>Please be constructive about this issue. Remember: Ubuntu is a community
project, and we have a well defined path for making decisions. The
appropriate avenues to make a change in Ubuntu on this point is to first
try to achieve consensus (eg. on mailing lists), and failing that, to
take the matter to the <a href="https://wiki.ubuntu.com/TechnicalBoard">Ubuntu Technical
Board</a>.</p>
<p>More discussion can be found in the <a href="https://bugs.php.net/bug.php?id=63520">upstream
bug</a> and <a href="http://www.reddit.com/r/PHP/comments/1ksnzw/php_json_removed_in_php_55/">reddit
thread</a>.</p>
<h3>Timezone handling</h3>
<p>In a distribution, users expect to set a system setting once, and to
have all applications to pick up that setting automatically. This
includes, for example, the system timezone setting.</p>
<p>This principle also applies to system timezone updates. When daylight
savings time rules change in some country or other, users expect to have
these picked up in a single system update, and for all applications to
immediately use this new data.</p>
<p>PHP upstream disagree on this point. PHP as shipped by upstream requires
the system timezone to be manually set in <code>php.ini</code>. PHP also uses its
own inbuilt timezone rule database, so users do not receive rule updates
without also bumping the PHP version with a new upstream tarball.</p>
<p>So Fedora, Debian, Ubuntu (and others?) all patch PHP to make it use the
system timezone setting and the system timezone database.</p>
<p>More information is available in a <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=618462#10">comment in Debian bug
618462</a>.</p>
<p>As upstream releases changed, we had a couple of regressions in the
past in <a href="https://launchpad.net/bugs/1069529">bug 1069529</a> and <a href="https://launchpad.net/bugs/1244343">bug
1244343</a>. I managed to sort these
out (and submitted patches to Debian also), so I think we now have this
functionality properly nailed in Trusty.</p>
<p>As all distributions are carrying this patch (written originally by Remi
Collet of Fedora, I think), it would be nice if upstream PHP could take
this patch too. But my understanding based on the Debian bug above is
that they are reluctant.</p>
<h3>libgd</h3>
<p>PHP includes a <a href="http://www.libgd.org/">GD</a> module, which provides an API
to various graphics operations. PHP upstream bundle this library. But
distributions prefer to use a single version of a library shipped
separately, rather than using the versions shipped bundled inside
upstreams.</p>
<p>The reasons for this are well documented in <a href="https://fedoraproject.org/wiki/Packaging:No_Bundled_Libraries">Fedora's No Bundled
Libraries
Policy</a>
and in <a href="https://wiki.debian.org/UpstreamGuide#No_inclusion_of_third_party_code">Debian's Upstream
Guide</a>,
so I won't repeat their justifications here.</p>
<p>In PHP's case, they effectively forked GD by adding functions that were
not available upstream, and then made these functions available in the
PHP GD API. Since distributions used the upstream GD library, this meant
that PHP programs written against this API failed to work on PHP as
shipped by distributions, since they used missing functions that were
not available.</p>
<p><a href="https://launchpad.net/bugs/74647">Bug 74647</a> contains the details. I
believe this is a mostly solved problem now, as PHP upstream did focus
on trying to get all of the required functions accepted in GD upstream.
There is one report that there is one remaining function (<code>imagerotate</code>)
that is still not available upstream, and thus not in distributions that
ship PHP.</p>
<p>I am pleased to note that it does appear that upstream are working on
the problem, so I hope that this issue will completely go away in a
future release.</p>
<h2>Thanks</h2>
<p>Thanks to Ondřej Surý for his relentless effort in keeping PHP
maintained in Debian, for monitoring PHP bugs in Ubuntu, and for
maintaining PPAs for PHP in Ubuntu.</p>
<h2>Getting help</h2>
<p>As always, see Ubuntu's main page on <a href="http://www.ubuntu.com/support/community">community support
options</a>.
<a href="https://askubuntu.com">askubuntu.com</a>, <code>#ubuntu-server</code> on IRC
(Freenode) and the <a href="https://lists.ubuntu.com/mailman/listinfo/ubuntu-server">Ubuntu Server mailing
list</a> are
appropriate venues.</p>New in Ubuntu 14.04 LTS: nginx 1.4 in main2014-05-01T17:04:16+00:00Robie Basaktag:www.justgohome.co.uk,2014-05-01:blog/2014/05/new-in-14-04-nginx.html<p>Ubuntu 14.04 LTS ships with nginx 1.4, which is now in main for the
first time. Packages in main are covered by the Ubuntu Security Team and
generally receive particular focus and attention in Ubuntu. This brings
nginx up to par with Apache as a first class citizen in Ubuntu.</p>
<p>This move also led us to closer collaboration with nginx upstream. This
is great to see happening in Ubuntu, and can only help to improve
quality in our ecosystem.</p>
<p>Note that it is only <code>nginx</code>, <code>nginx-core</code> and the other support
packages <code>nginx-doc</code> and <code>nginx-common</code> that are in main. The other
packages (extras, full, light, naxsi etc) contain third party plugins
and thus remain in universe. See below for details.</p>
<h2>Background</h2>
<p>Thomas Ward had been looking after the nginx packages in Ubuntu for
quite a while, so when I received requests to get nginx into main, I
made sure to contact him. One requirement for main inclusion is a team
commitment to look after the package. We concluded that Thomas would
carry on looking after nginx in general in Ubuntu, but that the rest of
the Ubuntu Server Team would be able to back him as necessary.</p>
<p>Following <a href="http://www.jorgecastro.org/2014/01/02/nginx-coming-to-main-in-14-dot-04/">Jorge's blog post about nginx plans for
main</a>,
Sarah Novotny from nginx upstream contacted us to see how we might be
able to collaborate. We are now all in touch so that we can work
together to make the nginx experience better for Ubuntu users. I made
sure that we all connected with the Debian nginx team also.</p>
<p>Thomas also <a href="http://lordoftime.info/?p=122">blogged about nginx in main</a>
as soon as it landed.</p>
<h2>Packaging notes</h2>
<p>There are a couple of notable differences in Ubuntu's nginx packaging
(inherited from Debian):</p>
<ol>
<li>
<p>The default path served is not <code>/var/www/</code> like it is with Apache.
Instead, it is <code>/usr/share/nginx/html/</code>. This directory contains the
<code>index.html</code> file that is served by default. However, <code>/usr/share/</code>
is not a suitable location to place your own files to serve, since
this area is maintained by packaging. Instead, you should configure
nginx to serve from a different path, and then use that. According
to the <a href="http://www.pathname.com/fhs/">Filesystem Hierarchy
Standard</a>,
<a href="http://www.pathname.com/fhs/pub/fhs-2.3.html#SRVDATAFORSERVICESPROVIDEDBYSYSTEM">/srv</a>
is a suitable path to use for this.</p>
<p>Placing your own files in <code>/usr/share/nginx/html/</code> is dangerous, as
they can be arbitrarily overwritten by package upgrades. This
unfortunate behaviour has been reported in <a href="https://launchpad.net/bugs/1194074">bug
1194074</a>, and there there has been
some discussion in <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=730382">Debian bug 730382</a>. But as
this is a consequence of the choice of default document root as a
deliberate decision by the Debian nginx maintainers, there isn't yet
any solution to stop users falling into this trap, except to know
about it. So please heed this warning and make sure that you change
your document root appropriately.</p>
</li>
<li>
<p>The nginx daemon does not start by default as soon as the package is
installed. You must do this by hand using <code>service nginx start</code>.
This makes sense since you will usually need to reconfigure nginx to
use a different document root first (see the previous point).</p>
</li>
</ol>
<h2>Adjustments for main inclusion</h2>
<p>A requirement for main inclusion in Ubuntu is quality and
maintainability from a security perspective. The security team reviewed
nginx and passed this requirement for nginx itself,
<a href="https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1262710/comments/6">noting</a>
that "Nginx is high-quality legible code, excellent explanatory comments
and platform notes, very useful utility functions, and defensive error
checking and logging".</p>
<p>However, some third party modules shipped with nginx in Debian varied in
quality, so did not pass this requirement for main inclusion.</p>
<p>Since nginx does not currently support dynamically loadable modules, it
is not possible for binary distributions such as Debian and Ubuntu to
independently build plugin modules using separate source packages. Since
module selection is done at build time, this makes it impossible for
users to select the precise set of modules they want to have enabled in
their nginx binaries, or to add modules written by third parties
afterwards, as the distribution has already built the nginx binaries as
part of the distribution.</p>
<p>So instead, Debian supplies a selection of third party modules as part
of the nginx packaging. This results in binary packages such as
<code>nginx-light</code>, <code>nginx-full</code> and <code>nginx-extras</code>, so that users can at
least pick from a list of predefined sets of modules, which include
common third party modules.</p>
<p>Since third party modules could not be included in main in Ubuntu, a new
binary package <code>nginx-core</code> was created which contains only modules
supplied in the nginx source itself. It is <code>nginx-core</code>, generated from
the nginx source only, and related support packages <code>nginx</code>,
<code>nginx-common</code>, and <code>nginx-doc</code>, that were promoted to main.</p>
<h2>nginx 1.6</h2>
<p>nginx 1.6 was released on 24 April, which was a week after the release
of 14.04 LTS. This means that it will not be available as part of Ubuntu
except in future releases. If you need nginx 1.6 on 12.04 or 14.04, you
can use a PPA or the upstream-provided package repository. Read on for
details.</p>
<h2>Multiple package sources</h2>
<p>You now have a variety of sources for nginx packages. You can install
nginx from the Ubuntu repository itself, use the <a href="https://launchpad.net/~nginx">Launchpad nginx team
PPA</a> or use the <a href="http://nginx.org/en/linux_packages.html">packages provided by
nginx upstream</a>.</p>
<p>When deciding which source to use, I suggest that you consider the
differences in release management, how security updates are handled and
by whom, and your deployment's external repository dependencies.</p>
<p>This sort of choice in package repository source seems to be becoming
increasingly common for key packages as the Free Software ecosystem
continues to develop. Ubuntu Server LTS releases remain the stable,
solid ground that production server deployments are built on, but the
faster development pace of upstreams means that there is constant demand
for newer upstream releases to be made available in older LTS releases.</p>
<p>Here are my own personal opinions on the pros and cons of the different
approaches, from an nginx perspective.</p>
<h3>nginx from Ubuntu itself</h3>
<p>nginx as shipped within Ubuntu follows the Ubuntu release cycle and
release management. You get the version available at the time the Ubuntu
release you're using entered feature freeze, with only high-impact
bugfixes and security updates issued as updates, as curated by the
Ubuntu Server Team and the Ubuntu Security Team.</p>
<p>This provides a stable platform, where by stable I mean that the package
does not functionally change in the lifetime of the Ubuntu release. From
a production perspective, this means that if you successfully deployed
last week, you can have maximum confidence in performing an identical
redeployment next week. If your workflow is to have a validated,
consistently reproducable deployment, then this approach minimises the
chance of your deployment regressing, by not changing it. More
information on Ubuntu's stable release policy can be found on the
<a href="https://wiki.ubuntu.com/StableReleaseUpdates">Stable Release Updates
page</a>.</p>
<p>The trade-off to this release stability is that the latest and greatest
is not available, except through six-monthly non-LTS releases, whose use
is less common on production servers.</p>
<p>You can see which nginx version ships with which Ubuntu release on the
<a href="https://launchpad.net/ubuntu/+source/nginx">Ubuntu nginx package page</a>.
If the version of nginx shipped with Ubuntu is suitable for your needs,
then I recommend using this option.</p>
<h3>nginx from the Launchpad nginx team PPAs</h3>
<p>The <a href="https://launchpad.net/~nginx">Launchpad nginx team PPAs</a> are mainly
maintained by Thomas Ward nowadays, who is the same person who generally
looks after the nginx packaging in Ubuntu itself. You have a choice of
two PPAs "stable" and "mainline", which follow the two lines of upstream
development.</p>
<p>The version of nginx in these PPAs move along with upstream releases.
For example: if you installed nginx 1.4 from this repository on 12.04
previously, it would automatically have upgraded to 1.6 when you
performed your regular system updates after the PPA was updated to 1.6.
This effectively gives you a "rolling release" of the latest nginx, but
based on the stable release of Ubuntu 12.04 LTS.</p>
<p>The advantage of this approach is that you get the latest version of
nginx, assuming that this matters to you. For example, if you need a
more recent feature that is not present in the version of nginx shipped
with the latest LTS release of Ubuntu, then this is useful.</p>
<p>If you want to continue to have the latest version, then this option
will work well for you.</p>
<p>However if you want the latest version but for it to subsequently not
change, then this is dangerous, since not updating your system from this
PPA having used it also means that you will not receive security
updates, and bugfixes will generally not be available to you unless you
also bump to the latest release version.</p>
<p>Assuming that you do stay up-to-date with the PPA, then instead of
managing regression risk by not changing things, regression risk must
now necessarily be managed by the nginx upstream team's QA process. In
this case, I know that they do have a <a href="http://hg.nginx.org/nginx-tests/">comprehensive test
suite</a> that they run before release,
but clearly the regression risk is higher than the approach of not
changing anything at all.</p>
<p>Also, note that as a PPA this does not receive the attention of the
Ubuntu Security Team. Thomas is very good at keeping this PPA up to
date, but clearly PPA maintanance primarily by one person does have a
very low <a href="http://en.wikipedia.org/wiki/Bus_factor">bus factor</a> in the
context of timely updates.</p>
<h3>nginx packages from upstream</h3>
<p><a href="http://nginx.org/en/linux_packages.html">nginx upstream also publish package repositories for
nginx</a>. The trade-offs in using
these are quite similar to using the Launchpad nginx team PPA from a
release management perspective.</p>
<p>The key difference is in packaging. nginx upstream packaging is designed
to appear largely the same regardless of which distribution you are
using, for more consistency across distribution families. This is
different from the nginx distribution packaging, which is generally
designed to follow the patterns commonly used across the Debian and
Ubuntu distributions. So if you move from distribution packaging to
upstream packaging or vice versa, you will probably need to adapt your
deployment configuration.</p>
<p>I assume that the bus factor for timely updates here is much higher than
for the PPA, since this repository is managed by the larger upstream
nginx team. Security updates generally originate from upstreams anyway,
so in general all nginx repositories are to some extent reliant on the
upstream nginx team for updates, of course, regardless of their direct
source.</p>
<p>Note that if you choose this option, your deployment will additionally
rely on the availability of the upstream nginx package archive. If you
use many upstream repositories for many different components in your
deployment, then this magnifies to many points of failure. You can
mitigate this risk entirely by mirroring the packages you are using. Of
course, this type of deployment dependency also applies to anything
based on the Ubuntu archive, in that your deployment already has a
dependency on the Ubuntu archive if you are not mirroring the Ubuntu
packages you use. I think that how you consider this trade-off, or
whether it is even a trade-off at all, is a matter of opinion.</p>
<h3>Backports</h3>
<p>I will note that the <a href="https://help.ubuntu.com/community/UbuntuBackports">Ubuntu Backports
repository</a> exists,
but it is not currently used for nginx, so I will not discuss this
option further here.</p>
<h2>Getting help</h2>
<p>As always, see Ubuntu's main page on <a href="http://www.ubuntu.com/support/community">community support
options</a>.
<a href="https://askubuntu.com">askubuntu.com</a>, <code>#ubuntu-server</code> on IRC
(Freenode) and the <a href="https://lists.ubuntu.com/mailman/listinfo/ubuntu-server">Ubuntu Server mailing
list</a> are
appropriate venues.</p>
<h2>Thanks</h2>
<p>A big shout out is due to Thomas Ward, who has been looking after both
nginx in Ubuntu and the Launchpad nginx team PPA for quite a while now.
Thomas was pivotal in getting nginx into main, and <a href="http://lordoftime.info/?p=122">blogged about
it</a> when it landed.</p>
<p>Thanks also to the Debian nginx packaging team. Ubuntu's nginx packaging
is based on their hard work.</p>
<p>And to Sarah Novotny of nginx upstream, for reaching out and
collaborating with us to help make the nginx experience of Ubuntu users
better.</p>New in Ubuntu 14.04: Apache 2.42014-04-24T13:13:10+00:00Robie Basaktag:www.justgohome.co.uk,2014-04-24:blog/2014/04/new-in-14-04-apache.html<p>Ubuntu 14.04 ships with Apache 2.4, which is a significant upgrade over
Apache 2.2 as found in 12.04.</p>
<p>Apache 2.4 actually first appeared in 13.10, though of course if you
intend to do an LTS to LTS upgrade, you won't notice this until now.</p>
<p>If you have a default configuration, then everything should upgrade
automatically.</p>
<p>Of course, server deployments typically do not run on defaults. In this
case, there are significant changes of which you should be aware. Expect
the apache2 postinst script to fail to restart Apache after the upgrade.
You'll need to fix up your own customisations to meet the requirements
in Apache 2.4 and then run <code>sudo dpkg --configure -a</code> and <code>sudo apt-get
-f install</code> to recover. Be sure to back up your system before you begin.</p>
<p>Instead of upgrading, you may want to consider this as an opportunity to
enter the new world of automated deployments. Codify your deployment,
and then test and deploy a fresh instance of Apache on 14.04 instead,
using virtual machines as needed. This is far less stressful than trying
to upgrade an existing production system!</p>
<h2>Upstream changes</h2>
<p>You will need to update any custom configuration according to latest
upstream configuration syntax.</p>
<p>See upstream's document "<a href="http://httpd.apache.org/docs/2.4/upgrading.html">Upgrading to 2.4 from
2.2</a>" for details of
required configuration changes. Authorization and access control
directives have changed, and will likely need adjustment. Various
defaults have also changed.</p>
<h2>Significant packaging changes</h2>
<p>The default path to served files has changed from <code>/var/www</code> to
<code>/var/www/html</code>, mainly for security reasons. See the debian-devel
thread "<a href="https://lists.debian.org/debian-devel/2012/04/msg00301.html">Changing the default
document root for HTTP
server</a>" for
details.</p>
<p>The packaging has been overhauled quite significantly.
<code>/etc/apache2/conf.d/</code> is now <code>/etc/apache2/conf-available/</code> and
<code>/etc/apache2/conf-enabled/</code>, to match the existing <code>sites-enabled/</code> and
<code>mods-enabled/</code> mechanisms.</p>
<p>Before you upgrade, I suggest that you first make sure that everything
in <code>/etc/apache2/*-available</code> is correctly a symlink to the corresponding
<code>/etc/apache2/*-enabled</code>. Note that all configurations in sites-enabled
and conf-enabled need a <code>.conf</code> suffix now.</p>
<p>Make use of the <code>a2enmod</code>, <code>a2ensite</code>, <code>a2enconf</code> series tools! These
help you easily manage the symlinks from <code>*-available</code> to <code>*-enabled</code>.</p>
<p>See <a href="http://sources.debian.net/src/apache2/2.4.9-1/debian/apache2.NEWS">Debian's apache2 packaging NEWS
file</a>
for full details.</p>
<h2>Other Notes</h2>
<p>Debian changed the default "It works!" page into a comprehensive page
explaining on where to go after an initial installation. Initially, I
imported this into Ubuntu without noticing this change. Thank you to
Andreas Hasenack for pointing out that the page referred to Debian and
the Debian bug tracker in a misleading way, in <a href="https://launchpad.net/bugs/1288690">bug
1288690</a>. I fixed this in Ubuntu by
essentially doing a <code>s/Debian/Ubuntu/g</code> and crediting Debian
appropriately instead.</p>
<h2>Thanks</h2>
<p>I think the Apache 2.4 packaging is a shining example of complex
packaging done well. All credit is due to Stefan Fritsch and Arno Töll,
the Debian maintainers of the Apache packaging. They have done the bulk
of the work involved in this update.</p>
<h2>Getting help</h2>
<p>As always, see Ubuntu's main page on <a href="http://www.ubuntu.com/support/community">community support
options</a>.
<a href="https://askubuntu.com">askubuntu.com</a>, <code>#ubuntu-server</code> on IRC
(Freenode) and the <a href="https://lists.ubuntu.com/mailman/listinfo/ubuntu-server">Ubuntu Server mailing
list</a> are
appropriate venues.</p>Mailcap, HTML and AppArmor2014-02-09T01:34:14+00:00Robie Basaktag:www.justgohome.co.uk,2014-02-09:blog/2014/02/mailcap-html-apparmor.html<p>Fed up of being unable to read legitimate <code>multipart/alternative</code> emails
in mutt because of badly formatted <code>text/plain</code> sections, I set about
fixing my mailcap to have mutt show me the <code>text/html</code> section rendered
sensibly.</p>
<p>The usual way to do this is to have the mailcap call out to lynx to
convert the HTML into text, and then have mutt display this instead. In
my case, I chose links, since by default it renders tables better.</p>
<p>Trouble is, many emails are malicious. They may contain <a href="http://en.wikipedia.org/wiki/Web_bug">web
bugs</a> or attempt to exploit my
HTML viewer in some way. How secure is links' (or lynx's) <code>-dump</code> mode
in protecting me from this? I wasn't sure, and it seemed like
unnecessary attack surface to me.
<a href="https://help.ubuntu.com/community/AppArmor">AppArmor</a> to the rescue!</p>
<p>With a custom AppArmor profile, I can arrange for my mailcap to run
links constrained. I only need the program to run, read the input file,
and write to stdout. No need for it to have network capability or access
any other files. So why let it?</p>
<p>To arrange this, I had to edit three files.</p>
<p>The first file is the AppArmor profile itself. I created
<code>/etc/apparmor.d/links-local</code> to define a non-attaching profile as
follows:</p>
<div class="highlight"><pre><span class="cp">#include</span> <span class="cpf"><tunables/global></span><span class="cp"></span>
<span class="n">profile</span> <span class="n">links</span><span class="o">-</span><span class="n">local</span> <span class="p">{</span>
<span class="cp">#include</span> <span class="cpf"><abstractions/base></span><span class="cp"></span>
<span class="o">/</span><span class="n">tmp</span><span class="o">/</span><span class="n">links</span><span class="o">-</span><span class="n">local</span><span class="o">-*</span> <span class="n">r</span><span class="p">,</span>
<span class="p">}</span>
</pre></div>
<p>For now, I loaded the profile with the command <code>sudo apparmor_parser -r
/etc/apparmor.d/links-local</code>. I think this will automatically load on
reboot; I will find out when I reboot next.</p>
<p>The <code>#include <abtractions/base></code> line pulls in the standard definitions
that allow regular programs to run, including things like loading shared
libraries. These depend on the tunables I pulled in earlier. Finally, I
permit the profile to load a single file in /tmp which I will write to
with a wrapper. The prefix should help to ensure that the profile cannot
even load some other file in /tmp.</p>
<p>The next file is a wrapper I will use to set up the file in /tmp, and
then run links constrained with the profile I defined above. I put this
in <code>~/bin/links-local</code>:</p>
<table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre>1
2
3
4
5
6
7
8</pre></div></td><td class="code"><div class="highlight"><pre><span class="ch">#!/bin/sh</span>
<span class="nb">set</span> -e
<span class="nv">tmpfile</span><span class="o">=</span><span class="sb">`</span>mktemp --tmpdir links-local-XXXXXXXXXX<span class="sb">`</span>
clean<span class="o">()</span> <span class="o">{</span> rm -f <span class="s2">"</span>$<span class="s2">tmpfile"</span><span class="p">;</span> <span class="o">}</span>
<span class="nb">trap</span> clean EXIT
cp <span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span> <span class="s2">"</span>$<span class="s2">tmpfile"</span>
aa-exec -p links-local -- links -dump -force-html <span class="s2">"</span>$<span class="s2">tmpfile"</span>
</pre></div>
</td></tr></table>
<p>Note that I use <code>mktemp</code> for secure /tmp file creation, named using a
template that my profile will permit. Then I call <code>aa-exec</code> to constrain
links with the profile I loaded earlier.</p>
<p>Finally, I added this entry to <code>~/.mailcap</code>:</p>
<div class="highlight"><pre>text/html; links-local '%s'; copiousoutput; description=HTML Text
</pre></div>
<p>(<code>~/bin/</code> is already in my <code>PATH</code>)</p>
<p>That's it. Now, from mutt, I can view a <code>text/html</code> section and it'll be
rendered securely and appear directly in mutt's pager. I am protected in
two ways. First, I hope that links is doing the right thing securely,
anyway. Second, even if it is compromised in some way, AppArmor will
constrain any compromise from being able to do much on my computer.</p>
<p>Incidentally, I noticed that links does try to do some things that are
being blocked by AppArmor. I can see this in <code>dmesg</code> output. It mainly
seems to relate to <code>~/.links2</code> but doesn't seem to stop the program from
operating correctly. This is fine with me. I have no need for it to be
touching my home directory when called like this.</p>Continuous integration of your Ubuntu-based server deployments2013-09-19T10:45:11+00:00Robie Basaktag:www.justgohome.co.uk,2013-09-19:blog/2013/09/continuous-integration-of-your-ubuntu-based-server-deployments.html<p>Yesterday I posted a call for testing to the <a href="https://lists.ubuntu.com/mailman/listinfo/ubuntu-server">ubuntu-server
list</a>:</p>
<blockquote>
<p>There has been much work in Debian since wheezy was released, including
a major transition to Apache 2.4[1]. The maintainers used this
opportunity to overhaul the packaging, which also affected dependencies
such as PHP[2].</p>
<p>Ubuntu has picked this up. We're now well into feature freeze, and
expect to release in October with Apache 2.4 and PHP 5.5.</p>
<p>I have done some testing. Everything seems to work, but I am aware that
users do quite radically different things with their Apache
configurations.</p>
<p>So if you intend to use Apache and/or PHP in a future release, please
take some time to check that your use cases still work on our current
development release (Saucy).</p>
<p>If you have automated deployments and testing, then please adapt, run
and test your deployments against the development release. This will put
you in great shape for our upcoming LTS release next year.</p>
<p>It would be great to identify and fix any issues before release, when
bugs are much easier to fix.</p>
</blockquote>
<p>I know that there are many users who use Ubuntu Server as an easy
platform on which to deploy a LAMP stack. It would be a shame if we
broke something and were unaware of it. Both success and failure reports
are appreciated.</p>
<p>A generally accepted key goal for best practice production use is an
automated, reproducible and automatically tested deployment. Creating
such an environment is a separate topic and out of the scope of this
post. But if you're already doing this, or you intend to do this in the
future, then I have a feature suggestion for you to note.</p>
<p>How about a branch that targets the Ubuntu development release? This way
your continuous integration can flag any action you need to take to make
your deployment work against the next release (and thus the next LTS, if
you deploy on LTS). If we make a change that makes your life difficult,
then you will be able to feed this information back to us, and we will
be able to take this into account.</p>
<p>Bugs and design changes are particularly hard to fix after release,
since we have to worry about not causing regressions for existing users.
During development, this is not a concern. So doing this helps you too:
more bugs get fixed quicker in time for the release that you want to
use.</p>
<p>While I'm talking about continuous integration on deployments, it's
worth noting that it's worthwhile running your tests on the -proposed
branch of the stable release you're using, too. If you also test your
deployment with <a href="https://wiki.ubuntu.com/Testing/EnableProposed">the -proposed pocket
enabled</a>, then you can
detect and flag any failures on proposed stable updates before we
release them. If we know that a proposed update causes a regression in
your deployment, then we won't release it. To flag a proposed update as
causing a regression, just tag the bug with <code>regression-proposed</code> and
add an explanation. You can find the bug number from the affected
package's changelog in
<code>/usr/share/doc/<em>package</em>/changelog.Debian.gz</code>.</p>Better two factor ssh authentication on Ubuntu2013-07-31T02:27:50+00:00Robie Basaktag:www.justgohome.co.uk,2013-07-31:blog/2013/07/better-two-factor-ssh-authentication-on-ubuntu.html<p>In the past, true two factor authentication in ssh has been something of
a hack to set up. Now that OpenSSH 6.2 has now been released with <a href="https://bugzilla.mindrot.org/show_bug.cgi?id=983">full
and proper support</a>, the next release of Ubuntu (Saucy Salamander)
will include it.</p>
<h2>Quick Start</h2>
<p>All you have to do is:</p>
<ol>
<li>
<p><code>apt-get install libpam-google-authenticator</code>.</p>
</li>
<li>
<p>Users who want to continue using ssh must each run the command
<code>google-authenticator</code>. This tool interatively helps you to create the
file <code>~/.google_authenticator</code>, which contains a shared secret and
emergency passcodes. It's a terminal application, but it does still
display a QR code for quick loading of the shared secret into your two
factor device (in my case, this is the Google Authenticator app on my
Android smartphone).</p>
</li>
<li>
<p>Edit <code>/etc/ssh/sshd_config</code>. Set:</p>
<div class="highlight"><pre>ChallengeResponseAuthentication yes
PasswordAuthentication no
AuthenticationMethods publickey,keyboard-interactive
</pre></div>
<p>In case you have changed them in the past, you should also check the
following two settings (these are both defaults on Ubuntu):</p>
<div class="highlight"><pre>UsePAM yes
PubkeyAuthentication yes
</pre></div>
</li>
<li>
<p>Run <code>sudo service ssh reload</code> to pick up your changes to
<code>/etc/ssh/sshd_config</code>.</p>
</li>
<li>
<p>Edit <code>/etc/pam.d/sshd</code> and replace the line:</p>
<div class="highlight"><pre>@include common-auth
</pre></div>
<p>with:</p>
<div class="highlight"><pre>auth required pam_google_authenticator.so
</pre></div>
</li>
</ol>
<p>That's it! Now ssh logins will require a key, and after your key is
verified will additionally require proof that you hold your second
factor device.</p>
<p>Your existing ssh session should not be affected by these changes. So
before you continue, make sure that you can still access the machine by
authenticating with your new two factor system in another session. If
you're using shared connections (or aren't sure), be sure to use the
<code>-Snone</code> option to ssh in order to make sure that you don't accidentally
skip authentication by re-using an existing connection.</p>
<h2>How It Works</h2>
<p>Traditionally, ssh only verified one thing. That was either your
password, or verification that you held your private key, or something
else (GSSAPI/Kerberos and other things I won't cover here). Multiple
methods were allowed, and success using any one method was considered a
successful authentication.</p>
<p>ssh continues to be able to defer to PAM for authentication. However,
authentication with an ssh key happens outside PAM, and ssh treats any
one of these (key or PAM) as successful. This means that it checks for
your password, or that you hold your private key, but not both. This
makes it difficult to use a key together with PAM for a second factor,
since ssh will just go ahead and never consult PAM for the second factor
if your client proves that you hold your key.</p>
<p>With the new feature, <code>AuthenticationMethods</code> can be used to specify two
methods that are <em>both</em> required. This means that you can require both
ssh key authentication and PAM, and adjust your PAM configuration to be
the second factor device in the case of the ssh service.</p>
<h2>Why is this more secure?</h2>
<p>It's always a good idea to reduce your attack surface, and limit
authentication code only to areas designed with security in mind.
Previous methods to implement this kind of support involved hacks or
patches. With this new support, the only security sensitive code used is
PAM and ssh itself, being used as they were designed to be used, and
without any third party patches. Both were designed for security from
the beginning.</p>
<h2>Any catches?</h2>
<p>The Google Authenticator PAM module (<code>libpam-google-authenticator</code>) is
in universe, which means that it is only community supported for
security updates. But this can be changed through the <a href="https://wiki.ubuntu.com/MainInclusionProcess">main inclusion
process</a>. If this method to implement two factor security in Ubuntu
becomes popular, then I'd love to see this happen, and would be happy to
drive it.</p>
<h2>Variations</h2>
<p>This is a clean and flexible mechanism. For example: you can leave
<code>/etc/pam.d/sshd</code> alone, and then you'll get a system which requires
both a key and the user's password. Or leave the <code>@include common-auth</code>
line in place, insert <code>auth required pam_google_authenticator.so</code> before
it, and you'll get a system which requires all three: a key, the
password and the code from the second factor device.</p>