Debian/Ubuntu: Package Building

Introduction

Modern Linux system management centers around the care and feeding of a configuration management system like Chef or Puppet. These systems do a fine job of maintaining complex configurations, but frequently the complete description of a system will include custom software of varying complexity. Configuration management systems are less than ideal when used as a system for developing and maintaining this ancillary software. In addition to being a superior way of deploying software, Debian package building tools provide excellent methods for developing and testing software. State of the art system management practices employ a configuration management system to control system properties and packages to deploy the software on the system, including custom software.

There is no one way to build a Debian package. This document describes a single work flow and does not attempt to delve into any alternatives. The flow described here uses git-buildpackage configured to build the package in a chroot. Using a chroot ensures cleaner builds because the build is not affected by the packages that are locally installed on the local system.

Packages used in conjunction with a configuration management system usually consist of a collection of utility scripts. These types of packages are very simple and basically consist of copying files on to a system and avoid the complexities of building binary images. While creating and running tests are an optional part of package building it is highly recommended to create tests as part of creating the package. Even trivial tests, like ensuring that a help messages display correctly, are very valuable and uncover a significant number of errors.

There are several excellent packaging tutorials on both Debian and Ubuntu web sites. The definitive guide for what must be in a package is the Debian Policy Manual.

---

Package Building using Build Server

A “build server” is a server that already has the chroots constructed for package building and has git-buildpackage installed. To use the server all that is required is to add personal user configuration elements. The current CorpSRE build server is shelter-apt.ca-zephyr.internal and is configured supporting building packages for several Debian versions.

Setup Personal Environment

Most of the files required to setup a personal package building environment are create by the script ubuild-setup script.

1. Edit the login script and create the environment variables DEBEMAIL, DEBFULLNAME, DIST, and ARCH. For bash users adding the following to .bashrc is sufficient.

    export DEBEMAIL=bill@ca-zephyr.org
    export DEBFULLNAME="Bill MacAllister"
    export DIST=bionic
    export ARCH=amd64
   

2. Create ~/.gbp.conf and ~/.dput.cf files. See below for details of these files.

3. Make sure a gpg key is available on the host that for the DEBEMAIL user. This will be used for signing packages.

4. Make sure ssh keys are available and you have a valid Kerberos ticket cache to allow pushing packages to the debian.ca-zephyr.org package repository.

Package Building

There is plenty of documentation on Debian web sites about package building. The simplest way to create a new package is to start with the pack source of an existing package and rename it. The example below details rebuilding an existing package.

• Clone an existing repository with:

    $ git clone git@github.com:whm/cz-xen-utils.git
  

• Or, create a new repository with:

    $ mkdir cz-package-name
    $ cd cz-package-name
    $ git init
  

• Make any modifications to the existing package or create the new package structures. If a new package is being created it is easiest to just copy the debian directory from an existing package and modify that.

• Update the debian/changelog file by adding a new section and increment the version number. Commit the changes. (Many editors have helper macros to format debian changelog files. Consider using debchange(1). The helpers for emacs have been installed on shelter-apt.ca-zephyr.org.)

• Run git-buildpackage.

    $ gbp buildpackage
  

• Copy the resulting .deb file to a test system and install it. This step is not absolutely required, but since the package version must be incremented for uploads to the debian repository it is a good way to prevent the excessive churn of the version number during development and testing.

• Once the changes have been tested sign the *.changes file and push the package to the debian.ca-zephyr.org debian repositiory server. (Note, the example below assumes that the build-area directory is cleaned out before building the package. If this is not the case then the full name of the changes files needs to be specified.)

    $ cd ../build-area
    $ debsign *.changes
    $ dput cz-afs *.changes
  

• Add a version tag to the git repository and push the changes to the remote debian repository.

    $ cd ../cz-package-name
    $ git tag debian/1
    $ git push --all
    $ git push --tags
  

• Copy the package to the desired distributions using reprepro.

    $ remctl debian reprepro help
    $ remctl debian reprepro copy dropbox bookworm sid dropbox-corp-server
  

  OB

  Package Building Variations

Building Release Specific Packages

Many times packages contain only scripts that are copied on to the target systems. These packages are frequently architecture and release independent and can be copied from on release to another without have to build the package for a specific OS release. This is not always the case. The most notable exception are packages that contain C source. To build a package for a specific OS release set the environment variable DIST to the target release. For example:

  $ DIST=bullseye gbp buildpackage

Backporting and Patching Packages

When patching a package it sometimes means that gbp does not recognize as the first build of a release. This is because the version does not end in "-1". The result is that the changes files that gbp creates does not include the tar file with the source. When this happens reprepro will refuse accept the package into the repository. If this happens the reprepro managed repository needs to be "cleaned" and the package rebuilt with the "-sa" switch.

Here is an example of the upload failure and recovery:

  $ debsign *.changes
  $ dput cz-afs *.changes
  ... lots of output ...
  file 'cyrus-sasl2_2.1.27+dfsg.orig.tar.xz' is needed ...
  There have been errors!
  ... more output ...
  $ remctl cz-afs reprepro clean dropbox
  $ rm *.upload

Here is an example of the gbp command line required to solve this problem:

  $  gbp buildpackage -sa

---

Package Building Environment From Scratch

This is a description of creating a build environment suitable for use with git-buildpackage on a Ubuntu system.

• Install the cowbuilder package.

    $ apt install build-essential \
                  cowbuilder \
                  debhelper \
                  debian-el \
                  debian-goodies \
                  dgit \
                  dh-autoreconf \
                  dh-buildinfo \
                  dh-exec \
                  dh-linktree \
                  dh-make \
                  dh-make-perl \
                  dh-make-php \
                  dh-metainit \
                  dh-python \
                  dpkg-dev-el \
                  git-buildpackage \
                  git-debrebase \
                  packaging-dev \
                  pbuilder-scripts \
                  quilt 
  

• Create the file /etc/pbuilderrc to describe a debian mirror. For example:

    MIRRORSTIE=http://ftp.us.debian.org/debian/
  

• Create a chroot for the package build. For building scripts only one chroot needs to be built because scripts are frequently portable across Debian releases. reprepro can be used to copy the scripts to the desired release once the package has been uploaded. The following script can be used to create chroots for pbuilder.

    #!/bin/bash
    if [ "$1" = "" ]
    then
        echo "Usage: make-chroot <dist>"
        exit 1
    fi
    base_dir="/var/cache/pbuilder/${1}-amd64"
    if [ ! -e $base_dir ]
    then
        mkdir -p $base_dir
    fi
    cowbuilder --create \
        --basepath $base_dir/base.cow \
        --distribution ${1} \
        --debootstrapopts \
        --arch \
        --debootstrapopts amd64
  

• Create symlinks of the form:

    ln -s /var/cache/pbuilder/bookworm-amd64/base.cow \
          /var/cache/pbuilder/base-bookworm.cow
  

• Create the following sudoers configuration.

    Cmnd_Alias PBUILDERS = /usr/bin/git-pbuilder, /usr/sbin/pbuilder, /usr/sbin/cowbuilder
    Defaults! PBUILDERS setenv
    %pack ALL = NOPASSWD: PBUILDERS
  

• Add users that will be package builders to the "pack" group.

• Set environment variables DIST, DEBMAIL, and DEBFULLNAME. For example, update the .bashrc with the following:

    export DEBEMAIL=bill@ca-zephyr.org
    export DEBFULLNAME="Bill MacAllister"
    export DIST=trusty
    export ARCH=amd64
  

• Create a git build package configuration file, ~/.gbp.conf.

    [DEFAULT]
    builder = /usr/bin/git-pbuilder
    cleaner = fakeroot debian/rules clean
    pristine-tar = True
  
    [buildpackage]
    export-dir = ../build-area/
    tarball-dir = ../tarballs/
  
    [import-orig]
    dch = False
  

• Create a ~/.dput.cf for uploading packages.

    # File: ~/.dput.cf
    [DEFAULT]
    method = scp
    hash = md5
    allow_unsigned_uploads = 0
    run_lintian = 0
    run_dinstall = 0
    check_version = 0
  
    [apt]
    method = scp
    fqdn = apt-01.corp.dropbox.com
    allowed_distributions = ^(unstable|stable|oldstable|bullseye|bookworm)$
    incoming = /afs/@cell/public/repo/ca-zephyr/incoming
    post_upload_command = krenew & remctl shelter-apt cz-upload upload
  

• Create a quilt configuration file.

    for where in ./ ../ ../../ ../../../ ../../../../ ../../../../../; do
        if [ -e ${where}debian/rules -a -d ${where}debian/patches ]; then
            export QUILT_PATCHES=debian/patches
            break
        fi
    done
  

At this point the packing build environment is ready to be used to build packages using gbp.

Importing an existing package into a new repo for managing with GBP

You will need the DSC file for the package, and whatever it references (.orig, .debian, and the .dsc itself). For example, if you want to import “mypackage” you would need:

    mypackage_1.0.0.orig.tar.gz
    mypackage_1.0.0-1.debian.tar.gz
    mypackage_1.0.0-1.dsc

Start by creating a directory where your repo will live

    $ mkdir ~/git-ubuild/mypackage

go into this directory and initialize it up as a git repo

    $ cd ~/git-ubuild/mypackage
    $ git init

Now, we need to “import” that version of the package into the repo:

    $ gbp import-dsc path/to/mypackage_1.0.0-1.dsc \
           --pristine-tar \
           --create-missing-branches

At this point, you should be able to build the package as mentioned above.