Tuesday, 11 December 2012

Upstart with "stateful re-exec" now in Ubuntu Raring

After a lot of testing, we have today landed Upstart 1.6.1 in Ubuntu Raring with "stateful re-exec" goodness.

What this means is that for all future Upstart upgrades, no reboot is required to make use of the newly-installed version: Upstart will restart itself with no interruption to service such that existing jobs will continue to run.

Additionally, it means that an upgrade to any of Upstarts dependent libraries (such as libc and libnih) can also trigger Upstart to restart itself so that it will automatically start using those new library versions whilst maintaining continuity of service.

For those of us who dislike downtime and want to remain current, this is known as a "win" ;-)

Thursday, 29 November 2012

procenv now in Debian and Ubuntu: how you can use it

The procenv utility covered in a previous post is now at version 0.15 and also available in Debian Sid and Ubuntu Raring. So, you can do the usual "sudo apt-get install procenv" followed by simply "procenv" to run it. However, due to the way this utility is built, there is a little more you can do. Read on...

procenv runs itself as part of its build (at the end of course :-). What this means is that you can now see what that build environment is like by looking at the build logs:

If you haven't already heard, for the Ubuntu Raring cycle, autopkgtest (DEP-8) tests -- where package builds automatically trigger tests that run on an environment very similar to a "normal" system -- are a hot topic. procenv is DEP-8 enabled, so again, we get to see exactly what such an autopkgtest environment provides.

With this information, you can perform some rather interesting analysis:

  • See the environment a "buildd" provides.
  • Compare a buildd environment versus your "normal" workstation environment.
  • Compare a Debian buildd environment with an Ubuntu build environment.
  • Compare an Ubuntu buildd environment with an autopkgtest environment.
Where this comes in particularly useful is understanding what is going on when your package "FTBFS" (fails to build from source) even though it builds perfectly on your dev box. The same logic applies if your "make check" tests are failing on the builders. By looking at the procenv logs for the same platform and release as the package you are working on, you may find clues as to what is going on. Maybe the tests are being run as an unexpected user. Maybe the package expects a controlling terminal but isn't being given one. Maybe the limits are being set too low for your tests to perform adequately. All of this and more can be gleaned from looking at the procenv build log output.

Please report any bugs you find here.

Friday, 16 November 2012

Procenv 0.8 released

Changes: expanded man page, more sizeof types shown and resource usage details added.

Code is available here:
If anyone is interested in contributing patches for the following, let me know:

  • code to show network details.
  • support for other platforms (I'd like to add AIX, HP-UX, Solaris, i5O/S, MVS/zO/S, VMS) but no longer have access to such systems).

Thursday, 15 November 2012

Upstart 1.6 released

New Upstart Release

Today we released Upstart 1.6, which includes the "stateful re-exec" feature, allowing Upstart to continue to supervise jobs after an upgrade of either itself, or any of its dependent libraries:

This will be available in Ubuntu Raring very soon.

Thanks to all the contributors:

  • A.Costa
  • Colin Watson
  • Eric S. Raymond
  • Jason Conti
  • Loïc Minier
  • Steve Langasek


Like to get involved?

If you'd like to get involved with Upstart Development...

Here's an early draft of what's planned for the next cycle:

If you're interested in contributing, let me know!

Update on Upstart in Debian

... But wait - there's more! Thanks to the efforts of Steve Langasek, Upstart 1.5 is now available in Debian:

Thursday, 1 November 2012

Wednesday, 31 October 2012

Hacking Upstart Together

If you missed the Hacking Upstart Together session at UDS-R in Copenhagen, here are the slides:

This gives a very gentle introduction to Upstart development and covers NIH, common gotchas and more.

... and if you'd like the LaTeX source....
We are always looking for new contributors and have some very interesting work lined up this cycle. So, if you're interested in contributing even a small patch or feature, please feel free to contact me or ask on the mailing list.

Monday, 29 October 2012

Procenv and the Process Environment


How many attributes of a processes "environment" can you think of besides its environment variables? I'm using the term "environment" in a very loose sense here to mean "any system or process level attribute that can be queried via system calls or library calls (or "/proc-picking") that affects a running process". I'm also including details of the process itself and details of the program that came to be a running process.

We're not talking about installed packages or application configuration files in /etc here, but purely low-level program, process and system meta-data.

I've got over 20 items in my list excluding environment variables.

Whilst you're pondering on that...

Compare and Contrast

If you've been involved with computers for any appreciable length of time, chances are you have come across the scenario where some program fails to run in a particular environment, but works "perfectly" in another (generally a developers cosy interactive shell environment).

Ignoring the fact that the development environment should always mirror the production environment as closely as possible, this problem tends to crop up in certain common scenarios. The two most common (from my observations) being:

  • system "services"

    Such programs (often daemons of course) run -- and indeed should run -- in an exceedingly sparse environment: they get given just enough of an environment to do their job safely and efficiently.

    And yes, I get asked quite a lot why some new Upstart job won't start, but runs in a desktop session environment "perfectly" :-)
  • build systems

    Sometimes, the environment of such systems is quite different to what the developer expected. Maybe the build system is a virtualised or chrooted system, or it might be running on a host which has a different architecture to the expected one.


Back to the poser. How many have you got? Here's my list of "groups":

  • pid details.
  • session details.
  • credentials information.
  • password database details for user running the program.
  • groups user running application belongs to.
  • SELinux details.
  • CPU details.
  • scheduler details.
  • capabilities.
  • oom details.
  • cgroups.
  • stat details for the binary that is being run.
  • details of libraries that said binary has loaded.
  • compiler details for the binary being run.
  • uname details.
  • open file descriptors.
  • process limits.
  • system limits.
  • configuration-dependent limits.
  • timezone details.
  • clock details.
  • time details.
  • terminal details.
  • signal dispositions.
  • mount details.
  • networking details.
  • sizeof details.
  • ranges of standard system data types.


The answer to the two posers above is procenv. This is a simple utility, licensed under the GPL, that essentially dumps every conceivable aspect of a processes environment (*) that it can (without needing super-user privs). It unashamedly emulates a number of existing system utilities as it is attempting to be all-encompassing: I wrote it with the aim of being able to dump "everything" that a process may care about by simply running a single program (by default). Also, the line of demarcation between "process", "program" and  "system" is slightly blurry in some aspects (for example sysconf(3) variables could arguably be considered system attributes, but procenv shows these too since they are obviously meant to be querable by applications.

(*) - well, it doesn't currently query the network environment and I may have overlooked something, so patches are of course welcome!


"But wait!" I hear you cry, "this procenv is redundant: I can get all the information I need from /proc!"

Well, that's half correct - you can get a lot of information out of /proc, but there are some caveats with that approach:

  • /proc might not be mounted.
  • /proc might not be readable (it could legitimately have permissions 0000).
  • the format and location of files in /proc may change in the future.
  • the format of certain files in /proc is sometimes extremely terse and cryptic.
As such, where possible, procenv uses system and library calls rather than grubbing around in /proc for the answers it needs since system calls and library calls don't generally change :-)

"Well, I could just write a shell script to massage the data in /proc into a consistent format, or just make use of other system utilities to produce an aggregated report" you say. Yes, you could do that, but procenv is striving to be cross-platform and additional to the /proc concerns, writing such a tool in shell comes with a bag of gotchas if it's going to work on anything but GNU Linux:

  • which shell are you going to support?
  • which version of standard utilities like (awk (or gawk or even nawk) are you going to support?
If you did write such a shell script, it could end up being somewhat of a monster if it was to work on all platforms and massage utility output to be consistent across these platforms.

Possible Uses

There are quite a few I suspect. Here are a few thoughts on some of the environments it could be run in to give a better understanding of what that enviroment is:

  • See what nohup(1) does to your process:

    $ nohup procenv
  • See what shell back-grounding using does to your process:

    $ procenv &
  • Exploring the environment available to you when you login to a system with restricted access (what has been restricted?)
  • Seeing the environment the kernel gives to the initramfs.
    Boot with options including something like:

    rdinit=/usr/bin/procenv PROCENV_FILE=/dev/ttyS0 PROCENV_EXEC="/init"
  • Exploring the environment the initramfs gives to PID 1 aka "init"(aka Upstart for the majority of folks I hope ;-)

    init=/usr/bin/procenv PROCENV_FILE=/dev/ttyS0 PROCENV_EXEC="/sbin/init"
  • Investigating chroots
  • Jails (yes, it runs on FreeBSD and Debian GNU/kFreeBSD too :-)
  • Explore a container environment (LXC, OpenVZ, VServer, etc).
  • Explore an OpenStack or EC2 instance.
  • Understanding the environment of an Upstart job:

    $ cat << EOT
    exec /usr/bin/procenv
    EOT | sudo tee /etc/init/procenv.conf

  • Those alluded-to build environments such as those provided by pbuilder(8) and sbuild(1).

  • Summary

    So, if you want to know if you are a Process Leader or a Session Leader (or both!), or you want to know what your stderr/stdout/stderr are connected to, give procenv a try.

    Example Output

    Here's some sample output from procenv shown running in an LXC container running Ubuntu Quantal:

    $ procenv --misc
    umask: 0002
    current directory (cwd): '/home/james'
    root: '/'
    chroot: yes
    selinux context: lxc-container-default (enforce)
    container: lxc
    cpu: 4 of 4
    scheduler: SCHED_OTHER
    scheduler priority: process=0, process group=0, user=-11
    memory page size: 4096 bytes

    Here's an example showing the full details (running procenv with no options):
      version: procenv 0.3
      platform: Linux (Intel)
      kernel bits: 32
      CLOCK_REALTIME: resolution: 0.000000001s
      CLOCK_MONOTONIC: resolution: 0.000000001s
      CLOCK_MONOTONIC_RAW: resolution: 0.000000001s
      CLOCK_PROCESS_CPUTIME_ID: resolution: 0.000000001s
      CLOCK_THREAD_CPUTIME_ID: resolution: 0.000000001s
      name: GCC
      version: 4.7.2
      compile date: Oct 27 2012
      compile time: 14:57:03
      __STRICT_ANSI__: not defined
      _POSIX_C_SOURCE: 200809
      _POSIX_SOURCE: defined
      _XOPEN_SOURCE: 700
      _ISOC95_SOURCE: defined
      _ISOC99_SOURCE: defined
      _ISOC11_SOURCE: not defined
      _LARGEFILE64_SOURCE: defined
      _FILE_OFFSET_BITS: not defined
      _BSD_SOURCE: defined
      _SVID_SOURCE: defined
      _ATFILE_SOURCE: defined
      _GNU_SOURCE: defined
      _REENTRANT: not defined
      _THREAD_SAFE: not defined
      _FORTIFY_SOURCE: defined
      _CS_GNU_LIBC_VERSION: 'glibc 2.15'
      _CS_PATH: '/bin:/usr/bin'
      fd 0: terminal=yes ('/dev/console')
      fd 1: terminal=yes ('/dev/console')
      fd 2: terminal=yes ('/dev/console')
      fd 3: terminal=yes ('/dev/tty')
      fd 4: terminal=no
      fd 17: terminal=no
    fds (linux/proc):
      '/proc/self/fd/0' -> '/dev/console' (terminal=yes, valid=yes)
      '/proc/self/fd/1' -> '/dev/console' (terminal=yes, valid=yes)
      '/proc/self/fd/2' -> '/dev/console' (terminal=yes, valid=yes)
      '/proc/self/fd/3' -> '/dev/tty' (terminal=yes, valid=yes)
      '/proc/self/fd/4' -> '/tmp/procenv-lxc.log' (terminal=no, valid=yes)
      RLIMIT_AS (soft=4294967295 (max), hard=4294967295 (max))
      RLIMIT_CORE (soft=4294967295 (max), hard=4294967295 (max))
      RLIMIT_CPU (soft=4294967295 (max), hard=4294967295 (max))
      RLIMIT_DATA (soft=4294967295 (max), hard=4294967295 (max))
      RLIMIT_FSIZE (soft=4294967295 (max), hard=4294967295 (max))
      RLIMIT_LOCKS (soft=4294967295 (max), hard=4294967295 (max))
      RLIMIT_MEMLOCK (soft=65536, hard=65536)
      RLIMIT_MSGQUEUE (soft=819200, hard=819200)
      RLIMIT_NICE (soft=0, hard=0)
      RLIMIT_NOFILE (soft=10240, hard=10240)
      RLIMIT_NPROC (soft=63489, hard=63489)
      RLIMIT_RSS (soft=4294967295 (max), hard=4294967295 (max))
      RLIMIT_RTPRIO (soft=0, hard=0)
      RLIMIT_SIGPENDING (soft=63489, hard=63489)
      RLIMIT_STACK (soft=8388608, hard=4294967295 (max))
      umask: 0002
      current directory (cwd): '/home/james'
      root: '/'
      chroot: yes
      selinux context: lxc-container-default (enforce)
      container: lxc
      cpu: 4 of 4
      scheduler: SCHED_OTHER
      scheduler priority: process=0, process group=0, user=-11
      memory page size: 4096 bytes
      fsname='rootfs', dir='/', type='rootfs', opts='rw', dev=(major:8, minor:1)
        pathconf for path '/':
      fsname='/dev/disk/by-uuid/7ad192e9-7b26-49d1-8e1c-fefc7dc495cb', dir='/', type='ext4', opts='rw,relatime,errors=remount-ro,data=ordered', dev=(major:8, minor:1)
        pathconf for path '/':
      fsname='proc', dir='/proc', type='proc', opts='rw,nosuid,nodev,noexec,relatime', dev=(major:0, minor:36)
        pathconf for path '/proc':
      fsname='sysfs', dir='/sys', type='sysfs', opts='rw,relatime', dev=(major:0, minor:37)
        pathconf for path '/sys':
      fsname='devpts', dir='/dev/console', type='devpts', opts='rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000', dev=(major:0, minor:11)
        pathconf for path '/dev/console':
      fsname='devpts', dir='/dev/tty1', type='devpts', opts='rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000', dev=(major:0, minor:11)
        pathconf for path '/dev/tty1':
      fsname='devpts', dir='/dev/tty2', type='devpts', opts='rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000', dev=(major:0, minor:11)
        pathconf for path '/dev/tty2':
      fsname='devpts', dir='/dev/tty3', type='devpts', opts='rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000', dev=(major:0, minor:11)
        pathconf for path '/dev/tty3':
      fsname='devpts', dir='/dev/tty4', type='devpts', opts='rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000', dev=(major:0, minor:11)
        pathconf for path '/dev/tty4':
      fsname='devpts', dir='/dev/pts', type='devpts', opts='rw,relatime,mode=600,ptmxmode=666', dev=(major:0, minor:38)
        pathconf for path '/dev/pts':
      fsname='devpts', dir='/dev/ptmx', type='devpts', opts='rw,relatime,mode=600,ptmxmode=666', dev=(major:0, minor:38)
        pathconf for path '/dev/ptmx':
      fsname='none', dir='/proc/sys/fs/binfmt_misc', type='binfmt_misc', opts='rw,nosuid,nodev,noexec,relatime', dev=(major:0, minor:29)
        pathconf for path '/proc/sys/fs/binfmt_misc':
      fsname='none', dir='/sys/fs/fuse/connections', type='fusectl', opts='rw,relatime', dev=(major:0, minor:16)
        pathconf for path '/sys/fs/fuse/connections':
      fsname='none', dir='/sys/kernel/debug', type='debugfs', opts='rw,relatime', dev=(major:0, minor:6)
        pathconf for path '/sys/kernel/debug':
      fsname='none', dir='/sys/kernel/security', type='securityfs', opts='rw,relatime', dev=(major:0, minor:10)
        pathconf for path '/sys/kernel/security':
      fsname='none', dir='/run', type='tmpfs', opts='rw,nosuid,noexec,relatime,size=814240k,mode=755', dev=(major:0, minor:39)
        pathconf for path '/run':
      fsname='none', dir='/run/lock', type='tmpfs', opts='rw,nosuid,nodev,noexec,relatime,size=5120k', dev=(major:0, minor:40)
        pathconf for path '/run/lock':
      fsname='none', dir='/run/shm', type='tmpfs', opts='rw,nosuid,nodev,relatime', dev=(major:0, minor:41)
        pathconf for path '/run/shm':
      fsname='none', dir='/run/user', type='tmpfs', opts='rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755', dev=(major:0, minor:42)
        pathconf for path '/run/user':
      process id (pid): 344
      parent process id (ppid): 307
      session id (sid): 290 (leader=no)
      name: 'procenv'
      ancestry: 344 ('procenv'), 307 ('zsh'), 290 ('login'), 1 ('init')
      process group id: 344 (leader=yes)
      foreground process group: 344
      terminal: '/dev/tty'
      has controlling terminal: yes
      on console: no
      real user id (uid): 1000 ('james')
      effective user id (euid): 1000 ('james')
      saved set-user-id (suid): 1000 ('james')
      real group id (gid): 1000 ('james')
      effective group id (egid): 1000 ('james')
      saved set-group-id (sgid): 1000 ('james')
      login name: 'james'
      passwd: name='james', gecos='', dir='/home/james', shell='/bin/zsh'
      groups: 'james' (1000)
        unsigned: 0 to 255 (0.000000e+00 to 2.550000e+02, 0x00 to 0xff)
        signed: -128 to 127
      short int:
        unsigned: 0 to 65535 (0.000000e+00 to 6.553500e+04, 0x0000 to 0xffff)
        signed: -32768 to 32767
        unsigned: 0 to 4294967295 (0.000000e+00 to 4.294967e+09, 0x00000000 to 0xffffffff)
        signed: -2147483648 to 2147483647
      long int:
        unsigned: 0 to 4294967295 (0.000000e+00 to 4.294967e+09, 0x00000000 to 0xffffffff)
        signed: -2147483648 to 2147483647
      long long int:
        unsigned: 0 to 18446744073709551615 (0.000000e+00 to 1.844674e+19, 0x0000000000000000 to 0xffffffffffffffff)
        signed: -9223372036854775808 to 9223372036854775807
        signed: 1.175494e-38 to 3.402823e+38
        signed: 2.225074e-308 to 1.797693e+308
      long double:
        signed: 3.362103e-4932 to 1.189731e+4932
      SIGHUP ('Hangup', 1): blocked=no, ignored=no
      SIGINT ('Interrupt', 2): blocked=no, ignored=no
      SIGQUIT ('Quit', 3): blocked=no, ignored=no
      SIGILL ('Illegal instruction', 4): blocked=no, ignored=no
      SIGTRAP ('Trace/breakpoint trap', 5): blocked=no, ignored=no
      SIGABRT ('Aborted', 6): blocked=no, ignored=no
      SIGBUS ('Bus error', 7): blocked=no, ignored=no
      SIGFPE ('Floating point exception', 8): blocked=no, ignored=no
      SIGKILL ('Killed', 9): blocked=no, ignored=no
      SIGUSR1 ('User defined signal 1', 10): blocked=no, ignored=no
      SIGSEGV ('Segmentation fault', 11): blocked=no, ignored=no
      SIGUSR2 ('User defined signal 2', 12): blocked=no, ignored=no
      SIGPIPE ('Broken pipe', 13): blocked=no, ignored=no
      SIGALRM ('Alarm clock', 14): blocked=no, ignored=no
      SIGTERM ('Terminated', 15): blocked=no, ignored=no
      SIGSTKFLT ('Stack fault', 16): blocked=no, ignored=no
      SIGCHLD|SIGCLD ('Child exited', 17): blocked=no, ignored=no
      SIGCONT ('Continued', 18): blocked=no, ignored=no
      SIGSTOP ('Stopped (signal)', 19): blocked=no, ignored=no
      SIGTSTP ('Stopped', 20): blocked=no, ignored=no
      SIGTTIN ('Stopped (tty input)', 21): blocked=no, ignored=no
      SIGTTOU ('Stopped (tty output)', 22): blocked=no, ignored=no
      SIGURG ('Urgent I/O condition', 23): blocked=no, ignored=no
      SIGXCPU ('CPU time limit exceeded', 24): blocked=no, ignored=no
      SIGXFSZ ('File size limit exceeded', 25): blocked=no, ignored=no
      SIGVTALRM ('Virtual timer expired', 26): blocked=no, ignored=no
      SIGPROF ('Profiling timer expired', 27): blocked=no, ignored=no
      SIGWINCH ('Window changed', 28): blocked=no, ignored=no
      SIGIO ('I/O possible', 29): blocked=no, ignored=no
      SIGPWR ('Power failure', 30): blocked=no, ignored=no
      SIGSYS ('Bad system call', 31): blocked=no, ignored=no
      bits/byte (CHAR_BIT): 8
      sizeof (char): 1 byte
      sizeof (short int): 2 bytes
      sizeof (int): 4 bytes
      sizeof (long int): 4 bytes
      sizeof (long long int): 8 bytes
      sizeof (float): 4 bytes
      sizeof (double): 8 bytes
      sizeof (long double): 12 bytes
      sizeof (size_t): 4 bytes
      sizeof (ssize_t): 4 bytes
      sizeof (ptrdiff_t): 4 bytes
      sizeof (void *): 4 bytes
      sizeof (wchar_t): 4 bytes
      sizeof (intmax_t): 8 bytes
      sizeof (intptr_t): 4 bytes
      sizeof (uintptr_t): 4 bytes
      sizeof (time_t): 4 bytes
      sizeof (clock_t): 4 bytes
      sizeof (sig_atomic_t): 4 bytes
      sizeof (off_t): 4 bytes
      sizeof (fpos_t): 12 bytes
      sizeof (mode_t): 4 bytes
      sizeof (pid_t): 4 bytes
      sizeof (uid_t): 4 bytes
      sizeof (gid_t): 4 bytes
      sizeof (wint_t): 4 bytes
      sizeof (div_t): 8 bytes
      sizeof (ldiv_t): 8 bytes
      sizeof (lldiv_t): 16 bytes
      sizeof (mbstate_t): 8 bytes
      argv[0]: './procenv'
      real path: '/home/james/procenv'
      dev: major=8, minor=1
      inode: 10223742
      mode: 0775 (-rwxrwxr-x)
      hard links: 1
      user id (uid): 1000 ('james')
      group id (gid): 1000 ('james')
      size: 161516 bytes (320 512-byte blocks)
      atime: 1351542553 (Mon Oct 29 20:29:13 2012)
      mtime: 1351542549 (Mon Oct 29 20:29:09 2012)
      ctime: 1351542549 (Mon Oct 29 20:29:09 2012)
      raw: 1351542560.130094335
      local: Mon Oct 29 20:29:20 2012
      ISO: 2012-09-29T20:29
        [c_cflag:input baud speed=B38400]
        [c_cflag:output baud speed=B38400]
      sysname: Linux
      nodename: quantal
      release: 3.5.0-17-generic
      version: #28-Ubuntu SMP Tue Oct 9 19:32:08 UTC 2012
      machine: i686
      domainname: (none)


    Grab the code from:

    Monday, 15 October 2012

    out is out, utfout is out!

    My output utility (originally called 'out' - see previous post) is now available in debian sid as 'utfout':

    Call for Testing: New "stateful re-exec" build of Upstart

    It didn't quite make quantal, but if you'd like to help test Upstart "stateful re-exec" support, there's a new build (Upstart version 1.5-0ubuntu10~jh20121015) in my PPA:

    We have so far added 60 new tests specifically for stateful re-exec support.

    The plan is to add this new feature to Ubuntu early in the "R" cycle, so please give it a test by installing the new test version (on a non-critical system! ;-) and running "sudo telinit u" to force Upstart to restart.

    Monday, 10 September 2012

    out: an output utility


    Whilst writing the tests for the Job Logging feature in Upstart, I discovered that I needed some very specific job behaviour that was not easy to produce using any simple existing utility I could find.

    The Problem

    What I wanted was a utility that would produce specified amounts of data to either standard output, standard error or direct to the terminal without using shell redirection. The reason for eschewing the shell being that Upstart is clever and intelligently determines whether a shell should be involved. From a user perspective, that's fantastic, but for testing purposes, I needed to force Upstart down particular code paths where it would not for example automatically pass the job through a shell. In fact the constraint was even more restrictive; what I really wanted to do was this:
    Produce output (including null bytes) to stdout, stderr or the terminal without using any shell meta characters.
    Unless I used some extremely esoteric shell, that effectively meant "no shells" ;-) ksh and zsh actually came close as their printf/print shell built-ins allow output to a specified file descriptor, but not to the terminal. Anyway, by this stage, I'd decided I wanted a generic utility, not some specific shell feature.

    The Pragmatic Solution

    For the Upstart tests, I ended up using various different command-line utilities for different tests that could produce the data I wanted. However, I couldn't help thinking there should be a better way. Yes, I could have written a bespoke C program or Python/Perl script to achieve my goal, but in the spirit of Larry Walls view that "Easy things should be easy", I thought that there should be a simpler method.

    The Improved Solution

    In the end this niggling problem got the better of me so I decided to write a tool to satisfy the above constraint. What I came up with is a small utility which sits somewhere between echo(1) and printf(1) (with just a dash of seq(1) thrown in) which, since it produces output, I've called simply "out".

    Here's an example of how you could print "hello" direct to the terminal (the redirection here is just to show that you still get output even though we've discarded stdout and stderr output):
    $ out -t hello >/dev/null 2>&1
    This command below will echo "hello" to stdout, the terminal and stderr:
    $ out hello -t -r 1 -e -r 1
    Along with allowing output redirection to any file descriptor or the terminal without the support of shell redirection, it supports the standard C-style escape sequences (like '\n' and '\a'). It also allows you to redefine the escape character.

    Additionally, out supports printf(1)-style Unicode escape sequences such as '\uFFFFFFFF' . After adding the Unicode support, I thought it would be rather fun to add ranges (similar to bash "sequence expressions") so out also supports some new sequences that allow sets of characters to be generated. These are great fun for exploring Unicode. Here's an example of generating the alphabet in lower-case:
    $ out '\{a..z}\n'
    How about some Greek?
    $ out "\{α..ω}\n"
    Or we could specify this using Unicode characters to get the same result:
    $ out "\{\u03b1..\u03c9}\n"
    And here's how you could generate the Unicode braille block:
    $ out '\{\u2800..\u28FF}\n'
    out also allows arbitrary delays to be introduced, repeats, basic random character generation and a few other tidbits. Here is the man page including a lot of examples:

    OUT(1)         OUT(1)
           out - manual page for out
           out [OPTION]... [STRING]...
           Echo strings to specified output stream(s).
           -a, --intra-char=<char>
           Insert  specified  character  (which may be a 1-character escape
           character) between all output characters.
           -b, --intra-pause=<delay>
           Pause between writing each character.
           -e, --stderr
           Write subsequent strings to standard error (file descriptor 2).
           -h, --help
           This help text.
           -i, --interpret
           Interpret escape characters (default).
           -l, --literal
           Write literal strings only (disable escape characters).
           -o, --stdout
           Write subsequent strings to standard output (file descriptor 1).
           -p, --prefix=<prefix>
           Use <prefix> as escape prefix (default='\').
           -r, --repeat=<repeat>
           Repeat previous value <repeat> times.
           -s, --sleep=<delay>
           Sleep for <delay> amount of time.
           -t, --terminal
           Write subsequent strings directly to terminal.
           -u, --file-descriptor=<fd> Write to specified file descriptor.
           -x, --exit=<num>
           Exit with value <num>.
           out recognises C-style escape sequences as  used  by  printf(1) .   By
           default an  escape  sequence  is introduced by the backslash character
           ('\'), however this may be changed with the -p option.  out  also  sup‐
           ports some additional sequences:
           \0     - nul byte (hex value 0x00)
           \a     - alert (bell)
           \b     - backspace
           \c     - no further output
           \e     - escape character (used for changing terminal attributes)
           \f     - form feed
           \g     - generate pseudo-random printable character
           \n     - newline
           \oNNN  - byte with octal value NNN (1 to 3 digits)
           \r     - carriage return
           \t     - horizontal tab
           \uNNNN -  2  byte Unicode (ISO/IEC 10646) character with hex value NNNN
           (4 digits)
           - 4 byte Unicode  (ISO/IEC  10646)  character  with  hex  value
           NNNNNNNN (8 digits)
           \v     - vertical tab
           \xNN   - byte with hexadecimal value NN (1 to 2 digits)
           out also supports range escapes which allow a range of characters to be
           specified in a compact format.
           - specify a range by two 1-byte literal characters.
           - specify a range by two 3-byte octal values.
           - specify a range by two 2-byte Unicode values.
           - specify a range by two 4-byte Unicode values.
           - specify a range by two 2-byte hex values.
           Note that ranges take two values of the same type and the maximum width
           for that type must be specified.
           ·   Arguments are processed in order.
           ·   With the exception of '-x', arguments may be repeated any number of
           ·   All output will be sent to standard output until  an  output  redi‐
        rection  option is specified that changes the output stream (namely
        -e or -t (or their  long-option  equivalents),  or  if  output  has
        already been redirected -o (or its long-option equivalent)).
           ·   If  <str>  is  the empty string ("" or '') it will be treated as \0
        such that a nul byte will be displayed.
           ·   To cancel the effect of -a, specify a null string: -a ''.
           ·   If <repeat> is '-1', repeat forever.
           ·   Replace the 'Z' in the range formats  above with  the  appropriate
           ·   Ranges can be either ascending or descending.
           ·   <delay>  can  take  the  following  forms where <num> is a positive
        <num>ns : nano-seconds (1/1,000,000,000 second)
        <num>us : micro-seconds (1/1,000,000 second)
        <num>ms : milli-seconds (1/1,000 second)
        <num>cs : centi-seconds (1/100 second)
        <num>ds : deci-seconds (1/10 second)
        <num>s  : seconds
        <num>m  : minutes
        <num>h  : hours
        <num>h  : days
        <num>   : seconds
        If <num> is -1, wait until any signal is received.
        If signal is SIGNUM continue, else exit immediately.
           ·   Generated printable random characters may not  display  unless  you
        are using an appropriate font.
     # Print "foofoofoo" to stderr, followed by "barbar" to stdout.
     out "foo" -r 2 -o "bar" -r 1
     # Write 50 nul bytes direct to the terminal.
     out -t "" -r 49
     # Write continuous stream of nul bytes direct to the terminal,
     # 1 per second.
     out -b 1s -t '' -r -1
     # Display a greeting slowly (as a human might type)
     out -b 20cs "Hello, $USER.\n"
     # Display a "spinner" that loops 4 times.
     out -b 20cs -p % "%r|%r/%r-%r\%r" -r 3
     # Display all digits between zero and nine with a trailing
     # newline.
     out "\{0..9}\n"
     # Display slowly the lower-case letters of the alphabet,
     # backwards without a newline.
     out -b 1ds "\{z..a}"
     # Display upper-case 'ABC' with newline.
     out '\u0041\u0042\u0043\n'
     # Display 'foo' with newline.
     out '\o146\u006f\x6F\n'
     # Clear the screen.
     out '\n' -r $LINES
     # Write hello to stdout, stderr and the terminal.
     out 'hello' -t -r 1 -e -r 1
     # Display upper-case letters of the alphabet using octal
     # notation, plus a newline.
     out "\{\o101..\o132}"
     # Display 'h.e.l.l.o.' followed by a newline.
     out -a . "hello" -a '' "\n"
     # Display upper-case and lower-case letters of the alphabet
     # including the characters in-between, with a trailing newline.
     out "\{A..z}\n"
     # Display lower-case alphabet followed by reversed lower-case alphabet
     # with the digits zero to nine, then nine to zero on the next line.
     out "\{a..z}\{z..a}\n\{0..9}\{9..0}\n"
     # Display lower-case Greek letters of the alphabet.
     out "\{α..ω}"
     # Display cyrillic characters.
     out "\{Ѐ..ӿ}"
     # Display all printable ASCII characters using hex range:
     out "\{\x21..\x7e}"
     # Display all printable ASCII characters using 2-byte UTF-8 range:
     out "\{\u0021..\u007e}"
     # Display all printable ASCII characters using 4-byte UTF-8 range:
     out "\{\U00000021..\U0000007e}"
     # Display all braille characters.
     out "\{\u2800..\u28FF}"
     # Display 'WARNING' in white on red background.
     out '\e[37;41mWARNING\e[0m\n'
     # Generate 10 random characters.
     out '\g' -r 9
           Written by James Hunt <james.hunt@ubuntu.com>
           Copyright © 2012 James Hunt <james.hunt@ubuntu.com>
           GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
           This  is  free  software:  you  are free to change and redistribute it.
           There is NO WARRANTY, to the extent permitted by law.
           echo(1) printf(1)
    User Commands     2012-09-10    OUT(1)


    The TODO file in the distribution contains a number of ideas to enhance outs abilities.


    See Also

    Tuesday, 4 September 2012

    Call for Testing: Upstart stateful re-exec

    If you'd like to help out testing the new "stateful re-exec" feature of Upstart which we hope to land for Ubuntu Quantal, see the details here:

    In brief (more posts will follow), "stateful re-exec" allows Upstart to be restarted (or upgraded!) without rebooting your system. A basic restart or re-exec isn't difficult, but what stateful re-exec does is maintain Upstarts internal state over the re-exec such that it continues to track jobs and log job output.

    This is an early cut of the code so don't install it on any critical systems. Additionally, there are some caveats...

    ... but we plan for that list to dwindle quickly over the coming days and weeks.

    Tuesday, 24 July 2012

    simplified strace diffing

    strace is an extremely powerful tool. But have you ever attempted to compare strace log files? That can get tricky. How about diffing multiple strace logs of multi-process applications? That can be a world of pain. So, I wrote a simple shell script to make a life easier. Then I decided to rewrite it in Python and life got even better :-)

    My reason for diffing multi-process strace logs was not to see where an application was failing (that's what debuggers are for) but more to understand the flow of program execution. So what follows may have a fairly niche audience.

    The script is pretty simple: it's just simplifies the log files to allow easier diffing. Specifically it:
    • replaces all addresses with 0xADDR (or 0xNULL).
    • replaces all timestamps with HH:MM:SS.
    • replaces all datestamps with YYYY/MM/DD.
    • tracks PIDs and replaces each PID seen with a 1-based value (so the first PID seen will be assigned PID1, the second PID2 et cetera).
    These simple changes turn out to be pretty powerful. For example...

    $ strace -o /tmp/st1.log -fFv -s 1024 script -c 'whoami' /dev/null
    $ strace -o /tmp/st2.log -fFv -s 1024 script -c 'whoami' /dev/null
    $ diff /tmp/st?.log | wc -l

    1,718 lines of differences from running the same command twice. Ouch!

    As shown in the screenshot, meld cannot find any lines in common between the files (hence the blue background).

    But when we pre-process st1.log and st2.log through strace_summarise.py, the results are much improved:

    $ strace_summarise.py /tmp/st1.log > /tmp/st1.sum
    $ strace_summarise.py /tmp/st2.log > /tmp/st2.sum
    $ diff /tmp/st?.sum | wc -l
    Down to only 153 lines of difference - that's a lot more manageable as shown by the meld screenshot above.

    In actual fact, since meld is written in Python, you can configure it to ignore certain patterns (Edit -> Preferences -> Text Filters). However, I usually use the awesomely fast xxdiff which doesn't offer this feature (although unlike meld it actually highlights similarities in lines which are marked as non-matching).

    The script is here:

    Wednesday, 18 July 2012

    A Bazaar pre-commit hook to look for signs of unfinished work


    Programming involves a fair amount of mental juggling: jumping around a codebase working on a new feature or attempting to track down a bug in existing code, you have to keep track of a lot of pieces of information. However, more balls end up being thrown into the air when, passing through some file or function, you come across an area that needs a bit of lovin'. Often the niggles you spot are minor, but not minor enough to ignore (no Broken Windows here please!)

    The problem is that these minor imperfections -- which are probably entirely unrelated to the task at hand -- are both annoying and distracting. Imagine...
    • A needs fixing.
    • C, D, E and G need refactoring now that B exists.
    • Documentation for C, E and G need their function docs updating.
    • H introduces a new feature that needs to be documented in a man page.
    • I is inconsistently formatted.
    • J contains a couple of typos.
    • K lacks sufficient tests.
    • L is now giving warnings with the new stricter version of GCC you've just started using.

    Now... where was I?
    If you can't make the changes at the time (and unless they are extremely quick to make you probably shouldn't as it will derail your current activity), an obvious strategy is leave little "doc droppings" (aka FIXME's) around your code as reminders of things that need to be done "later". If this sounds like you, read on...

    Found some Fluff

    Recently, I discovered a file I'd changed a long time ago still had a "FIXME" in it. This irked me on a couple of levels. Firstly, that I hadn't spotted it before committing the change, but also that I shouldn't have to worry about checking for such things manually. And so, unto the world an unscratched itch was born.

    It's Not a Problem, it's an Opportunity

    Enter bzr which allows you to write plugins and hooks extremely easily in Python. I'd wanted an excuse to write a plugin for a while. And now I had discovered the need to be reminded of unfinished work before I commit it, hence the excuse to write the unimaginatively-named "find_fixmes_pre_commit_hook.py".

    Hook Overview

    The hook iterates through all modified files looking for a set of regular expressions I tend to use. The current list being:

    • FIXME
    • TODO
    • // 
    • #if 1
    • #if 0

    If you're not a C coder, the last two are preprocessor directives: the last regex being indicative of a commented out block of code (we all know these are bad to leave lying around right?), and the penultimate being indicative of a chunk of test or debug code that is enabled but should probably be removed. I also occasionally use C++-style comments for temporarily disabling a 1-liner.

    If the hook finds any matches, it will perform the following steps for each match:
    1. Display the line matching the regular expression in question.
    2. Display a number of lines of context.
    3. Interactively prompt the user to either proceed or abort the commit.


    The hook is currently extremely basic - it doesn't even have any form of packaging so to install it, all you need to do is copy the hook file to ~/.bazaar/plugins/ (create the directory if required). You could just sym link it to that directory if you prefer.


    Once you've installed the hook, whenever you run "bzr commit" in any branch, the hook will be called automatically.


    To disable the hook, just delete the file or move it out of that directory.


    • If you "bzr commit -F mychanges.txt", you'll still be prompted if any of the regex's are found.
    • If "bzr commit" is run with no associated terminal (for example if you redirect stdin to /dev/null), even if the hook finds issues, the changes will still be committed.


    • Add ability to show "post-context" (where available) to give the user a better indication of where the code issue is located.
    • Add in ability to toggle the hook on/off depending on the branch being committed to.
    • Add ability to restrict the hook to only operating on particular types of files (it currently attempts to ignore checking binary files by considering the mime-type).
    • Add command-line option to "bzr commit" to force the commit but still display the warnings.

    The Hook in Operation

    Lets create a branch of the "hello" application (yes, it really does exist!), make a change to it that introduces one of my chosen regular expressions ("FIXME") and then attempt to commit the change.

    $ cd /tmp
    $ bzr branch lp:ubuntu/hello
    $ cd hello
    $ echo "/* FIXME: still have stuff to do */" >> src/hello.c
    $ bzr commit -m "a change"
    Committing to: /tmp/hello/
    modified src/hello.c
    | Running pre_commit hooks - Stage 3/5
    ERROR: Found unexpected pattern '\bFIXME\b' at line 214 in file 'src/hello.c'
    (branch: name '', path '/tmp/hello', url 'file:///tmp/hello/'):
    209:License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>\n\
    210:This is free software: you are free to change and redistribute it.\n\
    211:There is NO WARRANTY, to the extent permitted by law.\n"),
    212:              "2011");
    214:/* FIXME: still have stuff to do */
    Abort commit to allow problem to be resolved? ([y]es, [n]o): yes
    Commit aborted.

    Show Me the Code

    The plugin is here: find_fixmes_pre_commit_hook.py

    To Infinity and Beyond

    Hooks are cool and not difficult to implement. Consider a hook that stopped you committing code unless:

    • a build is successful.
    • a test suite passes successfully.
    • a security / license auditing tool runs cleanly over the code.
    • a static analysis run is successful (or code is atleast within some predefined threshold).
    • the new code conforms to a suitable coding standard.
    (Those last 2 ideas might seem a little harsh but there are companies which mandate this (I've worked for some ;-))


    See also:

    Friday, 1 June 2012

    pull-debian-source with apt-get or "how to download a Debian package from Ubuntu the hard but fun way"

    Have you ever wanted to download a Debian package from an Ubuntu system? Well you are not alone: hidden away in the wonderful "ubuntu-dev-tools" package is the tool for you - the excellent pull-debian-source, written by tumbleweed. This is a really useful tool which is also fast!

    $ sudo apt-get install -y ubuntu-dev-tools
    $ pull-debian-source hello
    pull-debian-source: Downloading hello version 2.8-1
    pull-debian-source: Downloading hello_2.8.orig.tar.gz from ftp.debian.org (0.665 MiB)
    pull-debian-source: Downloading hello_2.8-1.debian.tar.gz from ftp.debian.org (0.006 MiB)
    dpkg-source: info: extracting hello in hello-2.8                   
    dpkg-source: info: unpacking hello_2.8.orig.tar.gz
    dpkg-source: info: unpacking hello_2.8-1.debian.tar.gz
    Or, to download the at package from the sid release:
    $ pull-debian-source at sid
    pull-debian-source: Downloading at version 3.1.13-1
    pull-debian-source: Downloading at_3.1.13.orig.tar.gz from ftp.debian.org (0.117 MiB)
    pull-debian-source: Downloading at_3.1.13-1.diff.gz from ftp.debian.org (0.009 MiB)
    dpkg-source: info: extracting at in at-3.1.13                      
    dpkg-source: info: unpacking at_3.1.13.orig.tar.gz
    dpkg-source: info: applying at_3.1.13-1.diff.gz
    However, I had cause to pore over the apt.conf(5) man page recently, and wondered if it would be possible to do the same thing, but using only apt-get(8). Turns out it is...

    The first thing to do is create a few directories:
    $ dir=~/.cache/pull-debian-source.sh
    $ mkdir -p $dir
    $ for d in etc/trusted.gpg.d etc/preferences.d var log; do mkdir -p $dir/$d; done
    Now, we need a sources.list file so apt-get knows where to download index and package files from. I'm creating one for Debian sid:
    $ export conf=$dir/etc/sources.list
    $ cat >>$conf<<EOT
    deb http://ftp.uk.debian.org/debian/ sid main
    deb-src http://ftp.uk.debian.org/debian/ sid main
    deb http://ftp.uk.debian.org/debian/ experimental main
    deb-src http://ftp.uk.debian.org/debian/ experimental main
    Next, we need to grab a copy of the package cache containing the index files. This is where the magic comes in as we use some of the amazing array of apt options:
    $ export options="\
      -o Dir=\"$dir\" \
      -o Dir::Etc=\"$dir/etc\" \
      -o Dir::State=\"$dir/var\" \
      -o Dir::Cache=\"$dir/var\" \
      -o Dir::Log=\"$dir/log\" \
      -o APT::Get::AllowUnauthenticated=yes \
      -o Debug::NoLocking=true"
    As you can see, we've specified alternative locations for the directories apt cares about. We've also turned off locking and authentication. The locking option isn't a problem unless you happen to be running multiple copies of apt-get as yourself. The authentication option needs might be a problem for you - check the man page for details. Now we've set the options, lets create the cache:
    $ eval "apt-get $options update"
    Get:1 http://ftp.uk.debian.org sid InRelease [234 kB]
    Get:2 http://ftp.uk.debian.org experimental InRelease [162 kB]
    Ign http://ftp.uk.debian.org sid InRelease                    
    Ign http://ftp.uk.debian.org experimental InRelease           
    Get:3 http://ftp.uk.debian.org sid/main Sources/DiffIndex [7876 B]
    Get:4 http://ftp.uk.debian.org sid/main i386 Packages/DiffIndex [7876 B]
    Hit http://ftp.uk.debian.org sid/main TranslationIndex
    Get:5 http://ftp.uk.debian.org experimental/main Sources/DiffIndex [7819 B]
    Get:6 http://ftp.uk.debian.org experimental/main i386 Packages/DiffIndex [7819 B]
    Hit http://ftp.uk.debian.org experimental/main TranslationIndex
    Hit http://ftp.uk.debian.org sid/main Translation-en
    Hit http://ftp.uk.debian.org experimental/main Translation-en
    Fetched 407 kB in 2s (194 kB/s)
    Reading package lists... Done
    W: GPG error: http://ftp.uk.debian.org sid InRelease: No keyring installed in /home/james/.cache/pull-debian-source.sh/etc/trusted.gpg.d/.
    W: GPG error: http://ftp.uk.debian.org experimental InRelease: No keyring installed in /home/james/.cache/pull-debian-source.sh/etc/trusted.gpg.d/.
    We got a couple of warnings ("W:" entries above), but they are not fatal. Now we've got a local cache of package meta-data, let's download a package from Debian:
    $ eval "apt-get $options source cron"
    Reading package lists... Done
    Building dependency tree... Done
    NOTICE: 'cron' packaging is maintained in the 'Git' version control system at:
    Need to get 158 kB of source archives.
    Get:1 http://ftp.uk.debian.org/debian/ sid/main cron 3.0pl1-121 (dsc) [1267 B]
    Get:2 http://ftp.uk.debian.org/debian/ sid/main cron 3.0pl1-121 (tar) [59.2 kB]
    Get:3 http://ftp.uk.debian.org/debian/ sid/main cron 3.0pl1-121 (diff) [97.2 kB]
    Fetched 158 kB in 0s (206 kB/s)
    dpkg-source: info: extracting cron in cron-3.0pl1
    dpkg-source: info: unpacking cron_3.0pl1.orig.tar.gz
    dpkg-source: info: applying cron_3.0pl1-121.diff.gz
    dpkg-source: info: upstream files that have been modified: 
    By the way, if you want to see all the options available to apt, run the following:
    $ apt-config dump
    APT "";
    APT::Architecture "i386";
    APT::Build-Essential "";
    APT::Build-Essential:: "build-essential";
    APT::Install-Recommends "1";
    APT::Install-Suggests "0";
    APT::Authentication "";
    APT::Authentication::TrustCDROM "true";
    APT::NeverAutoRemove "";
    APT::NeverAutoRemove:: "^firmware-linux.*";
    APT::NeverAutoRemove:: "^linux-firmware$";
    APT::NeverAutoRemove:: "^linux-image.*";
    APT::NeverAutoRemove:: "^kfreebsd-image.*";
    APT::NeverAutoRemove:: "^linux-restricted-modules.*";
    APT::NeverAutoRemove:: "^linux-ubuntu-modules-.*";
    APT::NeverAutoRemove:: "^gnumach$";
    APT::NeverAutoRemove:: "^gnumach-image.*";
    APT::Never-MarkAuto-Sections "";
    APT::Never-MarkAuto-Sections:: "metapackages";
    APT::Never-MarkAuto-Sections:: "restricted/metapackages";
    APT::Never-MarkAuto-Sections:: "universe/metapackages";
    APT::Never-MarkAuto-Sections:: "multiverse/metapackages";
    APT::Never-MarkAuto-Sections:: "oldlibs";
    APT::Never-MarkAuto-Sections:: "restricted/oldlibs";
    APT::Never-MarkAuto-Sections:: "universe/oldlibs";
    APT::Never-MarkAuto-Sections:: "multiverse/oldlibs";
    APT::Periodic "";
    APT::Periodic::Update-Package-Lists "1";
    APT::Periodic::Download-Upgradeable-Packages "0";
    APT::Periodic::AutocleanInterval "0";
    APT::Update "";
    APT::Update::Post-Invoke-Success "";
    APT::Update::Post-Invoke-Success:: "touch /var/lib/apt/periodic/update-success-stamp 2>/dev/null || true";
    APT::Update::Post-Invoke-Success:: "[ ! -f /var/run/dbus/system_bus_socket ] || /usr/bin/dbus-send --system --dest=org.debian.apt --type=signal /org/debian/apt org.debian.apt.CacheChanged || true";
    APT::Archives "";
    APT::Archives::MaxAge "30";
    APT::Archives::MinAge "2";
    APT::Archives::MaxSize "500";
    APT::Changelogs "";
    APT::Changelogs::Server "http://changelogs.ubuntu.com/changelogs";
    APT::Architectures "";
    APT::Architectures:: "i386";
    APT::Compressor "";
    APT::Compressor::. "";
    APT::Compressor::.::Name ".";
    APT::Compressor::.::Extension "";
    APT::Compressor::.::Binary "";
    APT::Compressor::.::Cost "1";
    APT::Compressor::.::CompressArg "";
    APT::Compressor::.::CompressArg:: "";
    APT::Compressor::.::UncompressArg "";
    APT::Compressor::.::UncompressArg:: "";
    APT::Compressor::gzip "";
    APT::Compressor::gzip::Name "gzip";
    APT::Compressor::gzip::Extension ".gz";
    APT::Compressor::gzip::Binary "gzip";
    APT::Compressor::gzip::Cost "2";
    APT::Compressor::gzip::CompressArg "";
    APT::Compressor::gzip::CompressArg:: "-9n";
    APT::Compressor::gzip::UncompressArg "";
    APT::Compressor::gzip::UncompressArg:: "-d";
    APT::Compressor::bzip2 "";
    APT::Compressor::bzip2::Name "bzip2";
    APT::Compressor::bzip2::Extension ".bz2";
    APT::Compressor::bzip2::Binary "bzip2";
    APT::Compressor::bzip2::Cost "3";
    APT::Compressor::bzip2::CompressArg "";
    APT::Compressor::bzip2::CompressArg:: "-9";
    APT::Compressor::bzip2::UncompressArg "";
    APT::Compressor::bzip2::UncompressArg:: "-d";
    APT::Compressor::lzma "";
    APT::Compressor::lzma::Name "lzma";
    APT::Compressor::lzma::Extension ".lzma";
    APT::Compressor::lzma::Binary "lzma";
    APT::Compressor::lzma::Cost "4";
    APT::Compressor::lzma::CompressArg "";
    APT::Compressor::lzma::CompressArg:: "-9";
    APT::Compressor::lzma::UncompressArg "";
    APT::Compressor::lzma::UncompressArg:: "-d";
    APT::Compressor::xz "";
    APT::Compressor::xz::Name "xz";
    APT::Compressor::xz::Extension ".xz";
    APT::Compressor::xz::Binary "xz";
    APT::Compressor::xz::Cost "5";
    APT::Compressor::xz::CompressArg "";
    APT::Compressor::xz::CompressArg:: "-6";
    APT::Compressor::xz::UncompressArg "";
    APT::Compressor::xz::UncompressArg:: "-d";
    Dir "/";
    Dir::State "var/lib/apt/";
    Dir::State::lists "lists/";
    Dir::State::cdroms "cdroms.list";
    Dir::State::mirrors "mirrors/";
    Dir::State::extended_states "extended_states";
    Dir::State::status "/var/lib/dpkg/status";
    Dir::Cache "var/cache/apt/";
    Dir::Cache::archives "archives/";
    Dir::Cache::srcpkgcache "srcpkgcache.bin";
    Dir::Cache::pkgcache "pkgcache.bin";
    Dir::Etc "etc/apt/";
    Dir::Etc::sourcelist "sources.list";
    Dir::Etc::sourceparts "sources.list.d";
    Dir::Etc::vendorlist "vendors.list";
    Dir::Etc::vendorparts "vendors.list.d";
    Dir::Etc::main "apt.conf";
    Dir::Etc::netrc "auth.conf";
    Dir::Etc::parts "apt.conf.d";
    Dir::Etc::preferences "preferences";
    Dir::Etc::preferencesparts "preferences.d";
    Dir::Etc::trusted "trusted.gpg";
    Dir::Etc::trustedparts "trusted.gpg.d";
    Dir::Bin "";
    Dir::Bin::methods "/usr/lib/apt/methods";
    Dir::Bin::solvers "";
    Dir::Bin::solvers:: "/usr/lib/apt/solvers";
    Dir::Bin::dpkg "/usr/bin/dpkg";
    Dir::Bin::lzma "/usr/bin/lzma";
    Dir::Bin::xz "/usr/bin/xz";
    Dir::Bin::bzip2 "/bin/bzip2";
    Dir::Media "";
    Dir::Media::MountPath "/media/apt";
    Dir::Log "var/log/apt";
    Dir::Log::Terminal "term.log";
    Dir::Log::History "history.log";
    Dir::Ignore-Files-Silently "";
    Dir::Ignore-Files-Silently:: "~$";
    Dir::Ignore-Files-Silently:: "\.disabled$";
    Dir::Ignore-Files-Silently:: "\.bak$";
    Dir::Ignore-Files-Silently:: "\.dpkg-[a-z]+$";
    Dir::Ignore-Files-Silently:: "\.save$";
    Dir::Ignore-Files-Silently:: "\.orig$";
    Dir::Ignore-Files-Silently:: "\.distUpgrade$";
    Acquire "";
    Acquire::cdrom "";
    Acquire::cdrom::mount "/media/cdrom/";
    Acquire::Languages "";
    Acquire::Languages:: "en";
    Acquire::Languages:: "none";
    Acquire::Languages:: "fr";
    DPkg "";
    DPkg::Pre-Install-Pkgs "";
    DPkg::Pre-Install-Pkgs:: "/usr/bin/apt-listchanges --apt || test $? -ne 10";
    DPkg::Pre-Install-Pkgs:: "/usr/sbin/dpkg-preconfigure --apt || true";
    DPkg::Tools "";
    DPkg::Tools::Options "";
    DPkg::Tools::Options::/usr/bin/apt-listchanges "";
    DPkg::Tools::Options::/usr/bin/apt-listchanges::Version "2";
    DPkg::Post-Invoke "";
    DPkg::Post-Invoke:: "if [ -d /var/lib/update-notifier ]; then touch /var/lib/update-notifier/dpkg-run-stamp; fi; if [ -e /var/lib/update-notifier/updates-available ]; then echo > /var/lib/update-notifier/updates-available; fi ";
    Unattended-Upgrade "";
    Unattended-Upgrade::Allowed-Origins "";
    Unattended-Upgrade::Allowed-Origins:: "${distro_id}:${distro_codename}-security";
    CommandLine "";
    CommandLine::AsString "apt-config dump";

    Admittedly, this technique is rather slower than pull-debian-source (atleast for the initial download of the index files), but it is an instructive example of what is possible when you RTFM ;-)

    I've bundled the above into a pull-debian-source.sh script. Run it like this:
    $ chmod 755 ./pull-debian-source.sh
    $ ./pull-debian-source.sh update
    $ ./pull-debian-source.sh source hello

    Friday, 25 May 2012

    Job Logging in Upstart

    Job Logging in Upstart

    The big Upstart feature in Ubuntu Precise is "job logging" (in fact, it turned out to be a significantly bigger feature than we'd originally envisaged :-). This had been a wishlist item for some time and a lot of folk were very keen to see this implemented. All system jobs now have their stdout and stderr logged automatically by default to a text file in directory /var/log/upstart/.

    Why did we make this the default? Surely daemons and services don't generally write any output? True, but when they do produce output, it is worth capturing since it has a very high chance of being an error message. And errors should not be ignored. For jobs that do not produce any output, there is minimal overhead and of course no log is written.

    The logger actually uses pseudo-ptys just like script(1), xterm(1), expect(1)screen(1) et al. This is advantageous for a number of reasons, but from the logging perspective the biggie is "no buffering" so Upstart is able to slurp the jobs data as soon as it is generated. So even if a job produces only a single byte of output, this will be detected immediately.

    Whilst developing this feature, we found a few services that had been generating warning and error messages for a looooong time but since the output was been discarded (implicit "console none"), we were not aware of them.

    Job Debugging

    The other great thing about job logging is that you don't need any special setup to use this feature as it's already enabled. All you need to do is add some calls to echo(1) and you are away.

    To help with debugging script sections, you might like to add a "set -x" immediately after the script stanza. This is particularly helpful if you have tests in your code as you can see clearly which one is failing (recall that Upstart runs the script sections using "/bin/sh -e" so any errors are fatal.

    $ cat <<EOT | sudo tee /etc/init/foo.conf
    > task
    > script
    >   set -x
    >   echo I am a job running with the following environment:
    >   env
    >   if [ 1 -gt 2 ]
    >   then
    >     echo impossible
    >   else
    >     echo sane
    >   fi
    > end script
    > EOT
    $ init-checkconf /etc/init/foo.conf && sudo start foo && sudo cat /var/log/upstart/foo.log
    + echo I am a job running with the following environment:
    I am a job running with the following environment:
    + env
    + [ 1 -gt 2 ]
    + echo sane

    Early Logging

    Another neat feature of job logging is that it works from Upstart startup. What this means is that you can have a job that specifies "start on startup" (where startup is the first event Upstart emits at the point the disks are still read-only) and still have its output logged. Upstart handles this situation by writing the log as soon as the disk becomes writeable. This makes debugging very early system startup issues significantly easier. If you want the data "immediately" you could always use the  "--logdir" command-line option and arrange to have Upstart write job logs to a tmpfs filesystem.

    Controlling Logging

    Logging can be disabled entirely using the "--no-log" command-line option which gives you the old Upstart behaviour where all jobs implicitly defaulted to "console none".

    You can of course disable logging for individual jobs by specifying an explicit "console none".

    But you can also invert the logic and disable logging for all jobs except those you choose. To do this, add "console log" to all jobs you wish Upstart to log output for and then boot adding the following command-line option:  "--default-console=none". Now, Upstart will honour any job that specifies "console" explicitly, but if not, it will default to "console none", and discard all job output.

    Further Reading

    As usual, for further details, consult init(5) and the Upstart Cookbook, specifically:

    If you haven't yet upgraded to precise, check out the release notes:

    Why not take a look in /var/log/upstart/ now and see what you can find?


    Thursday, 17 May 2012

    A quick libnih tutorial


    The NIH Utility Library (libnih) is a small, efficient and most importantly safe library of general purpose routines. It was written by Keybuk so you can be assured that it is extremely elegant, well-designed, well-tested (includes 2863 tests currently!) and well-written. NIH is used by Upstart, the event-based init daemon which is used by:

    That's a lot of deployments of Upstart and NIH around the world!! (And we're not even including mobile device operating systems in that list).

    But why not just use glib I hear you ask? Well, glib is a very large library whereas NIH is small and designed for low-level daemons and systems which may be resource-constrained. Also, lets not forget that NIH, like Upstart, comes with a very comprehensive test suite so bugs are rare.

    Other reasons to use NIH:
    • It handles garbage collection for you

      That's right, you don't need to free memory manually.

    • It uses an Object-Oriented-like Design... in C!

      This is extremely powerful and elegant. It's also quite easy to use once you understand the way the API works.

    Let's start with some basics...

    Garbage Collection

    /* WARNING! Contains bugs! */
    main (int argc, char *argv[])
        nih_local char *string;
        if (argc > 1) {
            string = nih_strdup (NULL, "hello, world");
            nih_message ("string is set to '%s'", string);

    This code nominally is trying to display a message if the user runs this application with one or more command-line arguments specified. However, there are a couple of problems with it:

    • No check is performed on the memory allocated by nih_strdup().
    • If no command-line argument is specified, chances are this program will crash.
    The first issue is easy to spot and easy to remedy, but what about this crash? Well, nih_local variables are garbage collected automatically when they go out of scope. The string variable will therefore be garbage collected when it goes out of scope, which is when main() exits. However, since string was never initialized, it will be pointing to a random location in memory such that when the program exits, the runtime will attempt to free that random memory address. That will probably result in a SIGSEGV caused by dereferencing an illegal pointer value. The fix is easy and you should chant this mantra whenever you use nih_local variables:

    Always assign nih_local variables to NULL.

    Here's a corrected version:

    /* Correct version */
    main (int argc, char *argv[])
        /* XXX: *ALWAYS* set nih_local variables to NULL */ 
        nih_local char *string = NULL;
        if ( argc > 1) {
            string = nih_strdup (NULL, "hello, world");
            if (string) 
                nih_message ("string is set to '%s'", string);
            else {
                nih_error ("failed to allocate space for string");
                exit (EXIT_FAILURE);

    However, there is an even better way to code that check to ensure nih_strdup() succeeded:

    /* Improved version */ 
    main (int argc, char *argv[])
        /* XXX: *ALWAYS* set nih_local variables to NULL */ 
        nih_local char *string = NULL;
        if ( argc > 1) {
            string = NIH_MUST (nih_strdup (NULL, "hello, world"));
            nih_message ("string is set to '%s'", string);

    So now, if the user specifies a command-line option, the program will print "hello, world" and automatically free the variable string. If the user does not specify a command-line option, no garbage collection will be performed since the string variable will never be associated with allocated memory.

    Note that the code is simpler and easier to understand as a result. Note too that we're now using NIH_MUST(). This is a macro which will call the block you pass to it ('nih_strdup (NULL, "hello, world")' in this case) repeatedly until it succeeds. You should exercise caution using NIH_MUST()though since if there is a high likelihood of the allocation never succeeding, the code will spin forever at this point. There is similar call "NIH_SHOULD()" that will call the block passed to it repeatedly until either the result is TRUE, or an error other than ENOMEM is raised.


    Let's take a closer look at that call to nih_strdup. The system version of strdup takes a single argument (the string to copy), so why does nih_strdup take two arguments?

    nih_strdup (NULL, "hello, world");

    Well that first NULL parameter is the parent pointer. Most NIH functions take a parent pointer as their first argument. Lets see these pointers in action before explaining the detail...

    #include <nih/macros.h>
    #include <nih/logging.h>
    #include <nih/string.h>
    #include <nih/alloc.h>
    main(int argc, char *argv[])
        typedef struct foo {
            char *str1;
            char *str2;
        } Foo;
        nih_local Foo *foo = NIH_MUST (nih_new (NULL, Foo));
        foo->str1 = NIH_MUST (nih_strdup (foo, "first string"));
        foo->str2 = NIH_MUST (nih_strdup (foo, "second string"));
        nih_message ("foo->str1='%s'", foo->str1); 
        nih_message ("foo->str2='%s'", foo->str2); 
    Here we see our first complete NIH program. There are a couple of important points to note:

    • The call to nih_new() is like malloc()except it too takes a parent pointer. Since the foo object we're creating doesn't have a parent, we set the pointer to NULL.
    • Note that there is no call to free the memory allocated by nih_new()because since we're using nih_local, the object and all its children will be freed automatically when the block (in this example the main() function) ends. This is incredibly powerful: we've made 3 memory allocations in the example (one call to nih_new() and two calls to nih_strdup()), and all that memory will be automatically garbage collected for us because NIH knows to free the foo object when it goes out of scope, but it also knows that the str1 and str2 elements also need to be freed (since we told nih_strdup() their parent is the foo object we previously created).
    So the parent pointer provided by most NIH calls is used to enable intelligent garbage collection: by effectively tagging objects with a reference to their parent you are assured of that object being automatically garbage collected when the parent is freed.


    The NIH list implementation is essentially the same as a "struct list_head" in the Linux kernel:

    typedef struct nih_list {
        struct nih_list *prev, *next;  
    } NihList;

    Lists are designed to be contained within other objects like this:
    typedef struct bar {
        NihList   entry;
        char     *str;
    } Bar;

    So you don't create a "list of Bar objects", you create a list of list objects which provide access to their containing types.

    Note that the list element is the first in the Bar structure. This allows a list pointer to be dereferenced to its containing type trivially.

    Let's look at an example of list usage by implementing echo(1):

    #include <nih/macros.h>
    #include <nih/logging.h>
    #include <nih/string.h>
    #include <nih/alloc.h>
    typedef struct bar {
        NihList  entry;
        char    *str;
    } Bar;
    main(int argc, char *argv[])
        int i;
        NihList *args;
        args = NIH_MUST (nih_list_new (NULL));
        /* store all arguments in a list */
        for (i = 1; i < argc; ++i) {
            Bar *bar = NIH_MUST (nih_new (args, Bar));
            nih_list_init (&bar->entry);
            bar->str = NIH_MUST (nih_strdup (bar, argv[i]));
            nih_list_add (args, &bar->entry);
        i = 1;
        /* display all arguments by iterating over list */
        NIH_LIST_FOREACH (args, iter) {
            Bar *bar = (Bar *)iter;
            nih_message ("argument %d='%s'", i, bar->str);
        nih_free (args);
        return (0);

    The new features introduced here are the calls to nih_list_init() to initialise a list, and nih_list_add(), which adds the second argument to the list specified by the first argument. Additionally, we have that rather funky NIH_LIST_FOREACH() macro which allows for easy (and fast!) list traversal. In this example we are not using nih_local so what happens when nih_free() is called? Well, all entries in the args list are freed, but before each is freed, the str string within each entry is freed. Then the list itself is freed. Neat huh?

    To build our version of echo:
    gcc -std=gnu99 -Wall -pedantic echo.c -o echo $(pkg-config --cflags --libs libnih)
    Now let's run it:
    $ ./echo a b c "hello world" "foo bar" wibble "the end"
    argument 1='a'
    argument 2='b'
    argument 3='c'
    argument 4='hello world'
    argument 5='foo bar'
    argument 6='wibble'
    argument 7='the end'

    We've really only scratched the surface of NIHs abilities in this post. Here are some of the other facilities it provides:

    • hashes
    • binary trees
    • string arrays
    • file watches
    • I/O handling
    • signal handling
    • timers
    • reference handling
    • error/exception handling
    • main loop handling
    • command-line option and usage handling
    • child process handling
    • config file handling
    • logging facilities
    • test facilities
    If you're interested to learn more, start hacking, or take a look at some of the projects already using NIH: