Posts

Showing posts from 2012

Upstart with "stateful re-exec" now in Ubuntu Raring

After a lot of testing, we have today landed Upstart 1.6.1 in Ubuntu Raring with " stateful re-exec " goodness. What this means is that for all future Upstart upgrades, no reboot is required to make use of the newly-installed version: Upstart will restart itself with no interruption to service such that existing jobs will continue to run. Additionally, it means that an upgrade to any of Upstarts dependent libraries (such as libc and libnih ) can also trigger Upstart to restart itself so that it will automatically start using those new library versions whilst maintaining continuity of service. For those of us who dislike downtime and want to remain current, this is known as a "win" ;-)

procenv now in Debian and Ubuntu: how you can use it

The procenv utility covered in a previous post is now at version 0.15 and also available in Debian Sid and Ubuntu Raring . So, you can do the usual " sudo apt-get install procenv " followed by simply " procenv " to run it. However, due to the way this utility is built, there is a little more you can do. Read on... procenv runs itself as part of its build (at the end of course :-). What this means is that you can now see what that build environment is like by looking at the build logs: Debian build logs Ubuntu build logs (you need to expand the "twistie" to see the log files. If you haven't already heard, for the Ubuntu Raring cycle, autopkgtest ( DEP-8 ) tests -- where package builds automatically trigger tests that run on an environment very similar to a "normal" system -- are a hot topic. procenv is DEP-8 enabled, so again, we get to see exactly what such an autopkgtest environment provides . With this information, you can

Procenv 0.8 released

Changes: expanded man page, more sizeof types shown and resource usage details added. Code is available here: https://launchpad.net/procenv If anyone is interested in contributing patches for the following, let me know: code to show network details. support for other platforms (I'd like to add AIX, HP-UX, Solaris, i5O/S, MVS/zO/S, VMS) but no longer have access to such systems).

Upstart 1.6 released

New Upstart Release Today we released Upstart 1.6, which includes the " stateful re-exec " feature, allowing Upstart to continue to supervise jobs after an upgrade of either itself, or any of its dependent libraries: https://launchpad.net/upstart/1.x/1.6 This will be available in Ubuntu Raring very soon. Thanks to all the contributors: A.Costa Colin Watson Eric S. Raymond Jason Conti Loïc Minier Steve Langasek   Like to get involved? If you'd like to get involved with Upstart Development... Take a look at the ideas page Study the Hacking Upstart Together presentation Read the Upstart Cookbook Read the code Subscribe to the mailing list Join us on #upstart on freenode Here's an early draft of what's planned for the next cycle: https://wiki.ubuntu.com/FoundationsTeam/Specs/RaringUpstartUserSessions If you're interested in contributing, let me know! Update on Upstart in Debian ... But wait - there's m

Upstart Cookbook Updated for Developers!

The Upstart Cookbook now contains most of the UDS workshop " Hacking Upstart Together " in the form of a vastly-expanded development section: http://upstart.ubuntu.com/cookbook/#development-and-testing Also covered is the "friendly menu" for Upstart Developers: http://upstart.ubuntu.com/cookbook/#upstart-menu-sh

Hacking Upstart Together

If you missed the Hacking Upstart Together session at UDS-R in Copenhagen, here are the slides: http://people.canonical.com/~jhunt/presentations/uds-r/2012-10-31/upstart-development.pdf This gives a very gentle introduction to Upstart development and covers NIH, common gotchas and more. ... and if you'd like the LaTeX source.... http://people.canonical.com/~jhunt/presentations/uds-r/2012-10-31   We are always looking for new contributors and have some very interesting work lined up this cycle. So, if you're interested in contributing even a small patch or feature, please feel free to contact me or ask on the mailing list .

Procenv and the Process Environment

Quiz How many attributes of a processes "environment" can you think of besides its environment variables? I'm using the term "environment" in a very loose sense here to mean "any system or process level attribute that can be queried via system calls or library calls (or " /proc -picking ") that affects a running process". I'm also including details of the process itself and details of the program that came to be a running process. We're not talking about installed packages or application configuration files in /etc here, but purely low-level program, process and system meta-data. I've got over 20 items in my list excluding environment variables. Whilst you're pondering on that... Compare and Contrast If you've been involved with computers for any appreciable length of time, chances are you have come across the scenario where some program fails to run in a particular environment, but works "perfectly&q

out is out, utfout is out!

My output utility (originally called ' out ' - see previous post ) is now available in debian sid as ' utfout ': http://packages.debian.org/sid/utfout

Call for Testing: New "stateful re-exec" build of Upstart

It didn't quite make quantal, but if you'd like to help test Upstart " stateful re-exec " support, there's a new build (Upstart version 1.5-0ubuntu10~jh20121015 ) in my PPA: https://launchpad.net/~jamesodhunt/+archive/upstart-testing/ We have so far added 60 new tests specifically for stateful re-exec support. The plan is to add this new feature to Ubuntu early in the "R" cycle, so please give it a test by installing the new test version (on a non-critical system! ;-) and running " sudo telinit u " to force Upstart to restart.

out: an output utility

Intro Whilst writing the tests for the Job Logging feature in Upstart, I discovered that I needed some very specific job behaviour that was not easy to produce using any simple existing utility I could find. The Problem What I wanted was a utility that would produce specified amounts of data to either standard output, standard error or direct to the terminal without using shell redirection. The reason for eschewing the shell being that Upstart is clever and intelligently determines whether a shell should be involved. From a user perspective, that's fantastic, but for testing purposes, I needed to force Upstart down particular code paths where it would not for example automatically pass the job through a shell. In fact the constraint was even more restrictive; what I really wanted to do was this: Produce output (including null bytes) to stdout, stderr or the terminal without using any shell meta characters. Unless I used some extremely esoteric shell, that effectively me

Call for Testing: Upstart stateful re-exec

If you'd like to help out testing the new "stateful re-exec" feature of Upstart which we hope to land for Ubuntu Quantal, see the details here: https://lists.ubuntu.com/archives/upstart-devel/2012-September/001948.html In brief (more posts will follow), "stateful re-exec" allows Upstart to be restarted (or upgraded!) without rebooting your system. A basic restart or re-exec isn't difficult, but what stateful re-exec does is maintain Upstarts internal state over the re-exec such that it continues to track jobs and log job output. This is an early cut of the code so don't install it on any critical systems . Additionally, there are some caveats... https://wiki.ubuntu.com/FoundationsTeam/Specs/QuantalUpstartStatefulReexec#Phase_1_Limitations ... but we plan for that list to dwindle quickly over the coming days and weeks.

simplified strace diffing

Image
strace is an extremely powerful tool. But have you ever attempted to compare strace log files? That can get tricky. How about diffing multiple strace logs of multi-process applications? That can be a world of pain. So, I wrote a simple shell script to make a life easier. Then I decided to rewrite it in Python and life got even better :-) My reason for diffing multi-process strace logs was not to see where an application was failing (that's what debuggers are for) but more to understand the flow of program execution. So what follows may have a fairly niche audience. The script is pretty simple: it's just simplifies the log files to allow easier diffing. Specifically it: replaces all addresses with 0xADDR (or 0xNULL ). replaces all timestamps with HH:MM:SS . replaces all datestamps with YYYY/MM/DD . tracks PIDs and replaces each PID seen with a 1-based value (so the first PID seen will be assigned PID1 , the second PID2  et cetera). These simple changes turn out

A Bazaar pre-commit hook to look for signs of unfinished work

Intro Programming involves a fair amount of mental juggling: jumping around a codebase working on a new feature or attempting to track down a bug in existing code, you have to keep track of a lot of pieces of information. However, more balls end up being thrown into the air when, passing through some file or function, you come across an area that needs a bit of lovin'. Often the niggles you spot are minor, but not minor enough to ignore (no Broken Windows here please!) The problem is that these minor imperfections -- which are probably entirely unrelated to the task at hand -- are both annoying and distracting. Imagine... A needs fixing. C, D, E and G need refactoring now that B exists. Documentation for C, E and G need their function docs updating. H introduces a new feature that needs to be documented in a man page. I is inconsistently formatted. J contains a couple of typos. K lacks sufficient tests. L is now giving warnings with the new stricter version of

pull-debian-source with apt-get or "how to download a Debian package from Ubuntu the hard but fun way"

Have you ever wanted to download a Debian  package from an Ubuntu system? Well you are not alone: hidden away in the wonderful " ubuntu-dev-tools " package is the tool for you - the excellent pull-debian-source , written by  tumbleweed . This is a really useful tool which is also fast ! $ sudo apt-get install -y ubuntu-dev-tools $ pull-debian-source hello pull-debian-source: Downloading hello version 2.8-1 pull-debian-source: Downloading hello_2.8.orig.tar.gz from ftp.debian.org (0.665 MiB) pull-debian-source: Downloading hello_2.8-1.debian.tar.gz from ftp.debian.org (0.006 MiB) dpkg-source: info: extracting hello in hello-2.8 dpkg-source: info: unpacking hello_2.8.orig.tar.gz dpkg-source: info: unpacking hello_2.8-1.debian.tar.gz Or, to download the at package from the sid release: $ pull-debian-source at sid pull-debian-source: Downloading at version 3.1.13-1 pull-debian-source: Downloading at_3.1.13.orig.tar.gz from ftp.debian.org (0.117 MiB) pu

Job Logging in Upstart

Job Logging in Upstart The big Upstart feature in Ubuntu Precise is "job logging" (in fact, it turned out to be a significantly bigger feature than we'd originally envisaged :-). This had been a wishlist item for some time and a lot of folk were very keen to see this implemented. All system jobs now have their stdout and stderr logged automatically  by default to a text file in directory  /var/log/upstart/ . Why did we make this the default? Surely daemons and services don't generally write any output? True, but when they do produce output, it is worth capturing since it has a very high chance of being an error message. And errors should not be ignored. For jobs that do not produce any output, there is minimal overhead and of course no log is written. The logger actually uses pseudo-ptys just like  script(1) , xterm(1) , expect(1) ,  screen(1)  et al . This is advantageous for a number of reasons, but from the logging perspective the biggie is &quo

A quick libnih tutorial

Introduction The NIH Utility Library ( libnih ) is a small, efficient and most importantly safe library of general purpose routines. It was written by Keybuk so you can be assured that it is extremely elegant, well-designed, well-tested (includes 2863 tests currently!) and well-written. NIH is used by Upstart, the event-based init daemon which is used by: Ubuntu Desktop Ubuntu Server Ubuntu Cloud RedHats RHEL 6 Chromium OS  (and Chrome OS) That's a lot of deployments of Upstart and NIH around the world!! (And we're not even including mobile device operating systems in that list). But why not just use glib I hear you ask? Well, glib is a very large library whereas NIH is small and designed for low-level daemons and systems which may be resource-constrained. Also, lets not forget that NIH, like Upstart , comes with a very comprehensive test suite so bugs are rare. Other reasons to use NIH: It handles garbage collection for you That's rig