Saturday, 28 May 2016

Procenv 0.46 - now with more platform godness


I have just released procenv version 0.46. Although this is a very minor release for the existing platforms (essentially 1 bug fix), this release now introduces support for a new platform...

Darwin

Yup - OS X now joins the ranks of supported platforms.

Although adding support for Darwin was made significantly easier as a result of the recent internal restructure of the procenv code, it did present a challenge: I don't own any Apple hardware. I could have borrowed a Macbook, but instead I decided to see this as a challenge:

  • Could I port procenv to Darwin without actually having a local Apple system?
 Well, you've just read the answer, but how did I do this?

Stage 1: Docker


Whilst surfing around I came across this interesting docker image:


It provides a Darwin toolchain that I could run under Linux. It didn't take very long to follow my own instructions on porting procenv to a new platform. But although I ended up with a binary, I couldn't actually run it, partly because Darwin uses a different binary file format to Linux: rather than ELF, it uses the Mach-O format.



Stage 2: Travis

The final piece of the puzzle for me was solved by Travis. I'd read the very good documentation on their site, but had initially assumed that you could only build Objective-C based projects on OSX with Travis. But a quick test proved my assumption to be incorrect: it didn't take much more than adding "osx" to the os list and "clang" to the compiler list in procenv's .travis.yml to have procenv building and running (it runs itself as part of its build) on OSX under Travis!

Essentially, the following YAML snippet from procenv's .travis.yml did most of the work:

language: c
compiler:
  - gcc
  - clang
os:
  - linux
  - osx



All that remained was to install the build-time dependencies to the same file with this additional snippet:

before_install:
  - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then brew update; fi
  - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then brew install expat check perl; fi


(Note that it seems Travis is rather picky about before_install - all code must be on a single line, hence the rather awkward-to-read "if; then ....; fi" tests).


Summary


Although I've never personally run procenv under OSX, I have got a good degree of confidence that it does actually work.

That said, it would be useful if someone could independently verify this claim on a real system!) Feel free to raise bugs, send code (or even Apple hardware :-) my way!



Tuesday, 22 March 2016

Procenv 0.45 released


Yesterday, I released procenv 0.45.

Although not a huge amount has changed on the outside in this release, rather
dramatic changes have occurred on the inside, as evinced by the following:

$ git diff 0.44..0.45 --stat|grep "src/.*\.[ch]"
 src/messages.h                          |    41 +
 src/output.c                            |  2001 ++++++
 src/output.h                            |   179 +
 src/platform-headers.h                  |   243 +
 src/platform.h                          |   167 +
 src/platform/android/platform.c         |    46 +
 src/platform/freebsd/platform-freebsd.h |    43 +
 src/platform/freebsd/platform.c         |   479 ++
 src/platform/hurd/platform-hurd.h       |    38 +
 src/platform/hurd/platform.c            |    91 +
 src/platform/linux/platform-linux.h     |   174 +
 src/platform/linux/platform.c           |  2176 ++++++
 src/platform/minix/platform-minix.h     |    25 +
 src/platform/minix/platform.c           |   126 +
 src/platform/netbsd/platform-netbsd.h   |    25 +
 src/platform/netbsd/platform.c          |   237 +
 src/platform/openbsd/platform-openbsd.h |    25 +
 src/platform/openbsd/platform.c         |   199 +
 src/platform/platform-generic.c         |   922 +++
 src/platform/platform-generic.h         |    66 +
 src/platform/unknown/platform-unknown.h |    25 +
 src/platform/unknown/platform.c         |    89 +
 src/pr_list.c                           |     2 +-
 src/pr_list.h                           |     2 +-
 src/procenv.c                           | 11575 +++++++-----------------------
 src/procenv.h                           |   918 +--
 src/pstring.c                           |   309 +
 src/pstring.h                           |    55 +
 src/string-util.c                       |   543 ++
 src/string-util.h                       |    60 +
 src/types.h                             |    34 +
 src/util.c                              |   288 +
 src/util.h                              |    57 +


Partly driven by "#ifdef hell", and part inspired by Hisham's Fosdem presentation on htop, "Going cross-platform: how htop was made portable", I decided to modularise the code and introduce platform drivers encapsulating the O/S specifics and allowing procenv.c itself to be considerably simplified. This allows for easier maintenance and easier porting to new operating systems.

To prove it, procenv now runs on OpenBSD, NetBSD and Minix 3!

There is still further refactoring to do, but the code is now in much better shape for others to work with.

So, if you'd like to get involved in porting procenv to [insert-your-favourite-as-yet-unsupported-OS-here], get involved! I even wrote a simple porting guide to help get your started:


Tuesday, 24 November 2015

Procenv 0.43 released

(Cough... This should probably be called the "I-almost-forgot-I-had-a-blog release"! :-)

Development has now moved to github:

So you can grab the release files here:


The move to github brings a few advantages, including Travis-CI integration:

Even more awesome being that, thanks to the wonder of webhooks, procenv is now building for lots of distros. Take a look at:

So you get to see the environment those builds run in and OBS is also providing procenv packages!

Here's the funky download page (just click your distro):

Caveat emptor: those packages are building off the tip, so are not necessarily releases - they are built from the latest commit! That said, since procenv runs a lot of tests at build-time, you should be reasonably safe.

If you'd rather opt for official releases, the new version should be in Debian and Ubuntu Xenial soon. It should also arrive in Clear Linux tomorrow.

As for what has changed since the last blog-post, just take a look at the NEWS file:



Tuesday, 25 March 2014

procenv update

Earlier today I released procenv 0.34. Quite a bit has changed since version 0.27 including:
  • Recognises AARCH64, SuperH (fix), PPC64, PPCspe, PPC64LE, OpenRISC systems.
  • Added symbolic names in '--ranges'.
  • Displays Linux binary personality (and flags).
  • Improved '--capabilities' output showing not only bounding set, but also whether each capability is supported, permitted, effective and inheritable values.
  • Added '--memory' which shows NUMA memory details.
  • Added '--cpu' which displays CPU affinity details.
  • Added rpm spec file allowing it to build on RHEL5, Fedora, etc.
  • Improved '--sizeof' which now shows lots more standard types.
  • Displays FreeBSD Capsicum capabilities.
  • Lots of fixes.
Version 0.34 is now available in Debian sid whilst Ubuntu Trusty will be released with procenv 0.33 (which lacks the binary personality information).

Take a look at the links on the main procenv page to see the different environments that it is currently building in (the build logs show the build environments as procenv runs itself as part of its build):

Monday, 14 October 2013

Procenv 0.27 released

Procenv 0.27 has been released. This release introduces a raft of new features...

IPC

 It is now possible to display details of the following IPC mechanisms:

  • message queues
  • semaphores
  • shared memory
Alas, this feature is not available on BSD's as yet partly since there appears to be no documented way to query these mechanisms.

Output Categories


The introduction of the IPC categories brings the total number of output categories to 32:
  • meta
  • arguments
  • capabilities
  • cgroups
  • clocks
  • compiler
  • confstr
  • environment
  • file descriptors
  • libraries
  • limits
  • locale
  • misc
  • message queues
  • mounts
  • network
  • oom
  • platform
  • process
  • ranges
  • rusage
  • semaphores
  • shared memory
  • signals
  • sizeof
  • stat
  • sysconf
  • threads
  • time
  • timezone
  • tty
  • uname

Highly-Structured Output

The code that handles procenv has been completely rewritten so that all output is highly structured. So, rather than displaying file descriptors like this:

$ procenv --fds
fds:
  fd 0: terminal=yes ('/dev/pts/12')
  fd 1: terminal=no
  fd 2: terminal=yes ('/dev/pts/12')
fds (linux/proc):
  '/proc/self/fd/0' -> '/dev/pts/12' (terminal=yes, valid=yes)
  '/proc/self/fd/1' -> 'procenv.log' (terminal=no, valid=yes)
  '/proc/self/fd/2' -> '/dev/pts/12' (terminal=yes, valid=yes)

... they are now displayed like this:

$ procenv --fds
file descriptors:
  0:
    terminal: yes
    valid: yes
    device: /dev/pts/25
  1:
    terminal: no
    valid: yes
    device: procenv.log
  2:
    terminal: yes
    valid: yes
    device: /dev/pts/25

Note that not only is the output more structured, but for file-descriptor output there is now only a single section combining details of both the generic and Linux-specific details.

Format Version


Since it may be necessary to modify the output format, procenv now includes a format-version field in the meta section. This value (currently 1), will be incremented for each output format change. The format-version combined with the version of procenv itself should ease any version transition issues.

Tweaking the Output


If the default space indenting offends your sensibilities, you'll be glad to know that you can now change it to use tabs instead (or in fact any other character you choose):

--indent-char="\t"

Maybe you'd prefer 3 tab indents:

--indent-char="\t" --indent=3

Output Formats


The new highly-structured text output is in fact a side-effect of the fact that procenv can now produce output in other formats. The default is still text, but it can now produce JSON and XML. This even works when running --version:

$ procenv --version
0.27

$ procenv --format=json --version
{
  "version" : {
    "name" : "procenv",
    "version" : "0.27",
    "author" : "James Hunt <james.hunt@ubuntu.com>"
  }
}

$ procenv --format=xml --version
<?xml version="1.0" encoding="UTF-8"?>
<procenv version="0.27" package_string="procenv 0.27" mode="non-privileged" format_version="1">
  <section name="version">
    <entry name="name">procenv</entry>
    <entry name="version">0.27</entry>
    <entry name="author">James Hunt &lt;james.hunt@ubuntu.com&gt;</entry>
  </section>
</procenv>

This makes procenv output much more consumable by standard tools.

Crumbs!


However, the new highly-structured output introduces a problem. Before its introduction, it was easy to use grep(1) to extract particular values. But now, some of those values are on different lines. Yes, you can use the magic grep(1) options -A, -B and -C, but if that isn't convenient for you, procenv does offer an alternative in the form of an output format I've called "crumb" (short for "breadcrumb"). Here's the file descriptor output in breadcrumb output:

$ procenv --format=crumb --fds
file descriptors:0:terminal:yes
file descriptors:0:valid:yes
file descriptors:0:device:/dev/pts/25
file descriptors:1:terminal:yes
file descriptors:1:valid:yes
file descriptors:1:device:/dev/pts/25
file descriptors:2:terminal:yes
file descriptors:2:valid:yes
file descriptors:2:device:/dev/pts/25

As you can see, "crumb mode" shows every value with the appropriate sections the particular value belongs prior to the value. So the unique path to each value is shown by way of a list of "breadcrumbs" or headings. Thus, to extract all details of stdin (aka file descriptor zero):

$ procenv --format=crumb --fds | grep "^file descriptors:0"
file descriptors:0:terminal:yes
file descriptors:0:valid:yes
file descriptors:0:device:/dev/pts/25

It's also really easy to generate CSV data if you wanted to import the output into a spreadsheet:

$ procenv --format=crumb --crumb-separator=',' --separator=','

Download it from:

Tuesday, 27 August 2013

procenv 0.26 released


Version 0.26 of the procenv utility is now available.

Changes:

  • Check to determine if running on a console now works for FreeBSD/kFreeBSD too.
  • Added ability to show all arguments (-A/--arguments)
    (useful when using --exec).
  • Added ability to display network details (-N/--network).
  • Added BSD/Hurd-specific signals.
  • Corrected output sort order.
  • Mount details now include block, inode and fsck details.
There are now 29 categories of environment information displayed when procenv is run (on Linux).

Grab it from:

This update should appear in Debian, Ubuntu and FreeBSD soon...

Friday, 23 August 2013

Upstart 1.10 released

Lots of goodness in this release (explanatory posts to follow):

  • upstart-local-bridge: New bridge for starting jobs on local socket connections.
  • upstart-dconf-bridge: New bridge for Session Inits to react to dconf/gsettings changes.
  • upstart-dbus-bridge: New '--bus-name' option to allow bus name variable to be included in dbus-event(7).
  • New "reload signal" stanza to allow jobs to specify a custom signal that will be sent to the main process (rather than the default SIGHUP).
  • Inclusion of Session Init sample jobs.
  • Re-exec fixes for handling chroot sessions.
  • Shutdown fix for Session Inits.
  • New python3 module and accompanying integration test suite for testing Upstart running as PID 1 and as a Session Init (privileged and non-privileged).

The Upstart cookbook has been updated for this release.