The proc connector is one of those interesting kernel features that most people rarely come across, and even more rarely find documentation on. Likewise the socket filter. This is a shame, because they’re both really quite useful interfaces that might serve a variety of purposes if they were better documented.

The proc connector allows you to receive notification of process events such fork and exec calls, as well as changes to a process’s uid, gid or sid (session id). These are provided through a socket-based interface by reading instances of struct proc_event defined in the kernel header.

#include <linux/cn_proc.h>

The interface is built on the more generic connector API, which itself is built on the generic netlink API. These interfaces add some complexity as they are intended to provide bi-directional communication between the kernel and userspace; the connector API appears to have been largely forgotten as newer such socket interfaces simply declare their own first-class socket classes. So we need the headers for those too.

#include <linux/netlink.h>
#include <linux/connector.h>

(For brevity, I’ll omit any standard boilerplate such as the headers you need for syscalls and library functions that you should be used to as well as function definitions, error checking, and so-forth.)

Ok, now we’re ready to create the connector socket. This is straight-forward enough, since we’re dealing with atomic messages rather than a stream, datagram is appropriate.

int sock;
sock = socket (PF_NETLINK, SOCK_DGRAM | SOCK_NONBLOCK | SOCK_CLOEXEC,
               NETLINK_CONNECTOR);

To select the proc connector we bind the socket using a struct sockaddr_nl object.

struct sockaddr_nl addr;
addr.nl_family = AF_NETLINK;
addr.nl_pid = getpid ();
addr.nl_groups = CN_IDX_PROC;

bind (sock, (struct sockaddr *)&addr, sizeof addr);

Unfortunately that’s not quite enough yet; the proc connector socket is a bit of a firehose, so it doesn’t in fact send any messages until a process has subscribed to it. So we have to send a subscription message.

As I mentioned before, the proc connector is built on top of the generic connector and that itself is on top of netlink so sending that subscription message also involves embedded a message, inside a message inside a message.  If you understood Christopher Nolan’s Inception, you should do just fine.

Since we’re nesting a proc connector operation message inside a connector message inside a netlink message, it’s easiest to use an iovec for this kind of thing.

struct iovec iov[3];
char nlmsghdrbuf[NLMSG_LENGTH (0)];
struct nlmsghdr *nlmsghdr = nlmsghdrbuf;
struct cn_msg cn_msg;
enum proc_cn_mcast_op op;

nlmsghdr->nlmsg_len = NLMSG_LENGTH (sizeof cn_msg + sizeof op);
nlmsghdr->nlmsg_type = NLMSG_DONE;
nlmsghdr->nlmsg_flags = 0;
nlmsghdr->nlmsg_seq = 0;
nlmsghdr->nlmsg_pid = getpid ();

iov[0].iov_base = nlmsghdrbuf;
iov[0].iov_len = NLMSG_LENGTH (0);

cn_msg.id.idx = CN_IDX_PROC;
cn_msg.id.val = CN_VAL_PROC;
cn_msg.seq = 0;
cn_msg.ack = 0;
cn_msg.len = sizeof op;

iov[1].iov_base = &cn_msg;
iov[1].iov_len = sizeof cn_msg;

op = PROC_CN_MCAST_LISTEN;

iov[2].iov_base = &op;
iov[2].iov_len = sizeof op;

writev (sock, iov, 3);

The netlink message length is the combined length of the following connector and proc connector operation messages, and is otherwise simply a message from our process id with no following messages.  However all of the interfaces to netlink take a lot of care to make sure the following structure in the message is aligned as wide as possible using the NLMSG_LENGTH macro, to avoid issues with platforms that have fixed alignment for data types, so we have to be careful of that too.

So we actually have a bit of padding between the struct nlmsghdr and the struct cn_msg, this is accomplished by actually using a character buffer of the right size for the first iovec element and accessing it through a struct nlmsghdr pointer.

The connector message indicates that it is relevant to the proc connector through the idx and val fields, and the length is the legnth of the proc connector operation message.

Finally the proc connector operation message (just an enum) says we want to subscribe. Why isn’t there padding between the connector and proc connector operation messages? Because the last element in struct cn_msg is a zero-width type which results in the right padding, this interface is rather newer than netlink.

iovec stitches it all together so it’s sent as a single message, visualized this message looks like this:

There’s a matching PROC_CN_MCAST_IGNORE message if you want to turn off the firehose without closing the socket.

Ok, the firehose is on now we need to read the stream of messages.  Just like the message we sent, the stream of messages we receive are actually netlink messages, and inside those netlink messages are connector messages, and inside those are proc connector messages.

Netlink allows for all sorts of things like multi-part messages, but in reality we can ignore most of that since connector doesn’t use the, but it’s worth future-protecting ourselves and being liberal in what we accept.

struct msghdr msghdr;
struct sockaddr_nl addr;
struct iovec iov[1];
char buf[PAGE_SIZE];
ssize_t len;

msghdr.msg_name = &addr;
msghdr.msg_namelen = sizeof addr;
msghdr.msg_iov = iov;
msghdr.msg_iovlen = 1;
msghdr.msg_control = NULL;
msghdr.msg_controllen = 0;
msghdr.msg_flags = 0;

iov[0].iov_base = buf;
iov[0].iov_len = sizeof buf;

len = recvmsg (sock, &msghdr, 0);

Why do we use recvmsg rather than just read? Because netlink allows arbitrary processes to send messages to each other, so we need to make sure the message actually comes from the kernel; otherwise you have a potential security vulnerability. recvfrom lets us receive the sender address as well as the data.

if (addr.nl_pid != 0)
        continue;

(I’m assuming you’re reading in a loop there.)

So now we have a netlink message package from the kernel, this may contain multiple individual netlink messages (it doesn’t, but it may). So we iterate over those.

for (struct nlmsghdr *nlmsghdr = (struct nlmsghdr *)buf;
     NLMSG_OK (nlmsghdr, len);
     nlmsghdr = NLMSG_NEXT (nlmsghdr, len))

And we should ignore error or no-op messages from netlink.

if ((nlmsghdr->nlmsg_type == NLMSG_ERROR)
    || (nlmsghdr->nlmsg_type == NLMSG_NOOP))
        continue;

Inside each individual netlink message is a connector message, we extract that and make sure it comes from the proc connector system.

struct cn_msg *cn_msg = NLMSG_DATA (nlmsghdr);

if ((cn_msg->id.idx != CN_IDX_PROC)
    || (cn_msg->id.val != CN_VAL_PROC))
        continue;

Now we can safely extract the proc connector message; this is a struct proc_event that we haven’t seen before. It’s quite a large structure definition so I won’t paste it here, since it contains a union for each of the different possible message types. Instead here’s code to actually print the relevant contents for an example message.

struct proc_event *ev = (struct proc_event *)cn_msg->data;

switch (ev->what) {
case PROC_EVENT_FORK:
        printf ("FORK %d/%d -> %d/%d\n",
                ev->event_data.fork.parent_pid,
                ev->event_data.fork.parent_tgid,
                ev->event_data.fork.child_pid,
                ev->event_data.fork.child_tgid);
        break;
/* more message types here */
}

As you can see, each message type has an associated member of the event_data union containing the information fields for it. And as you can see, this gives you information about each individual kernel task, not just the top-level processes you’re normally used to seeing. In other words, you see threads as well as processes.

Like I keep saying, it’s a firehose. It would be great if there was some way to filter the socket in the kernel so that our process doesn’t even get woken up for messages. Wake-ups are bad, especially in the embedded space.

Fortunately there is a way to filter sockets on the kernel-side, the kernel socket filter interface. Unfortunately this isn’t too well documented either; but let’s use this opportunity to document an example.

We’ll filter the socket so that we only receive fork notifications, discarding the other types of proc connector event type and most importantly discarding the messages that indicate new threads being created (those where the pid and tgid fields differ). One important part of filtering is that you should be careful so that only expected messages are filtered, and that unexpected messages are still passed through.

The filter machine consists of a set of machine language instructions added to the socket through a special socket option. Fortunately this machine language is copied from the Berkeley Packet Filter from BSD, so we can find documentation for it in the bpf(4) manual page there. Just ignore the structure definitions, because they are different on Linux.

So let’s get started with our example; first we need to add the right header.

#include <linux/filter.h>

And now we need to insert the filter into the socket creation, before the subscription message is sent is usually a good place. On Linux the instructions are given as an array of struct sock_filter members which we can construct using the BPF_STMT and BPF_JUMP macros.

Just to make sure everything is working, we’ll create a simple “no-op” filter.

struct sock_filter filter[] = {
        BPF_STMT (BPF_RET|BPF_K, 0xffffffff),
};

struct sock_fprog fprog;
fprog.filter = filter;
fprog.len = sizeof filter / sizeof filter[0];

setsockopt (sock, SOL_SOCKET, SO_ATTACH_FILTER, &fprog, sizeof fprog);

Not very useful, but it means we can now concentrate on writing the filter code itself. This filter consists of a single statement, BPF_RET that tells the kernel to deliver an amount of bytes of the packet to the receiving process and to return from the filter. The BPF_K option means that we give the amount of bytes as the argument to the statement, and in this case we give the largest possible value. In other words, this statement declares to deliver the whole packet and return from the filter.

To not wake up the process at all, and filter everything we deliver no bytes and return from the filter.

BPF_STMT (BPF_RET|BPF_K, 0);

You may want to test that too.

Ok, now let’s actually do some examination of the packets to filter out the noise. Recall that we’re dealing with nested messages here, messages inside messages, inside messages. Visualizing this is really important to understanding what you’re dealing with.

The most basic filter code consists of three operations: load a value from the packet into the machine’s accumulator, compare that against a value and jump to a different instruction if equal (or not equal), and then possibly return or perform another operation.

All of the following filter code replaces whatever you had in the filter[] array before.

So first we should examine the nlmsghdr on the start of the packet, we want to make sure that there is just one netlink message in this packet. If there are multiple, we just pass the whole packet to userspace for dealing with. We check the nlmsg_type field to make sure it contains the value NLMSG_DONE.

BPF_STMT (BPF_LD|BPF_H|BPF_ABS,
          offsetof (struct nlmsghdr, nlmsg_type));
BPF_JUMP (BPF_JMP|BPF_JEQ|BPF_K,
          htons (NLMSG_DONE),
          1, 0);
BPF_STMT (BPF_RET|BPF_K, 0xffffffff);

The first statement says to load (BPF_LD) a “halfword” (16-bit) value (BPF_H) from the absolute offset (BPF_ABS) equivalent to the position of the nlmsg_type member in struct nlmsghdr. Since we expect that structure to be the start of the message, this means the accumulator should now have that value.

The next statement is a jump (BPF_JMP), it says to compare the accumulator for equality (BPF_JEQ) against the constant argument (BPF_K). We only want to continue if this is the sole message, so the value we compare against is NLMSG_DONE – first remembering to deal with host and network ordering.

If true, the jump will jump one statement; if false the jump will not jump any statements. These are the third and fourth arguments to the BPF_JUMP macro.

Note that the error case is always to return the whole packet to the process, waking it up. And the success case is future processing of the packet. This makes sure that we don’t filter unexpected packets that userspace may really need to deal with. Don’t use the socket filter for security filtering, it’s for reducing wake-ups.

So let’s filter the next set of values, we want to make sure that this netlink message is from the connector interface. Again we load the right “word” (32-bit) values (BPF_W) from the appropriate offsets and check them against constants.

BPF_STMT (BPF_LD|BPF_W|BPF_ABS,
          NLMSG_LENGTH (0) + offsetof (struct cn_msg, id)
          + offsetof (struct cb_id, idx));
BPF_JUMP (BPF_JMP|BPF_JEQ|BPF_K,
          htonl (CN_IDX_PROC),
          1, 0);
BPF_STMT (BPF_RET|BPF_K, 0xffffffff);

BPF_STMT (BPF_LD|BPF_W|BPF_ABS,
          NLMSG_LENGTH (0) + offsetof (struct cn_msg, id)
          + offsetof (struct cb_id, idx));
BPF_JUMP (BPF_JMP|BPF_JEQ|BPF_K,
          htonl (CN_VAL_PROC),
          1, 0);
BPF_STMT (BPF_RET|BPF_K, 0xffffffff);

So after this filter code has executed, we know the packet contains a single netlink message from the proc connector. Now we want to make sure it’s a fork message; this is a bit different from before, because now we explicitly do filter out the other message types so the return case for non-equality is to return zero bytes.

BPF_STMT (BPF_LD|BPF_W|BPF_ABS,
          NLMSG_LENGTH (0) + offsetof (struct cn_msg, data)
          + offsetof (struct proc_event, what);
BPF_JUMP (BPF_JMP|BPF_JEQ|BF_K,
          htonl (PROC_EVENT_FORK),
          1, 0);
BPF_STMT (BPF_RET|BPF_K, 0);

And now we can compare the pid and tgid values for the parent process and the child process fields. This is again slightly interesting because we can’t compare against an absolute offset with the jump instruction so we use the second index register instead (BPF_X in the jump instruction). Of course it would be too easy if we could load directly into that, so we have to do it via the scratch memory store instead; this requires loading into the accumulator (BPF_LD), storing into scratch memory (BPF_ST) and loading the index register (BPF_LDX) from scratch memory (BPF_MEM).

BPF_STMT (BPF_LD|BPF_W|BPF_ABS,
          NLMSG_LENGTH (0) + offsetof (struct cn_msg, data)
          + offsetof (struct proc_event, event_data)
          + offsetof (struct fork_proc_event, parent_pid));
BPF_STMT (BPF_ST, 0);
BPF_STMT (BPF_LDX|BPF_W|BPF_MEM, 0);

Then we load the tgid value into the accumulator and we can compare and jump as before; if they are equal we want to continue, if they are inequal we want to filter the packet.

BPF_STMT (BPF_LD|BPF_W|BPF_ABS,
          NLMSG_LENGTH (0) + offsetof (struct cn_msg, data)
          + offsetof (struct proc_event, event_data)
          + offsetof (struct fork_proc_event, parent_tgid));
BPF_JUMP (BPF_JMP|BPF_JEQ|BPF_X,
          0,
          1, 0);
BPF_STMT (BPF_RET|BPF_K, 0);

Then we do the same for the child field.

BPF_STMT (BPF_LD|BPF_W|BPF_ABS,
          NLMSG_LENGTH (0) + offsetof (struct cn_msg, data)
          + offsetof (struct proc_event, event_data)
          + offsetof (struct fork_proc_event, parent_pid));
BPF_STMT (BPF_ST, 0);
BPF_STMT (BPF_LDX|BPF_W|BPF_MEM, 0);

BPF_STMT (BPF_LD|BPF_W|BPF_ABS,
          NLMSG_LENGTH (0) + offsetof (struct cn_msg, data)
          + offsetof (struct proc_event, event_data)
          + offsetof (struct fork_proc_event, parent_tgid));

BPF_JUMP (BPF_JMP|BPF_JEQ|BPF_X,
          0,
          1, 0);

BPF_STMT (BPF_RET|BPF_K, 0);

After all that filter hurdling, we have a packet that we want to pass through to the process, so the final instruction is a return of the largest packet size.

BPF_STMT (BPF_RET|BPF_K, 0xffffffff);

That’s it. Of course, what you do with this is up to you. One example could be a daemon that watches for excessive forks and kills fork bombs before they kill the machine. Since you get notification of changes of uid or gid, another example could be a security audit daemon, etc.

Upstart uses this interface for its own nefarious process tracking purposes.

In addition to the regular posts documenting features of 0.6 and giving hints and tips about it’s usage, release announcements and so-forth; I’ll also be posting insights and anecdotes about Upstart’s ongoing development.  A particular story cropped up again this month, and I thought I’d share it with you.

When I began work on Upstart, one of the earliest decisions I made was to make sure the code was very-well covered by a comprehensive test suite.  I’d been working with Robert Collins a lot in the previous couple of years and he is very much an advocate of practices such as Extreme Programming (XP) and Agile Development; especially the discipline of Test Driven Development.

I’d also recently seen a keynote by Andrew Tridgell in which he talked about some of the development of Samba 4, in particular the high use of both test cases and code generation in that code-base.  Something he said in the keynote stuck with me: “untested code is broken code”.

Statistics obviously depend on exactly how you count lines of code, but using a simple semi-colon count the combined source code of libnih and Upstart is slightly over 20,000 lines of code.  The combined source code of the test suite for both is slightly over 120,000 lines of code.

The init daemon is an extremely important part of a Linux system, if it crashes then you’re left with a kernel panic; if it simply misbehaves, you’re left with just severe problems.  Not only was I changing it, but I was replacing a very simple dumb system (Sys V init) with something comparatively complex with rules and behaviours that needed rigorous testing.

It would have been very scary to have developed it without the careful testing, and I would have been very worried if anyone had agreed to replace such a core component of the system without this test suite to back up its behaviour.

That being said, maintaining the test suite can be a huge burden.  Don’t believe what anybody tells you, if you’re writing test cases as well as code, then your pace of development slows as well.  They’re right that you spend a lot less time debugging of course, but unlike in the commercial software business free software developers tend to release first and debug later.   If you use a similarly high test to code ratio in your own project, then you’ll find that the time until your first release will be pretty long and the time between releases longer as well.

Another decision is whether to do Test Driven Development or not; that discipline requires that you always write the tests first, to fail, and only write code in order to make the tests pass.  I’m not a fan of TDD, and I’ve no problem admitting that I mostly did not use it for Upstart.  My gut feel is that TDD produces code that hangs, swings and loops just to deal with testing.  It also just doesn’t suit my coding style: I like to write code from the middle outwards, the function API is the last thing I tend to fix, where TDD forces it to be the first.

I’m also not convinced TDD is really suitable for a language like C; it’s pretty hard to get a test case to compile, run and fail without writing any supporting code such as a header file, etc.

I have found TDD useful when I have code that really does break down into a single unit with a well-defined and obvious API, and that while the inputs and outputs have been obvious, the algorithm for getting between them wasn’t at the time.

What I’ve tended to do instead is write code naturally how I would, and write test cases alongside to run the code and make sure it’s working.  As the code grows more complex, more test cases appear for it.  One big advantage to this is then I don’t need to reboot or fire up a VM as much, I can test a large proportion of Upstart’s operation through testing.

Now, onto the stories.  There are two similar ones.

One of the side-effects of testing Upstart so strongly is that the tests are not only driving the code I’ve written but also code in libraries and even in the Kernel.  One particular set of tests was covering the code in libnih and Upstart that handles watching the configuration directory for changes, it’s this code that means Upstart automatically reloads jobs when you edit them without needing an explicitly signal.

One day these test cases started failing without warning.  Investigation showed that they passed fine under older kernels, but with the newest kernel update to Ubuntu, they failed.

The inotify subsystem in the kernel had undergone a radical overhaul and rewrite.  Rather than being its own code, it was completely rebased onto the new fsnotify system.  Fortunately I was aware of this, and after careful checking that it was indeed the kernel behaviour that was now incorrect (and that it wasn’t incorrect before), I got in touch with the Eric Paris, the author of the new code, and was able to give him minimal example code to replicate the problem.

inotify: check filename before dropping repeat events

This was a while ago, but pretty much the same story happened again recently, just this time not with the kernel.

Again, the story started with Upstart’s test suite failing.  The engineer who first noticed it assumed it was an issue with the new build daemon and disabled the test for the time being.  The test was in the part of the code testing Upstart’s interaction with D-Bus.

Now, sometimes I tend to write tests to deal with corner-cases and “what if” scenarios that I dream up.  This isn’t always about testing my code, often it’s a case of finding out whether something is really possible or whether that thing misbehaves.  These tests still stay in the suite of course.

A particular set of tests were intended to find out what happened if the D-Bus daemon crashed during initial connection, I considered this fairly important because at times the libdbus library has called exit() or abort() when things happened that it didn’t like.  If you call that from the init daemon, the kernel panics.

These tests had worked fine for a couple of years (actually at the time I had to fix bugs in libdbus to make them pass) but now one of these tests was breaking.  The disconnection was causing SIGPIPE to be delivered to the test.

Again, this turned out to be due to a change to D-Bus.  Lennart Poettering had been working on some changes to avoid libdbus’s awkward SIGPIPE handling and replace it with the use of the MSG_NOSIGNAL flag.  Unfortunately he’d missed a case in the authentication code.  The side-effect was that if the D-Bus daemon had crashed, been killed, OOM’d, etc. during initial connection – the connecting application would have gone too.  Especially bad for an init daemon.

Fortunately Upstart’s test suite caught it, and the fix was simple.

sysdeps-unix: use MSG_NOSIGNAL when sending creds

(reposted from http://upstart.at/2010/12/20/the-importance-of-being-tested/ – post comments there)

In last week’s post I talked about how Events can be treated like Signals, this week we’ll be looking at how Events can be treated like Methods.  That might seem a little surprising, since normally one considers signals and methods as very different things, but to Upstart they are both just events.

What do I mean by Methods?  You’ve almost certainly done some kind of programming, even if just a little scripting, so you should know about methods or functions.

In contrast to signals, which are just a notification that something happened on the system, a method is a request for the system to do something on your behalf.  Usually to make some kind of change to the system state.

Likewise in contrast to the signals where you don’t care about the result, for a method you want to wait for the changes to be completed and perhaps even be notified if the method failed.

It’s just as easy to implement a method in Upstart as it is to implement something that considers an event a signal.  Here’s an example of how you might implement a suspend method:

start on suspend

task
exec pm suspend

Doesn’t look that much difference from a signal, the only new stanza in this is task (and that’s not necessary for a method either).  So what happens if we want to trigger a suspend?  We use the command:

root@worldofwarcraft:~# initctl emit suspend

The difference here from emitting a signal we demonstrated in the previous post is that we aren’t using the –no-wait flag.

So we emit the suspend event, and Upstart will start our job as a result; but initctl emit will not return immediately, it waits for the results of the event to complete before it returns.

Because we used the task stanza in the configuration, we’ve told Upstart that the process we execute is expected to take a limited amount of time and then finish by itself.  This means that Upstart will not believe the job is complete until the process has exited, and will continue to block the event while it is still running.

Finally if the command exited with an error, that error is propagated back to the event that started it, and the initctl emit command will exit with an error code.

So now we can use Upstart events and jobs for two different purposes; we can announce changes to the system, and we can use them as methods to make changes to the system.

The most typical event that is used as a methods on your system is the runlevel event used to change the runlevel for System-V compatibility and generally emitted by the telinit and shutdown tools.  The /etc/init/rc.conf script that handles it can be pretty simple and looks not unlike the suspend example above:

start on runlevel [0123456]

task
exec /etc/init.d/rc $RUNLEVEL

What happens if you don’t include task?  Well, that means Upstart will consider the job as ready when the process executed is running, and the event will be unblocked and initctl emit will return.  If the service fails to start, then initctl will return with an error.  This is great for methods that start (or stop) services.

Side-note: the start and stop commands act very much like method events, they block until the service is running or the task has finished and they return errors as well.  However they’re not actually implemented as events right now, an oversight I intend to correct in Upstart 2.

(reposted from http://upstart.at/2010/12/16/events-are-like-methods/ – post comments there)

A little while ago I was asked to solve a problem that somebody was having with Upstart, and I realised that people weren’t understanding how things were actually working and were just muddling along when doing event matching in jobs.  This is unfortunate, because it hides some of Upstart’s true power, so I thought it high time I actually explained this.

Let’s start with a simple example.  Fire up any Linux distribution with Upstart 0.6, Ubuntu or Fedora current releases will do, and create a file named /etc/init/example1.conf with the following content:

start on surprise

This is pretty simple, it’s a job that does nothing except declare that it’s started when the surprise event happens.  We can demonstrate that works by emitting the event ourselves and checking the status of the job before and afterwards:

root@angrybirds:/etc/init# status example1
example1 stop/waiting
root@angrybirds:/etc/init# initctl emit surprise
root@angrybirds:/etc/init# status example1
example1 start/running

Nothing too surprising after all, I hope.  The job did indeed start on the surprise event, and would now be running if we’d actually told Upstart to run something.

Incidentally I’m often asked why there isn’t a single list of events anywhere, that’s because you can match any event you like as long as you know something emits it.  Events are supposed to come from all manner of sources.  I do try and document them though, try running man 7 startup on your system to see an example of an event’s man page.

If events were just names, they’d be pretty boring.  Events can also have attached environment variables, and these get put into the environment of any job’s process started by the event.  Here’s /etc/init/example2.conf:

start on weather

script
    echo $KIND > /tmp/weather
end script

This will now run a small shell script that outputs the $KIND environment variable to a file.  This isn’t set anywhere, but we can pass it in the event.

root@angrybirds:/etc/init# cat /tmp/weather
cat: /tmp/weather: No such file or directory
root@angrybirds:/etc/init# initctl emit weather KIND=RAIN
root@angrybirds:/etc/init# cat /tmp/weather
RAIN

Ok, these are just examples but there are plenty of useful events on your system right now which carry environment variables such as which network interface just came up, and so on.

If you wanted to only run on a certain type of weather, you might think to check the value of $KIND within the script; you could do that, but it’s inefficient, ideally you don’t want your script run at all.  Fortunately we can match the environment of an event in the job easily enough, here’s /etc/init/example3.conf:

start on weather KIND=snow

Hopefully you’ll figure that this one will only start if it’s snowing, and you’d be right:

root@angrybirds:/etc/init# status example3
example3 stop/waiting
root@angrybirds:/etc/init# initctl emit weather KIND=hail
root@angrybirds:/etc/init# status example3
example3 stop/waiting
root@angrybirds:/etc/init# initctl emit weather KIND=snow
root@angrybirds:/etc/init# status example3
example3 start/running

Events can have more than one environment variable, and you can have more than one match:

start on weather KIND=rain INTENSITY=heavy

The matches are actually globs, so you can use * and ? in there and as well as =, there’s obviously !=.

One useful use for the latter is in the stop on stanza, as well as being available for the job’s processes you can also use these in other stanzas within the job.  Here’s a cute example for /etc/init/example4.conf:

start on weather KIND=rain or weather KIND=snow
stop on weather KIND!=$KIND

This one takes a bit of explaining.  First of all to start the job we match the weather event with $KIND set to either rain or snow.  Now we supply a condition to stop the job, and we also match the weather event with a given value of $KIND except this time we match what looks like itself.

In fact this expansion of $KIND is the value that variable had when the job was started, not the value in the new event.  It says to stop the job if it stops raining, or stops snowing depending on which of the two started it.  Most importantly, if an event simply repeats the same kind of weather, but maybe with a different intensity, the job carries on running (but it doesn’t have its environment updated – UNIX can’t do that).

root@angrybirds:/etc/init# status example4
example4 stop/waiting
root@angrybirds:/etc/init# initctl emit weather KIND=rain INTENSITY=heavy
root@angrybirds:/etc/init# status example4
example4 start/running
root@angrybirds:/etc/init# initctl emit weather KIND=rain INTENSITY=light
root@angrybirds:/etc/init# status example4
example4 start/running
root@angrybirds:/etc/init# initctl emit weather KIND=sun
root@angrybirds:/etc/init# status example4
example4 stop/waiting

Ok, last fake example before we get onto the fun bits.  Remember the example from above:

start on weather KIND=rain INTENSITY=heavy

Upstart lets us shortcut this a little, the environment variables are specified in an order on the initctl command-line and if we know what that order is, we can just assume what variable is in that position.  So as long as we know a weather event always has a KIND followed by an INTENSITY, we could shortcut that to:

start on weather rain heavy

If you’ve used Upstart at all, you’ve seen that shortcut before.  A lot.  You may not have even realised it was a shortcut at all, and that’s what I hope to fix here.

Here’s an example of where you’ve used that:

start on started dbus

You should hopefully now recognise that started is the name of the event there, an dbus is simply the value of its first argument, whatever that might be.  Remember I mentioned that events have man pages?  Take a look at man 7 started, which is the man page for this event.

It documents which environment variables are attached to the started event, and most importantly what order they come in.

started JOB=JOB INSTANCE=INSTANCE [ENV]...

So really when we wrote the previous, we were just using a shortcut to specify:

start on started JOB=dbus

You might wonder what difference this makes.  A good example of how to exploit this is the stopped event.  If you look at it’s man page (man 7 stopped) you’ll see it has a large number of environment variables specifying not only which job stopped but the reason for it stopping.  One of those is the exit signal, for example.

Now you know that you’re just matching the $JOB environment variable, it’s obvious that you don’t have to!  You can match any other environment variable or variables in the event, or none at all.

Here’s how to run a script if any other job on the system exits with a segmentation fault:

start on stopped EXIT_SIGNAL=SEGV

I said you didn’t have to match any variables, just like in the first examples we didn’t, there’s a neat use for that with the job events.  The starting event blocks the named job from actually starting until anything run by it is started; or, in the case of jobs marked task, finished.

Here’s a little job that runs every time another job is started, and blocks that job from actually starting until the script finishes.

start on starting
task

script
    ....
end script

Useful both for debugging and performance analysis.

Now for the really neat bit.  So far we’ve concentrated on the environment variables that come from events, and those that Upstart puts into the job events.  But we can influence these in rather useful ways.

Firstly we can declare a default value for an environment variable in a job, if no alternate value is given in the start event or command, then this default value wins:

start on mounted

env MOUNTPOINT=/tmp
script
    ....
end script

This script will run for each occurrence of the mounted event, and will hopefully get the value for $MOUNTPOINT from that event.  But should the value be missing from the event, or the script be started manually by a system administrator, a default value is provided.

This isn’t a false example, that’s from the job on your system that cleans up the /tmp directory on boot.  The default value wasn’t there in earlier versions of Ubuntu, and this had a rather disastrous side-effect when run by hand.

Ok, we can set the values of environment variables from a job, and we don’t have to match the job name in the usual job events.  We can combine these two facts in a very interesting way when we can export the value of a job’s environment variable into its job events.

Here’s the first job:

env AM_A_DISPLAY_MANAGER=1
export AM_A_DISPLAY_MANAGER

This sets the default value of $AM_A_DISPLAY_MANAGER, but this isn’t a variable we ever expect to be supplied by an event so it just gets passed into the environment of its processes.  It’s not that useful either on its own.

The export line is the useful one, it adds the value of the named environment variable to the job’s events.  That is the starting, started, stopping and stopped events.

Now, in another job, we can do:

start on started AM_A_DISPLAY_MANAGER=1

This is run when any job is started that has that environment variable in its events.  In other words, we can tag classes of services so we don’t have to list every single one.

And because everything in Upstart is the same fundamental type of thing, this can work in the opposite direction.  For example we can put in our job:

env NEED_PORTMAP=1
export NEED_PORTMAP

This means our events will have NEED_PORTMAP=1 in them, now remembering that the job waits for the side-effects of the starting event to complete, we can now write in /etc/init/portmap.conf:

start on starting NEED_PORTMAP=1

So we can implement a dependency-based init system with Upstart, an event-based init system.

I look forwards to finding out what else you can do with it.

(reposted from http://upstart.at/2010/12/03/event-matching-in-upstart/ – post comments there)