With the recent announcement of systemd, I’ve noticed some increased confusion around Upstart and what it means to be an event-based init daemon. Now seems as good a time as any to try and clear that up by describing what I mean by that.
Dependency-based init
Before Upstart came along, the state of the art of init daemon replacements were the dependency-based init daemons. The two most well-known at the time was the Service Management Facility (SMF) of Solaris, and initng on Linux.
The easiest way to understand how a dependency-based init daemon works is to look at another dependency-based system you’re probably more familiar with: the package manager of your Linux distribution.
When you want to install a package, for example the Apache Web Server, you tell the package manager to do that. The Apache package will list additional dependencies that it requires to be installed, and those in turn will list additional dependencies, and so on. The package manager will walk this dependency tree, eliminating those that you already have installed, and it will then flatten the remaining tree to get an order in which those remaining can be safely installed.
To put it simply: you say that you want Apache installed, but you may get more than that installed to ensure that Apache works.
A dependency-based init daemon works in fundamentally the same way. When you say that you want Apache started, it looks at the configuration for that service for the list of dependency services, and builds up a similar tree. Eliminating those already running, and flattening the tree, gives you a list of services that must be started in an order that they should be safe to start in.
You say you want Apache running, but you may get more than Apache running as a result.
Booting a system with a dependency-based init daemon, however, is a little strange. They need to know the target set of services that must be running, otherwise they would start nothing. SMF simply started all services that were not in manual start mode, initng had the concept of goal services whose dependencies were those that should be running — and used these to define the runlevels.
Once you have that list of goal services, you work out the dependency trees, and flatten them as normal – and thus you get an order that all services on the system should be started in.
Dependency-based init daemons work, but I believed there was a better way to do things. I invented the event-based init daemon instead.
Event-based init
An event-based init daemon isn’t really a great leap from a dependency-based init daemon, it simply does everything backwards. A simplistic view says that instead of starting Apache’s dependencies because Apache is started, it starts Apache because its dependencies are now running.
But it’s much more interesting than that, and much more flexible. Most people don’t get the epiphany.
A better description might be that services are started and stopped due to external influences on them. Those external influences can be anything, for example: hardware coming and going; changes in the time; and not least, other services.
The events represent changes in the system state, and services define the states in which they can be running, and the system reacts accordingly.
I’m still convinced this is the best way to work, not in the least because you can implement a dependency-based system with an event-based init daemon. Starting a service causes an event for each of its dependencies declaring a need for them, and the service waits for those events to complete; those events cause the dependencies to be started.
launchd
The other well-known init daemon out there is Apple’s launchd, of which Lennart’s recent systemd project is similar implementation in some ways but not in others.
launchd’s modus operandi is that it starts services on demand, and it does this on the assumption that all services communicate through sockets or through the Mach IPC model. For the socket-based services, launchd itself creates the listening sockets, and when it receives a connection it starts the service and hands off the listening socket to it.
This has a beautiful engineering elegance, and it’s easy to see why it appeals to us.
You don’t need to configure a service’s dependencies or requirements in the init daemon, instead the service causes its dependencies to be started through this on-demand activation. If the dependency isn’t ready to be started, the service simply blocks in the connect or open syscall until it is ready.
As launchd as matured, Apple have added support to watch for files on the disk and for cron-like schedule events. In many ways, this makes launchd kinda like an event-based init daemon, except with listening sockets.
systemd takes a similar approach with regard to the listening sockets, though my understanding so far is that it combines it with a dependency-based resolution procedure for other parts of the system, rather than an event-based one. I’m willing to be corrected on this though.
Upstart
Upstart is an event-based init daemon; it’s taken a little while to develop because it’s the first pure example of its kind, and I only replaced the working sysvinit cautiously. I basically had to prove to myself, and others, that an event-based init daemon can really work. That’s why Ubuntu 9.10 and 10.04 were the first versions to really start taking advantage of it.
I also wanted to keep it relatively stable to encourage adoption by other distributions, and I believe this has also paid off given that Fedora, RedHat and OpenSuSE have all adopted it now.
I’ve proven it works, and it’s been adopted, now the fun development can begin!
Two of the main complains about Upstart are that the start on and stop on mechanism to define services is complicated and exposes far too much of the event model, and that it’s not very well documented. Ironically, these two complaints are entirely related.
The start on/stop on mechanism is basically just a debug interface, it allowed me during early development to access the raw event queue and find out what types of service model we really needed. Since it’s a debug interface, it wasn’t documented; I knew that future versions of Upstart would have a much better model.
So to correct a common misconception, the hideous start on lines are not a side-effect of event-based init daemons; they’re a side-effect of developing an event-based init daemon in a release early open-source way.
I’ve also mentioned that events can be just about anything, not just directly from other services. This includes on-demand activation; I don’t see any reason why Upstart should not be able to create sockets as launchd does, a connection on those sockets would simply be an event that would cause a service to be started.
Likewise, I fully intend Upstart to take over activation of system and session bus services from D-Bus, using an event from the D-Bus daemon to start and manage the service on its behalf.
This latter example neatly illustrates how start on will be replaced. Take a system bus service, you might declare such a service like this:
dbus system-bus org.freedesktop.UDisks
exec /usr/lib/udisks-daemonThat initial line replaces a whole slew of previous verbs. It tells Upstart that this service should be activated from the D-Bus system bus when a message for the given name has no destination in the bus. It also tells Upstart that this service should not be considered “ready” until it actually registers that name on the bus.
Finally it tells Upstart that the service can only be run while the D-Bus system bus service is running. You might think this superfluous, but remember from above that an event-based init daemon can work both ways; starting this service manually as a system administrator would start the message bus for you, if it wasn’t already running. This can be done with either an event or through the service connecting to the message bus via a known socket.
It’s this flexibility that still leaves me convinced that Upstart is a better all-round approach than the purity of launchd (or systemd).
Take another service, for example, the printing service: CUPS. At first glance, you might believe that it can be on-demand activated when something connects to its socket.
And that would certainly appear to work, you’d click Print in an application and the printer service would be started.
But that’s not the full picture; what if there was a job in the queue from before you shut down? You also need the service started if there are any files in the named queue directory.
And that’s still not the full picture; CUPS performs remote printer discovery, you most certainly don’t want to click Print and see no printers because CUPS hasn’t had time to discover them, having only just been started. Users have short attention spans to wait, I know I certainly do.
You need a combination of different conditions to start CUPS; it should be started on demand, it should be started if there are files in the print queue, and it should be still started on boot (just low-priority once the system is idle) to discover remote printers.
A pure on-demand daemon just doesn’t cut it, you need something more flexible.
The last point about user impatience is also my other major disagreement here. launchd supposes that you should always optimise for the minimum system footprint, at a cost to interaction performance.
It assumes that it’s ok to wait for a service to start when you click a button the first time, or bogusly that all services start immediately!
While this might be true in many situations, it’s also not true in many others. I’ve met very few system administrators who think that their web server should only ever be started on demand, and shut down again once there are no users browsing it.
And if you’re going to do always-running services like this, you do need to be able to encode their dependencies and requirements in the init-daemon configuration, which negates the engineering precision of avoiding doing so through on-demand activation.
Nice post. So far I still haven’t fully grasped the reason/need for systemd, even after reading Leonards post. IMHO it would’ve been better to improve/extend upstart.
You mentioned that an event-based system can implement a dependency-based system. Can it also implement an on-demand system? Could you provide the socket-handoff that systemd/launchd do? How useful would that be?
@Ted: yes; I was attempting to illustrate that point in the blog post but it’s worth making explicit.
I think it’s useful and desirable for Upstart to support on-demand socket activation like launchd does, and such things are simply a new type of event that starts services (albeit with a file descriptor payload).
So… does upstart have a commandline utility to add/remove services from runlevels yet, or should we wait another 3 years for it?
Thank you for the very instructive post. It helped me learn more about both upstart and systemd and how they compare than the lenghty post by Lennart did. Much appreciated. I look forward to have socket based activation of services in upstart, while xinetd can do some of that today, it cannot, for instance, take the services down on certain events.
I also agree that while you only want to start some services on demand (eg the ssh server on my laptop that I use once a year), others need to be started as soon as possible, you pointed out cups, and avahi, I’d add ntpd, NetworkManager.
I recently wrote an upstart job definition to setup a cryptographic /tmp and /var/log. Because it sets up /var/log, the job must be started very early. Initially, “start on starting rsyslog” seemed pretty elegant.
I quickly fell down the rabbit hole of spotting services that (independently of rsyslog) wanted to write to /var/log and eventually the “start on” line got very ugly indeed. It currently looks like:
start on (stopped cryptdisks-enable and starting rsyslog and starting dmesg and starting udev-finish and starting networking and starting module-init-tools and starting rc RUNLEVEL=[2345])
The problem with this model is that I have to explicitly list all possible events that might touch /tmp or /var/log. This means I can’t create my job conf file in a generic way, rather it must be custom tailored to the system it’s on, because if I explicitly list an event for some service that’s not installed, mine will never get executed. It happens that the jobs I ended up listing were installed by default, but as sysvinit scripts start getting rewritten as upstart jobs, that may not be the case.
The alternative of explicitly modifying all existing jobs to be aware of mine is obviously infeasible.
Ideally what I’d like is for some way to prevent parallelism of my job. Something like: “start immediately after / is mounted rw and do not allow anything that writes to /tmp or /var/log to execute until I’m finished.” In practice (because there’s no reliable way for Upstart to know just what is going to write to /tmp or /var/log), this almost certainly means to forbid _all_ parallelization with my job.
Anyway, my use-case might not be that common, but I thought I’d share some of my pains with my first exposure to Upstart, and possibly provide some insight into potential future features.
Thanks!
Also, another possibility: explicitly parallel startup of several jobs.
For example, Avahi can be started as soon as DBUS socket is created, no need to wait until it’s initialized. Ditto for some other services.
Also, idea of using cgroups to confine daemons might be worth looking into.
The CUPS remote supposed problem is not really a problem, you can have the system to listen at 631 and dynamically open it, just like inetd.
The spool case is something valid, but on the other hand there is no point to always run cupsd to check it… as 99.9999% of users will not have that case. I’d just add a spool-checker service that would activate the required servers, doing nothing on vast majority of the cases.
For DBus, yes… you just reinvented the activation system inside Upstart. Not a new idea, or a flawed concept as I remember you talking about it with Lennart and Marcel at some UDS. I quite of like the idea, but we better keep with existing .service descriptions as you gain not much by providing a new syntax just for Upstart. However I fail to understand how is it a problem with systemd, it could do exactly the same as you suggest.
@Gustavo that only works If the printers announce, which most don’t – and still leaves you with an awful delay of starting CUPS when the user actually hits print
Also the idea is to have the initial daemon look in the spool directory for the files, and only start the service if they are there
Most do not see the beauty about systemd.
It lets the kernel to handle synchronization and queuing and starts a bunch of daemons together in parallel.
It doesn’t need to synchronize dependent daemons.
Let’s say we have 3 daemons, A B and C.
A needs B, and both A and B needs C.
So what systemd does is to first create the sockets of the 3 daemons and than start all the three daemons simultaneously and just let the kernel to queue up requests until the daemons are up and running.
Pingback: Linux News » Dependency-based & Event-based init daemons and launchd
I want “the system” to do what it does very well: pay attention to minute details of dependency and resource allocation and accomplish repetitive and error-prone tasks. I want to devote my attention and efforts to “big picture” aspects of getting my selected tasks completed rather than working as mentat [aka, human computer] sorting details.
To this end, I want a system start-up suite that takes a list of my preferences for big-picture needs like “printing” or “web server” and sorts this out using arcane knowledge from the relevant package developers and the idiosyncracies of the specific workstation in use.
I know that Upstart is flegling software new to life outside its creator’s nest. However, the rest of us need more visibility into what is going on. It might be useful if some data were available that could report what Upstart and friends does. The ‘bootchart’ tools do some of this. Perhaps Upstart and friends could offer more details that bootchart understands. A UML “sequence diagram” (see http://en.wikipedia.org/wiki/Sequence_diagram) or similar would be wonderful, but then I’m a geek.
~~~ 0;-Dan
FOLLOW-UP: When I said that I wanted the system to sort out details, I failed to mention that it should also communicate in a obvious, end-user way that one or more of my stated “big picture” preferences is not possible, potential reasons for this trouble, potential corrective actions to resolve this trouble, and so on. I would also like to see some sort of end-of-boot summary of things that happened and things that went wrong or that are unusual. For example, “network parts are running, but you cannot actually communicate with other hosts” (Yes, this is a problem that I have from time to time.)
This latter is an example of what I call and “installation validation process” or IVP. In our context, the event “network started” would start a job that somehow inspects (maybe other events?) network operations and discovers (maybe other jobs?) what is and is not working as desired. For this, Upstart fits the bill better than the rest.
~~~ 0;-Dan
A few comments:
First of all, you create the impression that systemd’s core design was about dependencies. Well, it is not. Dependencies are supported by systemd only for the purpose of early boot. Normal services should not really use that for much. One of the nice things in systemd is that the kernel will get the dependencies and ordering right for you, out of the box.
Also, it is misleading that the hw hotplug logic would only really make sense in an event-based systemd. But well, that’s not true. It fits really nicely in a system that does dependencies: for example, there is a dependency between “bluetooth dongle is plugged in” and “bluetoothd must be running”. Systems that support dependencies can express that dependency explicitly, while event-based systems cannot.
Did you never get the idea that maybe nobody else is using your event-based design, because it is simply broken? (For the reasons I pointed out in my original blog story.)
Also, claiming that launchd’s or systemd’s core design was around on-demand loading of services is misleading. While we support that too, we do socket-based activation mostly to parellelize boot-up and to get rid of explicitly configured dependencies. Only a few services will actually be started on-demand in a systemd system. Most will be started in any case, but fully in parallel due to the socket activation logic.
Yes, systemd does both socket-based activation and dependencies. Manually configured dependencies are however only really needed for early boot-up or late shut-down and a few other cases. launchd does not support dependencies at all, for the price that their early boot-up is completely seperate from the starting of daemons, while in systemd this can be interleaved quite a bit.
OpenSUSE has already given up all Upstart experiments btw. Right now only Ubuntu uses Upstart fully, and Fedora and RHEL use it in sysv compatbility mode only.
And generally I think it would be nice to talk more about stuff Upstart has actually already accomplished, instead of great plans. In my postings about systemd I kept the part about my plans very short. Almost all features I talked about in my various postings can be listed under “DONE”, not under “WILL BE DOING”.
Note that systemd supports quite a few triggers for starting a service. And they can be combined in various ways. For example, we can start CUPS as a dependency of “printer plugged in”, as well as a dependency of “client requests CUPS services”, as well as “there’s a file in the CUPS spool dir”. It’s all implemented, and works fine. CUPS fits perfectly in the systemd model, as all its triggers are available already. If one of the conditions holds, CUPS is started. I don’t see how the current Upstart could do anything like this… So, on this topic: systemd 1, upstart 0.
And finally, I don’t think you have really understood launchd or systemd. The focus is on using socket-activation for parallelizing and getting rid of explicit dependencies. It’s much less on on-demand loading.
@Saint DanBert: AFAIK an event log is part of the spec for the new version of Upstart. It’s also supposed to be machine readable, so that you can use tools to interpret them (e.g. create graphical “flow charts” of what happened). I assume that’s close to what you propose?
“An awful delay of starting CUPS when the user actually hits print”? Give me a break! The time needed to start cupsd on my system is less than 1% of the time that my printer needs to wake up and start printing.
@Michael Goetze “Give me a break! The time needed to start cupsd on my system is less than 1% of the time that my printer needs to wake up and start printing.”
If you only have local or statically configured printers, yes. If your printer is on the network and cups needs to first notice the printer’s announcements, the time from cups start up until the printer appears and can be selected is indeed quite long.
Well the child play between Scott and Lennart leaves me puzzled – please get back to real…
I can only tell from experience with upstart right now, systemd experience is pending.
Whatever is the theory behind … i need something
- which starts the application then certain partitions, network interface and specific devices are ready
- i need something which is starting when a network socket is available and also needs the sound subsystem as well as X to be ready
- only starts when A if B is not started
upstart fails in certain ways to fullfil this. If A is started, it doesnt mean that it provides the required service to B , the dependencies on all the requirements of a service don’t work.
It could maybe work out and i’m doing something wrong – then again it would have failed the proof that people can handle it, or it just doesn’t work in real life.
For systemd: Proof is pending that it actually works as advertized. I hope it has not to much in common with PA which in theory works nice – in real world it sucks (because of latency issues and not available simple pass through of audio, which both are basic requirement for audio).
Please both do a step back get in touch with people who will actually use it and doublecheck the requirements are met. Then lets see who will win (the usual way in OSS world i thought)
I instantly flip the bozo bit on anyone who claims to have a deep understanding of software and yet speaks English so terribly.
This is the 21st century. The old excuse of being from a remote corner of East Africa no longer washes. You either speak English or you don’t understand software well enough to comment.
Hi Guys
I’m a sysadmin who has been using upstart for a little while, for starting some of our in-house daemons. I deal mainly with servers not desktop, and these servers are in various roles from database servers, webservers, and application servers.
I’m slightly disappointed with upstart, I think it has much promise, but my niggles with it are mainly ‘polish’ related:-
an unordered list of requests and comments.
1) logging, I really want to know what upstart is doing, when, why, I want this in syslog, and I want to monitor it. I could not understand where console output of running daemons went (tried console and logger options). I want to control logging on demand for a service (troubeshooting a daemon that’s being killed but why?)
2) I want to run daemons as non root users, I do not want to use su -c ‘/our/app’ user hacks, I think this messes up the pid figuring out logic, really annoying as the pid file stanza does not seem to work.
3) resource limits are critical for me, an easy way to express them would be nice (rlimits, cgroups, all good things). The current limit stanza could do with a lot better docs. I want to know if a process has been killed as a result of them, see logging.
4) I want to easily put things into environment variables, I really hate the script blocks.
5) Puppet integration would be much appreciated (I know basic service started was added late in lucid, but proper start service stop service stuff so I can drop my exec hacks)
6) visibility into the upstart community, roadmap/todo, documentation, issue tracker, swap recipes with other admin, list of low hanging fruit that those with spare tuits can help out with …. (I want to know your rate of bug closing vs bug opening, how often are releases made how detailed are the changelogs, do commits go in bursts or is there sustained development pace, what gets discussed on the ml is it the devs talking do users have their own ml … etc)
I want to know when a new feature/bug fix is gonna make it into a release, and about when that release is gonna hit a mirror. Take a look at the puppet guys, they are very good at that.
I largely want processes supervision features, how quickly a server boots, how it gets there, and in what order is simply not important (as long as it does) to me. Cos all being well I should never have to directly experience it.
Hopefully some feedback from a real world user.
Gordon
Theoretically the concept behind the upstart may be sound, but in the praxis it is a nightmare. Before ubuntu 9.10 we never had at the boot-time unpredictable lock-ups with nfs, etc, etc. Now the boot process is both hard to understand and hard to configure. Actually the only viable solution for recent ubuntu was to switch back to dep. based init.
Years ago during my CS Master’s studies, I was taught that “system startup” was a layering: hardware power on –> various hardware resets –> initial program load (primary boot) –> kernel load (secondary boot) –> drivers take control of hardware –> kernel takes control of drivers –> services expose kernel and devices to other software (including the kernel and drivers) –> application initialization enables end-user processing.
This simplistic description does little justice to the 1000-cuts that a modern workstation (even a cheap one) goes through. I support the event-driven approach that is Upstart. Reality is much more like the conference hall filled with mouse traps and ping-pong balls than we all want to admit. The world is analog and messy that way. That said, something is missing.
It seems that a system start needs must have some clear sequencing and synchronism. In addition, the end-user admin needs tools that expose what is going on with all of those bouncing balls.
Upstart Team: Keep up the good work. Put new-feature work on hold until you have addressed some of the end-user aspects. You may be able to read dozens or more config files and keep the details in your head that knows Upstart inside and out. We need some tools to see likely results of what we have configured so that we might predict what is likely to happen and compare that prediction against our desires. We then need tools to show us what actually happened during a system start and subsequent init-centered processing.
~~~ 0;-Dan
Years ago during my time as CS Professor at MIT, we would frown upon people who would start a comment by citing their credentials.
Pingback: Upstart und sysVinit - Linux: Linux-Forum
Pingback: FOSDEM 2011 Highlights: systemd « /Users/dan/proj
Happened upon this blog post again. Despite the overly negative comments above, I’d like to say I really enjoyed reading this and I think you made a number of very good points.