daemon

Overview ¶

Package daemon provides utilities for programs that need to fork themselves into the background (for instance, a persistent server). Its primary function, MakeDaemonizable, adds the --daemonize option to a cobra.Command.

Usage ¶

These utilities are designed for use with cobra.Command. Clients of the package need to do five things:

1. Provide a boolean variable (we'll call it shouldDaemonize, though you can name it whatever you want) which will store whether the current process is the child server process. (If you're familiar with the fork() syscall, think of this as its zero return value.)

2. Call MakeDaemonizable(), which adds the --daemonize option to the command and sets up the use of the shouldDaemonize flag.

3. During the command's Run() function, if the shouldDaemonize flag has been set, call Daemonize() after the server has fully initialized and is ready to receive requests. This will disconnect the standard streams and signal the original process to exit.

4. Handle the ErrSuccessfullyDaemonized pseudo-error if it is returned from the command's Execute() method. This is an indication that the child has correctly started, and the parent should now exit with a zero exit code.

5. Write to standard error from the child server process if, and only if, it is about to terminate without calling Daemonize(). (For an explanation of this limitation, see "Implementation" below.)

All stdout/stderr contents emitted by the server during initialization will be bubbled up to the caller of the original binary, so just use those streams as you normally would.

Here's an example:

var shouldDaemonize bool
cmd := &cobra.Command{
	RunE: func(cmd *cobra.Command, args []string) error {
		err := listenOnServerPort()
		if err != nil {
			// Cobra will write an error message to stderr for us.
			return err
		}

		if shouldDaemonize {
			fmt.Println("started successfully; now daemonizing")
			daemon.Daemonize() // disconnect standard streams
		}

		return serverMainLoop()
	},
}

daemon.MakeDaemonizable(cmd, &shouldDaemonize) // add --daemonize

err := cmd.Execute()
if err != nil {
	if err == daemon.ErrSuccessfullyDaemonized {
		// not actually an "error", just exit cleanly
	} else {
		// handle actual error
		...
	}
}

Telling this example server to daemonize is easy:

$ example_server --daemonize
started successfully; now daemonizing

$ # at this point, the exit code indicates success/failure of server startup

We can also execute synchronously, just like before:

$ example_server
# ... serves until we Ctrl-C out

MakeDaemonizable() preempts the PreRun/PreRunE functions of the cobra.Command it is passed. If --daemonize is set, the parent process will only execute the PersistentPreRun[E] steps; ErrSuccessfullyDaemonized will be returned from the PreRun step, which will stop further steps from executing. In the child process, and during execution without the --daemonize option, all steps run as usual.

Motivation ¶

This package helps solve a common race condition when initializing systems that have servers and dependent clients. Consider the following setup, where a client starts a server that it will then make a request to:

Client
|
| ------- start server ------> Server
| ------- make request ------> | ?
|                              |

This situation has a race -- there's no guarantee that the server will have initialized to the point that it is even listening for requests by the time the client makes one. A common workaround is to tell the client to sleep, to "give the server time to come up":

Client
|
| ------- start server ------> Server
| sleeping...                  |
| sleeping...                  |  initializing...
| sleeping...                  |
| sleeping...                  |- listening
| sleeping...                  |
| ------- make request ------> |
|                              |

Not only does this not solve the actual race, but it tends to inject unnecessary latency as the authors try to tune the sleep time to the slowest known environment. A better solution is to periodically poll the server during startup, but a robust client then also needs some way to poll for early/abnormal server exit so that it doesn't wait too long for a server that never actually started.

For a distributed system, these problems come with the domain, but for cases in which the client and server are on the same machine, we can do better. To get rid of the race, we need the server to be able to signal the child as soon as it is ready to listen, before it enters its main loop:

Client
|
| ------- start server ------> Server
| waiting for signal...        |
|                              |  initializing...
|                              |
| <------------- ready ------- |- listening
| ------- make request ------> |
| <---------- response ------- |
|                              |

This way, the server is guaranteed to either 1) be listening or 2) have terminated unexpectedly by the time the client's request is made, and error handling is greatly simplified.

Implementation ¶

The daemon package implements this signaling with standard POSIX streams and process exit codes, so no special client libraries are required. An intermediate "server parent" process is introduced so that the client can easily wait for that process to exit before continuing. (The server process itself is supposed to persist after creation, so the client can't wait for it to exit.) If the parent process exits with a non-zero code, then the server has failed to launch; otherwise, it has been forked into the background and is ready to receive requests.

Child-to-parent signaling is implemented using the child's stdout/stderr streams. When both are closed (either deliberately or through an abnormal exit), the parent will continue. If any stderr contents are found, the parent will then wait for an exit code from the child. Otherwise, the parent will exit cleanly and allow the child to continue serving.

(This magic use of stderr does introduce one complication: the child server process must write to stderr *if and only if* it terminates abnormally before it is able to process requests.)

The full system looks like this:

Client
|
| ---- execute server -----> Server Parent
| waiting for process...     |
|                            | ---- start child -------> Server
|                            | waiting for child...      |
|                            |                           |  initializing...
|                            |                           |
|                            | <----- close streams ---- |- listening
| <---------- exit code ---- x parent exits              |
|                                                        |
| ----- make request ----------------------------------> |
| <-------------------------------------- response ----- |
|                                                        |