cmd: github.com/cybozu-go/cmd Index | Examples | Files

package well

import "github.com/cybozu-go/cmd"

Package well provides a framework that helps implementation of commands having these features:

Better logging:

By using github.com/cybozu-go/log package, logs can be structured in JSON or logfmt format. HTTP servers log accesses automatically.

Graceful exit:

The framework provides functions to manage goroutines and network server implementation that can be shutdown gracefully.

Signal handlers:

The framework installs SIGINT/SIGTERM signal handlers for graceful exit, and SIGUSR1 signal handler to reopen log files.

Environment

Environment is the heart of the framework. It provides a base context.Context that will be canceled before program stops, and methods to manage goroutines.

To use the framework easily, the framework provides an instance of Environment as the default, and functions to work with it.

The most basic usage of the framework.

Code:

flag.Parse()
err := well.LogConfig{}.Apply()
if err != nil {
    log.ErrorExit(err)
}

well.Go(func(ctx context.Context) error {
    err := doSomething()

    if err != nil {
        // non-nil error will be passed to Cancel
        // by the framework.
        return err
    }

    // on success, nil should be returned.
    return nil
})

// some more Go
//well.Go(func(ctx context.Context) error {})

// Stop declares no Go calls will be made from this point.
// Calling Stop is optional if Cancel is guaranteed to be called
// at some point.
well.Stop()

// Wait waits for all goroutines started by Go to complete,
// or one of such goroutine returns non-nil error.
err = well.Wait()
if err != nil {
    log.ErrorExit(err)
}

HTTP server that stops gracefully.

Code:

flag.Parse() // must precedes LogConfig.Apply
err := well.LogConfig{}.Apply()
if err != nil {
    log.ErrorExit(err)
}

// log accesses in JSON Lines format.
accessLog := log.NewLogger()
accessLog.SetFormatter(log.JSONFormat{})

// HTTP server.
serv := &well.HTTPServer{
    Server: &http.Server{
        Handler: http.FileServer(http.Dir("/path/to/directory")),
    },
    AccessLog: accessLog,
}

// ListenAndServe is overridden to start a goroutine by well.Go.
err = serv.ListenAndServe()
if err != nil {
    log.ErrorExit(err)
}

// Wait waits for SIGINT or SIGTERM.
// In this case, well.Stop can be omitted.
err = well.Wait()

// Use IsSignaled to determine err is the result of a signal.
if err != nil && !well.IsSignaled(err) {
    log.ErrorExit(err)
}

Index

Examples

Package Files

default.go doc.go env.go exec.go graceful.go graceful_unix.go http.go idgen.go jsonlog.go log.go logfile.go reqid.go server.go signal.go sigpipe.go stopsig.go systemd.go

Constants

const (
    // RequestIDContextKey is a context key for request ID.
    RequestIDContextKey contextKey = "request_id"
)

func BackgroundWithID Uses

func BackgroundWithID(ctx context.Context) context.Context

BackgroundWithID returns a new background context with an existing request ID in ctx, if any.

func Cancel Uses

func Cancel(err error) bool

Cancel cancels the base context of the global environment.

Passed err will be returned by Wait(). Once canceled, Go() will not start new goroutines.

Note that calling Cancel(nil) is perfectly valid. Unlike Stop(), Cancel(nil) cancels the base context and can gracefully stop goroutines started by Server.Serve or HTTPServer.ListenAndServe.

This returns true if the caller is the first that calls Cancel. For second and later calls, Cancel does nothing and returns false.

func FieldsFromContext Uses

func FieldsFromContext(ctx context.Context) map[string]interface{}

FieldsFromContext returns a map of fields containing context information. Currently, request ID field is included, if any.

func GenerateID Uses

func GenerateID() string

GenerateID genereates an ID using the default generator. Multiple goroutines can safely call this.

func Go Uses

func Go(f func(ctx context.Context) error)

Go starts a goroutine that executes f in the global environment.

f takes a drived context from the base context. The context will be canceled when f returns.

Goroutines started by this function will be waited for by Wait until all such goroutines return.

If f returns non-nil error, Cancel is called immediately with that error.

f should watch ctx.Done() channel and return quickly when the channel is closed.

func GoWithID Uses

func GoWithID(f func(ctx context.Context) error)

GoWithID calls Go with a context having a new request tracking ID.

func IsSignaled Uses

func IsSignaled(err error) bool

IsSignaled returns true if err returned by Wait indicates that the program has received SIGINT or SIGTERM.

func IsSystemdService Uses

func IsSystemdService() bool

IsSystemdService returns true if the program runs as a systemd service.

func Stop Uses

func Stop()

Stop just declares no further Go will be called.

Calling Stop is optional if and only if Cancel is guaranteed to be called at some point. For instance, if the program runs until SIGINT or SIGTERM, Stop is optional.

func SystemdListeners Uses

func SystemdListeners() ([]net.Listener, error)

SystemdListeners returns listeners from systemd socket activation.

func UTF8StringFromBytes Uses

func UTF8StringFromBytes(b []byte) string

UTF8StringFromBytes returns a valid UTF-8 string from maybe invalid slice of bytes.

func Wait Uses

func Wait() error

Wait waits for Stop or Cancel, and for all goroutines started by Go to finish.

The returned err is the one passed to Cancel, or nil. err can be tested by IsSignaled to determine whether the program got SIGINT or SIGTERM.

func WithRequestID Uses

func WithRequestID(ctx context.Context, reqid string) context.Context

WithRequestID returns a new context with a request ID as a value.

type AccessLog Uses

type AccessLog struct {
    Topic    string    `json:"topic"`
    LoggedAt time.Time `json:"logged_at"`
    Severity string    `json:"severity"`
    Utsname  string    `json:"utsname"`
    Message  string    `json:"message"`

    Type           string  `json:"type"`             // "access"
    Elapsed        float64 `json:"response_time"`    // floating point number of seconds.
    Protocol       string  `json:"protocol"`         // "HTTP/1.1" or alike
    StatusCode     int     `json:"http_status_code"` // 200, 404, ...
    Method         string  `json:"http_method"`
    RequestURI     string  `json:"url"`
    Host           string  `json:"http_host"`
    RequestLength  int64   `json:"request_size"`
    ResponseLength int64   `json:"response_size"`
    RemoteAddr     string  `json:"remote_ipaddr"`
    UserAgent      string  `json:"http_user_agent"`
    RequestID      string  `json:"request_id"`
}

AccessLog is to decode access log records from HTTPServer. The struct is tagged for JSON format.

type Environment Uses

type Environment struct {
    // contains filtered or unexported fields
}

Environment implements context-based goroutine management.

func NewEnvironment Uses

func NewEnvironment(ctx context.Context) *Environment

NewEnvironment creates a new Environment.

This does *not* install signal handlers for SIGINT/SIGTERM for new environments. Only the global environment will be canceled on these signals.

Barrier wait for gorutines.

Code:

// An independent environment.
env := well.NewEnvironment(context.Background())

env.Go(func(ctx context.Context) error {
    // do something
    return nil
})
// some more env.Go()

// wait all goroutines started by env.Go().
// These goroutines may also be canceled when the global
// environment is canceled.
env.Stop()
err := env.Wait()
if err != nil {
    log.ErrorExit(err)
}

// another environment for another barrier.
env = well.NewEnvironment(context.Background())

// env.Go, env.Stop, and env.Wait again.

func (*Environment) Cancel Uses

func (e *Environment) Cancel(err error) bool

Cancel cancels the base context.

Passed err will be returned by Wait(). Once canceled, Go() will not start new goroutines.

Note that calling Cancel(nil) is perfectly valid. Unlike Stop(), Cancel(nil) cancels the base context and can gracefully stop goroutines started by Server.Serve or HTTPServer.ListenAndServe.

This returns true if the caller is the first that calls Cancel. For second and later calls, Cancel does nothing and returns false.

func (*Environment) Go Uses

func (e *Environment) Go(f func(ctx context.Context) error)

Go starts a goroutine that executes f.

f takes a drived context from the base context. The context will be canceled when f returns.

Goroutines started by this function will be waited for by Wait until all such goroutines return.

If f returns non-nil error, Cancel is called immediately with that error.

f should watch ctx.Done() channel and return quickly when the channel is closed.

func (*Environment) GoWithID Uses

func (e *Environment) GoWithID(f func(ctx context.Context) error)

GoWithID calls Go with a context having a new request tracking ID.

func (*Environment) Stop Uses

func (e *Environment) Stop()

Stop just declares no further Go will be called.

Calling Stop is optional if and only if Cancel is guaranteed to be called at some point. For instance, if the program runs until SIGINT or SIGTERM, Stop is optional.

func (*Environment) Wait Uses

func (e *Environment) Wait() error

Wait waits for Stop or Cancel, and for all goroutines started by Go to finish.

The returned err is the one passed to Cancel, or nil. err can be tested by IsSignaled to determine whether the program got SIGINT or SIGTERM.

type ExecLog Uses

type ExecLog struct {
    Topic    string    `json:"topic"`
    LoggedAt time.Time `json:"logged_at"`
    Severity string    `json:"severity"` // "error" if exec failed.
    Utsname  string    `json:"utsname"`
    Message  string    `json:"message"`

    Type      string   `json:"type"`          // "exec"
    Elapsed   float64  `json:"response_time"` // floating point number of seconds.
    Command   string   `json:"command"`
    Args      []string `json:"args"`
    RequestID string   `json:"request_id"`
    Error     string   `json:"error"`
    Stderr    string   `json:"stderr"`
}

ExecLog is a struct to decode command execution log from LogCmd. The struct is tagged for JSON format.

type Graceful Uses

type Graceful struct {
    // Listen is a function to create listening sockets.
    // This function is called in the master process.
    Listen func() ([]net.Listener, error)

    // Serve is a function to accept connections from listeners.
    // This function is called in child processes.
    // In case of errors, use os.Exit to exit.
    Serve func(listeners []net.Listener)

    // ExitTimeout is duration before Run gives up waiting for
    // a child to exit.  Zero disables timeout.
    ExitTimeout time.Duration

    // Env is the environment for the master process.
    // If nil, the global environment is used.
    Env *Environment
}

Graceful is a struct to implement graceful restart servers.

On Windows, this is just a dummy to make porting easy.

func (*Graceful) Run Uses

func (g *Graceful) Run()

Run runs the graceful restarting server.

If this is the master process, Run starts a child process, and installs SIGHUP handler to restarts the child process.

If this is a child process, Run simply calls g.Serve.

Run returns immediately in the master process, and never returns in the child process.

type HTTPClient Uses

type HTTPClient struct {
    *http.Client

    // Severity is used to log successful requests.
    //
    // Zero suppresses logging.  Valid values are one of
    // log.LvDebug, log.LvInfo, and so on.
    //
    // Errors are always logged with log.LvError.
    Severity int

    // Logger for HTTP request.  If nil, the default logger is used.
    Logger *log.Logger
}

HTTPClient is a thin wrapper for *http.Client.

This overrides Do method to add the request tracking header if the passed request's context brings a request tracking ID. Do also records the request log to Logger.

Do not use Get/Head/Post/PostForm. They panics.

func (*HTTPClient) Do Uses

func (c *HTTPClient) Do(req *http.Request) (*http.Response, error)

Do overrides http.Client.Do.

req's context should have been set by http.Request.WithContext for request tracking and context-based cancelation.

func (*HTTPClient) Get Uses

func (c *HTTPClient) Get(url string) (*http.Response, error)

Get panics.

func (*HTTPClient) Head Uses

func (c *HTTPClient) Head(url string) (*http.Response, error)

Head panics.

func (*HTTPClient) Post Uses

func (c *HTTPClient) Post(url, bodyType string, body io.Reader) (*http.Response, error)

Post panics.

func (*HTTPClient) PostForm Uses

func (c *HTTPClient) PostForm(url string, data url.Values) (*http.Response, error)

PostForm panics.

type HTTPServer Uses

type HTTPServer struct {
    *http.Server

    // AccessLog is a logger for access logs.
    // If this is nil, the default logger is used.
    AccessLog *log.Logger

    // ShutdownTimeout is the maximum duration the server waits for
    // all connections to be closed before shutdown.
    //
    // Zero duration disables timeout.
    ShutdownTimeout time.Duration

    // Env is the environment where this server runs.
    //
    // The global environment is used if Env is nil.
    Env *Environment
    // contains filtered or unexported fields
}

HTTPServer is a wrapper for http.Server.

This struct overrides Serve and ListenAndServe* methods.

http.Server members are replaced as following:

- Handler is replaced with a wrapper handler.
- ReadTimeout is set to 30 seconds if it is zero.
- ConnState is replaced with the one provided by the framework.

func (*HTTPServer) ListenAndServe Uses

func (s *HTTPServer) ListenAndServe() error

ListenAndServe overrides http.Server's method.

Unlike the original, this method returns immediately just after starting a goroutine to accept connections. To stop listening, call the environment's Cancel.

ListenAndServe returns non-nil error if and only if net.Listen failed.

func (*HTTPServer) ListenAndServeTLS Uses

func (s *HTTPServer) ListenAndServeTLS(certFile, keyFile string) error

ListenAndServeTLS overrides http.Server's method.

Unlike the original, this method returns immediately just after starting a goroutine to accept connections. To stop listening, call the environment's Cancel.

Another difference from the original is that certFile and keyFile must be specified. If not, configure http.Server.TLSConfig manually and use Serve().

HTTP/2 is always enabled.

ListenAndServeTLS returns non-nil error if net.Listen failed or failed to load certificate files.

func (*HTTPServer) Serve Uses

func (s *HTTPServer) Serve(l net.Listener) error

Serve overrides http.Server's Serve method.

Unlike the original, this method returns immediately just after starting a goroutine to accept connections.

The framework automatically closes l when the environment's Cancel is called.

Serve always returns nil.

func (*HTTPServer) ServeHTTP Uses

func (s *HTTPServer) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP implements http.Handler interface.

func (*HTTPServer) TimedOut Uses

func (s *HTTPServer) TimedOut() bool

TimedOut returns true if the server shut down before all connections got closed.

type IDGenerator Uses

type IDGenerator struct {
    // contains filtered or unexported fields
}

IDGenerator generates ID suitable for request tracking.

func NewIDGenerator Uses

func NewIDGenerator() *IDGenerator

NewIDGenerator creates a new IDGenerator.

func (*IDGenerator) Generate Uses

func (g *IDGenerator) Generate() string

Generate generates an ID. Multiple goroutines can safely call this.

type LogCmd Uses

type LogCmd struct {
    *exec.Cmd

    // Severity is used to log successful requests.
    //
    // Zero suppresses logging.  Valid values are one of
    // log.LvDebug, log.LvInfo, and so on.
    //
    // Errors are always logged with log.LvError.
    Severity int

    // Fields is passed to Logger as log fields.
    Fields map[string]interface{}

    // Logger for execution results.  If nil, the default logger is used.
    Logger *log.Logger
}

LogCmd is a wrapper for *exec.Cmd to record command execution results. If command fails, log level will be log.LvError. If command succeeds, log level will be log.LvInfo.

In most cases, use CommandContext function to prepare LogCmd.

func CommandContext Uses

func CommandContext(ctx context.Context, name string, args ...string) *LogCmd

CommandContext is similar to exec.CommandContext, but returns *LogCmd with its Context set to ctx.

LogCmd.Severity is set to log.LvInfo.

LogCmd.Logger is left nil. If you want to use another logger, set it manually.

func (*LogCmd) CombinedOutput Uses

func (c *LogCmd) CombinedOutput() ([]byte, error)

CombinedOutput overrides exec.Cmd.CombinedOutput to record the result.

func (*LogCmd) Output Uses

func (c *LogCmd) Output() ([]byte, error)

Output overrides exec.Cmd.Output to record the result. If Cmd.Stderr is nil, Output logs outputs to stderr as well.

func (*LogCmd) Run Uses

func (c *LogCmd) Run() error

Run overrides exec.Cmd.Run to record the result. If both Cmd.Stdout and Cmd.Stderr are nil, this calls Output instead to log stderr.

func (*LogCmd) Wait Uses

func (c *LogCmd) Wait() error

Wait overrides exec.Cmd.Wait to record the result.

type LogConfig Uses

type LogConfig struct {
    Filename string `toml:"filename" json:"filename" yaml:"filename"`
    Level    string `toml:"level"    json:"level"    yaml:"level"`
    Format   string `toml:"format"   json:"format"   yaml:"format"`
}

LogConfig configures cybozu-go/log's default logger.

Filename, if not an empty string, specifies the output filename.

Level is the log threshold level name. Valid levels are "critical", "error", "warning", "info", and "debug". Empty string is treated as "info".

Format specifies log formatter to be used. Available formatters are "plain", "logfmt", and "json". Empty string is treated as "plain".

For details, see https://godoc.org/github.com/cybozu-go/log .

Load logging configurations from TOML file.

Code:

// compile-time defaults
config := &well.LogConfig{
    Level:  "error",
    Format: "json",
}

// load from TOML
_, err := toml.DecodeFile("/path/to/config.toml", config)
if err != nil {
    log.ErrorExit(err)
}

// Apply gives priority to command-line flags, if any.
flag.Parse()
err = config.Apply()
if err != nil {
    log.ErrorExit(err)
}

func (LogConfig) Apply Uses

func (c LogConfig) Apply() error

Apply applies configurations to the default logger.

Command-line flags take precedence over the struct member values.

When used with github.com/spf13/{pflag,viper}, pflag values are bound to viper database, and Apply look for following keys in the viper database:

- log.file
- log.level
- log.format

If they are not empty, they take precedence over the struct member values.

type RequestLog Uses

type RequestLog struct {
    Topic    string    `json:"topic"`
    LoggedAt time.Time `json:"logged_at"`
    Severity string    `json:"severity"` // "error" if request failed.
    Utsname  string    `json:"utsname"`
    Message  string    `json:"message"`

    Type         string    `json:"type"`             // "http"
    ResponseTime float64   `json:"response_time"`    // floating point number of seconds.
    StatusCode   int       `json:"http_status_code"` // 200, 404, 500, ...
    Method       string    `json:"http_method"`
    URLString    string    `json:"url"`
    StartAt      time.Time `json:"start_at"`
    RequestID    string    `json:"request_id"`
    Error        string    `json:"error"`
}

RequestLog is to decode request log from HTTPClient. The struct is tagged for JSON format.

type Server Uses

type Server struct {

    // Handler handles a connection.  This must not be nil.
    //
    // ctx is a derived context from the base context that will be
    // canceled when Handler returns.
    //
    // conn will be closed when Handler returns.
    Handler func(ctx context.Context, conn net.Conn)

    // ShutdownTimeout is the maximum duration the server waits for
    // all connections to be closed before shutdown.
    //
    // Zero duration disables timeout.
    ShutdownTimeout time.Duration

    // Env is the environment where this server runs.
    //
    // The global environment is used if Env is nil.
    Env *Environment
    // contains filtered or unexported fields
}

Server is a generic network server that accepts connections and invokes Handler in a goroutine for each connection.

In addition, Serve method gracefully waits all its goroutines to complete before returning.

func (*Server) Serve Uses

func (s *Server) Serve(l net.Listener)

Serve starts a managed goroutine to accept connections.

Serve itself returns immediately. The goroutine continues to accept and handle connections until the base context is canceled.

If the listener is *net.TCPListener, TCP keep-alive is automatically enabled.

The listener l will be closed automatically when the environment's Cancel is called.

func (*Server) TimedOut Uses

func (s *Server) TimedOut() bool

TimedOut returns true if the server shut down before all connections got closed.

type StdResponseWriter Uses

type StdResponseWriter interface {
    http.ResponseWriter
    io.ReaderFrom
    http.Flusher
    http.CloseNotifier
    http.Hijacker
    WriteString(data string) (int, error)
}

StdResponseWriter is the interface implemented by the ResponseWriter from http.Server.

HTTPServer's ResponseWriter implements this as well.

Package well imports 28 packages (graph) and is imported by 3 packages. Updated 2019-03-21. Refresh now. Tools for package owners.