ghostgres

package module
v0.0.0-...-edeec10 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 29, 2014 License: BSD-3-Clause Imports: 17 Imported by: 0

README

Ghostgres

GoDoc Build Status Coverage Status

Ghostgres helps you start and control fresh PostgreSQL clusters very fast. On a 2.000 GHz Intel i7-2630QM CPU machine (non-SSD) cloning a cluster took < 100ms. However, your mileage may vary.

Please note that this is a work in progress and the API and documentation are in need of some cleanup. The API will likely change in the future.

Running a unit test while talking to a database can be tricky and sometimes requires a fair amount of setup to ensure the right database is installed and configured on developer machines, continuous build machines and other places where the tests run.

Ghostgres helps solve this by providing utilities to create a standalone PostgreSQL cluster. It comes with the following features for ease of use

  • Initialization of a fresh PostgreSQL cluster at a chosen location using initdb.
  • Cloning an existing installation for fast test startup times.
  • Controlling configuration of the cluster at initialization time.

The easiest way to use Ghostgres is to allow it to create a default template for your version of postgres. This is done in the Ghostgres package directory and is never modified. All future clusters are created by copying this directory tree to a new location and starting a postgres server from there. This takes 10s of ms vs 2 to 3 seconds to run initdb.

Quick Start

// Fetch the package
go get -t github.com/surullabs/ghostgres

// Run tests and create a default postgres cluster that will be used
// as a template for future clusters. This will take a while since
// it creates a number of postgres clusters from scratch. However, it also
// reports the time taken to clone a single cluster which is what you will see
// in practice.
go test github.com/surullabs/ghostgres --gocheck.vv --ghostgres_pg_bin_dir=<path_to_your_postgres_bin_dir>

In your test code you can now use (with appropriate error checks)

// Create a cloned cluster from the default template in a temporary directory
cluster, err := ghostgres.FromDefault("")
if err != nil {
	// fail
}
// Start the postgres server
err = cluster.Start() // Handle error
// Remember to stop it! This will delete the temporary directory.
defer cluster.Stop()

// Connect to the running postgres server through a unix socket.
var connStr string
connStr, err = cluster.TestConnectString() // Handle error
db, err := sql.Open("postgres", fmt.Sprintf("%s dbname=postgres", connStr))

Documentation and Examples

Please consult the package GoDoc for detailed documentation.

Licensing and Usage

Ghostgres is licensed under a 3-Clause BSD license. Please consult the LICENSE file for details.

We also ask that you please file bugs and enhancement requests if you run into any problems. In additon, we're always happy to accept pull requests! If you do find this useful please share it with others who might also find it useful. The more users we have the better the software becomes.

Documentation

Overview

Package ghostgres is a utility to start and control a PostgreSQL database. The expected usage is in tests where it allows for easy startup and shutdown of a database. The easiest way is to have Ghostgres build a template from which it can clone a fresh database when you need one. In order to do this run 

// Fetch the package
go get -t github.com/surullabs/ghostgres

// Run tests and create a default postgres cluster that will be used
// as a template for future clusters.
go test github.com/surullabs/ghostgres --ghostgres_pg_bin_dir=<path_to_your_postgres_bin_dir>

In your test code you can now use (with appropriate error checks) 

// Create a cloned cluster from the default template in a temporary directory
cluster, err := ghostgres.FromDefault("")
if err != nil {
	// fail
}
// Start the postgres server
err = cluster.Start() // Handle error
// Remember to stop it! This will delete the temporary directory.
defer cluster.Stop()

// Connect to the running postgres server through a unix socket.
var connStr string
connStr, err = cluster.TestConnectString() // Handle error
db, err := sql.Open("postgres", fmt.Sprintf("%s dbname=postgres", connStr))

Please consult the examples for other sample usage.

Example 
// Using a postgres cluster with test defaults in a temporary directory

tempDir, err := ioutil.TempDir("", "ghostgres")
if err != nil {
	log.Fatal(err)
	return
}
defer func() { os.RemoveAll(tempDir) }()

// A postgres cluster which will be created in tempDir and use binaries
// from /usr/lib/postgresql/9.3/bin. log.Fatal will be run if any errors
// occur on any  of the exported methods of ghostgres.
// This can also be an instance of testing.T.Fatal to automatically abort
// tests on error
master := &PostgresCluster{
	Config:  TestConfig,
	DataDir: tempDir,
	BinDir:  "/usr/lib/postgresql/9.3/bin",
}

// Initialize the cluster
master.Init()
master.Start()
defer master.Stop()

// Now use the database
db, err := sql.Open("postgres", fmt.Sprintf("%s dbname=postgres", testcheck.Return(master.TestConnectString()).(string)))
if err != nil {
	log.Fatal(err)
	return
}
defer db.Close()

master.Stop()

Output:
Example (Cloning) 
// Using a postgres cluster with test defaults that is cloned from a previously known
// location.

// The temporary directory will be used for the clone
tempDir, err := ioutil.TempDir("", "ghostgres")
if err != nil {
	log.Fatal(err)
	return
}
defer func() { os.RemoveAll(tempDir) }()

// A postgres cluster which will be loaded from testdata/templatedb and use binaries
// from /usr/lib/postgresql/9.3/bin. log.Fatal will be run if any errors
// occur on any  of the exported methods of ghostgres.
// This can also be an instance of testing.T.Fatal to automatically abort
// tests on error
master := &PostgresCluster{
	Config:  TestConfig,
	DataDir: "testdata/templatedb",
	BinDir:  "/usr/lib/postgresql/9.3/bin",
}

// Initialize the cluster if needed. This allows you to create a template
// easily. You can then choose to store the template in version control
// but be warned that it takes up to 33 MB.
master.InitIfNeeded()

// Create a clone which we will use for tests.
clone, _ := master.Clone(tempDir)

defer clone.Stop()

// Now use the database
db, err := sql.Open("postgres", fmt.Sprintf("%s dbname=postgres", testcheck.Return(clone.TestConnectString())))
if err != nil {
	log.Fatal(err)
	return
}
defer db.Close()

clone.Stop()

Output:

Index

Examples

Constants

View Source 
const DefaultTemplate = ""

DefaultTemplate is a convenience value used to refer to a default template. If used the value of the --ghostgres_template flag will be used as the template name.

View Source 
const DefaultTemplateDir = ""

DefaultTemplateDir is a convenience value used to refer to the installed location of the ghostgres package. It is to be used as the root location if you would like to have ghostgres manage all template copies.

View Source 
const TestLogFileName = "postgresql-tests.log"

TestLogFileName is the file name to which PostgresSQL will log if TestConfigWithLogging is used. The path is relative to DataDir/pg_log

Variables

View Source 
var LoggingConfig = []ConfigOpt{
	{"logging_collector", "on", "Collecting query logs can be useful to debug tests"},
	{"log_filename", TestLogFileName, "Well known file name to make log parsing easy in tests"},
	{"log_statement", "all", "Log all statements"},
	{"log_directory", "pg_log", "Logging directory"},
}

LoggingConfig provides useful defaults for logging in tests.

View Source 
var TestConfig = []ConfigOpt{
	{"port", "5432", "Use the default port since we disable TCP listen"},
	{"listen_addresses", "''", "Do not listen on TCP. Instead use a unix domain socket for communication"},
	{"ssl", "false", "No ssl for unit tests"},
	{"shared_buffers", "10MB", "Smaller shared buffers to reduce resource usage"},
	{"fsync", "off", "Ignore system crashes, since tests will fail in that event anyway"},
	{"autovacuum", "off", "Don't run autovacuum for tests"},
	{"full_page_writes", "off", "Useless without fsync"},
}

TestConfig provides some sane defaults for a cluster to be used in unit tests.

View Source 
var TestConfigWithLogging = append(TestConfig, LoggingConfig...)

TestConfigWithLogging combines TestConfig and LoggingConfig

Functions

func Delete 

func Delete(dir, name string) (err error)

Delete will delete a saved template configuration. dir and name have the same behaviour as in Freeze.

Types

type ConfigOpt 

type ConfigOpt struct {
	Key     string
	Value   string
	Comment string
}

ConfigOpt represents a PostgreSQL configuration option It is used both to specify command line arguments as well as populate the postgresql.conf file.

type FailureHandler 

type FailureHandler func(...interface{})

FailureHandler defines a function to be called when errors occur. Setting one makes using PostgresCluster easier in tests.

type PostgresCluster 

type PostgresCluster struct {
	// Key value pairs used to create a postgresql.conf file. They are
	// written out as
	// 	key = value # comment
	Config []ConfigOpt
	// Directory in which to initialize the cluster.
	DataDir string
	// A set of options to be used when creating the cluster. These
	// will be passed directly to initdb. A example would be
	//  {"--auth", "trust", ""}, {"--nosync", "", ""} to enable easy testing.
	// For more details on the command line flags see
	// http://www.postgresql.org/docs/9.3/static/app-initdb.html
	InitOpts []ConfigOpt
	// A set of options to be used when running the postgres server.
	RunOpts []ConfigOpt
	// Directory containing postgres binaries
	BinDir string
	// The password for the super user
	Password string
	// contains filtered or unexported fields
}

PostgresCluster describes a single PostgreSQL cluster

func FromDefault 

func FromDefault(dest string) (p *PostgresCluster, err error)

FromDefault is equivalent to FromTemplate(DefaultTemplateDir, DefaultTemplate, dest)

func FromTemplate 

func FromTemplate(dir, name, dest string) (p *PostgresCluster, err error)

FromTemplate will attempt to clone a cluster from a template located at 

%dir%/%name%/%pg_version%/

where dir and name have the same behaviour as in Freeze(dir,name).

If the defaults don't exist an error will be returned. Please call Freeze(dir, name) first before calling FromTemplate.

If dest is empty a temporary directory is created for the clone and will be deleted when Stop() is called on the cluster.

func (*PostgresCluster) Clone 

func (p *PostgresCluster) Clone(dest string) (c *PostgresCluster, err error)

Clone clones a previous postgres database by copying the entire directory This currently only works on systems which have a cp command. This will not work if the destination directory exists.

func (*PostgresCluster) Freeze 

func (cluster *PostgresCluster) Freeze(dir, name string) (err error)

Freeze will save a template to 

%dir%/%name%/%pg_version%/

where 

%dir%		directory into which to freeze. This will
		create a copy of the cluster into %dir%/data
		If %dir% is empty <path_to_ghostgres>/testdata/template is used.
%name%		is the value of the parameter 'name'. If empty the value of
		the ghostgres_template flag is used.
%pg_version%	is the result of calling PostgresVersion()

If a frozen template exists it will return an error

func (*PostgresCluster) Init 

func (p *PostgresCluster) Init() (err error)

Init will run initdb to create the cluster in the specified directory. Init will return an error if the directory contains an existing cluster. Use InitIfNeeded() to skip initialization of existing clusters.

Please note that this can be time consuming and it is recommended that a golden version of a database is first initialized outside of the test system and then used as a source for cloning using Clone(string). A newly initialized cluster usually takes up about 33 MB of space. One potential option is to have the golden version be initialized in a location that will not be committed into a source repository. Use InitIfNeeded instead of Init and always use Clone(string) and only call Start() on the clone. This allows a single golden copy to be shared among multiple tests with fast start times.

func (*PostgresCluster) InitIfNeeded 

func (p *PostgresCluster) InitIfNeeded() (err error)

InitIfNeeded calls Init() if a call to Initialized returns false.

func (*PostgresCluster) Initialized 

func (p *PostgresCluster) Initialized() bool

Initialized checks if a cluster has been initialized in the data directory. It uses the existence of the postgresql.conf file as a signal that the cluster has been initialized.

func (*PostgresCluster) Port 

func (p *PostgresCluster) Port() (portVal int, err error)

Port attempts to parse a port from the provided config options and returns the parsed port or an error if no port could be parsed..

func (*PostgresCluster) Running 

func (p *PostgresCluster) Running() bool

Running will return true if the server is running. Please note that this is still not very accurate as it merely checks if the server has been started.

func (*PostgresCluster) SocketDir 

func (p *PostgresCluster) SocketDir() (str string, err error)

SocketDir returns the location of the postgres unix socket directory. Note: This will panic if it is unable to find the absolute path to the socket directory.

func (*PostgresCluster) SocketFile 

func (p *PostgresCluster) SocketFile() (socketFile string, err error)

SocketFile returns the location of the postgres socket file

func (*PostgresCluster) Start 

func (p *PostgresCluster) Start() (err error)

Start starts the postgres database. It will add the following extra flags in addition to the RunOpts provided. 

-D p.DataDir  // Use the specified data directory
-k p.DataDir  // Use the data directory as the socket directory for unix sockets.
-c config_file=p.DataDir/postgresql.confg // Custom config file.

It does not attempt to read the config file to determine the data directory or the socket directory.

func (*PostgresCluster) Stop 

func (p *PostgresCluster) Stop() (err error)

Stop stops the postgres cluster if it is running by sending it a SIGTERM signal. This will request a slow shutdown and the postgres server will wait for all existing connections to close. It is an error to call this if the server is not running.

func (*PostgresCluster) TestConnectString 

func (p *PostgresCluster) TestConnectString() (str string, err error)

TestConnectString returns a connect string to use when using TestConfig or an error if unable to build the string.

func (*PostgresCluster) Wait 

func (p *PostgresCluster) Wait() (err error)

Wait waits for a running postgres server to terminate. It is useful when you wish to freeze a test and inspect the database. Once it is frozen it can be stopped using 

pg_ctl -D p.DataDir stop

It will return an error if the server exits with any return code other than 0 or as a result of SIGTERM. It is an error to call this before calling Start.

func (*PostgresCluster) WaitTillServing 

func (p *PostgresCluster) WaitTillServing(timeout time.Duration) (err error)

WaitTillServing waits for a duration of timeout for the postgres server to start. It must be called after a call to Start() and before a call to Stop() or Wait() It polls for the existence of the socket file every 10ms to detect if the server is running and accessible and will return an error if it cannot detect the server within timeout.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.