migration

package module
v0.0.0-...-d4132e8 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 3, 2019 License: MIT Imports: 8 Imported by: 0

README

migration: Database Migrations

GoDoc License Build Status (Linux) Coverage Status GoReportCard

Package migration manages database schema migrations.

See the Godoc for usage details.

Features

  • Write database migrations in SQL or Go
  • Supports SQLite, Postgres and MySQL databases (support for MSSQL planned)
  • Migrations are performed in a transaction where possible
  • Up/Down migrations for applying and rolling back migrations
  • Replay previous migrations for restoring views, functions and stored procedures
  • Support for writing migrations on separate branches
  • Migrations are embedded in the executable
  • CLI package for easy integration with programs using cobra

Installation

go get -u github.com/jjeffery/migration

Example Usage

See the Godoc package example.

Documentation

Overview

Package migration manages database schema migrations. There many popular packages that provide database migrations, so it is worth listing how this package is different.

Write migrations in SQL or Go

Database migrations are usually written in SQL/DDL using the dialect specific to the target database. Most of the time this is sufficient, but there are times when it is more convenient to specify a migration using Go code.

Migrate up and migrate down

Each database schema version has an "up" migration, which migrates up from the previous database schema version. It also has a "down" migration, which should revert the changes applied in the "up" migration.

Use transactions for migrations

Database migrations are performed within a transaction if the database supports it.

Replay previous migrations to restore views, stored procedures, etc

When a migration drops and recreates a view, it is necessary to restore the previous version of the view in the down migration, which results in duplicated code. (This problem also exists for stored procedures, functions, triggers, etc).

This package provides a simple solution in the form af defining migrations as a replay of a previous "up" migration. The "down" migration for a create view can be defined as a replay of the prevous "up" migration that created the previous version of the view.

Write migrations on separate branches

Database schema versions are identified by positive 64-bit integers. Migrations can be defined in different VCS branches using an arbitrary naming convention, such as the current date and time. When branches are merged and a database migration is performed, any unapplied migrations will be applied in ascending order of database schema version.

Lock database schema versions

Once a set of migrations has been applied, the database schema version can be locked, which means that any attempt to migrate down past this version will fail. This is useful for avoiding accidents with production database schemas.

Embed migrations in the executable

Migrations are written as part of the Go source code, and are embedded in the resulting executable without the need for any embedding utility, or the need to deploy any separate text files.

Deploy as part of a larger executable

This package does not provide a stand-alone command line utility for managing database migrations. Instead it provides a simple API that can be utilized as part of a project-specific CLI for database management. The cli subdirectory contains a re-usable command line command based on the "github.com/spf13/cobra" package.

Alternatives

If this package does not meet your needs, refer to https://github.com/avelino/awesome-go#database for a list popular database migration packages.

Example
package main

import (
	"context"
	"database/sql"
	"fmt"
	"log"
	"os"

	"github.com/jjeffery/migration"
	_ "github.com/mattn/go-sqlite3"
)

// Schema contains all the information needed to migrate
// the database schema.
//
// See the init function  below for where the individual
// migrations are defined.
var Schema migration.Schema

func main() {
	// Setup logging. Don't print a timestamp so that the
	// output can be checked at the end of this function.
	log.SetFlags(0)
	log.SetOutput(os.Stdout)

	// Perform example operations on an SQLite, in-memory database.
	ctx := context.Background()
	db, err := sql.Open("sqlite3", ":memory:")
	checkError(err)

	// A worker does the work, and can optionally log its progress.
	worker, err := migration.NewWorker(db, &Schema)
	checkError(err)
	worker.LogFunc = log.Println

	// Migrate up to the latest version
	err = worker.Up(ctx)
	checkError(err)

	// Migrate down
	err = worker.Goto(ctx, 4)
	checkError(err)

}

func checkError(err error) {
	if err != nil {
		fmt.Println("error:", err)
		//log.Fatal(err)
	}
}

// init defines all of the migrations for the migration schema.
//
// In practice, the migrations would probably be defined in separate
// source files, each with its own init function.
func init() {
	Schema.Define(1).Up(`
		create table city (
			id integer not null,
			name text not null,
			countrycode character(3) not null,
			district text not null,
			population integer not null
		);
	`).Down(`drop table city;`)

	Schema.Define(2).Up(`
		create table country (
			code character(3) not null,
			name text not null,
			continent text not null,
			region text not null,
			surfacearea real not null,
			indepyear smallint,
			population integer not null,
			lifeexpectancy real,
			gnp numeric(10,2),
			gnpold numeric(10,2),
			localname text not null,
			governmentform text not null,
			headofstate text,
			capital integer,
			code2 character(2) not null
		);
	`).Down(`drop table country;`)

	Schema.Define(3).Up(`
		-- drop view first so we can replay this migration, see version 6
		drop view if exists city_country;

		create view city_country as 
			select city.id, city.name, country.name as country_name
			from city
			inner join country on city.countrycode = country.code;
	`).Down(`drop view city_country`)

	// Contrived example of a migration implemented in Go that uses
	// a database transaction.
	Schema.Define(4).UpAction(migration.TxFunc(func(ctx context.Context, tx *sql.Tx) error {
		_, err := tx.ExecContext(ctx, `
				insert into city(id, name, countrycode, district, population)
				values(?, ?, ?, ?, ?)`,
			1, "Kabul", "AFG", "Kabol", 1780000,
		)
		return err
	})).DownAction(migration.TxFunc(func(ctx context.Context, tx *sql.Tx) error {
		_, err := tx.ExecContext(ctx, `delete from city where id = ?`, 1)
		return err
	}))

	// Contrived example of a migration implemented in Go that does
	// not use a database transaction. If this migration fails, the
	// database will require manual intervention.
	Schema.Define(5).UpAction(migration.DBFunc(func(ctx context.Context, db *sql.DB) error {
		_, err := db.ExecContext(ctx, `
				insert into city(id, name, countrycode, district, population)
				values(?, ?, ?, ?, ?)`,
			2, "Qandahar", "AFG", "Qandahar", 237500,
		)
		return err
	})).DownAction(migration.DBFunc(func(ctx context.Context, db *sql.DB) error {
		_, err := db.ExecContext(ctx, `delete from city where id = ?`, 2)
		return err
	}))

	// This migration alters the view, and the down migration reverts to the
	// previous view definition.
	Schema.Define(6).Up(`
		drop view if exists city_country;

		create view city_country as 
			select city.id, city.name, country.name as country_name, district
			from city
			inner join country on city.countrycode = country.code;
	`).DownAction(migration.Replay(3))
}
Output:

migrated up version=1
migrated up version=2
migrated up version=3
migrated up version=4
migrated up version=5
migrated up version=6
migrate up finished version=6
migrated down version=6
migrated down version=5
migrate goto finished version=4

Index

Examples

Constants

View Source
const (
	// DefaultMigrationsTable is the default name of the database table
	// used to keep track of all applied database migrations. This name
	// can be overridden by the Schema.MigrationsTable field.
	DefaultMigrationsTable = "schema_migrations"
)

Variables

This section is empty.

Functions

This section is empty.

Types

type Action

type Action func(*action)

An Action defines the action performed during an up migration or a down migration.

func Command

func Command(sql string) Action

Command returns an action that executes the SQL/DDL command.

Command is by far the most common migration action. The Up() and Down() methods provide a quick way to define migration actions when they are SQL/DDL commands.

func DBFunc

func DBFunc(f func(context.Context, *sql.DB) error) Action

DBFunc returns an action that executes the function f.

The migration is performed outside of a transaction, so if the migration fails for any reason, the database will require manual repair before any more migrations can proceed. If possible, use TxFunc to perform migrations within a database transaction.

func Replay

func Replay(id VersionID) Action

Replay returns an action that replays the up migration for the specified database version. Replay actions are useful for restoring views, functions and stored procedures.

func TxFunc

func TxFunc(f func(context.Context, *sql.Tx) error) Action

TxFunc returns an action that executes function f.

The migration is performed inside a transaction, so if the migration fails for any reason, the database will rollback to its state at the start version.

type Definition

type Definition struct {
	// contains filtered or unexported fields
}

A Definition is used to define a database schema version, the action required to migrate up from the previous version, and the action required to migrate down to the previous version.

func (*Definition) Down

func (d *Definition) Down(sql string) *Definition

Down defines the SQL/DDL to migrate down to the previous version. Calling this method is identical to calling:

DownAction(Command(sql))

func (*Definition) DownAction

func (d *Definition) DownAction(a Action) *Definition

DownAction defines the action to perform during the migration down from this database schema version.

The Down() method handles the most common case for "down" migrations.

func (*Definition) Up

func (d *Definition) Up(sql string) *Definition

Up defines the SQL to migrate up to the version. Calling this function is identical to calling:

UpAction(Command(sql))

func (*Definition) UpAction

func (d *Definition) UpAction(a Action) *Definition

UpAction defines the action to perform during the migration up to this database schema version.

The Up() method handles the most common case for "up" migrations.

type Error

type Error struct {
	Version     VersionID
	Description string
}

Error describes a single error in the migration schema definition.

func (*Error) Error

func (e *Error) Error() string

Error implements the error interface

type Errors

type Errors []*Error

Errors describes one or more errors in the migration schema definition. If the Schema.Err() method reports a non-nil value, then it will be of type Errors.

func (Errors) Error

func (e Errors) Error() string

Error implements the error interface.

type Schema

type Schema struct {
	// MigrationsTable specifies the name of the database table used
	// to keep track of the database migrations performed.
	//
	// If not specified, defaults to the constant DefaultMigrationsTable.
	MigrationsTable string
	// contains filtered or unexported fields
}

A Schema contains all of the information required to perform database migrations for a database schema. It contains details about how to migrate up to a version from the previous version, and how to migrate down from a version to the previous version.

func (*Schema) Define

func (s *Schema) Define(id VersionID) *Definition

Define a database schema version along with the migration up from the previous version and the migration down to the previous version.

This method is typically called at program initialization, once for each database schema version. See the package example.

func (*Schema) Err

func (s *Schema) Err() error

Err reports a non-nil error if there are any errors in the migration schema definition, otherwise it returns nil.

If Err does report a non-nil value, it will be of type Errors.

One common use for this method is to create a simple unit test that verifies that checks for errors in the migration schema definitions:

func TestMigrationSchema(t *testing.T) {
    if err := schema.Err(); err != nil {
        t.Fatal(err)
    }
}

type Version

type Version struct {
	ID        VersionID  // Database schema version number
	AppliedAt *time.Time // Time migration was applied, or nil if not applied
	Failed    bool       // Did migration fail
	Locked    bool       // Is version locked (prevent down migration)
	Up        string     // SQL for up migration, or "<go-func>" if go function
	Down      string     // SQL for down migration or "<go-func>"" if a go function
}

Version provides information about a database schema version.

type VersionID

type VersionID int64

VersionID uniquely identifies a database schema version.

type Worker

type Worker struct {
	// LogFunc is a function for logging progress. If not specified then
	// no logging is performed.
	//
	// One common practice is to assign the log.Println function to LogFunc.
	LogFunc func(v ...interface{})
	// contains filtered or unexported fields
}

A Worker performs database migrations. It combines the information in the migration schema along with the database on which to perform migrations.

func NewWorker

func NewWorker(db *sql.DB, schema *Schema) (*Worker, error)

NewWorker creates a worker that can perform migrations for the specified database using the database migration schema.

func (*Worker) Down

func (m *Worker) Down(ctx context.Context) error

Down migrates the database down to the latest locked version. If there are no locked versions, all down migrations are performed.

func (*Worker) Force

func (m *Worker) Force(ctx context.Context, id VersionID) error

Force the database schema to a specific version.

This is used to manually fix a database after a non-transactional migration has failed.

func (*Worker) Goto

func (m *Worker) Goto(ctx context.Context, id VersionID) error

Goto migrates up or down to the specified version.

If id is zero, then all down migrations are applied to result in an empty database.

func (*Worker) Lock

func (m *Worker) Lock(ctx context.Context, id VersionID) error

Lock a database schema version.

This is used to prevent accidental down migrations. When a database version is locked, it is not possible to perform a down migration to the previous version.

func (*Worker) Unlock

func (m *Worker) Unlock(ctx context.Context, id VersionID) error

Unlock a database schema version.

func (*Worker) Up

func (m *Worker) Up(ctx context.Context) error

Up migrates the database to the latest version.

func (*Worker) Version

func (m *Worker) Version(ctx context.Context, id VersionID) (*Version, error)

Version returns details of the specified version.

func (*Worker) Versions

func (m *Worker) Versions(ctx context.Context) ([]*Version, error)

Versions lists all of the database schema versions.

Directories

Path Synopsis
Package cli provides a command line interface for database migrations using the popular cobra CLI package.
Package cli provides a command line interface for database migrations using the popular cobra CLI package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL