gopter: github.com/leanovate/gopter/commands Index | Examples | Files

package commands

import "github.com/leanovate/gopter/commands"

Package commands contains helpers to create stateful tests based on commands.

Testers have to implement the Commands interface providing generators for the initial state and the commands. For convenience testers may also use the ProtoCommands as prototype.

The commands themselves have to implement the Command interface, whereas testers might choose to use ProtoCommand as prototype.

Demonstrates the usage of the commands package to find a bug in a counter implementation that only occurs if the counter is above 3.

The output of this example will be

! buggy counter: Falsified after 45 passed tests.
ARG_0: initial=0 sequential=[INC INC INC INC DEC GET]
ARG_0_ORIGINAL (9 shrinks): initial=0 sequential=[DEC RESET GET GET GET
   RESET DEC DEC INC INC RESET RESET DEC INC RESET INC INC GET INC INC DEC
   DEC GET RESET INC INC DEC INC INC INC RESET RESET INC INC GET INC DEC GET
   DEC GET INC RESET INC INC RESET]

I.e. gopter found an invalid state with a rather long sequence of arbitrary commands/function calls, and then shrank that sequence down to

INC INC INC INC DEC GET

which is indeed the minimal set of commands one has to perform to find the bug.

Code:

package main

import (
    "github.com/leanovate/gopter"
    "github.com/leanovate/gopter/commands"
    "github.com/leanovate/gopter/gen"
)

type BuggyCounter struct {
    n int
}

func (c *BuggyCounter) Inc() {
    c.n++
}

func (c *BuggyCounter) Dec() {
    if c.n > 3 {
        // Intentional error
        c.n -= 2
    } else {
        c.n--
    }
}

func (c *BuggyCounter) Get() int {
    return c.n
}

func (c *BuggyCounter) Reset() {
    c.n = 0
}

var GetBuggyCommand = &commands.ProtoCommand{
    Name: "GET",
    RunFunc: func(systemUnderTest commands.SystemUnderTest) commands.Result {
        return systemUnderTest.(*BuggyCounter).Get()
    },
    PostConditionFunc: func(state commands.State, result commands.Result) *gopter.PropResult {
        if state.(int) != result.(int) {
            return &gopter.PropResult{Status: gopter.PropFalse}
        }
        return &gopter.PropResult{Status: gopter.PropTrue}
    },
}

var IncBuggyCommand = &commands.ProtoCommand{
    Name: "INC",
    RunFunc: func(systemUnderTest commands.SystemUnderTest) commands.Result {
        systemUnderTest.(*BuggyCounter).Inc()
        return nil
    },
    NextStateFunc: func(state commands.State) commands.State {
        return state.(int) + 1
    },
}

var DecBuggyCommand = &commands.ProtoCommand{
    Name: "DEC",
    RunFunc: func(systemUnderTest commands.SystemUnderTest) commands.Result {
        systemUnderTest.(*BuggyCounter).Dec()
        return nil
    },
    NextStateFunc: func(state commands.State) commands.State {
        return state.(int) - 1
    },
}

var ResetBuggyCommand = &commands.ProtoCommand{
    Name: "RESET",
    RunFunc: func(systemUnderTest commands.SystemUnderTest) commands.Result {
        systemUnderTest.(*BuggyCounter).Reset()
        return nil
    },
    NextStateFunc: func(state commands.State) commands.State {
        return 0
    },
}

var buggyCounterCommands = &commands.ProtoCommands{
    NewSystemUnderTestFunc: func(initialState commands.State) commands.SystemUnderTest {
        return &BuggyCounter{}
    },
    InitialStateGen: gen.Const(0),
    InitialPreConditionFunc: func(state commands.State) bool {
        return state.(int) == 0
    },
    GenCommandFunc: func(state commands.State) gopter.Gen {
        return gen.OneConstOf(GetBuggyCommand, IncBuggyCommand, DecBuggyCommand, ResetBuggyCommand)
    },
}

// Demonstrates the usage of the commands package to find a bug in a counter
// implementation that only occurs if the counter is above 3.
//
// The output of this example will be
//  ! buggy counter: Falsified after 45 passed tests.
//  ARG_0: initial=0 sequential=[INC INC INC INC DEC GET]
//  ARG_0_ORIGINAL (9 shrinks): initial=0 sequential=[DEC RESET GET GET GET
//     RESET DEC DEC INC INC RESET RESET DEC INC RESET INC INC GET INC INC DEC
//     DEC GET RESET INC INC DEC INC INC INC RESET RESET INC INC GET INC DEC GET
//     DEC GET INC RESET INC INC RESET]
// I.e. gopter found an invalid state with a rather long sequence of arbitrary
// commands/function calls, and then shrank that sequence down to
//  INC INC INC INC DEC GET
// which is indeed the minimal set of commands one has to perform to find the
// bug.
func main() {
    parameters := gopter.DefaultTestParameters()
    parameters.Rng.Seed(1234) // Just for this example to generate reproducible results

    properties := gopter.NewProperties(parameters)

    properties.Property("buggy counter", commands.Prop(buggyCounterCommands))

    // When using testing.T you might just use: properties.TestingRun(t)
    properties.Run(gopter.ConsoleReporter(false))
}

Kudos to @jamesd for providing this real world example. ... of course he did not implemented the bug, that was evil me

The bug only occures on the following conditions:

- the queue size has to be greater than 4
- the queue has to be filled entirely once
- Get operations have to be at least 5 elements behind put
- The Put at the end of the queue and 5 elements later have to be non-zero

Lets see what gopter has to say:

The output of this example will be

! circular buffer: Falsified after 96 passed tests.
ARG_0: initialState=State(size=7, elements=[]) sequential=[Put(0) Put(0)
   Get Put(0) Get Put(0) Put(0) Get Put(0) Get Put(0) Get Put(-1) Put(0)
   Put(0) Put(0) Put(0) Get Get Put(2) Get]
ARG_0_ORIGINAL (85 shrinks): initialState=State(size=7, elements=[])
   sequential=[Put(-1855365712) Put(-1591723498) Get Size Size
   Put(-1015561691) Get Put(397128011) Size Get Put(1943174048) Size
   Put(1309500770) Size Get Put(-879438231) Size Get Put(-1644094687) Get
   Put(-1818606323) Size Put(488620313) Size Put(-1219794505)
   Put(1166147059) Get Put(11390361) Get Size Put(-1407993944) Get Get Size
   Put(1393923085) Get Put(1222853245) Size Put(2070918543) Put(1741323168)
   Size Get Get Size Put(2019939681) Get Put(-170089451) Size Get Get Size
   Size Put(-49249034) Put(1229062846) Put(642598551) Get Put(1183453167)
   Size Get Get Get Put(1010460728) Put(6828709) Put(-185198587) Size Size
   Get Put(586459644) Get Size Put(-1802196502) Get Size Put(2097590857) Get
   Get Get Get Size Put(-474576011) Size Get Size Size Put(771190414) Size
   Put(-1509199920) Get Put(967212411) Size Get Put(578995532) Size Get Size
   Get]

Though this is not the minimal possible combination of command, its already pretty close.

Code:

package main

import (
    "fmt"

    "github.com/leanovate/gopter"
    "github.com/leanovate/gopter/commands"
    "github.com/leanovate/gopter/gen"
)

// *****************************************
// Production code (i.e. the implementation)
// *****************************************

type Queue struct {
    inp  int
    outp int
    size int
    buf  []int
}

func New(n int) *Queue {
    return &Queue{
        inp:  0,
        outp: 0,
        size: n + 1,
        buf:  make([]int, n+1),
    }
}

func (q *Queue) Put(n int) int {
    if q.inp == 4 && n > 0 { // Intentional spooky bug
        q.buf[q.size-1] *= n
    }
    q.buf[q.inp] = n
    q.inp = (q.inp + 1) % q.size
    return n
}

func (q *Queue) Get() int {
    ans := q.buf[q.outp]
    q.outp = (q.outp + 1) % q.size
    return ans
}

func (q *Queue) Size() int {
    return (q.inp - q.outp + q.size) % q.size
}

func (q *Queue) Init() {
    q.inp = 0
    q.outp = 0
}

// *****************************************
//               Test code
// *****************************************

// cbState holds the expected state (i.e. its the commands.State)
type cbState struct {
    size         int
    elements     []int
    takenElement int
}

func (st *cbState) TakeFront() {
    st.takenElement = st.elements[0]
    st.elements = append(st.elements[:0], st.elements[1:]...)
}

func (st *cbState) PushBack(value int) {
    st.elements = append(st.elements, value)
}

func (st *cbState) String() string {
    return fmt.Sprintf("State(size=%d, elements=%v)", st.size, st.elements)
}

// Get command simply invokes the Get function on the queue and compares the
// result with the expected state.
var genGetCommand = gen.Const(&commands.ProtoCommand{
    Name: "Get",
    RunFunc: func(q commands.SystemUnderTest) commands.Result {
        return q.(*Queue).Get()
    },
    NextStateFunc: func(state commands.State) commands.State {
        state.(*cbState).TakeFront()
        return state
    },
    // The implementation implicitly assumes that Get is never called on an
    // empty Queue, therefore the command requires a corresponding pre-condition
    PreConditionFunc: func(state commands.State) bool {
        return len(state.(*cbState).elements) > 0
    },
    PostConditionFunc: func(state commands.State, result commands.Result) *gopter.PropResult {
        if result.(int) != state.(*cbState).takenElement {
            return &gopter.PropResult{Status: gopter.PropFalse}
        }
        return &gopter.PropResult{Status: gopter.PropTrue}
    },
})

// Put command puts a value into the queue by using the Put function. Since
// the Put function has an int argument the Put command should have a
// corresponding parameter.
type putCommand int

func (value putCommand) Run(q commands.SystemUnderTest) commands.Result {
    return q.(*Queue).Put(int(value))
}

func (value putCommand) NextState(state commands.State) commands.State {
    state.(*cbState).PushBack(int(value))
    return state
}

// The implementation implicitly assumes that that Put is never called if
// the capacity is exhausted, therefore the command requires a corresponding
// pre-condition.
func (putCommand) PreCondition(state commands.State) bool {
    s := state.(*cbState)
    return len(s.elements) < s.size
}

func (putCommand) PostCondition(state commands.State, result commands.Result) *gopter.PropResult {
    st := state.(*cbState)
    if result.(int) != st.elements[len(st.elements)-1] {
        return &gopter.PropResult{Status: gopter.PropFalse}
    }
    return &gopter.PropResult{Status: gopter.PropTrue}
}

func (value putCommand) String() string {
    return fmt.Sprintf("Put(%d)", value)
}

// We want to have a generator for put commands for arbitrary int values.
// In this case the command is actually shrinkable, e.g. if the property fails
// by putting a 1000, it might already fail for a 500 as well ...
var genPutCommand = gen.Int().Map(func(value int) commands.Command {
    return putCommand(value)
}).WithShrinker(func(v interface{}) gopter.Shrink {
    return gen.IntShrinker(int(v.(putCommand))).Map(func(value int) putCommand {
        return putCommand(value)
    })
})

// Size command is simple again, it just invokes the Size function and
// compares compares the result with the expected state.
// The Size function can be called any time, therefore this command does not
// require a pre-condition.
var genSizeCommand = gen.Const(&commands.ProtoCommand{
    Name: "Size",
    RunFunc: func(q commands.SystemUnderTest) commands.Result {
        return q.(*Queue).Size()
    },
    PostConditionFunc: func(state commands.State, result commands.Result) *gopter.PropResult {
        if result.(int) != len(state.(*cbState).elements) {
            return &gopter.PropResult{Status: gopter.PropFalse}
        }
        return &gopter.PropResult{Status: gopter.PropTrue}
    },
})

// cbCommands implements the command.Commands interface, i.e. is
// responsible for creating/destroying the system under test and generating
// commands and initial states (cbState)
var cbCommands = &commands.ProtoCommands{
    NewSystemUnderTestFunc: func(initialState commands.State) commands.SystemUnderTest {
        s := initialState.(*cbState)
        q := New(s.size)
        for e := range s.elements {
            q.Put(e)
        }
        return q
    },
    DestroySystemUnderTestFunc: func(sut commands.SystemUnderTest) {
        sut.(*Queue).Init()
    },
    InitialStateGen: gen.IntRange(1, 30).Map(func(size int) *cbState {
        return &cbState{
            size:     size,
            elements: make([]int, 0, size),
        }
    }),
    InitialPreConditionFunc: func(state commands.State) bool {
        s := state.(*cbState)
        return len(s.elements) >= 0 && len(s.elements) <= s.size
    },
    GenCommandFunc: func(state commands.State) gopter.Gen {
        return gen.OneGenOf(genGetCommand, genPutCommand, genSizeCommand)
    },
}

// Kudos to @jamesd for providing this real world example.
// ... of course he did not implemented the bug, that was evil me
//
// The bug only occures on the following conditions:
//  - the queue size has to be greater than 4
//  - the queue has to be filled entirely once
//  - Get operations have to be at least 5 elements behind put
//  - The Put at the end of the queue and 5 elements later have to be non-zero
//
// Lets see what gopter has to say:
//
// The output of this example will be
//  ! circular buffer: Falsified after 96 passed tests.
//  ARG_0: initialState=State(size=7, elements=[]) sequential=[Put(0) Put(0)
//     Get Put(0) Get Put(0) Put(0) Get Put(0) Get Put(0) Get Put(-1) Put(0)
//     Put(0) Put(0) Put(0) Get Get Put(2) Get]
//  ARG_0_ORIGINAL (85 shrinks): initialState=State(size=7, elements=[])
//     sequential=[Put(-1855365712) Put(-1591723498) Get Size Size
//     Put(-1015561691) Get Put(397128011) Size Get Put(1943174048) Size
//     Put(1309500770) Size Get Put(-879438231) Size Get Put(-1644094687) Get
//     Put(-1818606323) Size Put(488620313) Size Put(-1219794505)
//     Put(1166147059) Get Put(11390361) Get Size Put(-1407993944) Get Get Size
//     Put(1393923085) Get Put(1222853245) Size Put(2070918543) Put(1741323168)
//     Size Get Get Size Put(2019939681) Get Put(-170089451) Size Get Get Size
//     Size Put(-49249034) Put(1229062846) Put(642598551) Get Put(1183453167)
//     Size Get Get Get Put(1010460728) Put(6828709) Put(-185198587) Size Size
//     Get Put(586459644) Get Size Put(-1802196502) Get Size Put(2097590857) Get
//     Get Get Get Size Put(-474576011) Size Get Size Size Put(771190414) Size
//     Put(-1509199920) Get Put(967212411) Size Get Put(578995532) Size Get Size
//     Get]
//
// Though this is not the minimal possible combination of command, its already
// pretty close.
func main() {
    parameters := gopter.DefaultTestParametersWithSeed(1234) // Example should generate reproducible results, otherwise DefaultTestParameters() will suffice

    properties := gopter.NewProperties(parameters)

    properties.Property("circular buffer", commands.Prop(cbCommands))

    // When using testing.T you might just use: properties.TestingRun(t)
    properties.Run(gopter.ConsoleReporter(false))
}

Index

Examples

Package Files

actions.go command.go commands.go doc.go

func Prop Uses

func Prop(commands Commands) gopter.Prop

Prop creates a gopter.Prop from Commands

type Command Uses

type Command interface {
    // Run applies the command to the system under test
    Run(systemUnderTest SystemUnderTest) Result
    // NextState calculates the next expected state if the command is applied
    NextState(state State) State
    // PreCondition checks if the state is valid before the command is applied
    PreCondition(state State) bool
    // PostCondition checks if the state is valid after the command is applied
    PostCondition(state State, result Result) *gopter.PropResult
    // String gets a (short) string representation of the command
    String() string
}

Command is any kind of command that may be applied to the system under test

type Commands Uses

type Commands interface {
    // NewSystemUnderTest should create a new/isolated system under test
    NewSystemUnderTest(initialState State) SystemUnderTest
    // DestroySystemUnderTest may perform any cleanup tasks to destroy a system
    DestroySystemUnderTest(SystemUnderTest)
    // GenInitialState provides a generator for the initial State.
    // IMPORTANT: The generated state itself may be mutable, but this generator
    // is supposed to generate a clean and reproductable state every time.
    // Do not use an external random generator and be especially vary about
    // `gen.Const(<pointer to some mutable struct>)`.
    GenInitialState() gopter.Gen
    // GenCommand provides a generator for applicable commands to for a state
    GenCommand(state State) gopter.Gen
    // InitialPreCondition checks if the initial state is valid
    InitialPreCondition(state State) bool
}

Commands provide an entry point for testing a stateful system

type ProtoCommand Uses

type ProtoCommand struct {
    Name              string
    RunFunc           func(systemUnderTest SystemUnderTest) Result
    NextStateFunc     func(state State) State
    PreConditionFunc  func(state State) bool
    PostConditionFunc func(state State, result Result) *gopter.PropResult
}

ProtoCommand is a prototype implementation of the Command interface

func (*ProtoCommand) NextState Uses

func (p *ProtoCommand) NextState(state State) State

NextState calculates the next expected state if the command is applied

func (*ProtoCommand) PostCondition Uses

func (p *ProtoCommand) PostCondition(state State, result Result) *gopter.PropResult

PostCondition checks if the state is valid after the command is applied

func (*ProtoCommand) PreCondition Uses

func (p *ProtoCommand) PreCondition(state State) bool

PreCondition checks if the state is valid before the command is applied

func (*ProtoCommand) Run Uses

func (p *ProtoCommand) Run(systemUnderTest SystemUnderTest) Result

Run applies the command to the system under test

func (*ProtoCommand) String Uses

func (p *ProtoCommand) String() string

type ProtoCommands Uses

type ProtoCommands struct {
    NewSystemUnderTestFunc     func(initialState State) SystemUnderTest
    DestroySystemUnderTestFunc func(SystemUnderTest)
    InitialStateGen            gopter.Gen
    GenCommandFunc             func(State) gopter.Gen
    InitialPreConditionFunc    func(State) bool
}

ProtoCommands is a prototype implementation of the Commands interface

func (*ProtoCommands) DestroySystemUnderTest Uses

func (p *ProtoCommands) DestroySystemUnderTest(systemUnderTest SystemUnderTest)

DestroySystemUnderTest may perform any cleanup tasks to destroy a system

func (*ProtoCommands) GenCommand Uses

func (p *ProtoCommands) GenCommand(state State) gopter.Gen

GenCommand provides a generator for applicable commands to for a state

func (*ProtoCommands) GenInitialState Uses

func (p *ProtoCommands) GenInitialState() gopter.Gen

GenInitialState provides a generator for the initial State

func (*ProtoCommands) InitialPreCondition Uses

func (p *ProtoCommands) InitialPreCondition(state State) bool

InitialPreCondition checks if the initial state is valid

func (*ProtoCommands) NewSystemUnderTest Uses

func (p *ProtoCommands) NewSystemUnderTest(initialState State) SystemUnderTest

NewSystemUnderTest should create a new/isolated system under test

type Result Uses

type Result interface{}

Result resembles the result of a command that may or may not be checked

type State Uses

type State interface{}

State resembles the state the system under test is expected to be in

type SystemUnderTest Uses

type SystemUnderTest interface{}

SystemUnderTest resembles the system under test, which may be any kind of stateful unit of code

Package commands imports 5 packages (graph). Updated 2019-11-08. Refresh now. Tools for package owners.