testproject

package
v0.10.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 2, 2024 License: MIT Imports: 18 Imported by: 1

Documentation

Overview

Package testproject implements locking of Keboola Projects for E2E parallel tests.

Project is locked: - at locker level (redislock) OR - at the host level (flock.Flock) AND at the goroutines level (sync.Mutex)

Only one test can access the project at a time. See GetTestProject function. If there is no unlocked project, the function waits until a project is released.

Package can be safely used in parallel tests that run on a single host. Use GetTestProjectForTest function to get a testing project in a test. Project lock is automatically released at the end of the test.

Locking between multiple hosts is not provided.

The state of the project does not change automatically, if you need an empty project use storageapi.CleanProjectRequest.

Index

Examples

Constants

View Source 
const (
	StagingStorageABS = "abs"
	StagingStorageGCS = "gcs"
	StagingStorageS3  = "s3"

	BackendSnowflake               = "snowflake"
	BackendBigQuery                = "bigquery"
	TestKbcProjectsKey             = "TEST_KBC_PROJECTS"
	TestKbcProjectsLockDirNameKey  = "TEST_KBC_PROJECTS_LOCK_DIR_NAME"
	TestKbcProjectsLockHostKey     = "TEST_KBC_PROJECTS_LOCK_HOST"
	TestKbcProjectsLockPasswordKey = "TEST_KBC_PROJECTS_LOCK_PASSWORD"
	TestKbcProjectsLockTLSKey      = "TEST_KBC_PROJECTS_LOCK_TLS"
)
View Source 
const QueueV1 = "v1"
View Source 
const (
	TTL = 60 * time.Second
)

Variables

This section is empty.

Functions

func GetTestProject 

func GetTestProject(opts ...Option) (*Project, UnlockFn, error)

GetTestProject locks and returns a testing project specified in TEST_KBC_PROJECTS environment variable. The returned UnlockFn function must be called to free project, when the project is no longer used (e.g. defer unlockFn()) If no project is available, the function waits until a project is released.

Example 
// Note: For real use call the "GetTestProject" function,
// to get a testing project from the "TEST_KBC_PROJECTS" environment variable.
// Here, the "projects.GetTestProject" method is called to make it testable and without global variables.
projects := MustGetProjectsFrom(projectsForTest())

// Acquire exclusive access to the project.
project1, unlockFn1, _ := projects.GetTestProject()
defer unlockFn1()
fmt.Printf("Project %d locked.\n", project1.ID())

project2, unlockFn2, _ := projects.GetTestProject()
defer unlockFn2()
fmt.Printf("Project %d locked.\n", project2.ID())

project3, unlockFn3, _ := projects.GetTestProject()
defer unlockFn3()
fmt.Printf("Project %d locked.\n", project3.ID())

// The returned UnlockFn function must be called to free project, when the project is no longer used (e.g. defer unlockFn())

// See also ExampleGetTestProjectForTest for usage in a test.

Output:

Project 1234 locked.
Project 3456 locked.
Project 5678 locked.
Example (Second) 
// Note: For real use call the "GetTestProject" function,
// to get a testing project from the "TEST_KBC_PROJECTS" environment variable.
// Provide also "TEST_KBC_PROJECTS_LOCK_HOST" and "TEST_KBC_PROJECTS_LOCK_PASSWORD" variables to connect into redis.
// Here, the "projects.GetTestProject" method is called to make it testable and without global variables.
//os.Setenv(TestKbcProjectsLockHostKey, "redis:6379")
// os.Setenv(TestKbcProjectsLockPasswordKey, "testing")
//defer func() {
//	os.Unsetenv(TestKbcProjectsLockHostKey)
//	os.Unsetenv(TestKbcProjectsLockPasswordKey)
//}()
projects, err := GetProjectsFrom(projectsForTest())
if err != nil {
	fmt.Println("redis is not up.")
	return
}

// Acquire exclusive access to the project.
project1, unlockFn1, _ := projects.GetTestProject()
defer unlockFn1()
fmt.Printf("Project %d locked.\n", project1.ID())

project2, unlockFn2, _ := projects.GetTestProject()
defer unlockFn2()
fmt.Printf("Project %d locked.\n", project2.ID())

project3, unlockFn3, _ := projects.GetTestProject()
defer unlockFn3()
fmt.Printf("Project %d locked.\n", project3.ID())

// The returned UnlockFn function must be called to free project, when the project is no longer used (e.g. defer unlockFn())

Output:

Project 1234 locked.
Project 3456 locked.
Project 5678 locked.

Types

type Definition added in v0.6.0 

type Definition struct {
	Host                 string `json:"host" validate:"required"`
	Token                string `json:"token" validate:"required"`
	StagingStorage       string `json:"stagingStorage" validate:"required"`
	Backend              string `json:"backend" validate:"required"`
	ProjectID            int    `json:"project" validate:"required"`
	LegacyTransformation bool   `json:"legacyTransformation"`
	Queue                string `json:"queue,omitempty"`
}

Definition is project Definition parsed from the ENV.

type Option added in v0.6.0 

type Option func(c *config)

Option for the GetTestProjectForTest and GetTestProject functions.

func WithBigQueryBackend added in v0.8.4 

func WithBigQueryBackend() Option

func WithLegacyTransformation added in v0.9.0 

func WithLegacyTransformation() Option

func WithQueueV1 added in v0.8.0 

func WithQueueV1() Option

func WithSnowflakeBackend added in v0.8.4 

func WithSnowflakeBackend() Option

func WithStagingStorage added in v0.6.0 

func WithStagingStorage(stagingStorage string) Option
Example 
// Note: For real use call the "GetTestProject" function,
// to get a testing project from the "TEST_KBC_PROJECTS" environment variable.
// Here, the "projects.GetTestProject" method is called to make it testable and without global variables.
projects := MustGetProjectsFrom(projectsForTest())

// Acquire exclusive access to the project.
project, unlockFn, _ := projects.GetTestProject(WithStagingStorage("abs"))
defer unlockFn()
fmt.Printf("Project %d locked.\n", project.ID())
fmt.Printf("Staging storage: %s.\n", project.StagingStorage())

Output:

Project 3456 locked.
Staging storage: abs.

func WithStagingStorageABS added in v0.6.1 

func WithStagingStorageABS() Option

func WithStagingStorageGCS added in v0.6.2 

func WithStagingStorageGCS() Option

func WithStagingStorageS3 added in v0.6.1 

func WithStagingStorageS3() Option

type Project 

type Project struct {
	// contains filtered or unexported fields
}

Project represents a testing project for E2E tests.

func GetTestProjectForTest added in v0.3.0 

func GetTestProjectForTest(t TInterface, opts ...Option) (*Project, error)

GetTestProjectForTest locks and returns a testing project specified in TEST_KBC_PROJECTS environment variable. Project lock is automatically released at the end of the test. If no project is available, the function waits until a project is released.

Example 
t := &mockedT{}

// Note: For real use call the "GetTestProject" function,
// to get a testing project from the "TEST_KBC_PROJECTS" environment variable.
// Here, the "projects.GetTestProject" method is called to make it testable and without global variables.
projects := MustGetProjectsFrom(projectsForTest())

// Acquire exclusive access to the project.
project1, _ := projects.GetTestProjectForTest(t)
fmt.Printf("Project %d locked.\n", project1.ID()) //nolint:forbidigo

project2, _ := projects.GetTestProjectForTest(t)
fmt.Printf("Project %d locked.\n", project2.ID()) //nolint:forbidigo

project3, _ := projects.GetTestProjectForTest(t)
fmt.Printf("Project %d locked.\n", project3.ID()) //nolint:forbidigo

// Project lock will be automatically released at the end of the test.
for _, f := range t.cleanup {
	f()
}

Output:

Project 1234 locked.
Project 3456 locked.
Project 5678 locked.

func (*Project) Backend added in v0.8.5 

func (p *Project) Backend() string

Backend returns backend of the project Definition.

func (*Project) ID 

func (p *Project) ID() int

ID returns id of the project.

func (*Project) LegacyTransformation added in v0.9.0 

func (p *Project) LegacyTransformation() bool

LegacyTransformation returns support of legacy transformations of the project Definition.

func (*Project) StagingStorage added in v0.6.0 

func (p *Project) StagingStorage() string

StagingStorage returns staging storage of the project Definition.

func (*Project) StorageAPIHost 

func (p *Project) StorageAPIHost() string

StorageAPIHost returns Storage API host of the project stack.

func (*Project) StorageAPIToken 

func (p *Project) StorageAPIToken() string

StorageAPIToken returns Storage API token of the project.

type ProjectsPool added in v0.6.0 

type ProjectsPool []*Project

ProjectsPool a group of testing projects.

func GetProjectsFrom added in v0.6.0 

func GetProjectsFrom(str string) (ProjectsPool, error)

func MustGetProjectsFrom added in v0.6.0 

func MustGetProjectsFrom(str string) ProjectsPool

func (ProjectsPool) GetTestProject added in v0.6.0 

func (v ProjectsPool) GetTestProject(opts ...Option) (*Project, UnlockFn, error)

GetTestProject locks and returns a testing project specified in TEST_KBC_PROJECTS environment variable. The returned UnlockFn function must be called to free project, when the project is no longer used (e.g. defer unlockFn()) If no project is available, the function waits until a project is released.

func (ProjectsPool) GetTestProjectForTest added in v0.6.0 

func (v ProjectsPool) GetTestProjectForTest(t TInterface, opts ...Option) (*Project, error)

GetTestProjectForTest locks and returns a testing project specified in TEST_KBC_PROJECTS environment variable. Project lock is automatically released at the end of the test. If no project is available, the function waits until a project is released.

type TInterface added in v0.6.0 

type TInterface interface {
	Cleanup(f func())
}

TInterface is cleanup part of the *testing.T.

type UnlockFn added in v0.3.0 

type UnlockFn func()

UnlockFn must be called if the project is no longer used.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.