havoc

package module
v0.2.5 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 3, 2024 License: MIT Imports: 30 Imported by: 0

README

Havoc

DISCLAIMER: This software is not even early Alpha, and still in development, use it on your own risk

Havoc is a tool that introspects your k8s namespace and generates a ChaosMesh CRDs suite for you

You can use havoc as a CLI to quickly test hypothesis or run it in "monkey" mode with your load tests and have Grafana annotations

How it works

img.png

Havoc generates groups of experiments based on your pods and labels found in namespace

In order to test your namespace you need to label pods accordingly:

  • havoc-component-group is like app.kubernetes.io/component (see recommendation) but should be set explicitly
  • havoc-network-group in most cases match havoc-component-group but sometimes you need to break network even inside component group, ex. distributed databases

Example:

      havoc-component-group: node
      havoc-network-group: nodes-1

Every pod without a group will be marked as no-group and experiments will be assigned accordingly

Single pod experiments:

  • PodFailure
  • NetworkChaos (Pod latency)
  • Stress (Memory)
  • Stress (CPU)
  • External service failure (Network partition)
  • Blockchain specific experiments

Group experiments:

  • Group failure
  • Group latency
  • Group CPU
  • Group memory
  • Group network partition
  • OpenAPI based HTTP experiments

You can generate default chaos suite by configuring havoc then set dir param and add your custom experiments, then run monkey to test your services

Why use it?

Without Havoc your workflow is
  • Inspect full rendered deployment of your namespace
  • Figure out multiple groups of components you can select by various labels or annotations to form experiments
  • If some components are not selectable - ask DevOps guys to change the manifests
  • Create set of experiments for each chaos experiment type by hand or copy from other product chaos tests
  • Calculate permutations of different groups and calculate composite experiments (network partitioning, latency)
  • Create experiment for each API in every OpenAPI spec
  • Compose huge ChaosMesh Workflow YAML that fails without proper validation errors if group has no match or label is invalid
  • Run the load test, then manually run the chaos suite
  • Check experiment logs to debug with kubectl
  • Figure out which failures are caused by which experiments
  • If you have more than one project, use some templating make experiments work for other projects
With Havoc
  • Have a simple labelling convention for your namespaces, fill 5 vars in TOML config
  • Run chaos testing with havoc -c havoc.toml run ${namespace}

Install

Please use GitHub releases of this repo Download latest release

You need kubectl to available on your machine

You also need ChaosMesh installed in your k8s cluster

Grafana integration

Set env variables

HAVOC_LOG_LEVEL={warn,info,debug,trace}
GRAFANA_URL="..."
GRAFANA_TOKEN="..."

Set dashboard names in havoc.toml

[havoc.grafana]
# UIDs of dashboard which should be annotated with chaos experiments metadata
# You can also try to use name as you see it in the top bar of your dashboard but that's not guaranteed to match
dashboard_uids = ["WaspDebug", "e98b5451-12dc-4a8b-9576-2c0b67ddbd0c"]

Manual usage

Generate default experiments for your namespace

havoc -c havoc.toml generate [namespace]

Check this section for ignore_pods and ignore_group_labels, default settings should be reasonable, however, you can tweak them

This will create havoc-experiments dir, then you can choose from recommended experiments

havoc -c havoc.toml apply

You can also apply your experiment directly, using absolute or relative path to experiment file

havoc -c havoc.toml apply ${experiment_file_path}

Monkey mode

You can run havoc as an automated sequential or randomized suite

havoc -c havoc.toml run [namespace]

See [havoc.monkey] config here

Programmatic usage

See how you can use recommended experiments from code in examples

Custom experiments

Havoc is just a generator and a module that reads your dir = $mydir from config

If you wish to add custom experiments written by hand create your custom directory and add experiments

Experiments will be executed in lexicographic order, however, for custom experiments there are 2 simple rules:

  • directory names must be in
    "external",
    "failure",
    "latency",
    "cpu",
    "memory",
    "group-failure",
    "group-latency",
    "group-cpu",
    "group-memory",
    "group-partition",
    "blockchain_rewind_head",
    "http"
  • metadata.name should be equal to your experiment filename

When you are using run monkey command, if directory is not empty havoc won't automatically generate experiments, so you can extend generated experiments with your custom modifications

Developing

We are using nix

Enter the shell

nix develop

Documentation

Index

Constants

View Source
const (
	DefaultCMDTimeout = "3m"

	ErrNoSelection       = "no selection, exiting"
	ErrInvalidNamespace  = "first argument must be a valid k8s namespace"
	ErrAutocompleteError = "autocomplete file walk errored"
)
View Source
const (
	ErrReadSethConfig      = "failed to read TOML config for havoc"
	ErrUnmarshalSethConfig = "failed to unmarshal TOML config for havoc"

	ErrFailureGroupIsNil      = "failure group must be specified in config"
	ErrLatencyGroupIsNil      = "latency group must be specified in config"
	ErrStressCPUGroupIsNil    = "stress cpu group must be specified in config"
	ErrStressMemoryGroupIsNil = "stress memory group must be specified in config"
	ErrFormat                 = "format error"
)
View Source
const (
	DefaultExperimentsDir           = "havoc-experiments"
	DefaultPodFailureDuration       = "1m"
	DefaultNetworkLatencyDuration   = "1m"
	DefaultNetworkPartitionDuration = "1m"
	DefaultHTTPDuration             = "1m"
	DefaultNetworkPartitionLabel    = "havoc-network-group"
	DefaultComponentGroupLabelKey   = "havoc-component-group"
	DefaultStressMemoryDuration     = "1m"
	DefaultStressMemoryWorkers      = 1
	DefaultStressMemoryAmount       = "512MB"
	DefaultStressCPUDuration        = "1m"
	DefaultStressCPUWorkers         = 1
	DefaultStressCPULoad            = 100
	DefaultNetworkLatency           = "300ms"
	DefaultMonkeyDuration           = "24h"
	DefaultMonkeyMode               = "seq"
	DefaultMonkeyCooldown           = "30s"
)
View Source
const (
	ErrParsingTemplate = "failed to parse Go text template"

	ErrExperimentTimeout = "waiting for experiment to finish timed out"
	ErrExperimentApply   = "error applying experiment manifest"
	ErrInvalidCustomKind = "invalid custom Kind of experiment"
)
View Source
const (
	ChaosTypeBlockchainSetHead = "blockchain_rewind_head"
	ChaosTypeFailure           = "failure"
	ChaosTypeGroupFailure      = "group-failure"
	ChaosTypeLatency           = "latency"
	ChaosTypeGroupLatency      = "group-latency"
	ChaosTypeStressMemory      = "memory"
	ChaosTypeStressGroupMemory = "group-memory"
	ChaosTypeStressCPU         = "cpu"
	ChaosTypeStressGroupCPU    = "group-cpu"
	ChaosTypePartitionExternal = "external"
	ChaosTypePartitionGroup    = "group-partition"
	ChaosTypeHTTP              = "http"
)
View Source
const (
	MonkeyModeSeq    = "seq"
	MonkeyModeRandom = "rand"

	ErrInvalidMode = "monkey mode is invalid, should be either \"seq\" or \"rand\""
)
View Source
const (
	ErrNoNamespace    = "no namespace found"
	ErrEmptyNamespace = "no pods found inside namespace, namespace is empty or check your filter"
)
View Source
const (
	DebugContainerImage = "curlimages/curl:latest"
)
View Source
const (
	ErrParsingOpenAPISpec = "failed to parse OpenAPISpec"
)
View Source
const (
	NoGroupKey = "no-group"
)

Variables

View Source
var (
	DefaultGroupPercentage                 = []string{"10", "20", "30"}
	DefaultGroupFixed                      = []string{"1", "2", "3"}
	DefaultNetworkPartitionGroupPercentage = []string{"100"}
)
View Source
var (
	DefaultIgnoreGroupLabels = []string{
		"mainnet",
		"release",
		"intents.otterize.com",
		"pod-template-hash",
		"rollouts-pod-template-hash",
		"chain.link/app",
		"chain.link/cost-center",
		"chain.link/env",
		"chain.link/project",
		"chain.link/team",
		"app.kubernetes.io/part-of",
		"app.kubernetes.io/managed-by",
		"app.chain.link/product",
		"app.kubernetes.io/version",
		"app.chain.link/blockchain",
		"app.kubernetes.io/instance",
		"app.kubernetes.io/name",
	}
)
View Source
var (
	ExperimentTypesToCRDNames = map[string]string{
		"PodChaos":     "podchaos.chaos-mesh.org",
		"StressChaos":  "stresschaos.chaos-mesh.org",
		"NetworkChaos": "networkchaos.chaos-mesh.org",
		"HTTPChaos":    "httpchaos.chaos-mesh.org",
	}
)
View Source
var (
	OpenAPIPathParam = regexp.MustCompile(`({.*})`)
)

Functions

func ExecCmd

func ExecCmd(command string) (string, error)

func InitDefaultLogging

func InitDefaultLogging()

func MarshalTemplate

func MarshalTemplate(jobSpec interface{}, name, templateString string) (string, error)

MarshalTemplate Helper to marshal templates

func RunCLI

func RunCLI(args []string) error

func SetGlobalLogger

func SetGlobalLogger(l zerolog.Logger)

Types

type ActionablePodInfo

type ActionablePodInfo struct {
	PodName  string
	Labels   []string
	HasGroup bool
}

ActionablePodInfo info about pod and labels for which we can generate a chaos experiment

type BlockchainNodeConfig added in v0.2.5

type BlockchainNodeConfig struct {
	ExecutorPodPrefix     string  `toml:"executor_pod_prefix"`
	ExecutorContainerName string  `toml:"executor_container_name"`
	NodeInternalHTTPURL   string  `toml:"node_internal_http_url"`
	Blocks                []int64 `toml:"blocks"`
}

type BlockchainRewindHead added in v0.1.3

type BlockchainRewindHead struct {
	Duration    string                  `toml:"duration"`
	NodesConfig []*BlockchainNodeConfig `toml:"nodes"`
}

type BlockchainRewindHeadExperiment added in v0.1.3

type BlockchainRewindHeadExperiment struct {
	ExperimentName        string    `yaml:"experimentName"`
	Metadata              *Metadata `yaml:"metadata"`
	Namespace             string    `yaml:"namespace"`
	PodName               string    `yaml:"podName"`
	ExecutorPodPrefix     string    `yaml:"executorPodPrefix"`
	ExecutorContainerName string    `yaml:"executorContainerName"`
	NodeInternalHTTPURL   string    `yaml:"nodeInternalHTTPURL"`
	Blocks                int64     `yaml:"blocks"`
}

func (BlockchainRewindHeadExperiment) String added in v0.1.3

type CRD added in v0.1.4

type CRD struct {
	Kind       string `yaml:"kind"`
	APIVersion string `yaml:"apiVersion"`
	Metadata   struct {
		Name      string `yaml:"name"`
		Namespace string `yaml:"namespace"`
	} `yaml:"metadata"`
	Spec interface{} `yaml:"spec"` // Use interface{} if the spec can have various structures
}

type ChaosSpecs

type ChaosSpecs struct {
	ExperimentsByType map[string]map[string]string
}

func (*ChaosSpecs) Dump

func (m *ChaosSpecs) Dump(dir string) error

type CommonExperimentMeta

type CommonExperimentMeta struct {
	Kind     string `yaml:"kind"`
	Metadata struct {
		Name      string `yaml:"name"`
		Namespace string `yaml:"namespace"`
	} `yaml:"metadata"`
}

type Config

type Config struct {
	Havoc *Havoc `toml:"havoc"`
}

func DefaultConfig

func DefaultConfig() *Config

func ReadConfig

func ReadConfig(path string) (*Config, error)

func (*Config) Validate

func (c *Config) Validate() []error

type Controller

type Controller struct {
	// contains filtered or unexported fields
}

func NewController

func NewController(cfg *Config) (*Controller, error)

func (*Controller) AnnotateExperiment

func (m *Controller) AnnotateExperiment(a *ExperimentAction) error

AnnotateExperiment sends annotation marker to Grafana dashboard

func (*Controller) ApplyAndAnnotate

func (m *Controller) ApplyAndAnnotate(exp *NamedExperiment) error

func (*Controller) ApplyCustomKindChaosFile added in v0.1.3

func (m *Controller) ApplyCustomKindChaosFile(exp *NamedExperiment, chaosType string, wait bool) error

func (*Controller) ApplyExperiment added in v0.1.4

func (m *Controller) ApplyExperiment(exp *NamedExperiment, wait bool) error

func (*Controller) GenerateSpecs

func (m *Controller) GenerateSpecs(ns string) error

GenerateSpecs generates specs from namespace, should be used programmatically in tests

func (*Controller) GetPodsInfo

func (m *Controller) GetPodsInfo(namespace string) (*PodsListResponse, error)

GetPodsInfo gets info about all the pods in the namespace

func (*Controller) ParseOpenAPISpecs added in v0.2.1

func (m *Controller) ParseOpenAPISpecs() ([]*OAPISpecData, error)

ParseOpenAPISpecs parses OpenAPI spec methods

func (*Controller) ReadExperimentsFromDir

func (m *Controller) ReadExperimentsFromDir(expTypes []string, dir string) ([]*NamedExperiment, error)

func (*Controller) Run

func (m *Controller) Run() error

func (*Controller) Stop

func (m *Controller) Stop() []error

func (*Controller) Wait

func (m *Controller) Wait() []error

type CurrentBlockResponse added in v0.1.3

type CurrentBlockResponse struct {
	Result string `json:"result"`
}

type EventJSONItemResponse

type EventJSONItemResponse struct {
	APIVersion     string    `json:"apiVersion"`
	Count          int       `json:"count"`
	EventTime      any       `json:"eventTime"`
	FirstTimestamp time.Time `json:"firstTimestamp"`
	InvolvedObject struct {
		APIVersion      string `json:"apiVersion"`
		Kind            string `json:"kind"`
		Name            string `json:"name"`
		Namespace       string `json:"namespace"`
		ResourceVersion string `json:"resourceVersion"`
		UID             string `json:"uid"`
	} `json:"involvedObject"`
	Kind          string    `json:"kind"`
	LastTimestamp time.Time `json:"lastTimestamp"`
	Message       string    `json:"message"`
	Metadata      struct {
		Annotations struct {
			ChaosMeshOrgType string `json:"chaos-mesh.org/type"`
		} `json:"annotations"`
		CreationTimestamp time.Time `json:"creationTimestamp"`
		Name              string    `json:"name"`
		Namespace         string    `json:"namespace"`
		ResourceVersion   string    `json:"resourceVersion"`
		UID               string    `json:"uid"`
	} `json:"metadata"`
	Reason             string `json:"reason"`
	ReportingComponent string `json:"reportingComponent"`
	ReportingInstance  string `json:"reportingInstance"`
	Source             struct {
		Component string `json:"component"`
	} `json:"source"`
	Type string `json:"type"`
}

type EventsJSONResponse

type EventsJSONResponse struct {
	APIVersion string                   `json:"apiVersion"`
	Items      []*EventJSONItemResponse `json:"items"`
	Kind       string                   `json:"kind"`
	Metadata   struct {
		ResourceVersion string `json:"resourceVersion"`
	} `json:"metadata"`
}

type ExperimentAction

type ExperimentAction struct {
	Name           string
	ExperimentKind string
	ExperimentSpec string
	TimeStart      int64
	TimeEnd        int64
}

type ExperimentAnnotationBody

type ExperimentAnnotationBody struct {
	DashboardUID string   `json:"dashboardUID"`
	Time         int64    `json:"time"`
	TimeEnd      int64    `json:"timeEnd"`
	Tags         []string `json:"tags"`
	Text         string   `json:"text"`
}

type ExternalTargets

type ExternalTargets struct {
	Duration string   `toml:"duration"`
	URLs     []string `toml:"urls"`
}

type Failure

type Failure struct {
	Duration        string   `toml:"duration"`
	GroupPercentage []string `toml:"group_percentage"`
	GroupFixed      []string `toml:"group_fixed"`
}

type Grafana

type Grafana struct {
	URL           string   `toml:"grafana_url"`
	Token         string   `toml:"grafana_token"`
	DashboardUIDs []string `toml:"dashboard_uids"`
}

type GroupInfo added in v0.2.0

type GroupInfo struct {
	Label        string
	PodsAffected int
}

type HTTPExperiment added in v0.2.1

type HTTPExperiment struct {
	ExperimentName string
	Metadata       *Metadata
	Namespace      string
	Mode           string
	ModeValue      string
	Selector       string
	PodName        string
	Port           int64
	Target         string
	Path           string
	Method         string
	Abort          bool
	Duration       string
}

func (HTTPExperiment) String added in v0.2.1

func (m HTTPExperiment) String() (string, error)

type Havoc

type Havoc struct {
	Dir                  string                `toml:"dir"`
	ExperimentTypes      []string              `toml:"experiment_types"`
	NamespaceLabelFilter string                `toml:"namespace_label_filter"`
	ComponentLabelKey    string                `toml:"component_label_key"`
	IgnoredPods          []string              `toml:"ignore_pods"`
	IgnoreGroupLabels    []string              `toml:"ignore_group_labels"`
	Failure              *Failure              `toml:"failure"`
	Latency              *Latency              `toml:"latency"`
	NetworkPartition     *NetworkPartition     `toml:"network_partition"`
	StressMemory         *StressMemory         `toml:"stress_memory"`
	StressCPU            *StressCPU            `toml:"stress_cpu"`
	ExternalTargets      *ExternalTargets      `toml:"external_targets"`
	BlockchainRewindHead *BlockchainRewindHead `toml:"blockchain_rewind_head"`
	OpenAPI              *OpenAPI              `toml:"openapi"`
	Monkey               *Monkey               `toml:"monkey"`
	Grafana              *Grafana              `toml:"grafana"`
}

type Latency

type Latency struct {
	Duration        string   `toml:"duration"`
	Latency         string   `toml:"latency"`
	GroupPercentage []string `toml:"group_percentage"`
	GroupFixed      []string `toml:"group_fixed"`
}

type ManifestPart

type ManifestPart struct {
	Kind              string
	Name              string
	FlattenedManifest map[string]interface{}
}

type Metadata added in v0.1.4

type Metadata struct {
	Name   string            `json:"name"`
	Labels map[string]string `json:"labels"`
}

type Monkey

type Monkey struct {
	Duration string `toml:"duration"`
	Cooldown string `toml:"cooldown"`
	Mode     string `toml:"mode"`
}

type NamedExperiment

type NamedExperiment struct {
	CRD
	Name     string
	Path     string
	CRDBytes []byte
}

func NewNamedExperiment added in v0.1.4

func NewNamedExperiment(expPath string) (*NamedExperiment, error)

type NetworkChaosExperiment

type NetworkChaosExperiment struct {
	ExperimentName string
	Mode           string
	ModeValue      string
	Namespace      string
	Duration       string
	Latency        string
	PodName        string
	Selector       string
}

func (NetworkChaosExperiment) String

func (m NetworkChaosExperiment) String() (string, error)

type NetworkChaosExternalPartitionExperiment

type NetworkChaosExternalPartitionExperiment struct {
	ExperimentName string
	Namespace      string
	Duration       string
	PodName        string
	ExternalURL    string
}

func (NetworkChaosExternalPartitionExperiment) String

type NetworkChaosGroupPartitionExperiment

type NetworkChaosGroupPartitionExperiment struct {
	ExperimentName string
	ModeTo         string
	ModeToValue    string
	ModeFrom       string
	ModeFromValue  string
	Direction      string
	Namespace      string
	Duration       string
	SelectorFrom   string
	SelectorTo     string
}

func (NetworkChaosGroupPartitionExperiment) String

type NetworkPartition

type NetworkPartition struct {
	Duration        string   `toml:"duration"`
	Label           string   `toml:"label"`
	GroupPercentage []string `toml:"group_percentage"`
	GroupFixed      []string `toml:"group_fixed"`
}

type OAPISpecData added in v0.2.1

type OAPISpecData struct {
	Port     int64
	RawPaths []string
	SpecData map[string]*openapi3.PathItem
}

type OpenAPI added in v0.2.1

type OpenAPI struct {
	Mapping         map[string]*OpenApiSpecInfo `toml:"mapping"`
	Duration        string                      `toml:"duration"`
	GroupPercentage []string                    `toml:"group_percentage"`
	GroupFixed      []string                    `toml:"group_fixed"`
}

type OpenApiSpecInfo added in v0.2.1

type OpenApiSpecInfo struct {
	SpecToPortMappings []*SpecToPort `toml:"spec_to_port"`
}

type PodFailureExperiment

type PodFailureExperiment struct {
	ExperimentName string
	Mode           string
	ModeValue      string
	Namespace      string
	Duration       string
	PodName        string
	Selector       string
}

func (PodFailureExperiment) String

func (m PodFailureExperiment) String() (string, error)

type PodResponse

type PodResponse struct {
	Metadata struct {
		Name   string            `json:"name"`
		Labels map[string]string `json:"labels"`
	} `json:"metadata"`
}

PodResponse pod info response from kubectl in JSON

type PodStressCPUExperiment

type PodStressCPUExperiment struct {
	ExperimentName string
	Mode           string
	ModeValue      string
	Namespace      string
	Workers        int
	Load           int
	Duration       string
	PodName        string
	Selector       string
}

func (PodStressCPUExperiment) String

func (m PodStressCPUExperiment) String() (string, error)

type PodStressMemoryExperiment

type PodStressMemoryExperiment struct {
	ExperimentName string
	Mode           string
	ModeValue      string
	Namespace      string
	Workers        int
	Memory         string
	Duration       string
	PodName        string
	Selector       string
}

func (PodStressMemoryExperiment) String

func (m PodStressMemoryExperiment) String() (string, error)

type PodsListResponse

type PodsListResponse struct {
	Items []*PodResponse `json:"items"`
}

PodsListResponse pod list response from kubectl in JSON

type SpecToPort added in v0.2.1

type SpecToPort struct {
	Port int64  `toml:"port"`
	Path string `toml:"path"`
}

type StressCPU

type StressCPU struct {
	Duration        string   `toml:"duration"`
	Workers         int      `toml:"workers"`
	Load            int      `toml:"load"`
	GroupPercentage []string `toml:"group_percentage"`
	GroupFixed      []string `toml:"group_fixed"`
}

type StressMemory

type StressMemory struct {
	Duration        string   `toml:"duration"`
	Workers         int      `toml:"workers"`
	Memory          string   `toml:"memory"`
	GroupPercentage []string `toml:"group_percentage"`
	GroupFixed      []string `toml:"group_fixed"`
}

Directories

Path Synopsis
k8schaos module
lib module

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL