memlimiter

package module
v0.0.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 4, 2022 License: MIT Imports: 11 Imported by: 0

README

Go Reference Go Report Card Coverage CI

MemLimiter

Library that helps to limit memory consumption of your Go service.

Working principles

As of today (Go 1.18), there is a possibility for any Go application to be eventually stopped by OOM killer. The memory leak is because Go runtime knows nothing about the limitations imposed on the process by the operating system (for instance, using cgroups). However, an unexpected termination of a process because of OOM is highly undesirable, as it can lead to cache resetting, data integrity violation, distributed transaction hanging and even cascading failure of a distributed backend. Therefore, services should degrade gracefully instead of immediate stop due to SIGKILL.

A universal solution for programming languages with automatic memory management comprises two parts:

  1. Garbage collection intensification. The more often GC starts, the more garbage will be collected, the fewer new physical memory allocations we have to make for the service’s business logic.
  2. Request throttling. By suppressing some of the incoming requests, we implement the backpressure: the middleware simply cuts off part of the load coming from the client in order to avoid too many memory allocations.

MemLimiter represents a memory budget automated control system that helps to keep the memory consumption of a Go service within a predefined limit.

Memory budget utilization

The core of the MemLimiter is a special object quite similar to P-controller, but with certain specifics (more on that below). Memory budget utilization value acts as an input signal for the controller. We define the $Utilization$ as follows:

$$ Utilization = \frac {NextGC} {RSS_{limit} - CGO} $$

where:

  • $NextGC$ (from here) is a target size for heap, upon reaching which the Go runtime will launch the GC next time;
  • $RSS_{limit}$ is a hard limit for service's physical memory (RSS) consumption (so that exceeding this limit will highly likely result in OOM);
  • $CGO$ is a total size of heap allocations made beyond Cgo borders (within C/C++/.... libraries).

A few notes about $CGO$ component. Allocations made outside of the Go allocator, of course, are not controlled by the Go runtime in any way. At the same time, the memory consumption limit is common for both Go and non-Go allocators. Therefore, if non-Go allocations grow, all we can do is shrink the memory budget for Go allocations (which is why we subtract $CGO$ from the denominator of the previous expression). If your service uses Cgo, you need to figure out how much memory is allocated “on the other side” – otherwise MemLimiter won’t be able to save your service from OOM.

If the service doesn't use Cgo, the $Utilization$ formula is simplified to: $$Utilization = \frac {NextGC} {RSS_{limit}}$$

Control function

The controller converts the input signal into the control signal according to the following formula:

$$ K_{p} = C_{p} \cdot \frac {1} {1 - Utilization} $$

This is not an ordinary definition for a proportional component of the PID-controller, but still the direct proportionality is preserved: the closer the $Utilization$ is to 1 (or 100%), the higher the control signal value. The main purpose of the controller is to prevent a situation in which the next GC launch will be scheduled when the memory consumption exceeds the hard limit (and this will cause OOM).

You can adjust the proportional component control signal strength using a coefficient $C_{p}$. In addition, there is optional exponential averaging of the control signal. This helps to smooth out high-frequency fluctuations of the control signal (but it hardly eliminates self-oscillations).

The control signal is always saturated to prevent extremal values:

$$ Output = \begin{cases} \displaystyle 100 \ \ \ K_{p} \gt 100 \ \displaystyle 0 \ \ \ \ \ \ \ K_{p} \lt 100 \ \displaystyle K_{p} \ \ \ \ otherwise \ \end{cases}$$

Finally we convert the dimensionless quantity $Output$ into specific $GOGC$ (for the further use in debug.SetGCPercent) and $Throttling$ (percentage of suppressed requests) values, however, only if the $Utilization$ exceeds the specified limits:

$$ GC = \begin{cases} \displaystyle Output \ \ \ Utilization \gt DangerZoneGC \ \displaystyle 100 \ \ \ \ \ \ \ \ \ \ otherwise \ \end{cases}$$

$$ Throttling = \begin{cases} \displaystyle Output \ \ \ Utilization \gt DangerZoneThrottling \ \displaystyle 0 \ \ \ \ \ \ \ \ \ \ \ \ \ \ otherwise \ \end{cases}$$

Architecture

The MemLimiter comprises two main parts:

  1. Core implementing the memory budget controller and backpressure subsystems. Core relies on actual statistics provided by stats.ServiceStatsSubscription. In a critical situation, core may gracefully terminate the application with utils.ApplicationTerminator.
  2. Middleware providing request throttling feature for various web frameworks. Every time the server receives a request, it uses middleware to ask the MemLimiter’s core for permission to process this request. Currently, only GRPC is supported, but Middleware is an easily extensible interface, and PRs are welcome.

Architecture

Quick start guide

Services without Cgo

Refer to the example service.

Services with Cgo

Refer to the example service.

You must also provide your own stats.ServiceStatsSubscription and stats.ServiceStats implementations. The latter one must return non-nil stats.ConsumptionReport instances if you want MemLimiter to consider allocations made outside of Go runtime allocator and estimate memory utilization correctly.

Tuning

There are several key settings in MemLimiter configuration:

  • RSSLimit
  • DangerZoneGC
  • DangerZoneThrottling
  • Period
  • WindowSize
  • Coefficient ($C_{p}$)

You have to pick them empirically for your service. The settings must correspond to the business logic features of a particular service and to the workload expected.

We made a series of performance tests with [Allocator][test/allocator] - an example service which does nothing but allocations that reside in memory for some time. We used different settings, applied the same load and tracked the RSS of a process.

Settings ranges:

  • $RSS_{limit} = {1G}$
  • $DangerZoneGC = 50%$
  • $DangerZoneThrottling = 90%$
  • $Period = 100ms$
  • $WindowSize = 20$
  • $C_{p} \in \{0, 0.5, 1, 5, 10, 50, 100\}$

These plots may give you some inspiration on how $C_{p}$ value affects the physical memory consumption other things being equal:

Control params

And the summary plot with RSS consumption dependence on $C_{p}$ value:

RSS

The general conclusion is that:

  • The higher the $C_{p}$ is, the lower the $RSS$ consumption.
  • Too low and too high $C_{p}$ values cause self-oscillation of control parameters.
  • Disabling MemLimiter causes OOM.

TODO

  • Extend middleware.Middleware to support more frameworks.
  • Add GOGC limitations to prevent death spirals.
  • Support popular Cgo allocators like Jemalloc or TCMalloc, parse their stats to provide information about Cgo memory consumption.

Your PRs are welcome!

Publications

  • Isaev V. A. Go runtime high memory consumption (in Russian). Evrone Go meetup. 2022. Preview

Documentation

Overview

Package memlimiter - memory budget control subsystem for Go services. It tracks memory budget utilization and tries to stabilize memory usage with backpressure (GC and request throttling) techniques.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Config 

type Config struct {
	// ControllerNextGC - NextGC-based controller
	ControllerNextGC *nextgc.ControllerConfig `json:"controller_nextgc"` //nolint:tagliatelle

}

Config - high-level MemLimiter config.

func (*Config) Prepare 

func (c *Config) Prepare() error

Prepare validates config.

type Option added in v0.0.2 

type Option interface {
	// contains filtered or unexported methods
}

Option - MemLimiter constructor options.

func WithBackpressureOperator added in v0.0.2 

func WithBackpressureOperator(val backpressure.Operator) Option

WithBackpressureOperator allows client to provide customized backpressure.Operator; that's especially useful when implementing backpressure logic on the application side.

func WithServiceStatsSubscription added in v0.0.2 

func WithServiceStatsSubscription(val stats.ServiceStatsSubscription) Option

WithServiceStatsSubscription allows client to provide own implementation of service stats subscription.

type Service 

type Service interface {
	Middleware() middleware.Middleware
	GetStats() (*stats.MemLimiterStats, error)
	// Quit terminates service gracefully.
	Quit()
}

Service - a high-level interface for a memory usage control subsystem.

func NewServiceFromConfig 

func NewServiceFromConfig(
	logger logr.Logger,
	cfg *Config,
	options ...Option,
) (Service, error)

NewServiceFromConfig - main entrypoint for MemLimiter.

Source Files

View all Source files

Directories

Path Synopsis
Package backpressure contains code applying control signals issued by controller to Go runtime and and to gRPC server.
 Package backpressure contains code applying control signals issued by controller to Go runtime and and to gRPC server.
Package controller contains common types for different possible implementations of memory usage controller.
 Package controller contains common types for different possible implementations of memory usage controller.
nextgc
Package nextgc provides the implementation of memory usage controller, which aims to keep Go Runtime NextGC value lower than the RSS consumption hard limit to prevent OOM errors.
 Package nextgc provides the implementation of memory usage controller, which aims to keep Go Runtime NextGC value lower than the RSS consumption hard limit to prevent OOM errors.
Package middleware provides code that helps to integrate MemLimiter's backpressure subsystem with modern web frameworks.
 Package middleware provides code that helps to integrate MemLimiter's backpressure subsystem with modern web frameworks.
Package stats contains various data types describing service statistics MemLimiter relies on, as well as its own statistics.
 Package stats contains various data types describing service statistics MemLimiter relies on, as well as its own statistics.
test
allocator
allocator/app
Package app contains all the necessary things to build simple web application using MemLimiter.
 Package app contains all the necessary things to build simple web application using MemLimiter.
allocator/perf
Package perf contains performance client.
 Package perf contains performance client.
allocator/schema
allocator/server
Package server is a simple GRPC service performing useless memory allocations.
 Package server is a simple GRPC service performing useless memory allocations.
allocator/tracker
Package tracker contains logic of service stats persistence.
 Package tracker contains logic of service stats persistence.
Package utils provides various utilities and helpers.
 Package utils provides various utilities and helpers.
breaker
Package breaker contains useful thread-safe abstraction the helps to control lifetime of actors, background tasks, pools etc.
 Package breaker contains useful thread-safe abstraction the helps to control lifetime of actors, background tasks, pools etc.
config/bytes
Package bytes helps to represent human-readable size values in JSON.
 Package bytes helps to represent human-readable size values in JSON.
config/duration
Package duration helps to represent human-readable duration values in JSON.
 Package duration helps to represent human-readable duration values in JSON.
config/prepare
Package prepare provides a function to validate configuration recursively.
 Package prepare provides a function to validate configuration recursively.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.