README
¶
go-heapdump-threshold
If a container uses more memory than assigned to it, Kubernetes will kill it with a OOMKilled error. The problem with this is that after the container was killed, it is hard for engineers to debug what caused the unexpected memory usage. This package helps solve this problem.
With go-heap-dump you can define a memory limit and a memory threshold. If the memory consupmtion of your Go application exceeds the threshold, pprof will be called to dump a memory profile to the filesystem.
Usage
For a entire code and Kubernetes pod example see the example directory of this project.
- Use the HeapProfiler in your Go application
// In this example this env variable is set through K8s downward API
// https://kubernetes.io/docs/concepts/workloads/pods/downward-api/
strMemoryLimit := os.Getenv("MEMORY_LIMIT")
limit, err := strconv.ParseUint(strMemoryLimit, 10, 64)
if err != nil {
fmt.Println(err)
}
// This create a NewHeapProfiler with following configuration
// - Start to dump heap profiles when used heap is greater than or equal to 80% of the limit
// - A limit defined by env variable MEMORY_LIMIT in bytes e.g. 100000000 for 100MB
// - Create files at the "./dump" directory
// - Wait for 1 Minute once a heap profile was written before writing another one
heapProfiler := heapprofiler.NewHeapProfiler(0.80, limit, "./dump", 1*time.Minute)
// Start a background process that checks the threshold every 30 seconds and dumps a heap profile if necessary
ctx := context.Background()
go heapProfiler.DumpHeapOnThreshhold(ctx, 30*time.Second)
- Use Kubernetes Downward API to propagate the memory limit
...
env:
- name: MEMORY_LIMIT
valueFrom:
resourceFieldRef:
containerName: example-app
resource: limits.memory
...
- Use a volume to store the heap profiles between container restarts
...
volumeMounts:
- mountPath: ./dump
name: dump
volumes:
- name: dump
emptyDir: {}
...
Once the allocated memory in the container process reaches the defined threshhold of 0.80*limit it will dump a pprof memory profile to the /dump/heapdump/<datetime>. By using an emptyDir this dump is persisted throughout container restarts. If you'd like to persist it throughout different Pods use a PV instead.
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( // DefaultHeapProfilerBackoff is the default setting for the time to wait between heap dumps when hitting threshold in seconds DefaultHeapProfilerBackoff = time.Second * 30 )
Functions ¶
This section is empty.
Types ¶
type HeapProfiler ¶
type HeapProfiler struct {
// contains filtered or unexported fields
}
HeapProfiler is used to start a ticker that periodically checks if heap memory consumption exceed the thresholdFraction, if so the heap gets dumped to a file in Directory subject to Backoff.
func NewHeapProfiler ¶
func NewHeapProfiler(thresholdFraction float64, limitBytes uint64, directory string, backoff time.Duration) *HeapProfiler
NewHeapProfiler creates a new instance of HeapProfiler setting the given values. It appends a subdirectory to the directory to prevent acidental deletion of user files. If 0 is provides as limitBytes the limitBytes is set to 1, thus the heap dump will always run. backoff limits the maximum frequency of creating heap dumps
func (*HeapProfiler) DumpHeapOnThreshhold ¶
func (p *HeapProfiler) DumpHeapOnThreshhold(ctx context.Context, interval time.Duration)
DumpHeapOnThreshhold runs for as long as ctx is valid, checking heap usage with the given interval. Maximum frequency of creating heap dumps is limited by the configured backoff.
Source Files
Directories