taskcluster-worker: github.com/taskcluster/taskcluster-worker/runtime/gc Index | Files

package gc

import "github.com/taskcluster/taskcluster-worker/runtime/gc"

Package gc contains the GarbageCollector which allows cacheable resources to register themselves for disposal when we run low on resources.

The idea is that cacheable resources like cache folders, docker images, downloaded artifacts. Any resource that can be recreated if needed, but that we would like to keep around for as long as possible, assuming it doesn't affect system resources.

To do this we need to be able to dispose resources when the system runs low on disk space or memory. This easily gets complicated, especially as we may want to optimize by diposing least-recently-used first. So to simplify it cacheable resources should implement the Disposable interface and be registered with the GarbageCollector, so it prioritize disposal, even when we have different types of cacheable resources.


Package Files

auxiliary.go disposable.go doc.go gc.go


var (
    // ErrDisposableSizeNotSupported is used by Disposable to indicate that a resource
    // doesn't support a size function like MemorySize() or DiskSize()
    ErrDisposableSizeNotSupported = errors.New("Disposable instance doesn't support")
    // ErrDisposableInUse is used by Disposable to indicate that a resource is currently in
    // use and cannot be disposed
    ErrDisposableInUse = errors.New("Disposable is currently in use")

type Disposable Uses

type Disposable interface {
    // Get memory size used by resource, return ErrDisposableSizeNotSupported
    // if you don't know how much memory it uses, but knows that it uses some
    // non-trivial amount. If it just uses a small fixed amount, you can return
    // 1 for simplicity.
    MemorySize() (uint64, error)
    // Get disk space used by resource, return ErrDisposableSizeNotSupported
    // if you don't know how much disk space it uses, but knows that it uses some
    // non-trivial amount. If it just uses a small fixed amount, you can return
    // 1 for simplicity.
    DiskSize() (uint64, error)
    // Clean up after this resource, return ErrDisposableInUse if the resource is
    // in use. Note that implementors should use this method to also remove any
    // references to this resource from the resource manager, which typically has
    // an internal list of cached resources.
    // Warning, return an error other than ErrDisposableInUse and the worker will
    // panic, abort tasks, alert operation and crash.
    Dispose() error
    // Last time the cache was used
    LastUsed() time.Time

The Disposable Interface to be implemented by resources that can be managed by the GarbageCollector. To have the resource garbage collected you must call GarbageCollector.Register(&resource). The garbage collector will call resource.Dispose() when it wants to remove your resource, you must check that the resource isn't in use and return ErrDisposableInUse if it is.

Warning, return any error other than ErrDisposableSizeNotSupported or ErrDisposableInUse and it will result in a internal error aborting all tasks, reporting to operators and crashing the worker. This is because we can't have workers leak resources, so errors freeing resources has to be critical feel free to do retries and other things to avoid returning an error that we can't handle.

Note, all methods on this must be thread-safe, an implementation of most of these methods can be obtained by extending DisposableResource.

type DisposableResource Uses

type DisposableResource struct {
    // contains filtered or unexported fields

The DisposableResource implements as thread-safe reference counted resource for the Disposable interface. Such that Acquire/Release updates the LastUsed time stamp.

Implementors of the Disposable interface using this base struct should implement Dispose such that it returns ErrDisposableSizeNotSupported if currently in use. This can be easily actieved using CanDispose.

Implementor should also implement EstimateMemorySize() and EstimateDiskSize(), but these are unrelated to this trait.

func (*DisposableResource) Acquire Uses

func (r *DisposableResource) Acquire()

Acquire the resource incrementing the reference count by one

func (*DisposableResource) CanDispose Uses

func (r *DisposableResource) CanDispose() error

CanDispose returns ErrDisposableInUse if the resource is currently being used. This is intended to be used by implementors of Dispose.

func (*DisposableResource) DiskSize Uses

func (r *DisposableResource) DiskSize() (uint64, error)

DiskSize is the stub implementation of Disposable.DiskSize returning ErrDisposableSizeNotSupported, implementors really ought to overwrite this.

func (*DisposableResource) LastUsed Uses

func (r *DisposableResource) LastUsed() time.Time

LastUsed returns the last time this resource was used, as an implementation for the Disposable interface

func (*DisposableResource) MemorySize Uses

func (r *DisposableResource) MemorySize() (uint64, error)

MemorySize is the stub implementation of Disposable.MemorySize returning ErrDisposableSizeNotSupported, implementors really ought to overwrite this.

func (*DisposableResource) Release Uses

func (r *DisposableResource) Release()

Release the resource decrementing the reference count by one and updating the lastUsed time stamp to now.

type GarbageCollector Uses

type GarbageCollector struct {
    // contains filtered or unexported fields

GarbageCollector can be used register Disposable resources which will then be diposed when not in use and the system is low on available disk space or memory.

func New Uses

func New(storageFolder string, minimumDiskSpace, minimumMemory int64) *GarbageCollector

New creates a GarbageCollector which uses storageFolder to test for available diskspace and tries to ensure that minimumDiskSpace and minimumMemory is satisfied after each call to Collect()

func (*GarbageCollector) Collect Uses

func (gc *GarbageCollector) Collect() error

Collect runs garbage collection and reclaims resources, attempting to satisfy minimumMemory and minimumDiskSpace, if possible.

func (*GarbageCollector) CollectAll Uses

func (gc *GarbageCollector) CollectAll() error

CollectAll disposes all resources that can be disposed.

All resources not returning: ErrDisposableInUse. This is useful for testing when implementing resources.

func (*GarbageCollector) Register Uses

func (gc *GarbageCollector) Register(resource Disposable)

Register takes a Disposable resource for the GarbageCollector to manage.

GarbageCollector will attempt to to call resource.Dispose() at any time, you should return ErrDisposableInUse if the resource is in use.

func (*GarbageCollector) Unregister Uses

func (gc *GarbageCollector) Unregister(resource Disposable) bool

Unregister will inform the GarbageCollector to stop tracking the given resource. Returns true if the resource was tracked, notice that the resource maybe have been disposed of while waiting for the GC lock. Say if the GC was running when you made this call.

Note, you don't have to use this method. When you resource is in a state where you don't want it to be disposed just ensure that Dispose() returns ErrDisposableInUse.

type ResourceTracker Uses

type ResourceTracker interface {
    Register(resource Disposable)
    Unregister(resource Disposable) bool

A ResourceTracker is an object capable of tracking resources.

This is the interface for the GarbageCollector that should be exposed to engines and plugins. So they can't initiate garbage collection.


Not sure this base struct is useful at all...

Package gc imports 6 packages (graph) and is imported by 8 packages. Updated 2017-12-25. Refresh now. Tools for package owners.