Documentation
¶
Overview ¶
Package antcache implements an HTTP client that caches responses.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Cache ¶
type Cache struct {
// contains filtered or unexported fields
}
Cache implements an HTTP cache.
func (*Cache) Do ¶
Do performs the given request.
The method initially checks if the request can be cached if so, it will lookup a response that matches the request and return it if it's fresh, or was validated.
When the request is not cacheable, the method simply calls the underlying client with the request and never caches the response.
When a response is not found, the method calls the underlying client and if the response can be stored, it will store it when its body is closed, if the body is not closed, the response is never stored.
If there was an error loading a cached response the method returns the error and discards the response's body, if an error occurs when storing the response body, the response's Close() method will return the error.
type Client ¶
type Client interface {
// Do performs the given request.
Do(req *http.Request) (*http.Response, error)
}
Client represents an HTTP client.
type DebugFunc ¶
type DebugFunc func(format string, args ...interface{})
DebugFunc represents a debug func.
By default the diskstore outputs no debug logs.
type DiskOption ¶
DiskOption represents a disk option.
func Compress ¶
func Compress() DiskOption
Compress makes the diskstore compress and uncompress all cached items.
Note that the diskstore will not check the cached item before attempting to de-compress therefore the items are not interchangeable between to disks where one has no compression and the other one has compression.
By default compression is turned off.
func Debug ¶
func Debug(f DebugFunc) DiskOption
Debug sets the debug logging func.
When set on the diskstore debug will be enabled and debug logs are written to it.
By default the func is set to nil which disables debug logging.
Example:
Open("root", Debug(log.Printf))
Debug logs are automatically prefixed with `"antcache/disk: "`.
func Maxage ¶
func Maxage(age time.Duration) DiskOption
Maxage sets the maxage to age.
When <= 0, the disk will not track file age and will not remove files that have expired.
Defaults to 24 hours.
func Maxsize ¶
func Maxsize(size int64) DiskOption
Maxsize sets the maxsize to size.
When <= 0, the disk will not track the disk the file sizes and remove files to ensure the disk usage stays constant.
Defaults to 1gb.
func SweepEvery ¶
func SweepEvery(d time.Duration) DiskOption
SweepEvery sweeps the files every d.
By default the disk will sweep all files every 5 minutes, when d <= 0, no background file sweeper is done and so the disk size may keep growing.
The disk will remove all files that have exceeded the maxage when maxsize is set, the sweeper will remove files until the maxsize is reached.
type Diskstore ¶
type Diskstore struct {
// contains filtered or unexported fields
}
Diskstore implements disk cache storage.
The storage is expected to be configured with an existing directory `path` where it will write all cached responses.
The storage ensures that store calls are not visible to load calls until the file is written to disk successfully and fsynced, it does this by writing a temporary file, fsyncing it and then renaming it to the expected filename.
When the disk is configured with invalid directory name all its method return the same error.
func Open ¶
func Open(path string, opts ...DiskOption) (*Diskstore, error)
Open opens a new disk storage.
It is up to the caller to ensure that the given path will not be changed by different processes, the diskstore doesn't implement any filesystem level locking.
func (*Diskstore) Wait ¶
Wait waits for the disk to read all files.
When the disk is initialized it will spawn a goroutine to read all files in the configured path, if there are many files it will typically take a while.
The method returns the context's error if canceled, otherwise it will block until the disk cache is warm.
type Option ¶
Option represents a cache option.
func Aggressive ¶
Aggressive returns an aggressive cache with age.
Unlike the default RFC7234 cache, this caching strategy will cache all GET/HEAD requests unless they specify a Range/Content-Range headers, the Authorization header and the "no-store" cache directive.
This makes the aggressive cache arguably better for crawling since it can cache responses up to a specific age but still allows you to bypass the cache if you set the "no-cache" directive.
The cacher will also ignore any "no-cache" and "no-store" or other directives from the response, since some websites never implement proper caching.
When age <= 0, the default of 24 hours is used.
type Storage ¶
type Storage interface {
// Store stores the given response.
//
// The method is called just after the response's body is
// closed, the value contains the full response including headers.
Store(ctx context.Context, key uint64, value []byte) error
// Load loads a response by its key.
//
// When the response is not found, the method returns a nil
// byteslice and a nil error.
//
// The method returns the full response, as stored by `Store()`.
Load(ctx context.Context, key uint64) ([]byte, error)
}
Storage represents the cache storage.
A storage must be safe to use from multiple goroutines.
Source Files