cos

package
v1.3.19 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 6, 2023 License: MIT Imports: 44 Imported by: 0

Documentation

Overview

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2021-2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package cmn provides common constants, types, and utilities for AIS clients and AIStore.

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2021-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2018-2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Package cmn provides common constants, types, and utilities for AIS clients and AIStore.

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2022-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2022, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects.

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Package cos provides common low-level types and utilities for all aistore projects

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Index

Constants

View Source 
const (
	ChecksumNone   = "none"
	ChecksumXXHash = "xxhash"
	ChecksumMD5    = "md5"
	ChecksumCRC32C = "crc32c"
	ChecksumSHA256 = "sha256" // crypto.SHA512_256 (SHA-2)
	ChecksumSHA512 = "sha512" // crypto.SHA512 (SHA-2)
)

checksums

View Source 
const (
	PermRWR   os.FileMode = 0o640
	PermRWRR  os.FileMode = 0o644 // (archived)
	PermRWXRX os.FileMode = 0o750
)

POSIX permissions

View Source 
const (
	CtxReadWrapper contextID = "readWrapper" // context key for ReadWrapperFunc
	CtxSetSize     contextID = "setSize"     // context key for SetSizeFunc
	CtxOriginalURL contextID = "origURL"     // context key for OriginalURL for HTTP cloud
)
View Source 
const (
	ContentJSON           = "application/json"
	ContentJSONCharsetUTF = "application/json; charset=utf-8"
	ContentMsgPack        = "application/msgpack"
	ContentXML            = "application/xml"
	ContentBinary         = "application/octet-stream"

	// not currently used:
	ContentZip = "application/zip"
	ContentTar = "application/x-tar" // not present in IANA reg, mozilla.org has it though
)

standard MIME types - https://www.iana.org/assignments/media-types/media-types.xhtml - https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Common_types

View Source 
const (
	// range to read:
	HdrRange          = "Range" // Ref: https://www.rfc-editor.org/rfc/rfc7233#section-2.1
	HdrRangeValPrefix = "bytes="
	// range read response:
	HdrContentRange          = "Content-Range"
	HdrContentRangeValPrefix = "bytes " // Ref: https://tools.ietf.org/html/rfc7233#section-4.2
	HdrAcceptRanges          = "Accept-Ranges"

	// content length & type
	HdrContentType        = "Content-Type"
	HdrContentTypeOptions = "X-Content-Type-Options"
	HdrContentLength      = "Content-Length"

	// misc. gen
	HdrUserAgent = "User-Agent"
	HdrAccept    = "Accept"
	HdrLocation  = "Location"
	HdrServer    = "Server"
	HdrETag      = "ETag" // Ref: https://developer.mozilla.org/en-US/docs/Web/HTTP/Hdrs/ETag
)

Ref: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers

View Source 
const (
	// https://cloud.google.com/storage/docs/xml-api/reference-headers
	GsCksumHeader   = "x-goog-hash"
	GsVersionHeader = "x-goog-generation"

	// https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html
	// https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html
	S3CksumHeader   = "ETag"
	S3VersionHeader = "x-amz-version-id"

	// s3 api request headers
	S3HdrObjSrc        = "x-amz-copy-source"
	S3HdrMptCnt        = "x-amz-mp-parts-count"
	S3HdrContentSHA256 = "x-amz-content-sha256"
	S3HdrBckRegion     = "x-amz-bucket-region"

	S3ChecksumCRC32  = "x-amz-checksum-crc32"
	S3ChecksumCRC32C = "x-amz-checksum-crc32c"
	S3ChecksumSHA1   = "x-amz-checksum-sha1"
	S3ChecksumSHA256 = "x-amz-checksum-sha256"
	S3LastModified   = "Last-Modified"

	S3MetadataChecksumType = "x-amz-meta-ais-cksum-type"
	S3MetadataChecksumVal  = "x-amz-meta-ais-cksum-val"

	// https://docs.microsoft.com/en-us/rest/api/storageservices/get-blob-properties#response-headers
	AzCksumHeader   = "Content-MD5"
	AzVersionHeader = "ETag"
)

provider-specific headers (=> custom props, and more)

View Source 
const (
	QuantityPercent = "percent"
	QuantityBytes   = "bytes"
)
View Source 
const (
	LetterRunes = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
	LenRunes    = len(LetterRunes)
)
View Source 
const (
	UnitsIEC = "iec" // default
	UnitsSI  = "si"  // NOTE: currently, SI system is CLI-only (compare with cmn/cos/size.go)
	UnitsRaw = "raw"
)
View Source 
const (
	KiB = 1024
	MiB = 1024 * KiB
	GiB = 1024 * MiB
	TiB = 1024 * GiB
)

IEC (binary) units

View Source 
const (
	KB = 1000
	MB = 1000 * KB
	GB = 1000 * MB
	TB = 1000 * GB
)

IS (metric) units

View Source 
const (
	StampMicro = "15:04:05.000000" // time.StampMicro without a date, same as glog
	StampSec   = "15:04:05"        // time.Stamp without a date

	// S3 ListObjectsV2
	// https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html
	ISO8601 = "2006-01-02T15:04:05.000Z"

	// S3 HeadObject
	// https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html#API_HeadObject_Examples
	RFC1123GMT = "Mon, 17 Dec 2012 02:14:10 GMT"
)

in addition to standard layouts at /usr/local/go/src/time/format.go

View Source 
const (
	SizeofI64 = int(unsafe.Sizeof(uint64(0)))
	SizeofI32 = int(unsafe.Sizeof(uint32(0)))
	SizeofI16 = int(unsafe.Sizeof(uint16(0)))
)

assorted common constants

View Source 
const (
	OnlyNice = "may only contain letters, numbers, dashes (-), underscores (_), and dots (.)"
	OnlyPlus = OnlyNice + ", and dots (.)"
)
View Source 
const ContentLengthUnknown = -1
View Source 
const MLCG32 = 1103515245 // xxhash seed
View Source 
const (
	// Number of sync maps
	MultiSyncMapCount = 0x40
)
View Source 
const SizeofLen = SizeofI32

Internally size of byte slice or string is packed as this integer type. Since binary packed structs are used for fairly small amount of data, it is possible to use int32 instead of int64 to keep the length of the data.

Variables

View Source 
var (
	ErrInvalidQuantityUsage = errors.New("invalid quantity, format should be '81%' or '1GB'")

	ErrInvalidQuantityPercent = errors.New("percent must be in the range (0, 100)")
	ErrInvalidQuantityBytes   = errors.New("value (bytes) must be non-negative")
)
View Source 
var ErrBufferUnderrun = errors.New("buffer underrun")
View Source 
var ErrEmptyTemplate = errors.New("empty range template")
View Source 
var JSON jsoniter.API

JSON is used to Marshal/Unmarshal API json messages and is initialized in init function.

View Source 
var NoneCksum = NewCksum(ChecksumNone, "")

Functions

func Abs 

func Abs(a int) int

func AbsI64 

func AbsI64(a int64) int64

func AnyHasPrefixInSlice 

func AnyHasPrefixInSlice(prefix string, arr []string) bool

func Assert 

func Assert(cond bool)

func AssertMsg 

func AssertMsg(cond bool, msg string)

NOTE: when using Sprintf and such, `if (!cond) { AssertMsg(false, msg) }` is the preferable usage.

func AssertNoErr 

func AssertNoErr(err error)

func Assertf 

func Assertf(cond bool, f string, a ...any)

NOTE: Not to be used in the datapath - consider instead one of the flavors below.

func BHead 

func BHead(b []byte) string

func CapitalizeString 

func CapitalizeString(s string) string

func CeilAlign 

func CeilAlign(val, align uint) uint

CeilAlign returns smallest number bigger or equal to val, which is divisible by align

func CeilAlignInt64 

func CeilAlignInt64(val, align int64) int64

func ClearfAtomic 

func ClearfAtomic(f *uint64, flags uint64) (ok bool)

func Close 

func Close(closer io.Closer)

func ConvertToString 

func ConvertToString(value any) (valstr string, err error)

func CopyStruct 

func CopyStruct(dst, src any)

shallow copy

func CreateDir 

func CreateDir(dir string) error

CreateDir creates directory if does not exist. Does not return error when directory already exists.

func CreateFile 

func CreateFile(fqn string) (*os.File, error)

CreateFile creates a new write-only (O_WRONLY) file with default cos.PermRWR permissions. NOTE: if the file pathname doesn't exist it'll be created. NOTE: if the file already exists it'll be also silently truncated.

func CryptoRandS 

func CryptoRandS(n int) string

func DamerauLevenstheinDistance 

func DamerauLevenstheinDistance(s, t string) int

based on Wikipedia https://en.wikipedia.org/wiki/Damerau–Levenshtein_distance

func DivCeil 

func DivCeil(a, b int64) int64

func DivRound 

func DivRound(a, b int64) int64

func DivRoundU64 

func DivRoundU64(a, b uint64) uint64

func DrainReader 

func DrainReader(r io.Reader)

DrainReader reads and discards all the data from a reader. No need for `io.CopyBuffer` as `io.Discard` has efficient `io.ReaderFrom` implementation.

func Either 

func Either(lhs, rhs string) string

Either returns either lhs or rhs depending on which one is non-empty

func Err2ClientURLErr 

func Err2ClientURLErr(err error) (uerr *url.Error)

func Errorf 

func Errorf(format string, a ...any)

func ExitLog 

func ExitLog(a ...any)

func ExitLogf 

func ExitLogf(f string, a ...any)

+glog

func Exitf 

func Exitf(f string, a ...any)

func ExpandPath 

func ExpandPath(path string) string

ExpandPath replaces common abbreviations in file path (eg. `~` with absolute path to the current user home directory) and cleans the path.

func FastLog2 

func FastLog2(c uint64) uint

FastLog2 returns floor(log2(c))

func FastLog2Ceil 

func FastLog2Ceil(c uint64) uint

func FloodWriter 

func FloodWriter(w io.Writer, n int64) error

FloodWriter writes `n` random bytes to provided writer.

func FlushClose 

func FlushClose(file *os.File) (err error)

func FormatMilli 

func FormatMilli(tm time.Duration) string

FormatMilli returns a duration formatted as milliseconds. For values bigger than millisecond, it returns an integer number "#ms". For values smaller than millisecond, the function returns fractional number "0.##ms"

func FormatNanoTime 

func FormatNanoTime(unixnano int64, format string) string

`unixnano` here is expected to be "nanoseconds since January 1, 1970 UTC"

func FormatNowStamp 

func FormatNowStamp() string

func FormatTime 

func FormatTime(t time.Time, format string) string

func FreeMemToOS 

func FreeMemToOS(d ...time.Duration)

FreeMemToOS calls GC and returns allocated memory to OS after that Use to clean up memory after a huge amount of memory becomes "free" to return it to OS immediately without waiting for GC does it automatically Params: 

d - a delay before starting memory cleanup

func GenDaemonID 

func GenDaemonID() string

func GenTestingDaemonID 

func GenTestingDaemonID(suffix string) string

(when config.TestingEnv)

func GenTie 

func GenTie() string

3-letter tie breaker (fast)

func GenUUID 

func GenUUID() (uuid string)

func HomeConfigDir 

func HomeConfigDir(subdir string) (configDir string)

func HomeDir 

func HomeDir() (string, error)

func Infof 

func Infof(format string, a ...any)

func InitShortID 

func InitShortID(seed uint64)

func IsAlphaNice 

func IsAlphaNice(s string) bool

letters and numbers w/ '-' and '_' permitted with limitations (below) (see OnlyNice above)

func IsAlphaPlus 

func IsAlphaPlus(s string) bool

alpha-numeric++ including letters, numbers, dashes (-), and underscores (_) period (.) is allowed except for '..' (see OnlyPlus above)

func IsAnySetfAtomic 

func IsAnySetfAtomic(f *uint64, flags uint64) (yes bool)

func IsAzureURL 

func IsAzureURL(u *url.URL) bool

func IsEOF 

func IsEOF(err error) bool

including "unexpecting EOF" to accommodate unsized streaming and early termination of the other side (prior to sending the first byte)

func IsErrBadCksum 

func IsErrBadCksum(err error) bool

func IsErrBrokenPipe 

func IsErrBrokenPipe(err error) (yes bool)

func IsErrClientURLTimeout 

func IsErrClientURLTimeout(err error) bool

func IsErrConnectionNotAvail 

func IsErrConnectionNotAvail(err error) (yes bool)

likely out of socket descriptors

func IsErrConnectionRefused 

func IsErrConnectionRefused(err error) (yes bool)

retriable conn errs

func IsErrConnectionReset 

func IsErrConnectionReset(err error) (yes bool)

func IsErrNotFound 

func IsErrNotFound(err error) bool

func IsErrOOS 

func IsErrOOS(err error) bool

func IsErrSyscallTimeout 

func IsErrSyscallTimeout(err error) bool

func IsErrXattrNotFound 

func IsErrXattrNotFound(err error) bool

func IsGoogleAPIURL 

func IsGoogleAPIURL(u *url.URL) bool

func IsGoogleStorageURL 

func IsGoogleStorageURL(u *url.URL) bool

func IsHTTP 

func IsHTTP(url string) bool

func IsHTTPS 

func IsHTTPS(url string) bool

func IsIOError 

func IsIOError(err error) bool

Checks if the error is generated by any IO operation and if the error is severe enough to run the FSHC for mountpath testing

For mountpath definition, see fs/mountfs.go

func IsParseBool 

func IsParseBool(s string) bool

func IsRetriableConnErr 

func IsRetriableConnErr(err error) (yes bool)

func IsS3URL 

func IsS3URL(link string) bool

func IsSetfAtomic 

func IsSetfAtomic(f *uint64, flags uint64) (yes bool)

func IsTimeZero 

func IsTimeZero(t time.Time) bool

func IsUnreachable 

func IsUnreachable(err error, status int) bool

func IsValidAtime 

func IsValidAtime(atime int64) bool

access time validity; prefetch special (negative) case - sets atime=-now 946771140000000000 = time.Parse(time.RFC3339Nano, "2000-01-01T23:59:00Z").UnixNano()

func IsValidUUID 

func IsValidUUID(uuid string) bool

func JoinPath 

func JoinPath(url, path string) string

JoinPath joins two path elements that may (or may not) be prefixed/suffixed with a slash.

func JoinWords 

func JoinWords(w string, words ...string) (path string)

JoinWords uses forward slash to join any number of words into a single path. Returned path is prefixed with a slash.

func Max 

func Max(xs ...int) int

Max returns max value from given ints.

func MaxDuration 

func MaxDuration(a, b time.Duration) time.Duration

MaxDuration returns min value of a and b time.Duration types

func MaxI32 

func MaxI32(a, b int32) int32

MaxI32 returns max value of a and b for int32 types

func MaxI64 

func MaxI64(a, b int64) int64

MaxI64 returns max value of a and b for int64 types

func MaxU64 

func MaxU64(a, b uint64) uint64

MaxU64 returns max value of a and b for uint64 types

func MaxUint 

func MaxUint(a, b uint) uint

func Min 

func Min(xs ...int) int

Min returns min value from given ints.

func MinDuration 

func MinDuration(a, b time.Duration) time.Duration

MinDuration returns min value of a and b time.Duration types

func MinF64 

func MinF64(a, b float64) float64

func MinI32 

func MinI32(a, b int32) int32

MinI32 returns min value of a and b for int32 types

func MinI64 

func MinI64(a, b int64) int64

MinI64 returns min value of a and b for int64 types

func MinTime 

func MinTime(a, b time.Time) time.Time

func MinU64 

func MinU64(a, b uint64) uint64

MinU64 returns min value of a and b for uint64 types

func MinUint 

func MinUint(a, b uint) uint

func MorphMarshal 

func MorphMarshal(data, v any) error

func MustMarshal 

func MustMarshal(v any) []byte

MustMarshal marshals v and panics if error occurs.

func MustMarshalToString 

func MustMarshalToString(v any) string

func MustMorphMarshal 

func MustMorphMarshal(data, v any)

func NewCRC32C 

func NewCRC32C() hash.Hash

func NewErrDataCksum 

func NewErrDataCksum(a, b *Cksum, context ...string) error

func NewErrMetaCksum 

func NewErrMetaCksum(a, b uint64, context ...string) error

func NopReader 

func NopReader(size int64) io.Reader

func NowRand 

func NowRand() *rand.Rand

func PackedStrLen 

func PackedStrLen(s string) int

PackedStrLen returns the size occupied by a given string in the output

func ParseBool 

func ParseBool(s string) (value bool, err error)

ParseBool converts string to bool (case-insensitive): 

y, yes, on -> true
n, no, off, <empty value> -> false

strconv handles the following: 

1, true, t -> true
0, false, f -> false

func ParseSize 

func ParseSize(size, units string) (int64, error)

func ParseURL 

func ParseURL(s string) (u *url.URL, valid bool)

func Plural 

func Plural(num int) (s string)

(common use)

func ProbingFrequency 

func ProbingFrequency(dur time.Duration) time.Duration

wait duration => probing frequency

func RandStringWithSrc 

func RandStringWithSrc(src rand.Source, n int) string

func Ratio 

func Ratio(high, low, curr int64) float32

func RatioPct 

func RatioPct(high, low, curr int64) int64

func ReadLines 

func ReadLines(filename string, cb func(string) error) error

Read a file line by line and call a callback for each line until the file ends or a callback returns io.EOF

func ReadOneInt64 

func ReadOneInt64(filename string) (int64, error)

Read only the first line of a file and return it as int64 Do not use for big files: it reads all the content and then extracts the first line. Use for files that may contains a few lines with trailing EOL

func ReadOneLine 

func ReadOneLine(filename string) (string, error)

Read only the first line of a file. Do not use for big files: it reads all the content and then extracts the first line. Use for files that may contains a few lines with trailing EOL

func ReadOneUint64 

func ReadOneUint64(filename string) (uint64, error)

Read only the first line of a file and return it as uint64 Do not use for big files: it reads all the content and then extracts the first line. Use for files that may contains a few lines with trailing EOL

func RemoveFile 

func RemoveFile(path string) (err error)

RemoveFile removes path; returns nil upon success or if the path does not exist.

func Rename 

func Rename(src, dst string) (err error)

(creates destination directory if doesn't exist)

func ReparseQuery 

func ReparseQuery(r *http.Request)

WARNING: `ReparseQuery` might affect non-tensorflow clients using S3-compatible API with AIStore. To be used with caution.

func S2Duration 

func S2Duration(s string) (time.Duration, error)

func S2UnixNano 

func S2UnixNano(s string) (int64, error)

func SHead 

func SHead(s string) string

func SetfAtomic 

func SetfAtomic(f *uint64, flags uint64) (ok bool)

func Stat 

func Stat(path string) error

syscall to check that path exists (see bench/lstat)

func StrSlicesEqual 

func StrSlicesEqual(lhs, rhs []string) bool

StrSlicesEqual compares content of two string slices. It is replacement for reflect.DeepEqual because the latter returns false if slices have the same values but in different order.

func StrToSentence 

func StrToSentence(str string) string

func StringInSlice 

func StringInSlice(s string, arr []string) bool

func StringSliceToIntSlice 

func StringSliceToIntSlice(strs []string) ([]int64, error)

func SupportedChecksums 

func SupportedChecksums() (types []string)

func ToSizeIEC 

func ToSizeIEC(b int64, digits int) string

(compare w/ CLI `ToSizeIS`)

func UnixNano2S 

func UnixNano2S(unixnano int64) string

func UnsafeB 

func UnsafeB(s string) []byte

cast string to immutable bytes

func UnsafeS 

func UnsafeS(b []byte) string

cast bytes to an immutable string

func UnwrapSyscallErr 

func UnwrapSyscallErr(err error) error

func ValidateCksumType 

func ValidateCksumType(ty string, emptyOK ...bool) (err error)

func ValidateDaemonID 

func ValidateDaemonID(id string) error

func ValidateNiceID 

func ValidateNiceID(id string, minlen int, tag string) (err error)

func Warningf 

func Warningf(format string, a ...any)

Types

type BitFlags 

type BitFlags uint64

func (BitFlags) Clear 

func (f BitFlags) Clear(flags BitFlags) BitFlags

func (BitFlags) IsAnySet 

func (f BitFlags) IsAnySet(flags BitFlags) bool

func (BitFlags) IsSet 

func (f BitFlags) IsSet(flags BitFlags) bool

func (BitFlags) Set 

func (f BitFlags) Set(flags BitFlags) BitFlags

type Bits 

type Bits uint8

func (*Bits) Clear 

func (b *Bits) Clear(flag Bits)

func (*Bits) Has 

func (b *Bits) Has(flag Bits) bool

func (*Bits) Set 

func (b *Bits) Set(flag Bits)

func (*Bits) Toggle 

func (b *Bits) Toggle(flag Bits)

type ByteHandle 

type ByteHandle struct {
	*bytes.Reader
	// contains filtered or unexported fields
}

ByteHandle is a byte buffer(made from []byte) that implements ReadOpenCloser interface

func NewByteHandle 

func NewByteHandle(bt []byte) *ByteHandle

func (*ByteHandle) Close 

func (*ByteHandle) Close() error

func (*ByteHandle) Open 

func (b *ByteHandle) Open() (ReadOpenCloser, error)

type BytePack 

type BytePack struct {
	// contains filtered or unexported fields
}

func NewPacker 

func NewPacker(buf []byte, bufLen int) *BytePack

func (*BytePack) Bytes 

func (bw *BytePack) Bytes() []byte

func (*BytePack) WriteAny 

func (bw *BytePack) WriteAny(st Packer)

func (*BytePack) WriteBool 

func (bw *BytePack) WriteBool(b bool)

func (*BytePack) WriteByte 

func (bw *BytePack) WriteByte(b byte)

func (*BytePack) WriteBytes 

func (bw *BytePack) WriteBytes(b []byte)

func (*BytePack) WriteInt16 

func (bw *BytePack) WriteInt16(i int16)

func (*BytePack) WriteInt32 

func (bw *BytePack) WriteInt32(i int32)

func (*BytePack) WriteInt64 

func (bw *BytePack) WriteInt64(i int64)

func (*BytePack) WriteMapStrUint16 

func (bw *BytePack) WriteMapStrUint16(mp MapStrUint16)

func (*BytePack) WriteString 

func (bw *BytePack) WriteString(s string)

func (*BytePack) WriteUint16 

func (bw *BytePack) WriteUint16(i uint16)

func (*BytePack) WriteUint32 

func (bw *BytePack) WriteUint32(i uint32)

func (*BytePack) WriteUint64 

func (bw *BytePack) WriteUint64(i uint64)

type ByteUnpack 

type ByteUnpack struct {
	// contains filtered or unexported fields
}

func NewUnpacker 

func NewUnpacker(buf []byte) *ByteUnpack

func (*ByteUnpack) Bytes 

func (br *ByteUnpack) Bytes() []byte

func (*ByteUnpack) Len 

func (br *ByteUnpack) Len() int

func (*ByteUnpack) ReadAny 

func (br *ByteUnpack) ReadAny(st Unpacker) error

func (*ByteUnpack) ReadBool 

func (br *ByteUnpack) ReadBool() (bool, error)

func (*ByteUnpack) ReadByte 

func (br *ByteUnpack) ReadByte() (byte, error)

func (*ByteUnpack) ReadBytes 

func (br *ByteUnpack) ReadBytes() ([]byte, error)

func (*ByteUnpack) ReadInt16 

func (br *ByteUnpack) ReadInt16() (int16, error)

func (*ByteUnpack) ReadInt32 

func (br *ByteUnpack) ReadInt32() (int32, error)

func (*ByteUnpack) ReadInt64 

func (br *ByteUnpack) ReadInt64() (int64, error)

func (*ByteUnpack) ReadMapStrUint16 

func (br *ByteUnpack) ReadMapStrUint16() (MapStrUint16, error)

func (*ByteUnpack) ReadString 

func (br *ByteUnpack) ReadString() (string, error)

func (*ByteUnpack) ReadUint16 

func (br *ByteUnpack) ReadUint16() (uint16, error)

func (*ByteUnpack) ReadUint32 

func (br *ByteUnpack) ReadUint32() (uint32, error)

func (*ByteUnpack) ReadUint64 

func (br *ByteUnpack) ReadUint64() (uint64, error)

type CallbackROC 

type CallbackROC struct {
	// contains filtered or unexported fields
}

readers

func NewCallbackReadOpenCloser 

func NewCallbackReadOpenCloser(r ReadOpenCloser, readCb func(int, error), reportedBytes ...int) *CallbackROC

func (*CallbackROC) Close 

func (r *CallbackROC) Close() error

func (*CallbackROC) Open 

func (r *CallbackROC) Open() (ReadOpenCloser, error)

func (*CallbackROC) Read 

func (r *CallbackROC) Read(p []byte) (n int, err error)

type Cksum 

type Cksum struct {
	// contains filtered or unexported fields
}

func ChecksumBytes 

func ChecksumBytes(b []byte, cksumType string) (cksum *Cksum, err error)

ChecksumBytes computes checksum of given bytes using additional buffer.

func NewCksum 

func NewCksum(ty, value string) *Cksum

func (*Cksum) Clone 

func (ck *Cksum) Clone() *Cksum

func (*Cksum) Equal 

func (ck *Cksum) Equal(to *Cksum) bool

func (*Cksum) Get 

func (ck *Cksum) Get() (string, string)

func (*Cksum) IsEmpty 

func (ck *Cksum) IsEmpty() bool

func (*Cksum) MarshalJSON 

func (ck *Cksum) MarshalJSON() ([]byte, error)

func (*Cksum) String 

func (ck *Cksum) String() string

func (*Cksum) Ty 

func (ck *Cksum) Ty() string

func (*Cksum) Type 

func (ck *Cksum) Type() string

func (*Cksum) Val 

func (ck *Cksum) Val() string

func (*Cksum) Value 

func (ck *Cksum) Value() string

type CksumHash 

type CksumHash struct {
	Cksum
	H hash.Hash
	// contains filtered or unexported fields
}

func CopyAndChecksum 

func CopyAndChecksum(w io.Writer, r io.Reader, buf []byte, cksumType string) (n int64, cksum *CksumHash, err error)

CopyAndChecksum reads from `r` and writes to `w`; returns num bytes copied and checksum, or error

func CopyFile 

func CopyFile(src, dst string, buf []byte, cksumType string) (written int64, cksum *CksumHash, err error)

and computes checksum if requested

func NewCksumHash 

func NewCksumHash(ty string) (ck *CksumHash)

func SaveReader 

func SaveReader(fqn string, reader io.Reader, buf []byte, cksumType string, size int64) (cksum *CksumHash, err error)

Saves the reader directly to `fqn`, checksums if requested

func SaveReaderSafe 

func SaveReaderSafe(tmpfqn, fqn string, reader io.Reader, buf []byte, cksumType string, size int64) (cksum *CksumHash,
	err error)

func (*CksumHash) Equal 

func (ck *CksumHash) Equal(to *Cksum) bool

func (*CksumHash) Finalize 

func (ck *CksumHash) Finalize()

func (*CksumHash) Init 

func (ck *CksumHash) Init(ty string)

func (*CksumHash) Sum 

func (ck *CksumHash) Sum() []byte

type CksumHashSize 

type CksumHashSize struct {
	CksumHash
	Size int64
}

func (*CksumHashSize) Write 

func (ck *CksumHashSize) Write(b []byte) (n int, err error)

type Duration 

type Duration time.Duration

func (Duration) D 

func (d Duration) D() time.Duration

func (Duration) MarshalJSON 

func (d Duration) MarshalJSON() ([]byte, error)

func (Duration) String 

func (d Duration) String() (s string)

func (*Duration) UnmarshalJSON 

func (d *Duration) UnmarshalJSON(b []byte) (err error)

type DynSemaphore 

type DynSemaphore struct {
	// contains filtered or unexported fields
}

DynSemaphore implements sempahore which can change its size during usage.

func NewDynSemaphore 

func NewDynSemaphore(n int) *DynSemaphore

func (*DynSemaphore) Acquire 

func (s *DynSemaphore) Acquire(cnts ...int)

func (*DynSemaphore) Release 

func (s *DynSemaphore) Release(cnts ...int)

func (*DynSemaphore) SetSize 

func (s *DynSemaphore) SetSize(n int)

func (*DynSemaphore) Size 

func (s *DynSemaphore) Size() int

type ErrBadCksum 

type ErrBadCksum struct {
	// contains filtered or unexported fields
}

func (*ErrBadCksum) Error 

func (e *ErrBadCksum) Error() string

type ErrNotFound 

type ErrNotFound struct {
	// contains filtered or unexported fields
}

func NewErrNotFound 

func NewErrNotFound(format string, a ...any) *ErrNotFound

func (*ErrNotFound) Error 

func (e *ErrNotFound) Error() string

type ErrSignal 

type ErrSignal struct {
	// contains filtered or unexported fields
}

func NewSignalError 

func NewSignalError(s syscall.Signal) *ErrSignal

func (*ErrSignal) Error 

func (e *ErrSignal) Error() string

func (*ErrSignal) ExitCode 

func (e *ErrSignal) ExitCode() int

https://tldp.org/LDP/abs/html/exitcodes.html

type ErrTemplate 

type ErrTemplate struct {
	// contains filtered or unexported fields
}

func (*ErrTemplate) Error 

func (e *ErrTemplate) Error() string

type ErrValue 

type ErrValue struct {
	atomic.Value
	// contains filtered or unexported fields
}

func (*ErrValue) Cnt 

func (ea *ErrValue) Cnt() int

func (*ErrValue) Err 

func (ea *ErrValue) Err() (err error)

func (*ErrValue) Load 

func (*ErrValue) Load() any

NOTE: hide atomic.Value.Load() - must use Err() below

func (*ErrValue) Store 

func (ea *ErrValue) Store(err error)

type FS 

type FS struct {
	Fs     string
	FsType string
	FsID   FsID
}

func (*FS) Equal 

func (fs *FS) Equal(otherFs FS) bool

func (*FS) String 

func (fs *FS) String() string

type FileHandle 

type FileHandle struct {
	*os.File
	// contains filtered or unexported fields
}

handles (and more readers)

func NewFileHandle 

func NewFileHandle(fqn string) (*FileHandle, error)

func (*FileHandle) Open 

func (f *FileHandle) Open() (ReadOpenCloser, error)

type FileSectionHandle 

type FileSectionHandle struct {
	// contains filtered or unexported fields
}

FileSectionHandle opens a file and reads a section of it with optional padding. It implements the ReadOpenCloser interface.

func NewFileSectionHandle 

func NewFileSectionHandle(fqn string, offset, size int64) (*FileSectionHandle, error)

NewFileSectionHandle opens file which is expected at `fqn` and defines a SectionHandle on it to only read a specified section.

func (*FileSectionHandle) Close 

func (f *FileSectionHandle) Close() error

func (*FileSectionHandle) Open 

func (f *FileSectionHandle) Open() (ReadOpenCloser, error)

func (*FileSectionHandle) Read 

func (f *FileSectionHandle) Read(buf []byte) (int, error)

type FsID 

type FsID [2]int32

FsID is unified and cross-platform syscall.Fsid type which implements JSON marshaling.

func (FsID) MarshalJSON 

func (d FsID) MarshalJSON() ([]byte, error)

func (FsID) String 

func (d FsID) String() string

func (*FsID) UnmarshalJSON 

func (d *FsID) UnmarshalJSON(b []byte) error

type JSONRawMsgs 

type JSONRawMsgs map[string]jsoniter.RawMessage

type LimitedWaitGroup 

type LimitedWaitGroup struct {
	// contains filtered or unexported fields
}

LimitedWaitGroup is helper struct which combines standard wait group and semaphore to limit the number of goroutines created.

func (*LimitedWaitGroup) Add 

func (lwg *LimitedWaitGroup) Add(n int)

func (*LimitedWaitGroup) Done 

func (lwg *LimitedWaitGroup) Done()

func (*LimitedWaitGroup) Wait 

func (lwg *LimitedWaitGroup) Wait()

type MapStrUint16 

type MapStrUint16 map[string]uint16

type MultiSyncMap 

type MultiSyncMap struct {
	M [MultiSyncMapCount]sync.Map
}

func (*MultiSyncMap) Get 

func (msm *MultiSyncMap) Get(idx int) *sync.Map

func (*MultiSyncMap) GetByHash 

func (msm *MultiSyncMap) GetByHash(hash uint32) *sync.Map

type NamedVal64 

type NamedVal64 struct {
	Name       string
	NameSuffix string // forces immediate send when non-empty (see NOTE below)
	Value      int64
}

type NopLocker 

type NopLocker struct{}

func (NopLocker) Lock 

func (NopLocker) Lock()

func (NopLocker) Unlock 

func (NopLocker) Unlock()

type OAH 

type OAH interface {
	SizeBytes(special ...bool) int64
	Version(special ...bool) string
	Checksum() *Cksum
	AtimeUnix() int64
	GetCustomMD() StrKVs
	GetCustomKey(key string) (val string, exists bool)
	SetCustomKey(k, v string)
	String() string
}

type Packer 

type Packer interface {
	Pack(packer *BytePack)
	PackedSize() int
}

type ParsedQuantity 

type ParsedQuantity struct {
	Type  string
	Value uint64
}

func ParseQuantity 

func ParseQuantity(quantity string) (ParsedQuantity, error)

func (ParsedQuantity) String 

func (pq ParsedQuantity) String() string

type ParsedTemplate 

type ParsedTemplate struct {
	Prefix string
	Ranges []TemplateRange
	// contains filtered or unexported fields
}

func NewParsedTemplate 

func NewParsedTemplate(template string) (ParsedTemplate, error)

func ParseAtTemplate 

func ParseAtTemplate(template string) (pt ParsedTemplate, err error)

e.g. multi-range template: "prefix-@00001-gap-@100-suffix" 

single range:         "prefix@00100suffix"

func ParseBashTemplate 

func ParseBashTemplate(template string) (pt ParsedTemplate, err error)

examples - single-range: "prefix{0001..0010}suffix" - multi-range: "prefix-{00001..00010..2}-gap-{001..100..2}-suffix" (both prefix and suffix are optional, here and elsewhere)

func ParseFmtTemplate 

func ParseFmtTemplate(template string) (pt ParsedTemplate, err error)

template: "prefix-%06d-suffix" (both prefix and suffix are optional, here and elsewhere)

func (*ParsedTemplate) Count 

func (pt *ParsedTemplate) Count() int64

func (*ParsedTemplate) InitIter 

func (pt *ParsedTemplate) InitIter()

func (*ParsedTemplate) Next 

func (pt *ParsedTemplate) Next() (string, bool)

func (*ParsedTemplate) ToSlice 

func (pt *ParsedTemplate) ToSlice(maxLen ...int) []string

maxLen specifies maximum objects to be returned

type ReadCloseSizer 

type ReadCloseSizer interface {
	io.ReadCloser
	Size() int64
}

ReadCloseSizer is the interface that adds Size method to io.ReadCloser.

func NewDeferRCS 

func NewDeferRCS(r ReadCloseSizer, cb func()) ReadCloseSizer

func NewSizedRC 

func NewSizedRC(r io.ReadCloser, size int64) ReadCloseSizer

type ReadOpenCloseSizer 

type ReadOpenCloseSizer interface {
	ReadOpenCloser
	Size() int64
}

ReadOpenCloseSizer is the interface that adds Size method to ReadOpenCloser.

type ReadOpenCloser 

type ReadOpenCloser interface {
	io.ReadCloser
	Open() (ReadOpenCloser, error)
}

readers

func NopOpener 

func NopOpener(r io.ReadCloser) ReadOpenCloser

type ReadReaderAt 

type ReadReaderAt interface {
	io.Reader
	io.ReaderAt
}

readers

type ReadSizer 

type ReadSizer interface {
	io.Reader
	Size() int64
}

ReadSizer is the interface that adds Size method to io.Reader.

func NewSizedReader 

func NewSizedReader(r io.Reader, size int64) ReadSizer

type ReadWrapperFunc 

type ReadWrapperFunc func(r io.ReadCloser) io.ReadCloser

type ReaderArgs 

type ReaderArgs struct {
	R       io.Reader
	ReadCb  func(int, error)
	DeferCb func()
	Size    int64
}

readers

type ReaderWithArgs 

type ReaderWithArgs struct {
	// contains filtered or unexported fields
}

readers

func NewReaderWithArgs 

func NewReaderWithArgs(args ReaderArgs) *ReaderWithArgs

func (*ReaderWithArgs) Close 

func (r *ReaderWithArgs) Close() (err error)

func (*ReaderWithArgs) Open 

func (*ReaderWithArgs) Open() (ReadOpenCloser, error)

func (*ReaderWithArgs) Read 

func (r *ReaderWithArgs) Read(p []byte) (n int, err error)

func (*ReaderWithArgs) Size 

func (r *ReaderWithArgs) Size() int64

type Runner 

type Runner interface {
	Name() string
	Run() error
	Stop(error)
}

type SectionHandle 

type SectionHandle struct {
	// contains filtered or unexported fields
}

SectionHandle is a section of reader with optional padding that implements ReadOpenCloser interface.

func NewSectionHandle 

func NewSectionHandle(r io.ReaderAt, offset, size, padding int64) *SectionHandle

func (*SectionHandle) Close 

func (*SectionHandle) Close() error

func (*SectionHandle) Open 

func (f *SectionHandle) Open() (ReadOpenCloser, error)

func (*SectionHandle) Read 

func (f *SectionHandle) Read(buf []byte) (n int, err error)

Reads a reader section. When the slice finishes but the buffer is not filled yet, act as if it reads a few more bytes from somewhere.

type Semaphore 

type Semaphore struct {
	// contains filtered or unexported fields
}

Semaphore implements sempahore which is just a nice wrapper on `chan struct{}`.

func NewSemaphore 

func NewSemaphore(n int) *Semaphore

func (*Semaphore) Acquire 

func (s *Semaphore) Acquire()

func (*Semaphore) Release 

func (s *Semaphore) Release()

func (*Semaphore) TryAcquire 

func (s *Semaphore) TryAcquire() <-chan struct{}

type SetSizeFunc 

type SetSizeFunc func(size int64)

type SimpleOAH 

type SimpleOAH struct {
	Size  int64
	Atime int64
}

convenience/shortcut

func (SimpleOAH) AtimeUnix 

func (s SimpleOAH) AtimeUnix() int64

func (SimpleOAH) Checksum 

func (SimpleOAH) Checksum() *Cksum

func (SimpleOAH) GetCustomKey 

func (SimpleOAH) GetCustomKey(string) (string, bool)

func (SimpleOAH) GetCustomMD 

func (SimpleOAH) GetCustomMD() StrKVs

func (SimpleOAH) SetCustomKey 

func (SimpleOAH) SetCustomKey(_, _ string)

func (SimpleOAH) SizeBytes 

func (s SimpleOAH) SizeBytes(...bool) int64

func (SimpleOAH) String 

func (SimpleOAH) String() string

func (SimpleOAH) Version 

func (SimpleOAH) Version(...bool) string

type SizeIEC 

type SizeIEC int64

func (SizeIEC) MarshalJSON 

func (siz SizeIEC) MarshalJSON() ([]byte, error)

func (SizeIEC) String 

func (siz SizeIEC) String() string

func (*SizeIEC) UnmarshalJSON 

func (siz *SizeIEC) UnmarshalJSON(b []byte) (err error)

type StatsUpdater 

type StatsUpdater interface {
	Inc(name string)
	Add(name string, val int64)
	Get(name string) int64
	AddMany(namedVal64 ...NamedVal64)
}

type StopCh 

type StopCh struct {
	// contains filtered or unexported fields
}

StopCh is specialized channel for stopping things.

func NewStopCh 

func NewStopCh() *StopCh

func (*StopCh) Close 

func (sch *StopCh) Close()

func (*StopCh) Init 

func (sch *StopCh) Init()

func (*StopCh) Listen 

func (sch *StopCh) Listen() <-chan struct{}

type StrKVs 

type StrKVs map[string]string

func NewStrKVs 

func NewStrKVs(pairs ...string) (kvs StrKVs)

func (StrKVs) Compare 

func (kvs StrKVs) Compare(other StrKVs) bool

func (StrKVs) Contains 

func (kvs StrKVs) Contains(key string) (ok bool)

func (StrKVs) ContainsAnyMatch 

func (kvs StrKVs) ContainsAnyMatch(in []string) string

func (StrKVs) KeyFor 

func (kvs StrKVs) KeyFor(value string) (key string)

func (StrKVs) Keys 

func (kvs StrKVs) Keys() []string

type StrSet 

type StrSet map[string]struct{}

func NewStrSet 

func NewStrSet(keys ...string) (ss StrSet)

func (StrSet) Add 

func (ss StrSet) Add(keys ...string)

func (StrSet) All 

func (ss StrSet) All(xs ...string) bool

func (StrSet) Clone 

func (ss StrSet) Clone() StrSet

func (StrSet) Contains 

func (ss StrSet) Contains(key string) (yes bool)

func (StrSet) Delete 

func (ss StrSet) Delete(key string)

func (StrSet) Intersection 

func (ss StrSet) Intersection(other StrSet) StrSet

func (StrSet) Set 

func (ss StrSet) Set(key string)

func (StrSet) String 

func (ss StrSet) String() string

func (StrSet) ToSlice 

func (ss StrSet) ToSlice() []string

type TemplateRange 

type TemplateRange struct {
	Gap        string // characters after the range (to the next range or end of the string)
	Start      int64
	End        int64
	Step       int64
	DigitCount int
}

type TimeoutGroup 

type TimeoutGroup struct {
	// contains filtered or unexported fields
}

TimeoutGroup is similar to sync.WaitGroup with the difference on Wait where we only allow timing out.

WARNING: It should not be used in critical code as it may have worse performance than sync.WaitGroup - use only if its needed.

WARNING: It is not safe to wait on completion in multiple threads!

WARNING: It is not recommended to reuse the TimeoutGroup - it was not designed for that and bugs can be expected, especially when previous group was not called with successful (without timeout) WaitTimeout.

func NewTimeoutGroup 

func NewTimeoutGroup() *TimeoutGroup

func (*TimeoutGroup) Add 

func (twg *TimeoutGroup) Add(n int)

func (*TimeoutGroup) Done 

func (twg *TimeoutGroup) Done()

Done decrements number of jobs left to do. Panics if the number jobs left is less than 0.

func (*TimeoutGroup) Wait 

func (twg *TimeoutGroup) Wait()

Wait waits until the Added pending count goes to zero. NOTE: must be invoked after _all_ Adds.

func (*TimeoutGroup) WaitTimeout 

func (twg *TimeoutGroup) WaitTimeout(timeout time.Duration) bool

Wait waits until the Added pending count goes to zero _or_ timeout. NOTE: must be invoked after _all_ Adds.

func (*TimeoutGroup) WaitTimeoutWithStop 

func (twg *TimeoutGroup) WaitTimeoutWithStop(timeout time.Duration, stop <-chan struct{}) (timed, stopped bool)

Wait waits until the Added pending count goes to zero _or_ timeout _or_ stop. NOTE: must be invoked after _all_ Adds.

type Unpacker 

type Unpacker interface {
	Unpack(unpacker *ByteUnpack) error
}

Every object that is going to use binary representation instead of JSON must implement two following methods

type WG 

type WG interface {
	Add(int)
	Done()
	Wait()
}

WG is an interface for wait group

func NewLimitedWaitGroup 

func NewLimitedWaitGroup(limit, have int) WG

usage: no more than `limit` goroutines in parallel

type WriteSizer 

type WriteSizer interface {
	io.Writer
	Size() int64
}

writers

type WriterAt 

type WriterAt interface {
	io.Writer
	io.WriterAt
}

writers

type WriterMulti 

type WriterMulti struct {
	// contains filtered or unexported fields
}

writers

func NewWriterMulti 

func NewWriterMulti(w ...io.Writer) *WriterMulti

func (*WriterMulti) Write 

func (mw *WriterMulti) Write(b []byte) (n int, err error)

type WriterOnly 

type WriterOnly struct{ io.Writer }

WriterOnly is helper struct to hide `io.ReaderFrom` implementation which can use some heuristics to improve performance but can result in not using `buffer` provided in `io.CopyBuffer`. See: https://golang.org/doc/go1.15#os.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.