x

package
v83.0.0-...-38e0eed Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 12, 2017 License: AGPL-3.0, Apache-2.0 Imports: 34 Imported by: 0

Documentation

Overview

  • Copyright (C) 2017 Dgraph Labs, Inc. and Contributors *
  • Licensed under the Apache License, Version 2.0 (the "License");
  • you may not use this file except in compliance with the License.
  • You may obtain a copy of the License at *
  • http://www.apache.org/licenses/LICENSE-2.0 *
  • Unless required by applicable law or agreed to in writing, software
  • distributed under the License is distributed on an "AS IS" BASIS,
  • WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  • See the License for the specific language governing permissions and
  • limitations under the License.

Package x contains some very common utilities used by Dgraph. These utilities are of "miscellaneous" nature, e.g., error checking.

  • Copyright (C) 2017 Dgraph Labs, Inc. and Contributors *

  • This program is free software: you can redistribute it and/or modify

  • it under the terms of the GNU Affero General Public License as published by

  • the Free Software Foundation, either version 3 of the License, or

  • (at your option) any later version. *

  • This program is distributed in the hope that it will be useful,

  • but WITHOUT ANY WARRANTY; without even the implied warranty of

  • MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the

  • GNU Affero General Public License for more details. *

  • You should have received a copy of the GNU Affero General Public License

  • along with this program. If not, see <http://www.gnu.org/licenses/>.

  • Copyright (C) 2017 Dgraph Labs, Inc. and Contributors *

  • Licensed under the Apache License, Version 2.0 (the "License");

  • you may not use this file except in compliance with the License.

  • You may obtain a copy of the License at *

  • http://www.apache.org/licenses/LICENSE-2.0 *

  • Unless required by applicable law or agreed to in writing, software

  • distributed under the License is distributed on an "AS IS" BASIS,

  • WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  • See the License for the specific language governing permissions and

  • limitations under the License.

  • Copyright (C) 2017 Dgraph Labs, Inc. and Contributors *

  • Licensed under the Apache License, Version 2.0 (the "License");

  • you may not use this file except in compliance with the License.

  • You may obtain a copy of the License at *

  • http://www.apache.org/licenses/LICENSE-2.0 *

  • Unless required by applicable law or agreed to in writing, software

  • distributed under the License is distributed on an "AS IS" BASIS,

  • WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  • See the License for the specific language governing permissions and

  • limitations under the License.

Index

Constants

View Source
const (
	// TODO(pawan) - Make this 2 bytes long. Right now ParsedKey has byteType and
	// bytePrefix. Change it so that it just has one field which has all the information.
	ByteData = byte(0x00)

	ByteIndex    = byte(0x02)
	ByteReverse  = byte(0x04)
	ByteCount    = byte(0x08)
	ByteCountRev = ByteCount | ByteReverse
)
View Source
const (
	TLSClientConfig tlsConfigType = iota
	TLSServerConfig
)
View Source
const (
	Success                 = "Success"
	ErrorUnauthorized       = "ErrorUnauthorized"
	ErrorInvalidMethod      = "ErrorInvalidMethod"
	ErrorInvalidRequest     = "ErrorInvalidRequest"
	ErrorMissingRequired    = "ErrorMissingRequired"
	Error                   = "Error"
	ErrorNoData             = "ErrorNoData"
	ErrorUptodate           = "ErrorUptodate"
	ErrorNoPermission       = "ErrorNoPermission"
	ErrorInvalidMutation    = "ErrorInvalidMutation"
	ErrorServiceUnavailable = "ErrorServiceUnavailable"
	ValidHostnameRegex      = "^(([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\\-]*[a-zA-Z0-9])\\.)*([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\\-]*[A-Za-z0-9])$"
	Star                    = "_STAR_ALL"
	GrpcMaxSize             = 256 << 20
)

Error constants representing different types of errors.

Variables

View Source
var (
	// These are cummulative
	PostingReads  *expvar.Int
	PostingWrites *expvar.Int
	BytesRead     *expvar.Int
	BytesWrite    *expvar.Int
	EvictedPls    *expvar.Int
	NumQueries    *expvar.Int
	CacheHit      *expvar.Int
	CacheMiss     *expvar.Int
	CacheRace     *expvar.Int

	// value at particular point of time
	PendingQueries   *expvar.Int
	PendingProposals *expvar.Int
	LcacheSize       *expvar.Int
	LcacheLen        *expvar.Int
	LcacheCapacity   *expvar.Int
	DirtyMapSize     *expvar.Int
	NumGoRoutines    *expvar.Int
	MemoryInUse      *expvar.Int
	HeapIdle         *expvar.Int
	TotalMemory      *expvar.Int
	TotalOSMemory    *expvar.Int
	ActiveMutations  *expvar.Int
	ServerHealth     *expvar.Int
	MaxPlSize        *expvar.Int
	MaxPlLength      *expvar.Int

	PredicateStats *expvar.Map
	Conf           *expvar.Map

	MaxPlSz int64
)
View Source
var (
	Logger = log.New(os.Stderr, "", log.Lshortfile|log.Flags())
)
View Source
var Nilbyte []byte
View Source
var WhiteSpace = strings.NewReplacer(" ", "", "\t", "")

WhiteSpace Replacer removes spaces and tabs from a string.

Functions

func AddCorsHeaders

func AddCorsHeaders(w http.ResponseWriter)

func AddInit

func AddInit(f func())

AddInit adds a function to be run in x.Init, which should be called at the beginning of all mains.

func AppendProtoMsg

func AppendProtoMsg(p []byte, msg ProtoMessage) ([]byte, error)

func AppendUvarint

func AppendUvarint(p []byte, x uint64) []byte

func AssertTrue

func AssertTrue(b bool)

AssertTrue asserts that b is true. Otherwise, it would log fatal.

func AssertTruef

func AssertTruef(b bool, format string, args ...interface{})

AssertTruef is AssertTrue with extra info.

func AssertTruefNoTrace

func AssertTruefNoTrace(b bool, format string, args ...interface{})

func Check

func Check(err error)

Check logs fatal if err != nil.

func Check2

func Check2(_ interface{}, err error)

Check2 acts as convenience wrapper around Check, using the 2nd argument as error.

func Checkf

func Checkf(err error, format string, args ...interface{})

Checkf is Check with extra info.

func CountKey

func CountKey(attr string, count uint32, reverse bool) []byte

func DataKey

func DataKey(attr string, uid uint64) []byte

func Errorf

func Errorf(format string, args ...interface{}) error

Errorf creates a new error with stack trace, etc.

func Fatalf

func Fatalf(format string, args ...interface{})

Fatalf logs fatal.

func GenerateTLSConfig

func GenerateTLSConfig(config TLSHelperConfig) (tlsCfg *tls.Config, reloadConfig func(), err error)

GenerateTLSConfig creates and returns a new *tls.Config with the configuration provided. If the ConfigType provided in TLSHelperConfig is TLSServerConfig, it's return a reload function. If any problem is found, an error is returned

func HealthCheck

func HealthCheck() error

HealthCheck returns whether the server is ready to accept requests or not Load balancer would add the node to the endpoint once health check starts returning true

func IndexKey

func IndexKey(attr, term string) []byte

func Init

func Init()

Init initializes flags and run all functions in initFunc.

func IsTestRun

func IsTestRun() bool

func LoadConfigFromYAML

func LoadConfigFromYAML(file string)

loadConfigFromYAML reads configurations from specified YAML file.

func NewTrace

func NewTrace(title string, ctx context.Context) (trace.Trace, context.Context)

func PageRange

func PageRange(count, offset, n int) (int, int)

PageRange returns start and end indices given pagination params. Note that n is the size of the input list.

func ParseRequest

func ParseRequest(w http.ResponseWriter, r *http.Request, data interface{}) bool

func PredicatePrefix

func PredicatePrefix(predicate string) []byte

PredicatePrefix returns the prefix for all keys belonging to this predicate except schema key.

func PrintVersionOnly

func PrintVersionOnly()

PrintVersionOnly prints version and other helpful information if --version.

func Printf

func Printf(format string, args ...interface{})

Printf does a log.Printf. We often do printf for debugging but has to keep adding import "fmt" or "log" and removing them after we are done. Let's add Printf to "x" and include "x" almost everywhere. Caution: Do remember to call x.Init. For tests, you need a TestMain that calls x.Init.

func Println

func Println(args ...interface{})

func ReadLine

func ReadLine(r *bufio.Reader, buf *bytes.Buffer) error

Reads a single line from a buffered reader. The line is read into the passed in buffer to minimize allocations. This is the preferred method for loading long lines which could be longer than the buffer size of bufio.Scanner.

func RemoveDuplicates

func RemoveDuplicates(s []string) (out []string)

sorts the slice of strings and removes duplicates. changes the input slice. this function should be called like: someSlice = x.RemoveDuplicates(someSlice)

func Reply

func Reply(w http.ResponseWriter, rep interface{})

func ReserveCap

func ReserveCap(p []byte, atLeast int) []byte

func ReverseKey

func ReverseKey(attr string, uid uint64) []byte

func Round

func Round(d time.Duration) time.Duration

Go doesn't have a round function for Duration and the duration value can be a bit ugly to look at sometimes. This is an attempt to round the value.

func SchemaKey

func SchemaKey(attr string) []byte

SchemaKey returns schema key for given attribute, schema keys are stored separately with unique prefix, since we need to iterate over all schema keys

func SchemaPrefix

func SchemaPrefix() []byte

SchemaPrefix returns the prefix for Schema keys.

func SetError

func SetError(prev *error, n error)

SetError sets the error logged in this package.

func SetStatus

func SetStatus(w http.ResponseWriter, code, msg string)

SetStatus sets the error code, message and the newly assigned uids in the http response.

func SetStatusWithData

func SetStatusWithData(w http.ResponseWriter, code, msg string)

In case an error was encountered after the query execution started, we have to return data key with null value according to GraphQL spec.

func SetTestRun

func SetTestRun()

func UpdateHealthStatus

func UpdateHealthStatus(ok bool)

func UpdateMemoryStatus

func UpdateMemoryStatus(ok bool)

func ValidateAddress

func ValidateAddress(addr string) bool

ValidateAddress checks whether given address can be used with grpc dial function

func Version

func Version() string

func Wrap

func Wrap(err error) error

Wrap wraps errors from external lib.

func Wrapf

func Wrapf(err error, format string, args ...interface{}) error

Wrapf is Wrap with extra info.

func WriteFileSync

func WriteFileSync(filename string, data []byte, perm os.FileMode) error

WriteFileSync is the same as bufio.WriteFile, but syncs the data before closing.

Types

type BytesBuffer

type BytesBuffer struct {
	// contains filtered or unexported fields
}

func (*BytesBuffer) CopyTo

func (b *BytesBuffer) CopyTo(o []byte) int

Caller should ensure that o is of appropriate length

func (*BytesBuffer) Length

func (b *BytesBuffer) Length() int

func (*BytesBuffer) Slice

func (b *BytesBuffer) Slice(n int) []byte

returns a slice of lenght n to be used to writing

func (*BytesBuffer) TruncateBy

func (b *BytesBuffer) TruncateBy(n int)

Always give back <= touched bytes

type EmbeddedUidAllocator

type EmbeddedUidAllocator struct {
	sync.Mutex
	// contains filtered or unexported fields
}

func (*EmbeddedUidAllocator) AssignUids

func (e *EmbeddedUidAllocator) AssignUids(ctx context.Context,
	num *protos.Num) (*protos.AssignedIds, error)

func (*EmbeddedUidAllocator) Init

func (e *EmbeddedUidAllocator) Init(kv *badger.KV)

Start lease from 2, 1 is used by _lease_

type Histogram

type Histogram struct {
	sync.Mutex
	// contains filtered or unexported fields
}

A Histogram collects observed values by keeping bucketed counts. For convenience, internally two sets of buckets are kept: A cumulative set (i.e. data is never evicted) and a windowed set (which keeps only recently collected samples).

Top-level methods generally apply to the cumulative buckets; the windowed variant is exposed through the Windowed method.

func NewHistogram

func NewHistogram(duration time.Duration, maxVal int64, sigFigs int) *Histogram

NewHistogram initializes a given Histogram. The contained windowed histogram rotates every 'duration'; both the windowed and the cumulative histogram track nonnegative values up to 'maxVal' with 'sigFigs' decimal points of precision.

func (*Histogram) RecordValue

func (h *Histogram) RecordValue(v int64)

RecordValue adds the given value to the histogram. Recording a value in excess of the configured maximum value for that histogram results in recording the maximum value instead.

func (*Histogram) Stats

func (h *Histogram) Stats()

type NonceCounter

type NonceCounter struct {
	// contains filtered or unexported fields
}

NonceCounter can generate unique 128-bit numbers. Only usable on a single thread. (All it would take to be multithreaded is to increment hi atomically.)

func NewNonceCounter

func NewNonceCounter() NonceCounter

NewNonceCounter reads from /dev/urandom (or such) and generates a new randomized NonceCounter.

func (*NonceCounter) Generate

func (nc *NonceCounter) Generate() [16]byte

Generate returns a new nonce value.

func (*NonceCounter) GenerationNumber

func (nc *NonceCounter) GenerationNumber(buf []byte) uint64

GenerationNumber returns N+C when buf was produced by the Nth call to Generate(), for some global constant C.

type Options

type Options struct {
	ConfigFile string
	Version    bool
	DebugMode  bool
	PortOffset int
}
var Config Options

type ParsedKey

type ParsedKey struct {
	Attr  string
	Uid   uint64
	Term  string
	Count uint32
	// contains filtered or unexported fields
}

func Parse

func Parse(key []byte) *ParsedKey

func (ParsedKey) CountPrefix

func (p ParsedKey) CountPrefix(reverse bool) []byte

CountPrefix returns the prefix for count keys.

func (ParsedKey) DataPrefix

func (p ParsedKey) DataPrefix() []byte

DataPrefix returns the prefix for data keys.

func (ParsedKey) IndexPrefix

func (p ParsedKey) IndexPrefix() []byte

IndexPrefix returns the prefix for index keys.

func (ParsedKey) IsCount

func (p ParsedKey) IsCount() bool

func (ParsedKey) IsData

func (p ParsedKey) IsData() bool

func (ParsedKey) IsIndex

func (p ParsedKey) IsIndex() bool

func (ParsedKey) IsReverse

func (p ParsedKey) IsReverse() bool

func (ParsedKey) IsSchema

func (p ParsedKey) IsSchema() bool

func (ParsedKey) IsType

func (p ParsedKey) IsType(typ byte) bool

func (ParsedKey) ReversePrefix

func (p ParsedKey) ReversePrefix() []byte

ReversePrefix returns the prefix for index keys.

func (ParsedKey) SkipPredicate

func (p ParsedKey) SkipPredicate() []byte

func (ParsedKey) SkipRangeOfSameType

func (p ParsedKey) SkipRangeOfSameType() []byte

func (ParsedKey) SkipSchema

func (p ParsedKey) SkipSchema() []byte

type ProtoMessage

type ProtoMessage interface {
	Size() int
	MarshalTo([]byte) (int, error)
}

type QueryResWithData

type QueryResWithData struct {
	Errors []errRes `json:"errors"`
	Data   *string  `json:"data"`
}

type RaftValue

type RaftValue struct {
	Group uint32
	Index uint64
}

RaftValue contains the raft group and the raft proposal id. This is attached to the context, so the information could be passed down to the many posting lists, involved in mutations.

type SafeMutex

type SafeMutex struct {
	// contains filtered or unexported fields
}

SafeLock can be used in place of sync.RWMutex

func (*SafeMutex) AssertLock

func (s *SafeMutex) AssertLock()

func (*SafeMutex) AssertRLock

func (s *SafeMutex) AssertRLock()

func (*SafeMutex) Lock

func (s *SafeMutex) Lock()

func (*SafeMutex) RLock

func (s *SafeMutex) RLock()

func (*SafeMutex) RUnlock

func (s *SafeMutex) RUnlock()

func (*SafeMutex) StartWait

func (s *SafeMutex) StartWait() *SafeWait

func (*SafeMutex) Unlock

func (s *SafeMutex) Unlock()

func (*SafeMutex) Wait

func (s *SafeMutex) Wait()

type SafeWait

type SafeWait struct {
	// contains filtered or unexported fields
}

func (*SafeWait) Done

func (s *SafeWait) Done()

type TLSHelperConfig

type TLSHelperConfig struct {
	ConfigType             tlsConfigType
	CertRequired           bool
	Cert                   string
	Key                    string
	KeyPassphrase          string
	ServerName             string
	Insecure               bool
	RootCACerts            string
	UseSystemRootCACerts   bool
	ClientAuth             string
	ClientCACerts          string
	UseSystemClientCACerts bool
	MinVersion             string
	MaxVersion             string
}

TLSHelperConfig define params used to create a tls.Config

type Throttle

type Throttle struct {
	// contains filtered or unexported fields
}

Throttle allows a limited number of workers to run at a time. It also provides a mechanism to wait for all workers to finish.

func NewThrottle

func NewThrottle(max int) *Throttle

NewThrottle creates a new throttle with a max number of workers.

func (*Throttle) Done

func (t *Throttle) Done()

Done should be called by workers when they finish working. It panics if there wasn't a corresponding Start call.

func (*Throttle) Start

func (t *Throttle) Start()

Start should be called by workers before they start working. It blocks if there are already the maximum number of workers working.

func (*Throttle) Wait

func (t *Throttle) Wait()

Wait waits until all workers have finished working.

type ValueTypeInfo

type ValueTypeInfo int32
const (
	ValueUnknown ValueTypeInfo = iota // unknown type of value
	ValueEmpty                        // no UID and no value
	ValueUid                          // UID
	ValuePlain                        // plain old value without defined language tag
	// Value which is part of a multi-value posting list (like language).
	ValueMulti
)

Type of a data inside DirectedEdge, Posting or NQuad

func ValueType

func ValueType(hasValue, hasLang, hasSpecialId bool) ValueTypeInfo

Helper function, to decide value type of DirectedEdge/Posting/NQuad

type WaterMark

type WaterMark struct {
	Name string
	// contains filtered or unexported fields
}

WaterMark is used to keep track of the minimum un-finished index. Typically, an index k becomes finished or "done" according to a WaterMark once Done(k) has been called

  1. as many times as Begin(k) has, AND
  2. a positive number of times.

An index may also become "done" by calling SetDoneUntil at a time such that it is not inter-mingled with Begin/Done calls.

func (*WaterMark) Begin

func (w *WaterMark) Begin(index uint64)

func (*WaterMark) BeginMany

func (w *WaterMark) BeginMany(indices []uint64)

func (*WaterMark) Done

func (w *WaterMark) Done(index uint64)

func (*WaterMark) DoneMany

func (w *WaterMark) DoneMany(indices []uint64)

func (*WaterMark) DoneUntil

func (w *WaterMark) DoneUntil() uint64

DoneUntil returns the maximum index until which all tasks are done.

func (*WaterMark) Init

func (w *WaterMark) Init()

Init initializes a WaterMark struct. MUST be called before using it.

func (*WaterMark) SetDoneUntil

func (w *WaterMark) SetDoneUntil(val uint64)

func (*WaterMark) WaitForMark

func (w *WaterMark) WaitForMark(ctx context.Context, index uint64) error

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL