peer

package
v1.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 25, 2017 License: MIT Imports: 28 Imported by: 1

Documentation

Index

Constants

View Source
const RequestChanLen = 8

Variables

View Source
var BoltDataBucketName = []byte("data") // bucket
View Source
var BoltMetaBucketName = []byte("meta") // bucket
View Source
var BoltSizeBucketName = []byte("size") // bucket
View Source
var ErrAmFollower = fmt.Errorf("LeadTransferCheckpoint error: I am follower, not transmitting checkpoint")
View Source
var ErrAmLead = fmt.Errorf("error: I am lead")
View Source
var ErrNoFollowers = fmt.Errorf("error: no followers")
View Source
var ErrShutdown = fmt.Errorf("shutting down")
View Source
var ErrTimedOut = fmt.Errorf("timed out")

Functions

func GenAddress

func GenAddress() string

GenAddress generates a local address by calling GetAvailPort() and GetExternalIP(), then prefixing them with 'tcp://'.

func GetExternalIP

func GetExternalIP() string

GetExternalIP tries to determine the external IP address used on this host.

func GetExternalIPAsInt

func GetExternalIPAsInt() int

GetExternalIPAsInt calls GetExternalIP() and then converts the resulting IPv4 string into an integer.

func IsRoutableIPv4

func IsRoutableIPv4(ip string) bool

IsRoutableIPv4 returns true if the string in ip represents an IPv4 address that is not private. See http://en.wikipedia.org/wiki/Private_network#Private_IPv4_address_spaces for the numeric ranges that are private. 127.0.0.1, 192.168.0.1, and 172.16.0.1 are examples of non-routables IP addresses.

func StripNanomsgAddressPrefix

func StripNanomsgAddressPrefix(nanomsgAddr string) (suffix string, err error)

StripNanomsgAddressPrefix removes the 'tcp://' prefix from nanomsgAddr.

func UtilCleanupTestUserDatabases

func UtilCleanupTestUserDatabases()

func UtilTestSetupThree

func UtilTestSetupThree(callmePreStart func(peer *Peer)) (p0, p1, p2 *Peer, peers *LeadAndFollowList)

Types

type BoltSaver

type BoltSaver struct {
	// contains filtered or unexported fields
}

func NewBoltSaver

func NewBoltSaver(filepath string, who string, numSetsBeforeCompact int64) (*BoltSaver, error)

func (*BoltSaver) Close

func (b *BoltSaver) Close()

func (*BoltSaver) Compact added in v0.9.9

func (b *BoltSaver) Compact(lockNeeded bool) error

Compact does these steps to reduce the space fragmentation that happens in a bolt database.

  1. open a fresh .compressed bolt db file.
  2. read each object from the bolt db and write it to the fresh -> .compressed bolt db file.
  3. close the both files.
  4. rename the .compressed file to be the original db file name. os.Rename is atomic.
  5. re-open the newly compact-ed db file

The routines below were adapted from the compaction code in https://github.com/boltdb/bolt/blob/master/cmd/bolt/main.go which is used under the following MIT license.

The MIT License (MIT)

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

INVAR: b.db must be already open.

func (*BoltSaver) InitDbIfNeeded

func (b *BoltSaver) InitDbIfNeeded() error

func (*BoltSaver) LocalGet

func (b *BoltSaver) LocalGet(key []byte, includeValue bool) (ki *api.KeyInv, err error)

func (*BoltSaver) LocalSet

func (b *BoltSaver) LocalSet(ki *api.KeyInv) error

LocalSet will set ki.Size from len(ki.Val) before saving.

type Inventory

type Inventory struct {
	*rbtree.Tree
}

func NewInventory

func NewInventory() *Inventory

func (*Inventory) String

func (t *Inventory) String() string

func (*Inventory) Upsert

func (t *Inventory) Upsert(j *api.KeyInv)

type LeadAndFollowList

type LeadAndFollowList struct {
	Members []health.AgentLoc
	LeadID  string `json:"LeadID"`
	MyID    string
}

type Peer

type Peer struct {
	Halt *idem.Halter

	LeadAndFollowBchan *bchan.Bchan
	MemberGainedBchan  *bchan.Bchan
	MemberLostBchan    *bchan.Bchan

	LeadStatus leadFlag

	GservCfg *gserv.ServerConfig

	Whoami string // as a host

	SshClientLoginUsername        string
	SshClientPrivateKeyPath       string
	SshClientClientKnownHostsPath string

	SshdReady                    chan bool
	SshClientAllowsNewSshdServer bool
	TestAllowOneshotConnect      bool

	SkipEncryption bool
	// contains filtered or unexported fields
}

Peer serves as a member of a replication cluster. One peer will be elected lead. The others will be followers. All peers will run a background receive session.

func NewPeer

func NewPeer(args, whoami string, numSetsBeforeCompact int64) (*Peer, error)

NewPeer should be given the same cmdflags as a hnatsd/gnatsd process.

"-routes=nats://localhost:9229 -cluster=nats://localhost:9230 -p 4223"

We auto-append "-health" if not provided, since that is essential for our peering network.

Each node needs its own -cluster address and -p port, and to form a cluster, the -routes of subsequent nodes need to point at one of the -cluster of an earlier started node.

After NewPeer() and before calling Start(), the Peer.SetGrpcPorts() method should be called to establish which port(s) to listen on.

If we save every 30 seconds, then numSetsBeforeCompact int64 == 1000 means compact the boltdb after every ~8 hours of writes.

func (*Peer) AmFollow

func (peer *Peer) AmFollow() bool

func (*Peer) BackgroundReceiveBcastSetAndWriteToBolt

func (peer *Peer) BackgroundReceiveBcastSetAndWriteToBolt()

func (*Peer) BcastSet

func (peer *Peer) BcastSet(ki *api.KeyInv) error

func (*Peer) ClientInitiateBcastGet

func (peer *Peer) ClientInitiateBcastGet(key []byte, includeValue bool, timeout time.Duration, who string) (kis []*api.KeyInv, err error)

func (*Peer) GetFollowSess

func (peer *Peer) GetFollowSess() (sessF *swp.Session)

func (*Peer) GetGrpcAddr

func (peer *Peer) GetGrpcAddr() string

func (*Peer) GetGrpcPorts

func (peer *Peer) GetGrpcPorts() (xport, iport int)

func (*Peer) GetLatest

func (peer *Peer) GetLatest(key []byte, includeValue bool) (ki *api.KeyInv, err error)

func (*Peer) GetPeerList

func (peer *Peer) GetPeerList(timeout time.Duration) (*LeadAndFollowList, error)

func (*Peer) LeadTransferCheckpoint

func (peer *Peer) LeadTransferCheckpoint(key, chkptData []byte, when time.Time) error

LeadTransferCheckpoint is the raison d'etre of the peer libary. This function is called called when we've just generated a checkpoint and need to propagate it out to our followers. It is safe to be called by followers, but that should be avoided since the background peer will already be checkpointing when it receives data from the lead.

The first time the lead calls this, it should be with chkptData == nil. This will signal to a new lead that it should get the most recent checkpoint from the cluster (including ourselves) so that new leads taking over don't ignore the very recent state from any previous lead.

func (*Peer) LocalGet

func (peer *Peer) LocalGet(key []byte, includeValue bool) (ki *api.KeyInv, err error)

func (*Peer) LocalSet

func (peer *Peer) LocalSet(ki *api.KeyInv) error

func (*Peer) ServerHandleBcastGet

func (peer *Peer) ServerHandleBcastGet(msg *nats.Msg) error

pull the local boltdb version of the key and send it back. User nats if only the key's metadata was requested. Use gRPC for sending the large file.

func (*Peer) SetFollowSess

func (peer *Peer) SetFollowSess(sessF *swp.Session)

func (*Peer) SetGrpcPorts

func (peer *Peer) SetGrpcPorts(xport, iport int, xhost string)

func (*Peer) Start

func (peer *Peer) Start() error

Start launches an embedded gnatsd instance in the background.

func (*Peer) StartBackgroundCheckpointRecv

func (peer *Peer) StartBackgroundCheckpointRecv(myID, myFollowSubj string)

StartBackroundCheckpointdRecv will keep a peer running in the background and always accepting and writing checkpoints (as long as we are not lead when they are received). Track these by their timestamps, and if we have a new one (recognized by a more recent timestamp), then save it to disk (this dedups if we get multiples of the same).

The host address bound will be taken from peer.serverOpts.Host which corresponds to the --addr in the nats options.

After NewPeer() and before calling this routine, the Peer.SetGrpcPorts() method should be called to establish which port(s) to listen on.

func (*Peer) StartPeriodicClusterAgentLocQueries

func (peer *Peer) StartPeriodicClusterAgentLocQueries()

func (*Peer) Stop

func (peer *Peer) Stop()

Stop shutsdown the embedded gnatsd instance.

func (*Peer) WaitForPeerCount

func (peer *Peer) WaitForPeerCount(n int, timeout time.Duration) (*LeadAndFollowList, error)

type Saver

type Saver interface {
	WriteKv(key, val []byte, timestamp time.Time) error
}

Directories

Path Synopsis
cmd
gRPC client
gRPC client
gRPC server
gRPC server
Package streambigfile is a generated protocol buffer package.
Package streambigfile is a generated protocol buffer package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL