peer

package
v1.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 25, 2017 License: MIT Imports: 28 Imported by: 0

Documentation

Index

Constants

View Source 
const RequestChanLen = 8

Variables

View Source 
var BoltDataBucketName = []byte("data") // bucket
View Source 
var BoltMetaBucketName = []byte("meta") // bucket
View Source 
var BoltSizeBucketName = []byte("size") // bucket
View Source 
var ErrAmFollower = fmt.Errorf("LeadTransferCheckpoint error: I am follower, not transmitting checkpoint")
View Source 
var ErrAmLead = fmt.Errorf("error: I am lead")
View Source 
var ErrNoFollowers = fmt.Errorf("error: no followers")
View Source 
var ErrShutdown = fmt.Errorf("shutting down")
View Source 
var ErrTimedOut = fmt.Errorf("timed out")

Functions

func GenAddress 

func GenAddress() string

GenAddress generates a local address by calling GetAvailPort() and GetExternalIP(), then prefixing them with 'tcp://'.

func GetExternalIP 

func GetExternalIP() string

GetExternalIP tries to determine the external IP address used on this host.

func GetExternalIPAsInt 

func GetExternalIPAsInt() int

GetExternalIPAsInt calls GetExternalIP() and then converts the resulting IPv4 string into an integer.

func IsRoutableIPv4 

func IsRoutableIPv4(ip string) bool

IsRoutableIPv4 returns true if the string in ip represents an IPv4 address that is not private. See http://en.wikipedia.org/wiki/Private_network#Private_IPv4_address_spaces for the numeric ranges that are private. 127.0.0.1, 192.168.0.1, and 172.16.0.1 are examples of non-routables IP addresses.

func StripNanomsgAddressPrefix 

func StripNanomsgAddressPrefix(nanomsgAddr string) (suffix string, err error)

StripNanomsgAddressPrefix removes the 'tcp://' prefix from nanomsgAddr.

func UtilCleanupTestUserDatabases 

func UtilCleanupTestUserDatabases()

func UtilTestSetupThree 

func UtilTestSetupThree(callmePreStart func(peer *Peer)) (p0, p1, p2 *Peer, peers *LeadAndFollowList)

Types

type BoltSaver 

type BoltSaver struct {
	// contains filtered or unexported fields
}

func NewBoltSaver 

func NewBoltSaver(filepath string, who string, numSetsBeforeCompact int64) (*BoltSaver, error)

func (*BoltSaver) Close 

func (b *BoltSaver) Close()

func (*BoltSaver) Compact added in v0.9.9 

func (b *BoltSaver) Compact(lockNeeded bool) error

Compact does these steps to reduce the space fragmentation that happens in a bolt database.

  1. open a fresh .compressed bolt db file.
  2. read each object from the bolt db and write it to the fresh -> .compressed bolt db file.
  3. close the both files.
  4. rename the .compressed file to be the original db file name. os.Rename is atomic.
  5. re-open the newly compact-ed db file

The routines below were adapted from the compaction code in https://github.com/boltdb/bolt/blob/master/cmd/bolt/main.go which is used under the following MIT license.

The MIT License (MIT)

Copyright (c) 2013 Ben Johnson

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

INVAR: b.db must be already open.

func (*BoltSaver) InitDbIfNeeded 

func (b *BoltSaver) InitDbIfNeeded() error

func (*BoltSaver) LocalGet 

func (b *BoltSaver) LocalGet(key []byte, includeValue bool) (ki *api.KeyInv, err error)

func (*BoltSaver) LocalSet 

func (b *BoltSaver) LocalSet(ki *api.KeyInv) error

LocalSet will set ki.Size from len(ki.Val) before saving.

type Inventory 

type Inventory struct {
	*rbtree.Tree
}

func NewInventory 

func NewInventory() *Inventory

func (*Inventory) String 

func (t *Inventory) String() string

func (*Inventory) Upsert 

func (t *Inventory) Upsert(j *api.KeyInv)

type LeadAndFollowList 

type LeadAndFollowList struct {
	Members []health.AgentLoc
	LeadID  string `json:"LeadID"`
	MyID    string
}

type Peer 

type Peer struct {
	Halt *idem.Halter

	LeadAndFollowBchan *bchan.Bchan
	MemberGainedBchan  *bchan.Bchan
	MemberLostBchan    *bchan.Bchan

	LeadStatus leadFlag

	GservCfg *gserv.ServerConfig

	Whoami string // as a host

	SshClientLoginUsername        string
	SshClientPrivateKeyPath       string
	SshClientClientKnownHostsPath string

	SshdReady                    chan bool
	SshClientAllowsNewSshdServer bool
	TestAllowOneshotConnect      bool

	SkipEncryption bool
	// contains filtered or unexported fields
}

Peer serves as a member of a replication cluster. One peer will be elected lead. The others will be followers. All peers will run a background receive session.

func NewPeer 

func NewPeer(args, whoami string, numSetsBeforeCompact int64) (*Peer, error)

NewPeer should be given the same cmdflags as a hnatsd/gnatsd process.

"-routes=nats://localhost:9229 -cluster=nats://localhost:9230 -p 4223"

We auto-append "-health" if not provided, since that is essential for our peering network.

Each node needs its own -cluster address and -p port, and to form a cluster, the -routes of subsequent nodes need to point at one of the -cluster of an earlier started node.

After NewPeer() and before calling Start(), the Peer.SetGrpcPorts() method should be called to establish which port(s) to listen on.

If we save every 30 seconds, then numSetsBeforeCompact int64 == 1000 means compact the boltdb after every ~8 hours of writes.

func (*Peer) AmFollow 

func (peer *Peer) AmFollow() bool

func (*Peer) BackgroundReceiveBcastSetAndWriteToBolt 

func (peer *Peer) BackgroundReceiveBcastSetAndWriteToBolt()

func (*Peer) BcastSet 

func (peer *Peer) BcastSet(ki *api.KeyInv) error

func (*Peer) ClientInitiateBcastGet 

func (peer *Peer) ClientInitiateBcastGet(key []byte, includeValue bool, timeout time.Duration, who string) (kis []*api.KeyInv, err error)

func (*Peer) GetFollowSess 

func (peer *Peer) GetFollowSess() (sessF *swp.Session)

func (*Peer) GetGrpcAddr 

func (peer *Peer) GetGrpcAddr() string

func (*Peer) GetGrpcPorts 

func (peer *Peer) GetGrpcPorts() (xport, iport int)

func (*Peer) GetLatest 

func (peer *Peer) GetLatest(key []byte, includeValue bool) (ki *api.KeyInv, err error)

func (*Peer) GetPeerList 

func (peer *Peer) GetPeerList(timeout time.Duration) (*LeadAndFollowList, error)

func (*Peer) LeadTransferCheckpoint 

func (peer *Peer) LeadTransferCheckpoint(key, chkptData []byte, when time.Time) error

LeadTransferCheckpoint is the raison d'etre of the peer libary. This function is called called when we've just generated a checkpoint and need to propagate it out to our followers. It is safe to be called by followers, but that should be avoided since the background peer will already be checkpointing when it receives data from the lead.

The first time the lead calls this, it should be with chkptData == nil. This will signal to a new lead that it should get the most recent checkpoint from the cluster (including ourselves) so that new leads taking over don't ignore the very recent state from any previous lead.

func (*Peer) LocalGet 

func (peer *Peer) LocalGet(key []byte, includeValue bool) (ki *api.KeyInv, err error)

func (*Peer) LocalSet 

func (peer *Peer) LocalSet(ki *api.KeyInv) error

func (*Peer) ServerHandleBcastGet 

func (peer *Peer) ServerHandleBcastGet(msg *nats.Msg) error

pull the local boltdb version of the key and send it back. User nats if only the key's metadata was requested. Use gRPC for sending the large file.

func (*Peer) SetFollowSess 

func (peer *Peer) SetFollowSess(sessF *swp.Session)

func (*Peer) SetGrpcPorts 

func (peer *Peer) SetGrpcPorts(xport, iport int, xhost string)

func (*Peer) Start 

func (peer *Peer) Start() error

Start launches an embedded gnatsd instance in the background.

func (*Peer) StartBackgroundCheckpointRecv 

func (peer *Peer) StartBackgroundCheckpointRecv(myID, myFollowSubj string)

StartBackroundCheckpointdRecv will keep a peer running in the background and always accepting and writing checkpoints (as long as we are not lead when they are received). Track these by their timestamps, and if we have a new one (recognized by a more recent timestamp), then save it to disk (this dedups if we get multiples of the same).

The host address bound will be taken from peer.serverOpts.Host which corresponds to the --addr in the nats options.

After NewPeer() and before calling this routine, the Peer.SetGrpcPorts() method should be called to establish which port(s) to listen on.

func (*Peer) StartPeriodicClusterAgentLocQueries 

func (peer *Peer) StartPeriodicClusterAgentLocQueries()

func (*Peer) Stop 

func (peer *Peer) Stop()

Stop shutsdown the embedded gnatsd instance.

func (*Peer) WaitForPeerCount 

func (peer *Peer) WaitForPeerCount(n int, timeout time.Duration) (*LeadAndFollowList, error)

type Saver 

type Saver interface {
	WriteKv(key, val []byte, timestamp time.Time) error
}

Source Files

View all Source files

Directories

Path Synopsis
cmd
viewdb
gRPC client
 gRPC client
gRPC server
 gRPC server
Package streambigfile is a generated protocol buffer package.
 Package streambigfile is a generated protocol buffer package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.