bootstrap

package module
v0.0.200 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 8, 2021 License: LGPL-3.0 Imports: 16 Imported by: 0

README

NekoQ-Bootstrap

NekoQ-Bootstrap is the core and fundamental service for NekoQ related services.

It is responsible for the bootstrap service discovery and basic configuration storage.

In order to make NekoQ-Bootstrap as simple as possible, the usecases are kept as essential ones.

1. Usage

  1. run go build in folder cmd/nekoq-bootstrap
  2. copy bootstrap.toml.example to bootstrap.toml
  3. put them in the same folder
  4. run nekoq-bootstrap

2. Architecture

                         DC1                                                    DC2
 ┌──────────────────────────────────────────────────┐   ┌──────────────────────────────────────────────────┐
 │                                                  │   │                                                  │
 │ ┌─────────────────┐         ┌─────────────────┐  │   │  ┌─────────────────┐         ┌─────────────────┐ │
 │ │                 │         │                 ├──┼───┼─►│                 │         │                 │ │
 │ │ Nekoq Bootstrap │◄────────┤ Nekoq Discovery │  │   │  │ Nekoq Discovery ├────────►│ Nekoq Bootstrap │ │
 │ │     Cluster     │         │                 │◄─┼───┼──┤                 │         │     Cluster     │ │
 │ └─────────────────┘◄┐     ┌►└─────────────────┘  │   │  └─────────────────┘◄┐     ┌►└─────────────────┘ │
 │                     │     │                      │   │                      │     │                     │
 │                     │     │                      │   │                      │     │                     │
 │                     │     │                      │   │                      │     │                     │
 │                     │     │                      │   │                      │     │                     │
 │                     │     │                      │   │                      │     │                     │
 │               ┌─────┴─────┴─────┐                │   │                ┌─────┴─────┴─────┐               │
 │               │                 │                │   │                │                 │               │
 │               │ Nekoq Services  │                │   │                │ Nekoq Services  │               │
 │               │                 │                │   │                │                 │               │
 │               └─────────────────┘                │   │                └─────────────────┘               │
 │                                                  │   │                                                  │
 │                                                  │   │                                                  │
 │                                                  │   │                                                  │
 └──────────────────────────────────────────────────┘   └──────────────────────────────────────────────────┘

NekoQ-Boostrap is the bootstrap for all nekoq related and user services to discover fundamental services.

It acts as a service discovery but only do basic static and dynamic discovery within one single datacenter.

These services can be registered in nekoq-bootstrap:

  • nekoq discovery service(embedded in nekoq)
  • NekoQ-Security
  • nekoq consistency system(embedded in nekoq)
  • And simple storage

The workflow for a service to start is:

  1. Query nekoq-bootstrap -> discovery + nekoq-security + consistency system + key configurations
  2. Prepare authentications using nekoq-security
  3. Find essential services using discovery
  4. Get essential configurations from consistency system
  5. Do initialization within the service
  6. Ready to serve

The design principles:

  1. Keep NekoQ-Bootstrap as simple as possible in order to get high availability
  2. Only for several key components
  3. Easy to configure & start
  4. Keep as little as possible data to persist

Shared component types should be:

  1. discovery
  2. security
  3. consistency
  4. batch
  5. message queue
  6. agent/service bus

3. Feature List

DNS module
  • DNS service discovery: A record
  • DNS over http - rfc8484
  • DNS over https - rfc8484
  • DNS service discovery - AAAA/MX/SRV/TXT/CNAME
  • DNS Sec
  • DNS TCP
  • Recursive DNS
  • Authority DNS Server
Http module
  • Register several types of service
    • Support register same node to several NekoQ-Bootstrap. Note: DO NOT use different data in this case. Otherwise only the latest registration will be effect under current HA strategy within the cluster.
  • Peer auth
High available cluster module
  • Peer data sync
  • Peer auth
  • Peer data sync: dns data
Simple KV store module
  • KV store
Management
  • web manager
  • Graceful shutdown
Misc
  • Combine dns module and http module

4. Design

Cluster design

Use simple replication model:

Copy local services to every node that requesting the data

In this case, when every node in the cluster listens to other nodes, the cluster will reach a consistent state in which every node has the full data set of the cluster.

In addition, one node can be easily configured to observer mode when it listens to the cluster but nobody else listens to itself.

However, the drawbacks of this design is:

  1. If network splits, it can cause brain split as no consistency protocol runs to guarantee the majority.
  2. Data sync may great impact the network infrastructure even when full data sync happens.

5. Changelog

Planning
  • Refactor HA module
  • Web manager
v0.0.200
  • DNS module: dns/http for A record
  • Http module: query/register service
  • HA module: data sync

Documentation

Index

Constants

View Source 
const (
	DomainTypeA = 1
)
View Source 
const (
	USER_AGENT = "DNS-over-HTTPS/1.0 NekoQ-Bootstrap"
)

Variables

View Source 
var ErrStorageNotFound = errors.New("not found")

Functions

This section is empty.

Types

type DnsEndpoint 

type DnsEndpoint struct {
	Storage Storage
	Server  *dns.Server

	Addr string

	DebugPrintDnsRequest bool
}

func NewDnsEndpoint 

func NewDnsEndpoint(addr string, storage Storage) (*DnsEndpoint, error)

func (*DnsEndpoint) ServeDNS 

func (this *DnsEndpoint) ServeDNS(w dns.ResponseWriter, r *dns.Msg)

func (*DnsEndpoint) StartSync 

func (this *DnsEndpoint) StartSync() error

type DnsHttp 

type DnsHttp struct {
	Storage Storage
	Router  *httprouter.Router

	Addr string

	DebugPrintDnsRequest bool
}

func NewHttpDns 

func NewHttpDns(addr string, storage Storage) (*DnsHttp, error)

func (*DnsHttp) StartSync 

func (this *DnsHttp) StartSync() error

type DomainType 

type DomainType int

type HaModule 

type HaModule struct {
	NodeId  string
	Listen  string
	Storage Storage

	NodePeerMapping   map[string]string
	ClientPeerMapping map[string]*struct {
		LastUpdate int64
	}
	ClientPeerMappingLock sync.Mutex

	ClusterName   string
	ClusterSecret string

	SyncQueue chan SyncReq

	DebugPrint bool
	// contains filtered or unexported fields
}

func NewHaModule 

func NewHaModule(node, listen, clusterName, clusterSecret string, peerMapping map[string]string, storage Storage) (*HaModule, error)

func (*HaModule) CheckPeerHealth 

func (this *HaModule) CheckPeerHealth()

func (*HaModule) HttpEndpoint 

func (this *HaModule) HttpEndpoint() error

func (*HaModule) PeerSyncWorker 

func (this *HaModule) PeerSyncWorker(addr, peerNodeId string)

func (*HaModule) ProcessSyncWorker 

func (this *HaModule) ProcessSyncWorker()

func (*HaModule) ServeHTTP 

func (this *HaModule) ServeHTTP(w http.ResponseWriter, r *http.Request)

func (*HaModule) StartSync 

func (this *HaModule) StartSync() error

type HttpEndpoint 

type HttpEndpoint struct {
	Storage Storage
	Router  *httprouter.Router

	EnableAuth     bool
	AccessPassword string

	Addr string

	DebugPrint bool
	// contains filtered or unexported fields
}

func NewHttpEndpoint 

func NewHttpEndpoint(addr string, storage Storage, enableAuth bool, accessPassword string) (*HttpEndpoint, error)

func (*HttpEndpoint) CheckPublishClients 

func (this *HttpEndpoint) CheckPublishClients()

func (*HttpEndpoint) StartSync 

func (this *HttpEndpoint) StartSync() error

type MemStore 

type MemStore struct {
	// contains filtered or unexported fields
}

func NewMemStore 

func NewMemStore() *MemStore

func (*MemStore) Abandon 

func (m *MemStore) Abandon(node string) error

func (*MemStore) DeleteService 

func (this *MemStore) DeleteService(service string, item *ServiceItem) error

func (*MemStore) FetchChangesForPeerNodeRequest 

func (m *MemStore) FetchChangesForPeerNodeRequest(node string) (add, del map[string][]byte, err error)

func (*MemStore) FetchFullAndWatch 

func (m *MemStore) FetchFullAndWatch(node string) (map[string][]byte, error)

func (*MemStore) FullFrom 

func (m *MemStore) FullFrom(node string, data map[string][]byte) error

func (*MemStore) GetServiceList 

func (this *MemStore) GetServiceList(service string) ([]*ServiceItem, error)

func (*MemStore) PublishService 

func (this *MemStore) PublishService(service string, item *ServiceItem) error

func (*MemStore) PutDomain 

func (m *MemStore) PutDomain(domain, resolve string, domainType DomainType)

func (*MemStore) ResolveDomain 

func (m *MemStore) ResolveDomain(domain string, domainType DomainType) (string, error)

func (*MemStore) SyncFrom 

func (m *MemStore) SyncFrom(node, origNode string, add, del map[string][]byte) error

func (*MemStore) Unwatch 

func (m *MemStore) Unwatch(node string) error

type ServiceItem 

type ServiceItem struct {
	Addr   string `json:"address"`
	NodeId string `json:"node_id"`
}

type Storage 

type Storage interface {
	ResolveDomain(domain string, domainType DomainType) (string, error)
	PutDomain(domain, resolve string, domainType DomainType)

	GetServiceList(service string) ([]*ServiceItem, error)
	PublishService(service string, item *ServiceItem) error
	DeleteService(service string, item *ServiceItem) error

	/*
		for High Availability - client(listener) side
	*/
	FullFrom(node string, data map[string][]byte) error // get and watch
	SyncFrom(node, origNode string, add, del map[string][]byte) error
	Abandon(node string) error
	/*
		for High Availability - server(source) side
	*/
	FetchFullAndWatch(node string) (map[string][]byte, error)
	FetchChangesForPeerNodeRequest(node string) (add, del map[string][]byte, err error)
	Unwatch(node string) error
}

type SyncReq 

type SyncReq struct {
	NodeId string
	Full   map[string][]byte

	Add map[string][]byte
	Del map[string][]byte

	Err chan error
}

Source Files

View all Source files

Directories

Path Synopsis
cmd
nekoq-bootstrap
sample_dns_client

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.