grasure

package module
v0.0.0-...-6c569f3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 16, 2022 License: MIT Imports: 33 Imported by: 0

README

GrasureOL

A distributed version of Grasure with powerful Web and DB support.

Targets

Basically, we adopt Grausre as the core fullfilling functionalities like writing, reading, and recovering. And a pretty-looking HTTP frontend.

This project is governed by Durant Thorvalds since 13th, April, 2022.

Our goal is to forge an atomic, simple-to-study, highly module-independant and integratable project.

Documentation

Overview

Package grasure is an Universal Erasure Coding Architecture in Go

For usage and examples, see https://github.com/DurantVivado/Grasure

Index

Constants

View Source 
const (
	DATA      = "DATA"
	CLIENT    = "CLIENT"
	SERVER    = "SERVER"
	COMPUTING = "COMPUTING"
	GATEWAY   = "GATEWAY"
	NAME      = "NAME"
)
View Source 
const (
	DEFAULT_MAGIC_NUMBER = 0x7fffffff
	READ_MAGIC_NUMBER    = 0xaaaaaaaa
	WRITE_MAGIC_NUMBER   = 0xbbbbbbbb
)

Variables

View Source 
var DefaultOption = &Option{
	MagicNumber:    DEFAULT_MAGIC_NUMBER,
	CodecType:      codec.GobType,
	ConnectTimeout: defaultConnectionTimeout,
	HandleTimeout:  defaultHandleTimeout,
}
View Source 
var DefaultRegister = NewRegistry(defaultRegistryTimeout)
View Source 
var DefaultServer = NewServer()
View Source 
var ErrShutdown = errors.New("connection is shut down")

Functions

func Accept 

func Accept(lis net.Listener)

Accept accepts connections on the listener and serves requests for each incoming connection.

func DiskUsage 

func DiskUsage(path string) (uint64, uint64)

return the total and free volume of a disk in bytes

func HandleHTTP 

func HandleHTTP()

HandleHTTP is a convenient approach for default server to register HTTP handlers

func Heartbeat 

func Heartbeat(ctx context.Context, registry, addr string, duration time.Duration)

Heartbeat send a heartbeat message every once in a while it's a helper function for a server to register or send heartbeat

func Register 

func Register(rcvr interface{}) error

Register publishes the receiver's methods in the DefaultServer.

func RegistryHandleHTTP 

func RegistryHandleHTTP()

Types

type ACL 

type ACL struct {
}

ACL acts as access control list

type BlockReadRequest 

type BlockReadRequest struct {
	Address string
	Offset  uint64
	Size    uint64
}

type BlockReadResponse 

type BlockReadResponse struct {
	Msg  string
	Data []byte
}

type Call 

type Call struct {
	Seq           uint64
	ServiceMethod string      // format "<service>.<method>"
	Args          interface{} // arguments to the function
	Reply         interface{} // reply from the function
	Error         error       // if error occurs, it will be set
	Done          chan *Call  // Strobes when call is complete.
}

Call represents an active RPC.

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client represents an RPC Client. There may be multiple outstanding Calls associated with a single Client, and a Client may be used by multiple goroutines simultaneously.

func Dial 

func Dial(network, address string, opts ...*Option) (*Client, error)

Dial connects to an RPC server at the specified network address

func DialHTTP 

func DialHTTP(network, address string, opts ...*Option) (*Client, error)

DialHTTP connects to an HTTP RPC server at the specified network address listening on the default HTTP RPC path.

func NewClient 

func NewClient(conn net.Conn, opt *Option) (*Client, error)

func NewHTTPClient 

func NewHTTPClient(conn net.Conn, opt *Option) (*Client, error)

NewHTTPClient new a Client instance via HTTP as transport protocol

func XDial 

func XDial(rpcAddr string, opts ...*Option) (*Client, error)

XDial calls different functions to connect to a RPC server according the first parameter rpcAddr. rpcAddr is a general format (protocol@addr) to represent a rpc server eg, http@10.0.0.1:7001, tcp@10.0.0.1:9999, unix@/tmp/geerpc.sock

func (*Client) Call 

func (c *Client) Call(ctx context.Context, serviceMethod string, args, reply interface{}) error

Call invokes the named function, waits for it to complete, and returns its error status.

func (*Client) Close 

func (c *Client) Close() error

Close the connection

func (*Client) Go 

func (c *Client) Go(serviceMethod string, args, reply interface{}, done chan *Call) *Call

Go invokes the function asynchronously. It returns the Call structure representing the invocation.

func (*Client) IsAvailable 

func (c *Client) IsAvailable() bool

IsAvailable return true if the client does work

type Cluster 

type Cluster struct {
	//UUID is the global unique id of a cluster
	UUID int64 `json:"UUID"`

	//InfoFilePath is the path of a textual file storing basic information of the cluster
	InfoFilePath string

	//NodeFilePath is the path of file includes all nodes address, provided by user
	NodeFilePath string
	// contains filtered or unexported fields
}

Cluster is an instance that is only created once at each node

func NewCluster 

func NewCluster(ctx context.Context, usedNodeNum int, hashfn Hash) *Cluster

NewCluster initializes a Cluster with customized ataShards, parityShards, usedNodeNum, replicateFactor and blockSize

func (*Cluster) AddErasurePools 

func (c *Cluster) AddErasurePools(pool ...*ErasurePool)

AddErasurePool adds an erasure-pool to the cluster's poolMap whose key should be like "RS-10-4-1024k", namely policy.

func (*Cluster) AddNode 

func (c *Cluster) AddNode(id int, node *Node)

AddNode adds a node into cluster using ConsistentHashAlgorithm

func (*Cluster) CheckIfExistPool 

func (c *Cluster) CheckIfExistPool(pattern string) (*ErasurePool, bool)

CheckIfExistPool checks if there exists pool of certain pattern, e.g. "RS-4-2-4096"

func (*Cluster) DelNode 

func (c *Cluster) DelNode(id int)

DelNode deletes a node from the map

func (*Cluster) GetIPsFromRole 

func (c *Cluster) GetIPsFromRole(role string) (addrs []string)

GetIPsFromRole returns IP address according to given role

func (*Cluster) GetLocalNode 

func (c *Cluster) GetLocalNode() (ret *Node)

GetLocalNode returns the local node if exists in the cluster

func (*Cluster) GetNodesFromRole 

func (c *Cluster) GetNodesFromRole(role string) (nodes []*Node)

GetNodesFromRole returns Node slice according to given role

func (*Cluster) ReadNodesAddr 

func (c *Cluster) ReadNodesAddr()

ReadNodesAddr reads the node information from file

func (*Cluster) SetNodeStatus 

func (c *Cluster) SetNodeStatus(duration time.Duration)

SetNodeStatus reset nodes' status every duration

func (*Cluster) SetOuput 

func (c *Cluster) SetOuput(level LogLevel, filename string)

func (*Cluster) StartDFSServer 

func (c *Cluster) StartDFSServer(port string)

StartDFSServer starts the server to handle HTTP requests

func (*Cluster) StartHeartbeatServer 

func (c *Cluster) StartHeartbeatServer(port string)

StartServer starts the server to handle HTTP requests

type ClusterOption 

type ClusterOption struct {
	// contains filtered or unexported fields
}

type ClusterStatus 

type ClusterStatus struct {
	UUID string
	// contains filtered or unexported fields
}

ClusterStatus is delpoyed on certain port of the server to inform the user of real-time status of the cluster

type Context 

type Context context.Context

type Discovery 

type Discovery interface {
	Refresh() error // refresh from remote registry
	Update(servers []string) error
	Get(mode SelectMode) (string, error)
	GetAll() ([]string, error)
}

type DiskArray 

type DiskArray struct {
	// contains filtered or unexported fields
}

DiskArray contains the low-level disk information

func NewDiskArray 

func NewDiskArray(diskFilePath string) *DiskArray

func (*DiskArray) ReadDiskPath 

func (d *DiskArray) ReadDiskPath()

ReadDiskPath reads the disk paths from diskFilePath. There should be exactly ONE disk path at each line.

This func can NOT be called concurrently.

type DiskInfo 

type DiskInfo struct {
	// contains filtered or unexported fields
}

func NewDiskInfo 

func NewDiskInfo(path string) *DiskInfo

NewDiskInfo news a disk with basic information

type DiskState 

type DiskState int
const (
	Normal DiskState = iota
	Fail
	BitRot
)

type ErasurePool 

type ErasurePool struct {
	//the redun is how the files are encoded in this pool
	Redun Redundancy `json:"redundancy"`

	// the number of data blocks in a stripe
	K int `json:"dataShards"`

	// the number of parity blocks in a stripe
	M int `json:"parityShards"`

	//the used node number for the pool
	NodeNum int `json:"nodeNum"`

	// the block size. default to 4KiB
	BlockSize int64 `json:"blockSize"`
	// contains filtered or unexported fields
}

func NewErasurePool 

func NewErasurePool(redun Redundancy, dataShards, parityShards, nodeNum int, blockSize int64, dataNodes []int, layout Pattern) *ErasurePool

NewErasurePool news an erasurePool with designated dataShards, parityShards, nodeNum and blockSize, When set dataNodes as nil by default, the pool uses the first nodeNum nodes, you can specify the nodes in the order of their ids (indexed from 0).

func (*ErasurePool) Write 

func (ep *ErasurePool) Write(src io.ReadCloser, addr string) error

Write writes a file or byte flow into certain node.

type Hash 

type Hash func(key []byte) uint32

Hash maps bytes to uint32

type Int64Arr 

type Int64Arr []int64

func (Int64Arr) Len 

func (arr Int64Arr) Len() int

func (Int64Arr) Less 

func (arr Int64Arr) Less(i, j int) bool

func (Int64Arr) Swap 

func (arr Int64Arr) Swap(i, j int)

type IntSet 

type IntSet map[int]struct{}

A Go-version Set

func (*IntSet) Clear 

func (is *IntSet) Clear()

func (*IntSet) Empty 

func (is *IntSet) Empty() bool

func (*IntSet) Erase 

func (is *IntSet) Erase(x int)

func (*IntSet) Exist 

func (is *IntSet) Exist(x int) bool

func (*IntSet) Insert 

func (is *IntSet) Insert(x int)

func (*IntSet) Size 

func (is *IntSet) Size() int

type Layout 

type Layout struct {
	// contains filtered or unexported fields
}

Layout determines the block location relative to the nodes, usually stored on name node

func NewLayout 

func NewLayout(pattern Pattern, stripeNum, dataShards, parityShards, nodeNum int) *Layout

NewLayout news a layout for certain patterns stripeNum = totalPoolSize // blockSize // stripeLen stripeLen = dataShards + parityShards

type LogLevel 

type LogLevel int
const (
	Disabled LogLevel = iota
	FatalLevel
	ErrorLevel
	InfoLevel
	DebugLevel
)

type Mode 

type Mode string
const (
	InitMode      Mode = "InitMode"
	NormalMode    Mode = "NormalMode"
	DegradedMode  Mode = "DegradedMode"
	RecoveryMode  Mode = "RecoveryMode"
	ScaleMode     Mode = "ScaleMode"
	PowerSaveMode Mode = "PowerSaveMode"
)

type MultiServersDiscovery 

type MultiServersDiscovery struct {
	// contains filtered or unexported fields
}

MultiServersDiscovery is a discovery for multi servers without a registry center user provides the server addresses explicitly instead

func NewMultiServerDiscovery 

func NewMultiServerDiscovery(servers []string) *MultiServersDiscovery

NewMultiServerDiscovery creates a MultiServersDiscovery instance

func (*MultiServersDiscovery) Get 

func (d *MultiServersDiscovery) Get(mode SelectMode) (string, error)

Get a server according to mode

func (*MultiServersDiscovery) GetAll 

func (d *MultiServersDiscovery) GetAll() ([]string, error)

GetAll returns all servers in discovery

func (*MultiServersDiscovery) Refresh 

func (d *MultiServersDiscovery) Refresh() error

Refresh doesn't make sense for MultiServersDiscovery, so ignore it

func (*MultiServersDiscovery) Update 

func (d *MultiServersDiscovery) Update(servers []string) error

Update the servers of discovery dynamically if needed

type Node 

type Node struct {

	//For name node:
	//FileMeta lists, indicating fileName, fileSize, fileHash, fileDist...
	FileMeta sync.Map
	// contains filtered or unexported fields
}

func NewNode 

func NewNode(ctx context.Context, id int, addr string, nodeType []NodeType, redun Redundancy) *Node

type NodeStat 

type NodeStat string
const (
	NodeInit     NodeStat = "NodeInit"
	HealthOK     NodeStat = "HealthOK"
	CPUFailed    NodeStat = "CPUFailed"
	DiskFailed   NodeStat = "DiskFailed"
	NetworkError NodeStat = "NetworkError"
)

type NodeStatus 

type NodeStatus struct {
	// contains filtered or unexported fields
}

type NodeType 

type NodeType string

type NodeWriter 

type NodeWriter struct {
	// contains filtered or unexported fields
}

type Option 

type Option struct {
	MagicNumber    int           // MagicNumber marks this's a geerpc request
	CodecType      codec.Type    // client may choose different Codec to encode body
	ConnectTimeout time.Duration // 0 means no limit
	HandleTimeout  time.Duration
}

type Output 

type Output int
const (
	CONSOLE Output = iota
	LOGFILE
	NONE
)

type ParallelReader 

type ParallelReader struct {
	// contains filtered or unexported fields
}

ParallelReader is a reader that handles parallel disk reads

func NewParallelReader 

func NewParallelReader(pool *ErasurePool, offset, size uint64, degrade bool) *ParallelReader

func (*ParallelReader) ReadBlock 

func (pr *ParallelReader) ReadBlock(offset, size uint64)

type ParallelWriter 

type ParallelWriter struct {
	// contains filtered or unexported fields
}

ParallelReader is a writer that write data bytes to multiple nodes

type Pattern 

type Pattern int
const (
	Random Pattern = iota
	// see http://www.accs.com/p_and_p/RAID/LinuxRAID.html for more infos
	LeftSymmetric
	LeftAsymmetric
	RightSymmetric
	RightAsymmetric
)

type Redundancy 

type Redundancy string
const (
	Erasure_RS  Redundancy = "Erasure_RS"
	Erasure_XOR Redundancy = "Erasure_XOR"
	Erasure_LRC Redundancy = "Erasure_LRC"
	Replication Redundancy = "Replication"
	None        Redundancy = "None"
)

type Registry 

type Registry struct {
	// contains filtered or unexported fields
}

Registry is a simple register center, provide following functions. add a server and receive heartbeat to keep it alive. returns all alive servers and delete dead servers sync simultaneously.

func NewRegistry 

func NewRegistry(timeout time.Duration) *Registry

NewRegistry create a registry instance with timeout setting

func (*Registry) HandleHTTP 

func (r *Registry) HandleHTTP(registryPath string)

HandleHTTP registers an HTTP handler for Registry messages on registryPath

func (*Registry) ServeHTTP 

func (r *Registry) ServeHTTP(w http.ResponseWriter, req *http.Request)

Runs at /_geerpc_/registry

type RegistryDiscovery 

type RegistryDiscovery struct {
	*MultiServersDiscovery
	// contains filtered or unexported fields
}

func NewRegistryDiscovery 

func NewRegistryDiscovery(registerAddr string, timeout time.Duration) *RegistryDiscovery

func (*RegistryDiscovery) Get 

func (d *RegistryDiscovery) Get(mode SelectMode) (string, error)

func (*RegistryDiscovery) GetAll 

func (d *RegistryDiscovery) GetAll() ([]string, error)

func (*RegistryDiscovery) Refresh 

func (d *RegistryDiscovery) Refresh() error

func (*RegistryDiscovery) Update 

func (d *RegistryDiscovery) Update(servers []string) error

type SelectMode 

type SelectMode int
const (
	RandomSelect     SelectMode = iota // select randomly
	RoundRobinSelect                   // select using Robbin algorithm
)

type Server 

type Server struct {
	// contains filtered or unexported fields
}

Server represents an RPC server.

func NewServer 

func NewServer() *Server

func (*Server) Accept 

func (s *Server) Accept(lis net.Listener)

Accept accepts connections on the listener and serves requests for each incoming connection.

func (*Server) HandleHTTP 

func (s *Server) HandleHTTP()

HandleHTTP registers an HTTP handler for RPC messages on rpcPath. It is still necessary to invoke http.Serve(), typically in a go statement.

func (*Server) Register 

func (s *Server) Register(rcvr interface{}) error

Register publishes in the server the set of methods of the

func (*Server) ServeConn 

func (s *Server) ServeConn(conn io.ReadWriteCloser)

func (*Server) ServeHTTP 

func (s *Server) ServeHTTP(w http.ResponseWriter, req *http.Request)

ServeHTTP implements an http.Handler that answers RPC requests.

type ServerItem 

type ServerItem struct {
	Addr string
	// contains filtered or unexported fields
}

type SimOptions 

type SimOptions struct {
	//switch between "diskFail" and "bitRot"
	Mode string
	// specify which disks to fail
	FailDisk string
	// specify number of disks to fail
	FailNum int
	//specify the fileName, used only for "bitRot" mode
	FileName string
}

SimOptions defines the parameters for simulation

type StorageAPI 

type StorageAPI interface {
	ListDir(ctx context.Context, dirPath string, level int) ([]string, error)
	ReadFile(ctx context.Context, path string, offset, size int64, buf []byte) (n int64, err error)
	ReadFileStream(ctx context.Context, path string, offset, length int64) (io.ReadCloser, error)

	WriteFile(ctx context.Context, path string, size int64, buf []byte) error
	WriteFileStream(ctx context.Context, path string, size int64, reader io.Reader) error
	RenameFile(ctx context.Context, srcPath, dstPath string) error

	Delete(ctx context.Context, path string, recursive bool) (err error)
}

Customer APIs

type Version 

type Version string

type Volume 

type Volume struct {
	Used  uint64 `json:"Used"`
	Free  uint64 `json:"Free"`
	Total uint64 `json:"Total"`
}

type XClient 

type XClient struct {
	// contains filtered or unexported fields
}

func NewXClient 

func NewXClient(d Discovery, mode SelectMode, opt *Option) *XClient

func (*XClient) Broadcast 

func (xc *XClient) Broadcast(ctx context.Context, serviceMethod string, args, reply interface{}) error

Broadcast invokes the named function for every server registered in discovery Once a server returns response, the function quickly quits by a cancelled context

func (*XClient) Call 

func (xc *XClient) Call(ctx context.Context, serviceMethod string, args, reply interface{}) error

Call invokes the named function, waits for it to complete, and returns its error status. xc will choose a proper server.

func (*XClient) Close 

func (xc *XClient) Close() error

Source Files

View all Source files

Directories

Path Synopsis
Package xlog is a user-defined log, include multi-level highlight support, output redirection support
 Package xlog is a user-defined log, include multi-level highlight support, output redirection support

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.