Documentation
¶
Index ¶
Constants ¶
const ( // BUFCHUNKNUM is num of chunk in buffer // Why it's 9? I think 126 % 9 = 0 // (In fact, 126 % 7 also equal 0) // BUF write to Packet will process well BUFCHUNKNUM = 9 // BUFSIZE is size of buffer used in write process BUFSIZE = CHUNKCONTENTSIZE * BUFCHUNKNUM // CHUNKCONTENTSIZE is size of chunk's content CHUNKCONTENTSIZE = 512 // CHUNKCHECKSUMSIZE is size of chunk's checksum CHUNKCHECKSUMSIZE = 4 // CHUNKTOTALSIZE equal size(checksum) + size(content) CHUNKTOTALSIZE = CHUNKCONTENTSIZE + CHUNKCHECKSUMSIZE // PACKETCHUNKNUM is num of chunk in packet PACKETCHUNKNUM = 126 // PACKETCHUNKSIZE is increasing offset in file when writing & reading PACKETCHUNKSIZE = PACKETCHUNKNUM * CHUNKCONTENTSIZE // PACKETTOTALSIZE is size of packet PACKETTOTALSIZE = PACKETCHUNKNUM * CHUNKTOTALSIZE )
Variables ¶
This section is empty.
Functions ¶
func Chunk2ChunkProto ¶
func Chunk2ChunkProto(chunk *Chunk) (chunkPb *pb.ChunkProto)
Chunk2ChunkProto is a translator
func ParseInstr ¶
ParseInstr is a function to process instrucion
Types ¶
type ChunkPb ¶
type ChunkPb struct {
// contains filtered or unexported fields
}
ChunkPb is a data structure defined by protobuf
func NewChunkPb ¶
NewChunkPb is a constructor of Chunk defined by protobuf
type HDFSOutputStream ¶
HDFSOutputStream can write
func (*HDFSOutputStream) Write ¶
func (ops *HDFSOutputStream) Write(localPath string)
Write is user interface a function, use a stream-object to upload file to hdfs
- communicate with namenode build a conn between name node and client, send localPath, query blocks and get metadata
- build a conn between data node and client
- for { get NewPacket as "pkt" from file descriptor. conn.Write(pkt) }
type Packet ¶
type Packet struct {
Header *pb.PacketHeaderProto
PacketData []Chunk
PacketOffset int
// contains filtered or unexported fields
}
Packet is base data structure in hdfs PacketData should be separate into checksumField and DataField, but I do not choose to implement that kind of packet format
PacketOffset means the number of which Write function has appended, if PacketOffset equal PACKETCHUNKNUM(126), listen fiber will flush it.
type PacketCtx ¶
type PacketCtx struct {
OffsetInBlock uint64
SeqNo uint32
LastPacketInBlock bool
DataLen uint32
Buf []byte
EmptyBuf []byte
// contains filtered or unexported fields
}
PacketCtx is a user interface rw(io.ReadWriter) is usually file descriptor Buf's length equal CHUNKCONTENTSIZE
func NewPacketCtx ¶
func NewPacketCtx(rw io.ReadWriter, OffsetInBlock uint64, SeqNo uint32, LastPacketInBlock bool, DataLen uint32) (ctx *PacketCtx)
NewPacketCtx is a constructor
type PacketQueue ¶
type PacketQueue struct {
// contains filtered or unexported fields
}
PacketQueue is a base data structure in hdfs
func (*PacketQueue) GetLength ¶
func (pq *PacketQueue) GetLength() int
GetLength is a method to return the num of elements
func (*PacketQueue) Pop ¶
func (pq *PacketQueue) Pop() (out *Packet, err error)
Pop is to pop a elem from queue
func (*PacketQueue) Push ¶
func (pq *PacketQueue) Push(elem Packet) error
Push is to push a elem into queue
type PacketReadWriter ¶
PacketReadWriter is an interface
Source Files