Documentation
¶
Overview ¶
Package shard implements sharded writer.
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type PrefixSum64Hash ¶
PrefixSum64Hash implements hash.Hash64 interface.
func (*PrefixSum64Hash) Sum64 ¶
func (h *PrefixSum64Hash) Sum64() uint64
Sun64 implements hash.Hash64 interface. It simply reads the first 8 bytes by big endian.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer writes data to one of the shard based on the hash function.
Example ¶
name, err := ioutil.TempDir("", "test")
if err != nil {
fmt.Println(err)
return
}
defer os.RemoveAll(name) //nolint:wsl
w := NewWriter(5, &PrefixSum64Hash{sha512.New()}, NewOSFileWriterFactory(path.Join(name, "test-")))
records := []string{
"test0",
"test1",
"test2",
"test3",
"test4",
"test5",
"test6",
"test7",
"test1",
}
for _, rec := range records {
if _, err := w.Write([]byte(rec)); err != nil {
fmt.Println(err)
}
}
w.Close()
for i := 0; i < 5; i++ {
filename := fmt.Sprintf("test-%05d-of-00005", i)
b, _ := ioutil.ReadFile(path.Join(name, filename))
fmt.Printf("%s:%s\n", filename, string(b))
}
Output: test-00000-of-00005:test0test4 test-00001-of-00005:test7 test-00002-of-00005:test5test6 test-00003-of-00005: test-00004-of-00005:test1test2test3test1
func NewWriter ¶
func NewWriter(n int, h hash64, wf WriterFactory) *Writer
NewWriter returns a new sharded writer.
type WriterFactory ¶
WriterFactory returns a writer for each i of n.
func NewOSFileWriterFactory ¶
func NewOSFileWriterFactory(prefix string) WriterFactory
NewOSFileWriterFactory returns a WriterFactory that returns os.File with the filenane prefixed by prefix and the index string such as 00000-of-00100, 00001-of-00100, and so on.
Source Files
Click to show internal directories.
Click to hide internal directories.