badgerutils

Badger Utils

Go package with utilities for interacting with Badger.

Table of Contents

• Getting Started
  • IO Stream to Badger
    • Example
• Development
  • Dependency Management
  • Format Code
  • Unit Tests

Getting Started

IO Stream to Badger

To stream data Badger, use badgerutils.WriteStream.

Example

Creates a CLI tool that streams data from stdin.

// examples/writer_cli.go
package main

import (
	"errors"
	"flag"
	"fmt"
	"log"
	"os"
	"strings"

	"github.com/Surfline/badgerutils"
)

type sampleRecord struct {
	Key   string
	Value string
}

func csvToKeyValue(line string) (*badgerutils.KeyValue, error) {
	values := strings.Split(line, ":")
	if len(values) < 2 {
		return nil, fmt.Errorf("%v has less than 2 values", line)
	}

	return &badgerutils.KeyValue{
		Key:   []byte(kv[0]),
		Value: []byte(kv[1]),
	}, nil
}

func main() {
	dir := flag.String("dir", "", "Directory to save DB files")
	batchSize := flag.Int("batch-size", 1000, "Number of records to write per transaction")
	flag.Parse()

	if *dir == "" {
		log.Fatal(errors.New("dir flag is required"))
	}

	log.Printf("Directory: %v", *dir)
	log.Printf("Batch Size: %v", *batchSize)

	if err := badgerutils.WriteStream(os.Stdin, *dir, *batchSize, csvToKeyValue); err != nil {
		log.Fatal(err)
	}
}

The code above can be called with the following flags:

• -dir - (required) The path to the directory to persist Badger files.
• -batch-size - (default: 1000) The size of each transaction (or batch of writes). This can be tuned for optimal performance depending on the machine.

For example:

$ for i in {1..10}; do echo "field${i}1,field${i}2,field${i}3,field${i}4"; done | go run main.go -dir=temp -batch-size=1
Directory: temp
Batch Size: 3
...
Records: 3
Records: 6
Records: 9
Records: 10
Inserted 10 records in 474.69µs

Development

Dependency Management

dep is required for dependency management.

$ make install

Format Code

Run this before opening pull requests to ensure code is properly formatted.

$ make fmt

Unit Tests

$ make test