Documentation
¶
Overview ¶
Package simpleio contains core utils that reading in data optimized for both memory allocation, speed, and performance
Index ¶
- Constants
- func BytesToBuffer(reader *SimpleReader) *bytes.Buffer
- func BytesToInt(b []byte) int
- func CatchErrThrowEOF(err error)
- func Float32ToString(num float32) string
- func Float64ToString(num float64) string
- func GetBuffer(bufferPool sync.Pool) bytes.Buffer
- func Int16ToString(num int16) string
- func IntSliceToString(nums []int) string
- func IntToString(i int) string
- func InvMap(num int) string
- func LookUpRoman(s string) int
- func PrintString(buf *bytes.Buffer)
- func PutBuffer(buf bytes.Buffer, bufferPool sync.Pool)
- func ReadBgzipFile(filename string) *bytes.Buffer
- func ReadFromFile(filename string) []string
- func ReadLine(reader *SimpleReader) (*bytes.Buffer, bool)
- func ReadLineBgzip(reader *BgzipReader) (*bytes.Buffer, bool)
- func Rm(filename string)
- func ScientificNotation(s string) float64
- func SimplyRun(f func())
- func StdError(err error)
- func StringToFloat(s string) float32
- func StringToFloat64(s string) float64
- func StringToInt(s string) int
- func StringToIntSlice(column string) []int
- func StringToUInt16(s string) uint16
- func StringToUInt32(s string) uint32
- func ToNumber(n string) int
- func ToRoman(n int) string
- func Touch(filename string) *os.File
- func Vim(filename string) *os.File
- func VimUrl(url string)
- func WriteLine(writer *SimpleWriter, s string)
- func WriteToFile(filename string, data []string)
- type BgzipReader
- type BufferPool
- type GunzipReader
- type OrderedMap
- type SimpleReader
- type SimpleWriter
- type SizedBufferPool
Constants ¶
const (
BufferSize = mb
)
Variables ¶
This section is empty.
Functions ¶
func BytesToBuffer ¶
func BytesToBuffer(reader *SimpleReader) *bytes.Buffer
BytesToBuffer will parse []byte and return a pointer to the same underlying bytes.Buffer
func BytesToInt ¶
BytesToInt a function that converts a byte slice and return a number, type int.
func CatchErrThrowEOF ¶
func CatchErrThrowEOF(err error)
CatchErrThrowEOF will silently handles and throws the EOF error and will log and exit any other errors.
func Float32ToString ¶
func Float64ToString ¶
func Int16ToString ¶
func IntSliceToString ¶
intListToString will process a slice of type int as an input and return a each value separated by a comma as a string.
func IntToString ¶
IntToString a function that converts a number of type int and return a string.
func LookUpRoman ¶
func PrintString ¶
func ReadBgzipFile ¶
func ReadFromFile ¶
func ReadLine ¶
func ReadLine(reader *SimpleReader) (*bytes.Buffer, bool)
ReadLine will return a bytes.Buffer pointing to the internal slice of bytes. Provided this function is called within a loop, the function will read one line at a time, and return bool to continue reading. Important to note the buffer return points to the internal slice belonging to the reader, meaning the slice will be overridden if the data is not copied. Please be aware the reader will call close on the file once the reader encounters EOF.
func ReadLineBgzip ¶
func ReadLineBgzip(reader *BgzipReader) (*bytes.Buffer, bool)
func ScientificNotation ¶
ScientificNotation will convert a string with scientific notation into a float64
func StringToFloat ¶
StringToFloat is a function that converts a string to a type float64.
func StringToFloat64 ¶
StringToFloat is a function that converts a string to a type float64.
func StringToInt ¶
StringToInt is a function that converts a string and return a number, type int.
func StringToIntSlice ¶
StringToInts will process strings (usually from column data) and return a slice of []int
func StringToUInt16 ¶
StringToUint16 parses a string into a uint16 and will exit on error
func StringToUInt32 ¶
StringToInt is a function that converts a string and return a number, type int.
func VimUrl ¶
func VimUrl(url string)
VimUrl is a basic function to procress a url link and print it out to stdout.
func WriteLine ¶
func WriteLine(writer *SimpleWriter, s string)
func WriteToFile ¶
Types ¶
type BgzipReader ¶
type BgzipReader struct {
*bufio.Reader
Buffer *bytes.Buffer
// contains filtered or unexported fields
}
func NewBgzipReader ¶
func NewBgzipReader(filename string) *BgzipReader
func (*BgzipReader) Close ¶
func (reader *BgzipReader) Close()
type BufferPool ¶
type BufferPool struct {
// contains filtered or unexported fields
}
BufferPool implements a pool of bytes.Buffers in the form of a bounded channel.
func NewBufferPool ¶
func NewBufferPool(size int) *BufferPool
NewBufferPool creates a new BufferPool bounded to the given size.
func (*BufferPool) Get ¶
func (bp *BufferPool) Get() *bytes.Buffer
Get gets a Buffer from the BufferPool, or creates a new one if none are available in the pool.
func (*BufferPool) Put ¶
func (bp *BufferPool) Put(b *bytes.Buffer)
Put returns the given Buffer to the BufferPool.
type GunzipReader ¶
GunzipReader uncompress the input using the system's gzip. Apparently, the system gzip is much much faster than the go library, so I wrote some benchmarks and tests
func NewGunzipReader ¶
func NewGunzipReader(filename string) (*GunzipReader, error)
type OrderedMap ¶
type OrderedMap struct {
// contains filtered or unexported fields
}
func NewOrderedMap ¶
func NewOrderedMap() *OrderedMap
func (*OrderedMap) Delete ¶
func (om *OrderedMap) Delete(key int)
func (*OrderedMap) Get ¶
func (om *OrderedMap) Get(key int) interface{}
func (*OrderedMap) Iter ¶
func (om *OrderedMap) Iter() []interface{}
func (*OrderedMap) IterKeys ¶
func (om *OrderedMap) IterKeys() []int
func (*OrderedMap) Len ¶
func (om *OrderedMap) Len() int
func (*OrderedMap) Set ¶
func (om *OrderedMap) Set(key int, val interface{})
type SimpleReader ¶
type SimpleReader struct {
*bufio.Reader
Gunzip *GunzipReader
Buffer *bytes.Buffer
// contains filtered or unexported fields
}
SimpleReader implements the io.Reader interface by providing the Read(b []byte) method. The struct contains an embedded *bufio.Reader and a pointer to os.File for closeure when reading is complete.
func HttpReader ¶
func HttpReader(url string) *SimpleReader
HttpReader will fetch data from files uploaded to an internet server and stream data into an io.Reader interface.
func NewReader ¶
func NewReader(filename string) *SimpleReader
NewSimpleReader will process a given file and performs error handling if an error occurs. SimpleReader will prcoess gzipped files accordinging by performing a check on the suffix of the provided file.
func (*SimpleReader) Close ¶
func (reader *SimpleReader) Close()
Close closes the File, rendering it unusable for I/O. On files that support SetDeadline, any pending I/O operations will be canceled and return immediately with an error. Close will return an error if it has already been called.
func (*SimpleReader) Read ¶
func (reader *SimpleReader) Read(b []byte) (n int, err error)
Read reads data into p and is a method required to implement the io.Reader interface. It returns the number of bytes read into p.
func (*SimpleReader) ToString ¶
func (reader *SimpleReader) ToString() string
type SimpleWriter ¶
type SimpleWriter struct {
*bufio.Writer
Gzip *gzip.Writer
Buffer *bytes.Buffer
// contains filtered or unexported fields
}
func NewWriter ¶
func NewWriter(filename string) *SimpleWriter
func (*SimpleWriter) Close ¶
func (writer *SimpleWriter) Close()
type SizedBufferPool ¶
type SizedBufferPool struct {
// contains filtered or unexported fields
}
func NewSizedBufferPool ¶
func NewSizedBufferPool(size int, alloc int) (bp *SizedBufferPool)
func (*SizedBufferPool) Get ¶
func (bp *SizedBufferPool) Get() (b *bytes.Buffer)
Get gets a Buffer from the SizedBufferPool, or creates a new one if none are available in the pool. Buffers have a pre-allocated capacity.
func (*SizedBufferPool) Put ¶
func (bp *SizedBufferPool) Put(b *bytes.Buffer)
Put returns the given Buffer to the SizedBufferPool.
Source Files