Documentation
¶
Index ¶
- Constants
- func ConvertToStruct(record, recordOutput interface{}) error
- type ContentReader
- func MergeReaders(arr []*ContentReader, arrayKey string) (contentReader *ContentReader, err error)
- func MergeSortedReaders(readerRecord SortableContentItem, sortedReaders []*ContentReader, ...) (contentReader *ContentReader, err error)
- func NewContentReader(filePath string, arrayKey string) *ContentReader
- func NewEmptyContentReader(arrayKey string) *ContentReader
- func NewMultiSourceContentReader(filePaths []string, arrayKey string) *ContentReader
- func SortAndSaveBufferToFile(keysToContentItems map[string]SortableContentItem, allKeys []string, ...) (contentReader *ContentReader, err error)
- func SortContentReader(readerRecord SortableContentItem, reader *ContentReader, ascendingOrder bool) (*ContentReader, error)
- func SortContentReaderByCalculatedKey(reader *ContentReader, getKeyFunc keyCalculationFunc, ascendingOrder bool) (contentReader *ContentReader, err error)
- func (cr *ContentReader) Close() error
- func (cr *ContentReader) GetError() error
- func (cr *ContentReader) GetFilesPaths() []string
- func (cr *ContentReader) IsEmpty() bool
- func (cr *ContentReader) Length() (int, error)
- func (cr *ContentReader) NextRecord(recordOutput interface{}) error
- func (cr *ContentReader) Reset()
- type ContentWriter
- func (rw *ContentWriter) Close() error
- func (rw *ContentWriter) GetArrayKey() string
- func (rw *ContentWriter) GetError() error
- func (rw *ContentWriter) GetFilePath() string
- func (rw *ContentWriter) IsEmpty() bool
- func (rw *ContentWriter) RemoveOutputFilePath() error
- func (rw *ContentWriter) SetArrayKey(arrKey string) *ContentWriter
- func (rw *ContentWriter) Write(record interface{})
- type SortRecord
- type SortableContentItem
Constants ¶
const (
DefaultKey = "results"
)
Variables ¶
This section is empty.
Functions ¶
func ConvertToStruct ¶ added in v0.21.0
func ConvertToStruct(record, recordOutput interface{}) error
Types ¶
type ContentReader ¶
type ContentReader struct {
// contains filtered or unexported fields
}
Open and read JSON files, find the array key inside it and load its value into the memory in small chunks. Currently, 'ContentReader' only support extracting a single value for a given key (arrayKey), other keys are ignored. The value must be of type array. Each array value can be fetched using 'NextRecord' (thread-safe). This technique solves the limit of memory size which may be too small to fit large JSON.
func MergeReaders ¶ added in v0.13.0
func MergeReaders(arr []*ContentReader, arrayKey string) (contentReader *ContentReader, err error)
func MergeSortedReaders ¶ added in v0.17.0
func MergeSortedReaders(readerRecord SortableContentItem, sortedReaders []*ContentReader, ascendingOrder bool) (contentReader *ContentReader, err error)
Merge a slice of sorted content-readers into a single sorted content-reader.
func NewContentReader ¶
func NewContentReader(filePath string, arrayKey string) *ContentReader
func NewEmptyContentReader ¶ added in v0.13.0
func NewEmptyContentReader(arrayKey string) *ContentReader
func NewMultiSourceContentReader ¶ added in v0.20.1
func NewMultiSourceContentReader(filePaths []string, arrayKey string) *ContentReader
func SortAndSaveBufferToFile ¶ added in v0.17.0
func SortAndSaveBufferToFile(keysToContentItems map[string]SortableContentItem, allKeys []string, increasingOrder bool) (contentReader *ContentReader, err error)
func SortContentReader ¶ added in v0.17.0
func SortContentReader(readerRecord SortableContentItem, reader *ContentReader, ascendingOrder bool) (*ContentReader, error)
Sort a content-reader in the required order (ascending or descending). Performs a merge-sort on the reader, splitting the reader to multiple readers of size 'utils.MaxBufferSize'. Sort each of the split readers, and merge them into a single sorted reader. In case of multiple items with the same key - all the items will appear in the sorted reader, but their order is not guaranteed to be preserved.
func SortContentReaderByCalculatedKey ¶ added in v0.21.0
func SortContentReaderByCalculatedKey(reader *ContentReader, getKeyFunc keyCalculationFunc, ascendingOrder bool) (contentReader *ContentReader, err error)
Sort a ContentReader, according to a key generated by getKeyFunc. getKeyFunc gets an item from the reader and returns the key of the item. Attention! In case of multiple items with the same key - only the first item in the original reader will appear in the sorted one! The other items will be removed. Also pay attention that the order of the fields inside the objects might change.
func (*ContentReader) GetError ¶
func (cr *ContentReader) GetError() error
func (*ContentReader) GetFilesPaths ¶ added in v0.20.1
func (cr *ContentReader) GetFilesPaths() []string
func (*ContentReader) IsEmpty ¶
func (cr *ContentReader) IsEmpty() bool
func (*ContentReader) Length ¶ added in v0.13.0
func (cr *ContentReader) Length() (int, error)
Number of element in the array.
func (*ContentReader) NextRecord ¶
func (cr *ContentReader) NextRecord(recordOutput interface{}) error
Each call to 'NextRecord()' will return a single element from the channel. Only the first call invokes a goroutine to read data from the file and push it into the channel. 'io.EOF' will be returned if no data is left.
func (*ContentReader) Reset ¶
func (cr *ContentReader) Reset()
Prepare the reader to read the file all over again (not thread-safe).
type ContentWriter ¶
type ContentWriter struct {
// contains filtered or unexported fields
}
Write a JSON file in small chunks. Only a single JSON key can be written to the file, and array as its value. The array's values could be any JSON value types (number, string, etc...). Once the first 'Write" call is made, the file will stay open, waiting for the next struct to be written (thread-safe). Finally, 'Close' will fill the end of the JSON file and the operation will be completed.
func NewContentWriter ¶
func NewContentWriter(arrayKey string, isCompleteFile, useStdout bool) (*ContentWriter, error)
func (*ContentWriter) GetArrayKey ¶ added in v0.13.0
func (rw *ContentWriter) GetArrayKey() string
func (*ContentWriter) GetError ¶
func (rw *ContentWriter) GetError() error
func (*ContentWriter) GetFilePath ¶
func (rw *ContentWriter) GetFilePath() string
func (*ContentWriter) IsEmpty ¶ added in v0.13.0
func (rw *ContentWriter) IsEmpty() bool
func (*ContentWriter) RemoveOutputFilePath ¶
func (rw *ContentWriter) RemoveOutputFilePath() error
func (*ContentWriter) SetArrayKey ¶
func (rw *ContentWriter) SetArrayKey(arrKey string) *ContentWriter
func (*ContentWriter) Write ¶
func (rw *ContentWriter) Write(record interface{})
Write a single item to the JSON array.
type SortRecord ¶ added in v0.21.0
type SortRecord struct {
Key string `json:"key,omitempty"`
Record interface{} `json:"record,omitempty"`
}
func (SortRecord) GetSortKey ¶ added in v0.21.0
func (sr SortRecord) GetSortKey() string
type SortableContentItem ¶ added in v0.17.0
type SortableContentItem interface {
GetSortKey() string
}
Source Files