csv

package
v0.0.0-...-d6cb97c Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 14, 2018 License: Apache-2.0 Imports: 11 Imported by: 0

README

CSV Importer

Imports a CSV file as List<T> where T is a struct with fields corresponding to the CSV's column headers. The struct spec can also be set manually with the -header flag.

Usage

$ cd csv-import
$ go build
$ ./csv-import <PATH> http://localhost:8000::foo

Some places for CSV files

CSV Exporter

Export a dataset in CSV format to stdout with column headers.

Usage

$ cd csv-export
$ go build
$ ./csv-export http://localhost:8000:foo

Documentation

Index

Constants

This section is empty.

Variables

View Source
var StringToKind = func(kindMap map[types.NomsKind]string) map[string]types.NomsKind {
	m := map[string]types.NomsKind{}
	for k, v := range kindMap {
		m[v] = k
	}
	return m
}(types.KindToString)

StringToKind maps names of valid NomsKinds (e.g. Bool, Number, etc) to their associated types.NomsKind

Functions

func EscapeStructFieldFromCSV

func EscapeStructFieldFromCSV(input string) string

EscapeStructFieldFromCSV removes special characters and replaces spaces with camelCasing (camel case turns to camelCase)

func FindPrimaryKeys

func FindPrimaryKeys(r *csv.Reader, numSamples, maxLenPrimaryKeyList, numFields int) [][]int

FindPrimaryKeys reads numSamples from r, using the first numFields and returns slices of []int indices that are primary keys for those samples

func GetFieldNamesFromIndices

func GetFieldNamesFromIndices(headers []string, indices []int) []string

func GetListElemDesc

func GetListElemDesc(l types.List, vr types.ValueReader) types.StructDesc

GetListElemDesc ensures that l is a types.List of structs, pulls the types.StructDesc that describes the elements of l out of vr, and returns the StructDesc.

func GetMapElemDesc

func GetMapElemDesc(m types.Map, vr types.ValueReader) types.StructDesc

GetMapElemDesc ensures that m is a types.Map of structs, pulls the types.StructDesc that describes the elements of m out of vr, and returns the StructDesc. If m is a nested types.Map of types.Map, then GetMapElemDesc will descend the levels of the enclosed types.Maps to get to a types.Struct

func KindsToStrings

func KindsToStrings(kinds KindSlice) []string

KindsToStrings looks up each element of kinds in the types.KindToString map and returns a slice of answers

func MakeStructTemplateFromHeaders

func MakeStructTemplateFromHeaders(headers []string, structName string, kinds KindSlice) (temp types.StructTemplate, fieldOrder []int, kindMap []types.NomsKind)

MakeStructTemplateFromHeaders creates a struct type from the headers using |kinds| as the type of each field. If |kinds| is empty, default to strings.

func NewCSVReader

func NewCSVReader(res io.Reader, comma rune) *csv.Reader

NewCSVReader returns a new csv.Reader that splits on comma

func ReadToColumnar

func ReadToColumnar(r *csv.Reader, structName string, headers []string, kinds KindSlice, vrw types.ValueReadWriter, limit uint64) (s types.Struct)

ReadToColumnar takes a CSV reader and reads data from each column into a separate list. Values from columns in each successive row are appended to the column-specific lists whose type is described by headers. Finally, a new "Columnar" struct is created that consists of one field for each column and each field contains a list of values. If the original data contained headers it is expected that the input reader has already read those and are pointing at the first data row. If kinds is non-empty, it will be used to type the fields in the generated structs; otherwise, they will be left as string-fields. In addition to the list, ReadToList returns the typeDef of the structs in the list.

func ReadToList

func ReadToList(r *csv.Reader, structName string, headers []string, kinds KindSlice, vrw types.ValueReadWriter, limit uint64) (l types.List)

ReadToList takes a CSV reader and reads data into a typed List of structs. Each row gets read into a struct named structName, described by headers. If the original data contained headers it is expected that the input reader has already read those and are pointing at the first data row. If kinds is non-empty, it will be used to type the fields in the generated structs; otherwise, they will be left as string-fields. In addition to the list, ReadToList returns the typeDef of the structs in the list.

func ReadToMap

func ReadToMap(r *csv.Reader, structName string, headersRaw []string, primaryKeys []string, kinds KindSlice, vrw types.ValueReadWriter, limit uint64) types.Map

ReadToMap takes a CSV reader and reads data into a typed Map of structs. Each row gets read into a struct named structName, described by headers. If the original data contained headers it is expected that the input reader has already read those and are pointing at the first data row. If kinds is non-empty, it will be used to type the fields in the generated structs; otherwise, they will be left as string-fields.

func SkipRecords

func SkipRecords(r *csv.Reader, n uint) error

func StringToRune

func StringToRune(delimiter string) (rune, error)

StringToRune returns the rune contained in delimiter or an error.

func StringToValue

func StringToValue(s string, k types.NomsKind) (types.Value, error)

StringToValue takes a piece of data as a string and attempts to convert it to a types.Value of the appropriate types.NomsKind.

func WriteList

func WriteList(l types.List, sd types.StructDesc, comma rune, output io.Writer)

WriteList takes a types.List l of structs (described by sd) and writes it to output as comma-delineated values.

func WriteMap

func WriteMap(m types.Map, sd types.StructDesc, comma rune, output io.Writer)

WriteMap takes a types.Map m of structs (described by sd) and writes it to output as comma-delineated values.

Types

type KindSlice

type KindSlice []types.NomsKind

KindSlice is an alias for []types.NomsKind. It's needed because types.NomsKind are really just 8 bit unsigned ints, which are what Go uses to represent 'byte', and this confuses the Go JSON marshal/unmarshal code -- it treats them as byte arrays and base64 encodes them!

func GetSchema

func GetSchema(r *csv.Reader, numSamples int, numFields int) KindSlice

func StringsToKinds

func StringsToKinds(strs []string) KindSlice

StringsToKinds looks up each element of strs in the StringToKind map and returns a slice of answers

func (KindSlice) MarshalJSON

func (ks KindSlice) MarshalJSON() ([]byte, error)

func (*KindSlice) UnmarshalJSON

func (ks *KindSlice) UnmarshalJSON(value []byte) error

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL