split_csv

Split csv

Fast and efficient Golang package for splitting large csv files on smaller chunks by size in bytes.

Features:

• Super-fast splitting. Splitting of 700MB+ file takes less than 1 sec!
• Allocates minimum memory regardless file size.
• Supports multiline cells and headers (csv should follow the basic rules https://en.wikipedia.org/wiki/Comma-separated_values).
• Configurable destination folder.
• Disabling/enabling of copying a header in chunk files.

Installation

Install:

go get -u github.com/tolik505/split-csv

Import:

import splitCsv "github.com/tolik505/split-csv"

Quickstart

func ExampleSplitCsv() {
	splitter := splitCsv.New()
	splitter.Separator = ";"     // "," is by default
	splitter.FileChunkSize = 100000000 //in bytes (100MB)
	result, _ := splitter.Split("testdata/test.csv", "testdata/")
	fmt.Println(result)
	// Output: [testdata/test_1.csv testdata/test_2.csv testdata/test_3.csv]
}

If copying of a header in chunks is not needed then:

func ExampleSplitCsv() {
	splitter := splitCsv.New()
	splitter.Separator = ";"     // "," is by default
	splitter.FileChunkSize = 20000000 //in bytes (20MB)
	s.WithHeader = false //copying of header in chunks is disabled
	result, _ := splitter.Split("testdata/test.csv", "testdata/")
	fmt.Println(result)
	// Output: [testdata/test_1.csv testdata/test_2.csv testdata/test_3.csv]
}

License