mongo-tools: github.com/mongodb/mongo-tools/mongoimport Index | Files | Directories

package mongoimport

import "github.com/mongodb/mongo-tools/mongoimport"

Package mongoimport allows importing content from a JSON, CSV, or TSV into a MongoDB instance.

Index

Package Files

common.go csv.go json.go mongoimport.go options.go tsv.go typed_fields.go

Constants

const (
    CSV  = "csv"
    TSV  = "tsv"
    JSON = "json"
)

Input format types accepted by mongoimport.

Variables

var (
    // ErrNoOpeningBracket means that the input source did not contain any
    // opening brace - returned only if --jsonArray is passed in.
    ErrNoOpeningBracket = errors.New("bad JSON array format - found no " +
        "opening bracket '[' in input source")

    // ErrNoClosingBracket means that the input source did not contain any
    // closing brace - returned only if --jsonArray is passed in.
    ErrNoClosingBracket = errors.New("bad JSON array format - found no " +
        "closing bracket ']' in input source")
)
var (
    UTF8_BOM = []byte{0xEF, 0xBB, 0xBF}
)
var Usage = "" /* 205 byte string literal not displayed */

func ColumnNames Uses

func ColumnNames(fs []ColumnSpec) (s []string)

ColumnNames maps a ColumnSpec slice to their associated names

type CSVConverter Uses

type CSVConverter struct {
    // contains filtered or unexported fields
}

CSVConverter implements the Converter interface for CSV input.

func (CSVConverter) Convert Uses

func (c CSVConverter) Convert() (b bson.D, err error)

Convert implements the Converter interface for CSV input. It converts a CSVConverter struct to a BSON document.

func (CSVConverter) Print Uses

func (c CSVConverter) Print()

type CSVInputReader Uses

type CSVInputReader struct {
    // contains filtered or unexported fields
}

CSVInputReader implements the InputReader interface for CSV input types.

func NewCSVInputReader Uses

func NewCSVInputReader(colSpecs []ColumnSpec, in io.Reader, rejects io.Writer, numDecoders int, ignoreBlanks bool) *CSVInputReader

NewCSVInputReader returns a CSVInputReader configured to read data from the given io.Reader, extracting only the specified columns using exactly "numDecoders" goroutines.

func (*CSVInputReader) ReadAndValidateHeader Uses

func (r *CSVInputReader) ReadAndValidateHeader() (err error)

ReadAndValidateHeader reads the header from the underlying reader and validates the header fields. It sets err if the read/validation fails.

func (*CSVInputReader) ReadAndValidateTypedHeader Uses

func (r *CSVInputReader) ReadAndValidateTypedHeader(parseGrace ParseGrace) (err error)

ReadAndValidateHeader reads the header from the underlying reader and validates the header fields. It sets err if the read/validation fails.

func (*CSVInputReader) StreamDocument Uses

func (r *CSVInputReader) StreamDocument(ordered bool, readDocs chan bson.D) (retErr error)

StreamDocument takes a boolean indicating if the documents should be streamed in read order and a channel on which to stream the documents processed from the underlying reader. Returns a non-nil error if streaming fails.

type ColumnSpec Uses

type ColumnSpec struct {
    Name       string
    Parser     FieldParser
    ParseGrace ParseGrace
    TypeName   string
}

ColumnSpec keeps information for each 'column' of import.

func ParseAutoHeaders Uses

func ParseAutoHeaders(headers []string) (fs []ColumnSpec)

ParseAutoHeaders converts a list of header items to ColumnSpec objects, with automatic parsers.

func ParseTypedHeader Uses

func ParseTypedHeader(header string, parseGrace ParseGrace) (f ColumnSpec, err error)

ParseTypedHeader produces a ColumnSpec from a header item, extracting type information from the it. The parseGrace is passed along to the new ColumnSpec.

func ParseTypedHeaders Uses

func ParseTypedHeaders(headers []string, parseGrace ParseGrace) (fs []ColumnSpec, err error)

ParseTypedHeaders performs ParseTypedHeader on each item, returning an error if any single one fails.

type Converter Uses

type Converter interface {
    Convert() (document bson.D, err error)
}

Converter is an interface that adds the basic Convert method which returns a valid BSON document that has been converted by the underlying implementation. If conversion fails, err will be set.

type FieldAutoParser Uses

type FieldAutoParser struct{}

func (*FieldAutoParser) Parse Uses

func (ap *FieldAutoParser) Parse(in string) (interface{}, error)

type FieldBinaryParser Uses

type FieldBinaryParser struct {
    // contains filtered or unexported fields
}

func NewFieldBinaryParser Uses

func NewFieldBinaryParser(arg string) (*FieldBinaryParser, error)

func (*FieldBinaryParser) Parse Uses

func (bp *FieldBinaryParser) Parse(in string) (interface{}, error)

type FieldBooleanParser Uses

type FieldBooleanParser struct{}

func (*FieldBooleanParser) Parse Uses

func (bp *FieldBooleanParser) Parse(in string) (interface{}, error)

type FieldDateParser Uses

type FieldDateParser struct {
    // contains filtered or unexported fields
}

func (*FieldDateParser) Parse Uses

func (dp *FieldDateParser) Parse(in string) (interface{}, error)

type FieldDecimalParser Uses

type FieldDecimalParser struct{}

func (*FieldDecimalParser) Parse Uses

func (ip *FieldDecimalParser) Parse(in string) (interface{}, error)

type FieldDoubleParser Uses

type FieldDoubleParser struct{}

func (*FieldDoubleParser) Parse Uses

func (dp *FieldDoubleParser) Parse(in string) (interface{}, error)

type FieldInt32Parser Uses

type FieldInt32Parser struct{}

func (*FieldInt32Parser) Parse Uses

func (ip *FieldInt32Parser) Parse(in string) (interface{}, error)

type FieldInt64Parser Uses

type FieldInt64Parser struct{}

func (*FieldInt64Parser) Parse Uses

func (ip *FieldInt64Parser) Parse(in string) (interface{}, error)

type FieldParser Uses

type FieldParser interface {
    Parse(in string) (interface{}, error)
}

FieldParser is the interface for any parser of a field item.

func NewFieldParser Uses

func NewFieldParser(t columnType, arg string) (parser FieldParser, err error)

NewFieldParser yields a FieldParser corresponding to the given columnType. arg is passed along to the specific type's parser, if it permits an argument. An error will be raised if arg is not valid for the type's parser.

type FieldStringParser Uses

type FieldStringParser struct{}

func (*FieldStringParser) Parse Uses

func (sp *FieldStringParser) Parse(in string) (interface{}, error)

type IngestOptions Uses

type IngestOptions struct {
    // Drops target collection before importing.
    Drop bool `long:"drop" description:"drop collection before inserting documents"`

    // Ignores fields with empty values in CSV and TSV imports.
    IgnoreBlanks bool `long:"ignoreBlanks" description:"ignore fields with empty values in CSV and TSV"`

    // Indicates that documents will be inserted in the order of their appearance in the input source.
    MaintainInsertionOrder bool `long:"maintainInsertionOrder" description:"insert the documents in the order of their appearance in the input source. By default the insertions will be performed in an arbitrary order. Setting this flag also enables the behavior of --stopOnError and restricts NumInsertionWorkers to 1."`

    // Sets the number of insertion routines to use
    NumInsertionWorkers int `short:"j" value-name:"<number>" long:"numInsertionWorkers" description:"number of insert operations to run concurrently" default:"1" default-mask:"-"`

    // Forces mongoimport to halt the import operation at the first insert or upsert error.
    StopOnError bool `long:"stopOnError" description:"halt after encountering any error during importing. By default, mongoimport will attempt to continue through document validation and DuplicateKey errors, but with this option enabled, the tool will stop instead. A small number of documents may be inserted after encountering an error even with this option enabled; use --maintainInsertionOrder to halt immediately after an error"`

    // Modify the import process.
    // Always insert the documents if they are new (do NOT match --upsertFields).
    // For existing documents (match --upsertFields) in the database:
    // "insert": Insert only, skip existing documents.
    // "upsert": Insert new documents or replace existing ones.
    // "merge": Insert new documents or modify existing ones; Preserve values in the database that are not overwritten.
    Mode string `long:"mode" choice:"insert" choice:"upsert" choice:"merge" description:"insert: insert only. upsert: insert or replace existing documents. merge: insert or modify existing documents. (default: insert)"`

    Upsert bool `long:"upsert" hidden:"true" description:"(deprecated; same as --mode=upsert) insert or update objects that already exist"`

    // Specifies a list of fields for the query portion of the upsert; defaults to _id field.
    UpsertFields string `long:"upsertFields" value-name:"<field>[,<field>]*" description:"comma-separated fields for the query part when --mode is set to upsert or merge"`

    // Sets write concern level for write operations.
    // By default mongoimport uses a write concern of 'majority'.
    // Cannot be used simultaneously with write concern options in a URI.
    WriteConcern string `long:"writeConcern" value-name:"<write-concern-specifier>" default-mask:"-" description:"write concern options e.g. --writeConcern majority, --writeConcern '{w: 3, wtimeout: 500, fsync: true, j: true}'"`

    // Indicates that the server should bypass document validation on import.
    BypassDocumentValidation bool `long:"bypassDocumentValidation" description:"bypass document validation"`

    // Specifies the number of threads to use in processing data read from the input source
    NumDecodingWorkers int `long:"numDecodingWorkers" default:"0" hidden:"true"`

    BulkBufferSize int `long:"batchSize" default:"1000" hidden:"true"`
}

IngestOptions defines the set of options for storing data.

func (*IngestOptions) Name Uses

func (_ *IngestOptions) Name() string

Name returns a description of the IngestOptions struct.

type InputOptions Uses

type InputOptions struct {
    // Fields is an option to directly specify comma-separated fields to import to CSV.
    Fields *string `long:"fields" value-name:"<field>[,<field>]*" short:"f" description:"comma separated list of fields, e.g. -f name,age"`

    // FieldFile is a filename that refers to a list of fields to import, 1 per line.
    FieldFile *string `long:"fieldFile" value-name:"<filename>" description:"file with field names - 1 per line"`

    // Specifies the location and name of a file containing the data to import.
    File string `long:"file" value-name:"<filename>" description:"file to import from; if not specified, stdin is used"`

    // Treats the input source's first line as field list (csv and tsv only).
    HeaderLine bool `long:"headerline" description:"use first line in input source as the field list (CSV and TSV only)"`

    // Indicates that the underlying input source contains a single JSON array with the documents to import.
    JSONArray bool `long:"jsonArray" description:"treat input source as a JSON array"`

    // Indicates how to handle type coercion failures
    ParseGrace string `long:"parseGrace" value-name:"<grace>" default:"stop" description:"controls behavior when type coercion fails - one of: autoCast, skipField, skipRow, stop"`

    // Specifies the file type to import. The default format is JSON, but it’s possible to import CSV and TSV files.
    Type string `long:"type" value-name:"<type>" default:"json" default-mask:"-" description:"input format to import: json, csv, or tsv"`

    // Indicates that field names include type descriptions
    ColumnsHaveTypes bool `long:"columnsHaveTypes" description:"indicated that the field list (from --fields, --fieldsFile, or --headerline) specifies types; They must be in the form of '<colName>.<type>(<arg>)'. The type can be one of: auto, binary, boolean, date, date_go, date_ms, date_oracle, double, int32, int64, string. For each of the date types, the argument is a datetime layout string. For the binary type, the argument can be one of: base32, base64, hex. All other types take an empty argument. Only valid for CSV and TSV imports. e.g. zipcode.string(), thumbnail.binary(base64)"`

    // Indicates that the legacy extended JSON format should be used to parse JSON documents. Defaults to false.
    Legacy bool `long:"legacy" default:"false" description:"use the legacy extended JSON format"`
}

InputOptions defines the set of options for reading input data.

func (*InputOptions) Name Uses

func (_ *InputOptions) Name() string

Name returns a description of the InputOptions struct.

type InputReader Uses

type InputReader interface {
    // StreamDocument takes a boolean indicating if the documents should be streamed
    // in read order and a channel on which to stream the documents processed from
    // the underlying reader.  Returns a non-nil error if encountered.
    StreamDocument(ordered bool, read chan bson.D) error

    // ReadAndValidateHeader reads the header line from the InputReader and returns
    // a non-nil error if the fields from the header line are invalid; returns
    // nil otherwise. No-op for JSON input readers.
    ReadAndValidateHeader() error

    // ReadAndValidateTypedHeader is the same as ReadAndValidateHeader,
    // except it also parses types from the fields of the header. Parse errors
    // will be handled according parseGrace.
    ReadAndValidateTypedHeader(parseGrace ParseGrace) error
    // contains filtered or unexported methods
}

type JSONConverter Uses

type JSONConverter struct {
    // contains filtered or unexported fields
}

JSONConverter implements the Converter interface for JSON input.

func (JSONConverter) Convert Uses

func (c JSONConverter) Convert() (bson.D, error)

Convert implements the Converter interface for JSON input. It converts a JSONConverter struct to a BSON document.

type JSONInputReader Uses

type JSONInputReader struct {
    // contains filtered or unexported fields
}

JSONInputReader is an implementation of InputReader that reads documents in JSON format.

func NewJSONInputReader Uses

func NewJSONInputReader(isArray bool, legacyExtJSON bool, in io.Reader, numDecoders int) *JSONInputReader

NewJSONInputReader creates a new JSONInputReader in array mode if specified, configured to read data to the given io.Reader.

func (*JSONInputReader) ReadAndValidateHeader Uses

func (r *JSONInputReader) ReadAndValidateHeader() error

ReadAndValidateHeader is a no-op for JSON imports; always returns nil.

func (*JSONInputReader) ReadAndValidateTypedHeader Uses

func (r *JSONInputReader) ReadAndValidateTypedHeader(parseGrace ParseGrace) error

ReadAndValidateTypedHeader is a no-op for JSON imports; always returns nil.

func (*JSONInputReader) StreamDocument Uses

func (r *JSONInputReader) StreamDocument(ordered bool, readChan chan bson.D) (retErr error)

StreamDocument takes a boolean indicating if the documents should be streamed in read order and a channel on which to stream the documents processed from the underlying reader. Returns a non-nil error if encountered

type MongoImport Uses

type MongoImport struct {

    // generic mongo tool options
    ToolOptions *options.ToolOptions

    // InputOptions defines options used to read data to be ingested
    InputOptions *InputOptions

    // IngestOptions defines options used to ingest data into MongoDB
    IngestOptions *IngestOptions

    // SessionProvider is used for connecting to the database
    SessionProvider *db.SessionProvider

    // the tomb is used to synchronize ingestion goroutines and causes
    // other sibling goroutines to terminate immediately if one errors out
    tomb.Tomb
    // contains filtered or unexported fields
}

MongoImport is a container for the user-specified options and internal state used for running mongoimport.

func New Uses

func New(opts Options) (*MongoImport, error)

New constructs a new MongoImport instance from the provided options. This will fail if the options are invalid or if it cannot establish a new connection to the server.

func (*MongoImport) Close Uses

func (imp *MongoImport) Close()

Close disconnects the server.

func (*MongoImport) ImportDocuments Uses

func (imp *MongoImport) ImportDocuments() (uint64, uint64, error)

ImportDocuments is used to write input data to the database. It returns the number of documents successfully imported to the appropriate namespace and any error encountered in doing this

type Options Uses

type Options struct {
    *options.ToolOptions
    *InputOptions
    *IngestOptions
    ParsedArgs []string
}

Options contains all the possible options that can be used to configure mongoimport.

func ParseOptions Uses

func ParseOptions(rawArgs []string, versionStr, gitCommit string) (Options, error)

ParseOptions reads command line arguments and converts them into options used to configure mongoimport.

type ParseGrace Uses

type ParseGrace int

func ParsePG Uses

func ParsePG(pg string) (res ParseGrace)

ParsePG interprets the user-provided parseGrace, assuming it is valid.

func ValidatePG Uses

func ValidatePG(pg string) (ParseGrace, error)

ValidatePG ensures the user-provided parseGrace is one of the allowed values.

type TSVConverter Uses

type TSVConverter struct {
    // contains filtered or unexported fields
}

TSVConverter implements the Converter interface for TSV input.

func (TSVConverter) Convert Uses

func (c TSVConverter) Convert() (b bson.D, err error)

Convert implements the Converter interface for TSV input. It converts a TSVConverter struct to a BSON document.

func (TSVConverter) Print Uses

func (c TSVConverter) Print()

type TSVInputReader Uses

type TSVInputReader struct {
    // contains filtered or unexported fields
}

TSVInputReader is a struct that implements the InputReader interface for a TSV input source.

func NewTSVInputReader Uses

func NewTSVInputReader(colSpecs []ColumnSpec, in io.Reader, rejects io.Writer, numDecoders int, ignoreBlanks bool) *TSVInputReader

NewTSVInputReader returns a TSVInputReader configured to read input from the given io.Reader, extracting the specified columns only.

func (*TSVInputReader) ReadAndValidateHeader Uses

func (r *TSVInputReader) ReadAndValidateHeader() (err error)

ReadAndValidateHeader reads the header from the underlying reader and validates the header fields. It sets err if the read/validation fails.

func (*TSVInputReader) ReadAndValidateTypedHeader Uses

func (r *TSVInputReader) ReadAndValidateTypedHeader(parseGrace ParseGrace) (err error)

ReadAndValidateTypedHeader reads the header from the underlying reader and validates the header fields. It sets err if the read/validation fails.

func (*TSVInputReader) StreamDocument Uses

func (r *TSVInputReader) StreamDocument(ordered bool, readDocs chan bson.D) (retErr error)

StreamDocument takes a boolean indicating if the documents should be streamed in read order and a channel on which to stream the documents processed from the underlying reader. Returns a non-nil error if streaming fails.

Directories

PathSynopsis
csvPackage csv reads and writes comma-separated values (CSV) files.
dateconv
mainMain package for the mongoimport tool.

Package mongoimport imports 32 packages (graph) and is imported by 12 packages. Updated 2019-08-24. Refresh now. Tools for package owners.