ratchet: github.com/dailyburn/ratchet/util Index | Files

package util

import "github.com/dailyburn/ratchet/util"

Package util has various helper functions used by components of ratchet.


Package Files

csv.go csv_writer.go doc.go err.go s3.go sftp.go sql.go timer.go

func CSVProcess Uses

func CSVProcess(params *CSVParameters, d data.JSON, outputChan chan data.JSON, killChan chan error)

CSVProcess writes the contents to the file and optionally sends the written bytes upstream on outputChan

func CSVString Uses

func CSVString(v interface{}) string

CSVString returns an empty string for nil values to make sure that the text "null" is not written to a file

func DeleteS3Objects Uses

func DeleteS3Objects(client *s3.S3, bucket string, objKeys []string) (*s3.DeleteObjectsOutput, error)

DeleteS3Objects deletes the objects specified by the given object keys

func ExecuteSQLQuery Uses

func ExecuteSQLQuery(db *sql.DB, query string) error

ExecuteSQLQuery allows you to execute arbitrary SQL statements

func GetDataFromSQLQuery Uses

func GetDataFromSQLQuery(db *sql.DB, query string, batchSize int, structDest interface{}) (chan data.JSON, error)

GetDataFromSQLQuery is a util function that, given a properly intialized sql.DB and a valid SQL query, will handle executing the query and getting back data.JSON objects. This function is asynch, and data.JSON should be received on the return data channel. If there was a problem setting up the query, then an error will also be returned immediately. It is also possible for errors to occur during execution as data is retrieved from the query. If this happens, the object returned will be a JSON object in the form of {"Error": "description"}.

func GetS3Object Uses

func GetS3Object(client *s3.S3, bucket, objKey string) (*s3.GetObjectOutput, error)

GetS3Object returns the object output for the given object key

func KillPipelineIfErr Uses

func KillPipelineIfErr(err error, killChan chan error)

KillPipelineIfErr is an error-checking helper.

func ListS3Objects Uses

func ListS3Objects(client *s3.S3, bucket, keyPrefix string) ([]string, error)

ListS3Objects returns all object keys matching the given prefix. Note that delimiter is set to "/". See http://docs.aws.amazon.com/AmazonS3/latest/dev/ListingKeysHierarchy.html

func SQLInsertData Uses

func SQLInsertData(db *sql.DB, d data.JSON, tableName string, onDupKeyUpdate bool, onDupKeyFields []string, batchSize int) error

SQLInsertData abstracts building and executing a SQL INSERT statement for the given Data object.

Note that the Data must be a valid JSON object (or an array of valid objects all with the same keys), where the keys are column names and the the values are SQL values to be inserted into those columns.

func SftpClient Uses

func SftpClient(server string, username string, authMethod []ssh.AuthMethod, opts ...func(*sftp.Client) error) (*sftp.Client, error)

SftpClient sets up and return the client

func SftpKeyAuth Uses

func SftpKeyAuth(privateKeyPath string) (auth ssh.AuthMethod, err error)

SftpKeyAuth generates an ssh.AuthMethod given the path of a private key

func WriteS3Object Uses

func WriteS3Object(data []string, config *aws.Config, bucket string, key string, lineSeparator string, compress bool) (string, error)

WriteS3Object writes the data to the given key, optionally compressing it first

type CSVParameters Uses

type CSVParameters struct {
    Writer        *CSVWriter
    WriteHeader   bool
    HeaderWritten bool
    Header        []string
    SendUpstream  bool
    QuoteEscape   string
    Comma         rune

CSVParameters allows you to define all of your csv writing preferences in a single struct for reuse in multiple processors

type CSVWriter Uses

type CSVWriter struct {
    Comma   rune
    UseCRLF bool

    AlwaysEncapsulate bool   // If the content should be encapsulated independent of its type
    QuoteEscape       string // String to use to escape a quote character
    // contains filtered or unexported fields

CSVWriter reimplements the standard library csv.Writer adding AlwaysEncapsulate and QuoteEscape

func NewCSVWriter Uses

func NewCSVWriter() *CSVWriter

NewCSVWriter instantiates a new instance of CSVWriter

func (*CSVWriter) Error Uses

func (w *CSVWriter) Error() error

Error reports any error that has occurred during a previous Write or Flush.

func (*CSVWriter) Flush Uses

func (w *CSVWriter) Flush()

Flush writes any buffered data to the underlying io.Writer. To check if an error occurred during the Flush, call Error.

func (*CSVWriter) SetWriter Uses

func (w *CSVWriter) SetWriter(writer io.Writer)

SetWriter allows you to change the writer (which is not directly exposed)

func (*CSVWriter) Write Uses

func (w *CSVWriter) Write(record []string) (err error)

Write writes a single CSV record to w along with any necessary quoting. A record is a slice of strings with each string being one field.

func (*CSVWriter) WriteAll Uses

func (w *CSVWriter) WriteAll(records [][]string) (err error)

WriteAll writes multiple CSV records to w using Write and then calls Flush.

type SftpParameters Uses

type SftpParameters struct {
    Server      string
    Username    string
    Path        string
    AuthMethods []ssh.AuthMethod

SftpParameters is used for storing connection parameters for later executing sftp commands

type SftpPath Uses

type SftpPath struct {
    Path string `json:"path,omitempty"`

SftpPath is a simple struct for storing the full path of an object

func (SftpPath) FileName Uses

func (t SftpPath) FileName() string

FileName defers to filepath.Base

type Timer Uses

type Timer struct {
    // contains filtered or unexported fields

Timer is a basic mechanism for measuring execution time.

func StartTimer Uses

func StartTimer() (t *Timer)

StartTimer returns a new Timer that's already "started".

func (*Timer) Duration Uses

func (t *Timer) Duration() time.Duration

Duration returns either the total executino duration (if Timer stopped) or the duration until time.Now() if timer is still running.

func (*Timer) Stop Uses

func (t *Timer) Stop() *Timer

Stop sets the end time for the Timer and returns itself.

func (*Timer) Stopped Uses

func (t *Timer) Stopped() bool

Stopped returns true if Stop() has been called on the timer.

func (*Timer) String Uses

func (t *Timer) String() string

Package util imports 22 packages (graph) and is imported by 10 packages. Updated 2017-04-14. Refresh now. Tools for package owners.