Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Formatter ¶ added in v0.3.0
type Formatter interface {
Source(project string, config config.Config, year int) string
Partitions(source string) string
Partition(row map[string]bigquery.Value) string
Marshal(rows []map[string]bigquery.Value) ([]byte, error)
}
Formatter is the interface for all types that format table sources and queries used during the export process.
type JSONExporter ¶
type JSONExporter struct {
// contains filtered or unexported fields
}
JSONExporter is a JSON exporter for histogram data on BigQuery.
func (*JSONExporter) Export ¶
func (exporter *JSONExporter) Export(ctx context.Context, config config.Config, queryTpl *template.Template, year int) error
Export runs the provided SQL query and, for each row in the result, uploads a file to the provided config.OutputPath on GCS. This file contains the JSON representation of the rows. config.OutputPath is a template whose parameters are provided by the BigQuery row's fields. If a field is present in the output path, it's removed from the BigQuery row's JSON representation, to reduce redundancy.
e.g. if outputPath is "{{ .year }}/output.json" and we have a row per year, the histograms will be uploaded to: - 2010/output.json - 2020/output.json - etc.
If any of the steps (running the query, reading the result, marshalling, uploading) fails, this function returns the corresponding error.
Note: config.OutputPath should not start with a "/".
type QueryJob ¶
type QueryJob struct {
// contains filtered or unexported fields
}
QueryJob is a job for running queries on BQ.
type UploadJob ¶
type UploadJob struct {
// contains filtered or unexported fields
}
UploadJob is a job for uploading data to a GCS bucket.
type UploadResult ¶
type UploadResult struct {
// contains filtered or unexported fields
}
UploadResult is the result of a GCS upload.
Source Files