Documentation
¶
Overview ¶
Package bq is responsible for all interactions with BigQuery.
Note that it uses the low level google.golang.org/api/bigquery/v2 for reasons that may longer be valid, but at the time this was written certain functionality was not supported by the higher level cloud.google.com/go/bigquery, specifically it had to do with how to create a client without a GOOGLE_APPLICATION_CREDENTIALS environment variable, see newBqApiClient().
Index ¶
- func ParseTableSpec(spec string) (proj, ds, table string)
- type BigQuery
- func (b *BigQuery) GetJob(jobId string) (*bigquery.Job, error)
- func (b *BigQuery) GetTable(dataset, table string) (*bigquery.Table, error)
- func (b *BigQuery) NewExtractJobConfiguration(dataset, table string) *bigquery.JobConfiguration
- func (b *BigQuery) NewLoadJobConfiguration(table, dataset, wdisp string, gsUrls []string, format string) *bigquery.JobConfiguration
- func (b *BigQuery) NewQueryJobConfiguration(sql, dataset, table, disp string, legacy, partitioned bool) *bigquery.JobConfiguration
- func (b *BigQuery) ProjectId() string
- func (b *BigQuery) SignedStorageUrl(filename, method string) (string, error)
- func (b *BigQuery) StartJob(conf *bigquery.JobConfiguration) (*bigquery.Job, error)
- func (b *BigQuery) TableColumnNames(dataset, table string) ([]string, error)
- func (b *BigQuery) TableData(dataset, table string, f func([]string) error) error
- type Config
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ParseTableSpec ¶
Parse a qualified BigQuery table name, such as "[orgname:project.table]" (Legacy) or `orgname.project.table` (Standard). Returns three strings: project, dataset and table. (Blank if not specified).
Types ¶
type BigQuery ¶
type BigQuery struct {
// contains filtered or unexported fields
}
BigQuery is a struct containing the information necessary to communicate with the BigQuery API.
func NewBigQuery ¶
NewBigQuery returns a BigQuery instance given a Config pointer.
func (*BigQuery) GetTable ¶
Retrieve BigQuery table information form the BigQuery API given a dataset and a table.
func (*BigQuery) NewExtractJobConfiguration ¶
Create an extract JobConfiguration to be able to start an extract job.
func (*BigQuery) NewLoadJobConfiguration ¶
func (*BigQuery) NewQueryJobConfiguration ¶
func (b *BigQuery) NewQueryJobConfiguration(sql, dataset, table, disp string, legacy, partitioned bool) *bigquery.JobConfiguration
Create a query JobConfiguration given the SQL, the destination dataset and table, disposition ("WRITE_APPEND" or "WRITE_TRUNCATE), whether legacy SQL is used and whether the resulting table is DAY partitioned."
func (*BigQuery) SignedStorageUrl ¶
Construct and return a cryptographically signed URL to a GCS file. This URL can be used to retrieve the file without any authentication. The signature expires after duration provided to NewBigQuery().
func (*BigQuery) TableColumnNames ¶
Get table column names as a []string
func (*BigQuery) TableData ¶
Given a function which accepts a []string, call it repeatdly for every row of data. This is meant for tables that do not have repeated fields, but if a repeated field is encountered, it will be simply marshalled as JSON. This uses the BigQuery paging API which is slow and only suitable for small tables.
type Config ¶
type Config struct {
ProjectId string // ProjectId
Email string // Email (for authentication)
Key string // Key (for authentication)
GcsBucket string // Bucket used for GCS exports
GcsExpiration time.Duration // Expire GCS URL after (defaults to 4 hours)
}
Configration parameters to instantiate a BigQuery connection.
Source Files