optimus

package

v0.10.4 Latest Latest Go to latest Published: Apr 18, 2024 License: Apache-2.0 Imports: 13 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

source:
  type: optimus
  config:
    host: optimus.com:80

Key	Value	Example	Description
`host`	`string`	`optimus.com:80`	Optimus' GRPC host	required
`max_size_in_mb`	`int`	`45`	Max megabytes for GRPC client to receive message. Default to 45.

Field	Sample Value
`resource.urn`	`optimus::https://optimus-host.com/project.namespace.job`
`resource.name`	`job-name`
`resource.service`	`optimus`
`resource.description`	`Sample job description`
`ownership.owners[0].urn`	`john_doe@example.com`
`ownership.owners[0].name`	`john_doe@example.com`
`lineage.upstreams[].urn`	`bigquery::project/dataset/table`
`lineage.upstreams[].type`	`table`
`lineage.upstreams[].service`	`bigquery`
`lineage.downstreams[0].urn`	`bigquery::project/dataset/table`
`lineage.downstreams[0].type`	`table`
`lineage.downstreams[0].service`	`bigquery`
`properties.attributes`	`{}`

Refer to the contribution guidelines for information on contributing to this module.

This section is empty.

This section is empty.

This section is empty.

type Config struct {
	Host        string `mapstructure:"host" validate:"required"`
	MaxSizeInMB int    `mapstructure:"max_size_in_mb"`
}

Config holds the set of configuration for the bigquery extractor

type Extractor struct {
	plugins.BaseExtractor
	// contains filtered or unexported fields
}

Extractor manages the communication with the bigquery service

func New(l log.Logger, c client.Client) *Extractor

func (e *Extractor) Extract(ctx context.Context, emit plugins.Emit) error

Extract checks if the table is valid and extracts the table schema

func (e *Extractor) Init(ctx context.Context, config plugins.Config) error

Init initializes the extractor

Path	Synopsis
client