lib

package

v0.0.0-...-c0cdd26 Latest Latest Go to latest Published: Mar 29, 2024 License: Apache-2.0 Imports: 11 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/datacommonsorg/tools

Links

Open Source Insights

Documentation ¶

Overview ¶

Package gcf runs a GCF function that triggers in 2 scenarios:

completion of prophet-flume job in borg. On triggering it sets up new cloud BT table, scales up BT cluster (if needed) and starts a dataflow job.
completion of BT cache ingestion dataflow job. It scales BT cluster down (if needed).

There are two set of trigger functions defined: - ProdBTImportController - PrivateBTImportController which targets on production imports and private imports. The folder structure are different for the two scenarios. The environment variables for deployments are stored in (prod|private)/*.yaml

Dataflow launcher helper functions. launchDataflowJob will either start a Dataflow job using a Flex Template or a Classic Template depending on the template path. Flex templates are json based Dataflow templates and MUST end with ".json".

Index ¶

Constants
func DeleteBTTable(ctx context.Context, projectID, instance, table string) error
func DoesObjectExist(ctx context.Context, path string) (bool, error)
func JoinURL(base string, paths ...string) string
func LaunchDataflowJob(ctx context.Context, projectID string, instance string, tableID string, ...) error
func ParsePath(path string) (string, string, error)
func SetupBT(ctx context.Context, btProjectID, btInstance, tableID string) error
func WriteToGCS(ctx context.Context, path, data string) error
type GCSEvent

Constants ¶

View Source

const (

	// NOTE: The following three files represents the state of a BT import. They
	// get written under:
	//
	//		<controlPath>/<TableID>/
	//
	// Init: written by borg to start BT import.
	InitFile = "init.txt"
	// Launched: written by this cloud function to mark launching of BT import job.
	LaunchedFile = "launched.txt"
	// Completed: written by dataflow to mark completion of BT import.
	CompletedFile = "completed.txt"
)

Variables ¶

This section is empty.

Functions ¶

func DeleteBTTable ¶

func DeleteBTTable(ctx context.Context, projectID, instance, table string) error

func DoesObjectExist ¶

func DoesObjectExist(ctx context.Context, path string) (bool, error)

func JoinURL ¶

func JoinURL(base string, paths ...string) string

joinURL joins url components. path.Join does work well for url, for example gs:// is changaed to gs:/

func LaunchDataflowJob ¶

func LaunchDataflowJob(
	ctx context.Context,
	projectID string,
	instance string,
	tableID string,
	dataPath string,
	controlPath string,
	dataflowTemplate string,
	appProfileID string,
	maxNumWorkers string,
) error

func ParsePath ¶

func ParsePath(path string) (string, string, error)

parsePath returns the GCS bucket and object from path in the form of gs://<bucket>/<object>

func SetupBT ¶

func SetupBT(ctx context.Context, btProjectID, btInstance, tableID string) error

func WriteToGCS ¶

func WriteToGCS(ctx context.Context, path, data string) error

Types ¶

type GCSEvent ¶

type GCSEvent struct {
	Name   string `json:"name"` // File name in the control folder
	Bucket string `json:"bucket"`
}

GCSEvent is the payload of a GCS event.

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL