gcf

package module

v0.0.0-...-d2d3fdd Latest Latest Go to latest Published: Jan 21, 2023 License: Apache-2.0 Imports: 11 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/datacommonsorg/tools

README ¶

Bigtable Automation

Background

This directory contains the Google Cloud Function involved in BT cache generation. The Cloud Function first gets notified by a Borg pipeline, via init.txt, when the cache files have been uploaded to GCS. As a result, it creates a new BT table, scales-up the node-limit (for faster loads, only on base cache), kicks off the CsvImport Dataflow job and registers launched.txt.

After the dataflow job has run to completion, it notifies this Cloud Function again, via completed.txt. As a result, it scales-down the node-limit (only on base cache). In future, for branch cache, it will notify Mixer.

All of the above .txt files are created in a per-cache directory. So, they allow for concurrent cache builds and tracking past state.

There are two distince GCF, one used for production imports with entry point ProdBTImportController and the other used for private imports with entry point PrivateBTImportController. The two functions has the same workflow as described below, except the GCS folder structure are different. The production entry point can be tested locally through local/main.go

Validate BT Import End-to-end using (GCS | BT) Test Environment

First start the Cloud Function locally, as follows:
```
./local/deploy.sh
```
A test branch cache exists in this folder.

Just in case a test was run with that cache before, clean-up first:
```
./local/test.sh cleanup
```
Fake an init trigger from Google pipeline:
```
./local/test.sh init
```
To validate this step:
- prophet-test BT instance should have 3 nodes (instead of 1) and a new table dcbranch_2022_05_06_16_16_13 under Tables
- A dataflow job dcbranch_2022_05_06_16_16_13 runs for ~4-5 minutes here.
- A directory for the cache name within gs://automation_control_test/dcbranch_2022_05_06_16_16_13 containing a launched.txt file. When the dataflow job above ends, there should be a completed.txt file.
Fake a completion trigger from Dataflow job:
```
./local/test.sh completed
```
Validate this step by confirming that the prophet-test BT instance now has 1 node.

Deployment

After validating the change in test environment, deploy to PROD by running:

./prod/deploy.sh base
./prod/deploy.sh branch

When this completes, look at the prophet-cache-trigger on GCP console to version.

To deploy private GCF, identify the environment, pick the corresponding yaml files in private/*.yaml and run

./private/deploy.sh <env>

Documentation ¶

Overview ¶

Package gcf runs a GCF function that triggers in 2 scenarios:

completion of prophet-flume job in borg. On triggering it sets up new cloud BT table, scales up BT cluster (if needed) and starts a dataflow job.
completion of BT cache ingestion dataflow job. It scales BT cluster down (if needed).

There are two set of trigger functions defined: - ProdBTImportController - PrivateBTImportController which targets on production imports and private imports. The folder structure are different for the two scenarios. The environment variables for deployments are stored in (prod|private)/*.yaml

Dataflow launcher helper functions. launchDataflowJob will either start a Dataflow job using a Flex Template or a Classic Template depending on the template path. Flex templates are json based Dataflow templates and MUST end with ".json".

Index ¶

func CustomBTImportController(ctx context.Context, e GCSEvent) error
func ProdBTImportController(ctx context.Context, e GCSEvent) error
type GCSEvent

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

func CustomBTImportController ¶

func CustomBTImportController(ctx context.Context, e GCSEvent) error

CustomBTImportController consumes a GCS event and runs an import state machine.

func ProdBTImportController ¶

func ProdBTImportController(ctx context.Context, e GCSEvent) error

ProdBTImportController consumes a GCS event and runs an import state machine.

Types ¶

type GCSEvent ¶

type GCSEvent struct {
	Name   string `json:"name"` // File name in the control folder
	Bucket string `json:"bucket"`
}

GCSEvent is the payload of a GCS event.

Source Files ¶

View all Source files

Directories ¶

Path	Synopsis
local

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL