custom

package

v0.0.0-...-c0cdd26 Latest Latest Go to latest Published: Mar 29, 2024 License: Apache-2.0 Imports: 18 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/datacommonsorg/tools

Links

Open Source Insights

README ¶

Custom DC BT automation

Backgroud

Custom DC BT automation also triggers KG builders owned by the DC team by writing to a blob in the resource bucket.

The blob written to the following path will trigger a cache build.

gs://<resource-bucket>/../<import name>/process/<import id>/trigger.txt

How to test

Run ./local/deploy.sh to start custom DC gcf locally.

Testing csv trigger

Run ./local/test.sh publish.

Documentation ¶

Overview ¶

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Index ¶

func ComputeManifest(ctx context.Context, reader Reader, bucket string, layout *Layout) (*pb.DataCommonsManifest, error)
func HandleTriggerFlow(ctx context.Context, bucket, root string) error
func Publish(ctx context.Context, topicName string, attributes map[string]string) error
type GCSReader
- func (g *GCSReader) ListObjects(ctx context.Context, bucket, rootDir string) ([]string, error)
- func (g *GCSReader) ReadObject(ctx context.Context, bucket, object string) ([]byte, error)
type Import
type Layout
- func BuildLayout(root string, objects []string) (*Layout, error)
type Reader
type Table

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

func ComputeManifest ¶

func ComputeManifest(
	ctx context.Context,
	reader Reader,
	bucket string,
	layout *Layout,
) (*pb.DataCommonsManifest, error)

func HandleTriggerFlow ¶

func HandleTriggerFlow(ctx context.Context, bucket, root string) error

func Publish ¶

func Publish(
	ctx context.Context,
	topicName string,
	attributes map[string]string,
) error

Publish publishes the current import config to a topic.

Types ¶

type GCSReader ¶

type GCSReader struct{}

func (*GCSReader) ListObjects ¶

func (g *GCSReader) ListObjects(
	ctx context.Context, bucket, rootDir string) ([]string, error)

func (*GCSReader) ReadObject ¶

func (g *GCSReader) ReadObject(
	ctx context.Context, bucket, object string) ([]byte, error)

type Import ¶

type Import struct {
	// contains filtered or unexported fields
}

type Layout ¶

type Layout struct {
	// contains filtered or unexported fields
}

func BuildLayout ¶

func BuildLayout(root string, objects []string) (*Layout, error)

type Reader ¶

type Reader interface {
	ListObjects(context.Context, string, string) ([]string, error)
	ReadObject(context.Context, string, string) ([]byte, error)
}

type Table ¶

type Table struct {
	// contains filtered or unexported fields
}

Only one of (mcf, (tmcf, csv)) should be set.

Source Files ¶

View all Source files

Directories ¶

Path	Synopsis
local

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL