ds

package
v0.0.0-...-f1eeddb Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 24, 2024 License: BSD-3-Clause Imports: 12 Imported by: 44

Documentation

Overview

ds is a package for using Google Cloud Datastore.

Index

Constants

View Source 
const (
	// Android Compile
	ANDROID_COMPILE_NS = "android-compile"

	// Leasing
	LEASING_SERVER_NS = "leasing-server"

	// CT
	CT_NS = "cluster-telemetry"

	// Autoroll
	AUTOROLL_NS          = "autoroll"
	AUTOROLL_INTERNAL_NS = "autoroll-internal"

	// AlertManager
	ALERT_MANAGER_NS = "alert-manager"
)

Namespaces that are used in production, and thus might be backed up.

View Source 
const (
	// Maximum number of entities which may be inserted or deleted at once.
	MAX_MODIFICATIONS = 500
)

Global constants.

Variables

View Source 
var (
	// DS is the Cloud Datastore client. Valid after Init() has been called.
	DS *datastore.Client

	// Namespace is the datastore namespace that data will be stored in. Valid after Init() has been called.
	Namespace string
)
View Source 
var (
	// KindsToBackup is a map from namespace to the list of Kinds to backup.
	// If this value is changed then remember to push a new version of /ds/go/datastore_backup.
	//
	// Note that we try to backup all kinds and all namespaces for every project
	// even if that app isn't running there, which has a better failure mode of
	// possibly backing up too much data rather than too little.
	KindsToBackup = map[string][]Kind{
		AUTOROLL_NS:          {KIND_AUTOROLL_MODE, KIND_AUTOROLL_MODE_ANCESTOR, KIND_AUTOROLL_ROLL, KIND_AUTOROLL_ROLL_ANCESTOR, KIND_AUTOROLL_STATUS, KIND_AUTOROLL_STATUS_ANCESTOR, KIND_AUTOROLL_STRATEGY, KIND_AUTOROLL_STRATEGY_ANCESTOR, KIND_AUTOROLL_UNTHROTTLE, KIND_AUTOROLL_UNTHROTTLE_ANCESTOR},
		AUTOROLL_INTERNAL_NS: {KIND_AUTOROLL_MODE, KIND_AUTOROLL_MODE_ANCESTOR, KIND_AUTOROLL_ROLL, KIND_AUTOROLL_ROLL_ANCESTOR, KIND_AUTOROLL_STATUS, KIND_AUTOROLL_STATUS_ANCESTOR, KIND_AUTOROLL_STRATEGY, KIND_AUTOROLL_STRATEGY_ANCESTOR, KIND_AUTOROLL_UNTHROTTLE, KIND_AUTOROLL_UNTHROTTLE_ANCESTOR},
		ANDROID_COMPILE_NS:   {COMPILE_TASK, ANDROID_COMPILE_INSTANCES},
		LEASING_SERVER_NS:    {TASK},
		CT_NS:                {CAPTURE_SKPS_TASKS, CHROMIUM_ANALYSIS_TASKS, CHROMIUM_BUILD_TASKS, CHROMIUM_PERF_TASKS, LUA_SCRIPT_TASKS, METRICS_ANALYSIS_TASKS, PIXEL_DIFF_TASKS, RECREATE_PAGESETS_TASKS, RECREATE_WEBPAGE_ARCHIVES_TASKS, CLUSTER_TELEMETRY_IDS},
		ALERT_MANAGER_NS:     {INCIDENT_AM, INCIDENT_ACTIVE_PARENT_AM, SILENCE_AM, SILENCE_ACTIVE_PARENT_AM, REMINDER_AM, AUDITLOG_AM},
	}
)

Functions

func DeleteAll 

func DeleteAll(client *datastore.Client, kind Kind, wait bool) (int, error)

DeleteAll removes all entities of the given kind. If wait is true it waits until an eventually consistent query of the Kind returns a count of 0. Upon success the number of deleted entities is returned.

Note: This is a very expensive operation if there are many entities of this kind and should be run as an 'offline' task.

func EnsureNotEmulator 

func EnsureNotEmulator()

EnsureNotEmulator will panic if it detects the Datastore Emulator is configured.

func Init 

func Init(project string, ns string) error

Init the Cloud Datastore Client (DS).

project - The project name, i.e. "google.com:skia-buildbots". ns - The datastore namespace to store data into.

func InitForTesting 

func InitForTesting(project string, ns string, kinds ...Kind) error

InitForTesting is an init to call when running tests. It doesn't do any auth as it is expecting to run against the Cloud Datastore Emulator. See https://cloud.google.com/datastore/docs/tools/datastore-emulator

project - The project name, i.e. "google.com:skia-buildbots". ns - The datastore namespace to store data into.

func InitWithOpt 

func InitWithOpt(project string, ns string, opts ...option.ClientOption) error

InitWithOpt the Cloud Datastore Client (DS).

project - The project name, i.e. "google.com:skia-buildbots". ns - The datastore namespace to store data into. opt - Options to pass to the client.

func IterKeys 

func IterKeys(client *datastore.Client, kind Kind, pageSize int) (<-chan *IterKeysItem, error)

IterKeys iterates all keys of the specified kind in slices of pageSize length.

func NewKey 

func NewKey(kind Kind) *datastore.Key

NewKey creates a new indeterminate key of the given kind.

func NewKeyWithParent 

func NewKeyWithParent(kind Kind, parent *datastore.Key) *datastore.Key

func NewQuery 

func NewQuery(kind Kind) *datastore.Query

NewQuery creates a new query of the given kind with the right namespace.

Types

type IterKeysItem 

type IterKeysItem struct {
	Keys []*datastore.Key
	Err  error
}

IterKeysItem is the item returned by the IterKeys function via a channel.

type Kind 

type Kind string

Kind of datastore entry. 

const (
	// Predict
	FAILURES     Kind = "Failures"
	FLAKY_RANGES Kind = "FlakyRanges"

	// Android Compile
	COMPILE_TASK              Kind = "CompileTask"
	ANDROID_COMPILE_INSTANCES Kind = "AndroidCompileInstances"

	// Leasing
	TASK Kind = "Task"

	// CT
	CAPTURE_SKPS_TASKS              Kind = "CaptureSkpsTasks"
	CHROMIUM_ANALYSIS_TASKS         Kind = "ChromiumAnalysisTasks"
	CHROMIUM_BUILD_TASKS            Kind = "ChromiumBuildTasks"
	CHROMIUM_PERF_TASKS             Kind = "ChromiumPerfTasks"
	LUA_SCRIPT_TASKS                Kind = "LuaScriptTasks"
	METRICS_ANALYSIS_TASKS          Kind = "MetricsAnalysisTasks"
	PIXEL_DIFF_TASKS                Kind = "PixelDiffTasks"
	RECREATE_PAGESETS_TASKS         Kind = "RecreatePageSetsTasks"
	RECREATE_WEBPAGE_ARCHIVES_TASKS Kind = "RecreateWebpageArchivesTasks"
	CLUSTER_TELEMETRY_IDS           Kind = "ClusterTelemetryIDs"

	// Autoroll
	KIND_AUTOROLL_MODE                Kind = "AutorollMode"
	KIND_AUTOROLL_MODE_ANCESTOR       Kind = "AutorollModeAncestor" // Fake; used to force strong consistency for testing's sake.
	KIND_AUTOROLL_ROLL                Kind = "AutorollRoll"
	KIND_AUTOROLL_ROLL_ANCESTOR       Kind = "AutorollRollAncestor" // Fake; used to force strong consistency for testing's sake.
	KIND_AUTOROLL_STATUS              Kind = "AutorollStatus"
	KIND_AUTOROLL_STATUS_ANCESTOR     Kind = "AutorollStatusAncestor" // Fake; used to force strong consistency for testing's sake.
	KIND_AUTOROLL_STRATEGY            Kind = "AutorollStrategy"
	KIND_AUTOROLL_STRATEGY_ANCESTOR   Kind = "AutorollStrategyAncestor" // Fake; used to force strong consistency for testing's sake.
	KIND_AUTOROLL_UNTHROTTLE          Kind = "AutorollUnthrottle"
	KIND_AUTOROLL_UNTHROTTLE_ANCESTOR Kind = "AutorollUnthrottleAncestor" // Fake; used to force strong consistency for testing's sake.

	// AlertManager
	INCIDENT_AM               Kind = "IncidentAm"
	INCIDENT_ACTIVE_PARENT_AM Kind = "IncidentActiveParentAm"
	SILENCE_ACTIVE_PARENT_AM  Kind = "SilenceActiveParentAm"
	SILENCE_AM                Kind = "SilenceAm"
	REMINDER_AM               Kind = "ReminderAm"
	AUDITLOG_AM               Kind = "AuditLogAm"
)

Below are all the Kinds used in all applications. New Kinds should be listed here, and they should all have unique values, because when defining indexes for Cloud Datastore the index config is per project, not pre-namespace. Remember to add them to KindsToBackup if they are to be backed up, and push a new version of /ds/go/datastore_backup.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.