bigproto

package module
v0.8.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 22, 2024 License: Apache-2.0 Imports: 17 Imported by: 0

README

BigProto: Streamlined CRUD Operations with Google Bigtable and Protocol Buffers

The bigproto package offers a collection of CRUD operations for interaction with Google Bigtable, utilizing Protocol Buffers for serialization purposes.

Documentation

Overview

Package bigproto package offers a collection of CRUD operations for interaction with Google Bigtable, utilizing Protocol Buffers for serialization purposes.

Index

Examples

Constants

View Source 
const (
	DefaultColumnName = "0"
)

Variables

View Source 
var ErrInvalidFieldMask = errors.New("invalid field mask")

Functions

func Now added in v0.2.0 

func Now() *timestamppb.Timestamp

Now returns the time using Bigtable's time method.

func SetupAndUseBigtableEmulator added in v0.2.1 

func SetupAndUseBigtableEmulator(googleProject string, bigTableInstance string, tableName string,
	columnFamilies []string, createIfNotExist bool, resetIfExist bool,
)

SetupAndUseBigtableEmulator ensures that any other calls from the bigtable client are made to the gcloud bigtable emulator running on your local machine. This makes it possible to test your code without needing to set up an actual bigtable instance in the cloud. Prerequisites: You need to have the gcloud cli installed, including the bigtable emulator extension which might not be installed by default with gcloud. You also need to run "gcloud beta emulators bigtable start" once in any terminal on your pc and keep that terminal open while using the emulator. For debugging content in the local table, you can use the google cbt cli exactly as you would for a cloud bigtable instance, except that you need to run "export BIGTABLE_EMULATOR_HOST=localhost:8086" in your terminal session before running any cbt commands.

Example 
// Create a connection to a Table using bigproto
table := NewClient(context.Background(), "your-project", "bigtable-instance", "your-table")

// If not running on cloudrun use bigtable emulator
SetupAndUseBigtableEmulator("your-project", "bigtable-instance", "your-table", []string{"0", "1"}, true, true)

// use table
_ = table

Output:

Types

type BigProto 

type BigProto struct {
	// contains filtered or unexported fields
}

func New added in v0.0.2 

func New(client *bigtable.Client, tableName string) *BigProto

New does the same as NewClient, except that it allows you to pass in the bigtable client directly, instead of passing in the project, instance and table name.

Example 
// Instantiate a Google Bigtable Client
ctx := context.Background()
client, _ := bigtable.NewClient(ctx, "your-project", "bigtable-instance")

// Create a connection to a Table using bigproto
table := New(client, "your-table")

// Read a single row
row, err := table.ReadRow(ctx, "row-key-1")
if err != nil {
	// Handle the specific errors here
	if errors.As(err, &ErrNotFound{}) {
		// TODO: handle error.
	}
}

// use the bigtable row object.
_ = row

Output:

func NewClient added in v0.2.1 

func NewClient(ctx context.Context, googleProject string, bigTableInstance string, tableName string) *BigProto

NewClient returns a bigproto object, containing an initialized bigtable connection using the project,instance and table name as connection parameters It is recommended that you call this function once in your package's init function and then store the returned object as a global variable, instead of making new connections with every read/write.

Example 
// Create a connection to a Table using bigproto
table := NewClient(context.Background(), "your-project", "bigtable-instance", "your-table")

// Read a single row
row, _ := table.ReadRow(context.Background(), "row-key-1")

// use the row object.
_ = row

Output:

func (*BigProto) BatchReadProtos added in v0.5.0 

func (b *BigProto) BatchReadProtos(ctx context.Context, rowKeys []string, columnFamily string, messageType proto.Message,
	readMask *fieldmaskpb.FieldMask, opts ...bigtable.ReadOption,
) ([]proto.Message, error)

BatchReadProtos returns the list of rows for a specified set of rowKeys. The order of the response is consistent with the order of the rowKeys. Also, if a particular rowKey is not found, the corresponding response will be a nil entry in the list of messages returned.

func (*BigProto) BatchUpdateProtos added in v0.7.0 

func (b *BigProto) BatchUpdateProtos(ctx context.Context, rowKeys []string, columnFamily string, messageType proto.Message, messages []proto.Message,
	updateMasks []*fieldmaskpb.FieldMask, allowMissing []bool,
) ([]error, error)

BatchUpdateProtos obtains Bigtable row entries and unmarshalls the value at the given columnFamily to the type provided. It then merges the updates as specified in the provided message, into the current type, in line with the update mask and writes the updated protos back to Bigtable. The updated protos are also stored in the provided message pointers.

Two types of failures may occur. If the entire process fails, (nil, err) will be returned. If specific mutations fail to apply, ([]err, nil) will be returned, and the errors will correspond to the relevant rowKeys arguments.

func (*BigProto) BatchWriteProtos added in v0.6.0 

func (b *BigProto) BatchWriteProtos(ctx context.Context, rowKeys []string, columnFamily string, messages []proto.Message) ([]error, error)

BatchWriteProtos writes the provided proto messages to Bigtable by marshaling it to bytes and storing the data at the given row keys, and column family. Two types of failures may occur. If the entire process fails, (nil, err) will be returned. If specific mutations fail to apply, ([]err, nil) will be returned, and the errors will correspond to the relevant rowKeys arguments.

func (*BigProto) DeleteRow added in v0.1.0 

func (b *BigProto) DeleteRow(ctx context.Context, rowKey string) error

DeleteRow deletes an entire row from bigtable at the given rowKey.

Example 
// Instantiate a Google Bigtable Client
ctx := context.Background()
client, _ := bigtable.NewClient(ctx, "your-project", "bigtable-instance")

// Create a connection to a Table using bigproto
table := New(client, "your-table")

// Delete the row
_ = table.DeleteRow(ctx, "your-row-key")

Output:

func (*BigProto) ListProtos 

func (b *BigProto) ListProtos(ctx context.Context, columnFamily string, messageType proto.Message,
	readMask *fieldmaskpb.FieldMask, rowSet bigtable.RowSet, opts ...bigtable.ReadOption,
) ([]proto.Message, string, error)

ListProtos returns the list of rows for a specified set of rows

Example 
// Instantiate a Google Bigtable Client
ctx := context.Background()
client, _ := bigtable.NewClient(ctx, "your-project", "bigtable-instance")

// Create a connection to a Table using bigproto
table := New(client, "your-table")

// Prepare arguments for ListProtos method.
readMask := &fieldmaskpb.FieldMask{Paths: []string{"name", "display_name", "state"}}
var rowSet bigtable.RowSet
var opts []bigtable.ReadOption

// In this example, we'll list all the child resources for a given parent, filtered by provided
// column family and row key filter, only returning the latest version of a cell.
rowSet = bigtable.PrefixRange("your-parent-row-key")
opts = append(opts, bigtable.RowFilter(bigtable.ChainFilters(bigtable.LatestNFilter(1),
	bigtable.FamilyFilter("column-family"),
	bigtable.RowKeyFilter("rowkey-filter"),
)))
protos, lastRowKey, _ := table.ListProtos(ctx, "column-family", nil, readMask, rowSet, opts...)

// use the messages and lastRowKey
_ = protos
_ = lastRowKey

// convert the list of protos into your defined resources.
//resources := make([]*pb.YourMessage, len(protos))
//for i, proto := range protos {
//	resources[i] = proto.(*pb.YourMessage)
//}

Output:

func (*BigProto) ListProtosWithPolicies added in v0.8.0 

func (b *BigProto) ListProtosWithPolicies(ctx context.Context, columnFamily string, messageType proto.Message,
	readMask *fieldmaskpb.FieldMask, rowSet bigtable.RowSet, policyColumnFamily string, opts ...bigtable.ReadOption,
) ([]*RowWithPolicy, string, error)

ListProtosWithPolicies returns the list of results where each result has a row and poicy for a specified set of rows

func (*BigProto) PageProtos added in v0.3.0 

func (b *BigProto) PageProtos(ctx context.Context, columnFamily string, messageType proto.Message,
	opts PageOptions,
) ([]proto.Message, string, error)

PageProtos enables paginated list requests. if opts.maxPageSize is 0 (default value), 100 will be used.

Example 
// Instantiate a Google Bigtable Client
ctx := context.Background()
client, _ := bigtable.NewClient(ctx, "your-project", "bigtable-instance")

// Create a connection to a Table using bigproto
table := New(client, "your-table")

// Prepare arguments for PageProtos method.
readMask := &fieldmaskpb.FieldMask{Paths: []string{"name", "display_name", "state"}}

// page through protos (messageType should be a valid proto message, set to nil just for example)
protos, newNextToken, _ := table.PageProtos(ctx, "column-family", nil, PageOptions{
	RowKeyPrefix: "prefix",
	PageSize:     10,
	MaxPageSize:  100,
	ReadMask:     readMask,
})
//handle protos
_ = protos
//go to next page
_, newNextToken, _ = table.PageProtos(ctx, "column-family", nil, PageOptions{
	RowKeyPrefix: "prefix",
	PageSize:     10,
	NextToken:    newNextToken,
	MaxPageSize:  100,
	ReadMask:     nil,
})
if newNextToken == "" {
	println("No more pages")
}

Output:

func (*BigProto) PageProtosWithPolicies added in v0.8.0 

func (b *BigProto) PageProtosWithPolicies(ctx context.Context, columnFamily string, messageType proto.Message, policyColumnFamily string,
	opts PageOptions,
) ([]*RowWithPolicy, string, error)

PageProtosWithPolicies enables paginated list requests that include the policy of each row in the response. if opts.maxPageSize is 0 (default value), 100 will be used

func (*BigProto) ReadProto 

func (b *BigProto) ReadProto(ctx context.Context, rowKey string, columnFamily string, message proto.Message,
	readMask *fieldmaskpb.FieldMask,
) error

ReadProto obtains a Bigtable row entry, unmarshalls the value at the given columnFamily, applies the read mask and stores the result in the provided message pointer.

func (*BigProto) ReadProtoWithPolicy added in v0.8.0 

func (b *BigProto) ReadProtoWithPolicy(ctx context.Context, rowKey string, columnFamily string, message proto.Message,
	readMask *fieldmaskpb.FieldMask, policyColumnFamily string,
) (*iampb.Policy, error)

ReadProtoWithPolicy obtains a Bigtable row entry, unmarshalls the value at the given columnFamily, applies the read mask and stores the result in the provided message pointer. It also returns the IAM policy for the row.

func (*BigProto) ReadRow 

func (b *BigProto) ReadRow(ctx context.Context, rowKey string) (bigtable.Row, error)

ReadRow returns the row from bigtable at the given rowKey. This allows for more custom read functionality to be implemented on the row that is returned. This is useful for reading multiple columns from a row, or reading a row with a filter. It also allows for things like "Source Prioritisation" whereby data may be duplicated across column families for different sources and the sources are used in order of prior

func (*BigProto) StreamProtos added in v0.4.0 

func (b *BigProto) StreamProtos(ctx context.Context, stream chan<- proto.Message, columnFamily string, messageType proto.Message,
	readMask *fieldmaskpb.FieldMask, rowSet bigtable.RowSet, opts ...bigtable.ReadOption,
) error

StreamProtos returns the list of rows for a specified set of rows

Example 
// Instantiate a Google Bigtable Client
ctx := context.Background()
client, _ := bigtable.NewClient(ctx, "your-project", "bigtable-instance")

// Create a connection to a Table using bigproto
table := New(client, "your-table")

// Prepare arguments for ListProtos method.
var rowSet bigtable.RowSet
var opts []bigtable.ReadOption

// In this example, we'll list all the child resources for a given parent, filtered by provided
// column family and row key filter, only returning the latest version of a cell.
rowSet = bigtable.PrefixRange("your-parent-row-key")
opts = append(opts, bigtable.RowFilter(bigtable.ChainFilters(bigtable.LatestNFilter(1),
	bigtable.FamilyFilter("column-family"),
	bigtable.RowKeyFilter("rowkey-filter"),
)))

// Create a new channel to receive the streamed messages.
ch := make(chan proto.Message)

// Start the db.StreamProtos goroutine.
go func() {
	defer close(ch)
	err := table.StreamProtos(ctx, ch, "0", nil, nil, rowSet, opts...)
	if err != nil {
		log.Fatal(err)
	}
}()

// Read messages from the channel, and send to the gRPC stream server
for message := range ch {
	_ = message
	//err := stream.Send(message.(*pb.YourMessage))
	//if err != nil {
	//	return err
	//}
}

Output:

func (*BigProto) UpdateProto added in v0.1.2 

func (b *BigProto) UpdateProto(ctx context.Context, rowKey string, columnFamily string, message proto.Message,
	updateMask *fieldmaskpb.FieldMask,
) error

UpdateProto obtains a Bigtable row entry and unmarshalls the value at the given columnFamily to the type provided. It then merges the updates as specified in the provided message, into the current type, in line with the update mask and writes the updated proto back to Bigtable. The updated proto is also stored in the provided message pointer.

func (*BigProto) WriteIamPolicy added in v0.8.0 

func (b *BigProto) WriteIamPolicy(ctx context.Context, rowKey string, policyColumnFamily string, policy *iampb.Policy) error

WriteProtoIamPolicy writes the provided IAM policy to Bigtable by marshaling it to bytes and storing the data at the given row key, and column family.

func (*BigProto) WriteMutation added in v0.1.2 

func (b *BigProto) WriteMutation(ctx context.Context, rowKey string, mut *bigtable.Mutation) error

WriteMutation writes a mutation to bigtable at the given rowKey. This allows for more custom write functionality to be implemented on the row that is written. This is useful for writing multiple columns to a row, or writing a row with a filter. It also allows for things like "Source Prioritisation" whereby data may be duplicated across column families for different sources and the sources are used in order of prior

func (*BigProto) WriteProto 

func (b *BigProto) WriteProto(ctx context.Context, rowKey string, columnFamily string, message proto.Message) error

WriteProto writes the provided proto message to Bigtable by marshaling it to bytes and storing the data at the given row key, and column family.

type ErrInvalidNextToken added in v0.3.0 

type ErrInvalidNextToken struct {
	// contains filtered or unexported fields
}

func (ErrInvalidNextToken) Error added in v0.3.0 

func (e ErrInvalidNextToken) Error() string

type ErrMismatchedTypes added in v0.1.2 

type ErrMismatchedTypes struct {
	Expected reflect.Type
	Actual   reflect.Type
}

func (ErrMismatchedTypes) Error added in v0.1.2 

func (e ErrMismatchedTypes) Error() string

type ErrNegativePageSize added in v0.3.0 

type ErrNegativePageSize struct{}

func (ErrNegativePageSize) Error added in v0.3.0 

func (e ErrNegativePageSize) Error() string

type ErrNotFound 

type ErrNotFound struct {
	RowKey string // unavailable locations
}

ErrNotFound is returned when the desired resource is not found in Bigtable.

func (ErrNotFound) Error 

func (e ErrNotFound) Error() string

type PageOptions added in v0.3.0 

type PageOptions struct {
	RowKeyPrefix string
	PageSize     int32
	NextToken    string
	MaxPageSize  int32
	ReadMask     *fieldmaskpb.FieldMask
}

type RowWithPolicy added in v0.8.0 

type RowWithPolicy struct {
	Row    proto.Message
	Policy *iampb.Policy
}

type Stream added in v0.4.0 

type Stream[T any] interface {
	Send(T) error
	grpc.ServerStream
}

Stream is a generic interface for types that can send specific type objects.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.