cockroach: Index | Files | Directories

package blobs

import ""

Package blobs contains a gRPC service to be used for remote file access.

It is used for bulk file reads and writes to files on any CockroachDB node. Each node will run a blob service, which serves the file access for files on that node. Each node will also have a blob client, which uses the nodedialer to connect to another node's blob service, and access its files. The blob client is the point of entry to this service and it supports the `BlobClient` interface, which includes the following functionalities:

- ReadFile
- WriteFile
- List
- Delete
- Stat


Package Files

client.go local_storage.go service.go stream.go testutils.go


var TestEmptyBlobClientFactory = func(
    ctx context.Context, dialing roachpb.NodeID,
) (BlobClient, error) {
    return nil, nil

TestEmptyBlobClientFactory can be used as a mock BlobClient in tests that create ExternalStorage but do not use nodelocal storage.

type BlobClient Uses

type BlobClient interface {
    // ReadFile fetches the named payload from the requested node,
    // and stores it in memory. It then returns an io.ReadCloser to
    // read the contents.
    ReadFile(ctx context.Context, file string) (io.ReadCloser, error)

    // WriteFile sends the named payload to the requested node.
    // This method will read entire content of file and send
    // it over to another node, based on the nodeID.
    WriteFile(ctx context.Context, file string, content io.ReadSeeker) error

    // List lists the corresponding filenames from the requested node.
    // The requested node can be the current node.
    List(ctx context.Context, pattern string) ([]string, error)

    // Delete deletes the specified file or empty directory from a remote node.
    Delete(ctx context.Context, file string) error

    // Stat gets the size (in bytes) of a specified file from a remote node.
    Stat(ctx context.Context, file string) (*blobspb.BlobStat, error)

BlobClient provides an interface for file access on all nodes' local storage. Given the nodeID of the node on which the operation should occur, the a blob client should be able to find the correct node and call its blob service API.

type BlobClientFactory Uses

type BlobClientFactory func(ctx context.Context, dialing roachpb.NodeID) (BlobClient, error)

BlobClientFactory creates a blob client based on the nodeID we are dialing.

func NewBlobClientFactory Uses

func NewBlobClientFactory(
    localNodeID roachpb.NodeID, dialer *nodedialer.Dialer, externalIODir string,
) BlobClientFactory

NewBlobClientFactory returns a BlobClientFactory

func TestBlobServiceClient Uses

func TestBlobServiceClient(externalIODir string) BlobClientFactory

TestBlobServiceClient can be used as a mock BlobClient in tests that use nodelocal storage.

type LocalStorage Uses

type LocalStorage struct {
    // contains filtered or unexported fields

LocalStorage wraps all operations with the local file system that the blob service makes.

func NewLocalStorage Uses

func NewLocalStorage(externalIODir string) (*LocalStorage, error)

NewLocalStorage creates a new LocalStorage object and returns an error when we cannot take the absolute path of `externalIODir`.

func (*LocalStorage) Delete Uses

func (l *LocalStorage) Delete(filename string) error

Delete prepends IO dir to filename and deletes that local file.

func (*LocalStorage) List Uses

func (l *LocalStorage) List(pattern string) ([]string, error)

List prepends IO dir to pattern and glob matches all local files against that pattern.

func (*LocalStorage) ReadFile Uses

func (l *LocalStorage) ReadFile(filename string) (res io.ReadCloser, err error)

ReadFile prepends IO dir to filename and reads the content of that local file.

func (*LocalStorage) Stat Uses

func (l *LocalStorage) Stat(filename string) (*blobspb.BlobStat, error)

Stat prepends IO dir to filename and gets the Stat() of that local file.

func (*LocalStorage) WriteFile Uses

func (l *LocalStorage) WriteFile(filename string, content io.Reader) (err error)

WriteFile prepends IO dir to filename and writes the content to that local file.

type Service Uses

type Service struct {
    // contains filtered or unexported fields

Service implements the gRPC BlobService which exchanges bulk files between different nodes.

func NewBlobService Uses

func NewBlobService(externalIODir string) (*Service, error)

NewBlobService instantiates a blob service server.

func (*Service) Delete Uses

func (s *Service) Delete(
    ctx context.Context, req *blobspb.DeleteRequest,
) (*blobspb.DeleteResponse, error)

Delete implements the gRPC service.

func (*Service) GetStream Uses

func (s *Service) GetStream(req *blobspb.GetRequest, stream blobspb.Blob_GetStreamServer) error

GetStream implements the gRPC service.

func (*Service) List Uses

func (s *Service) List(
    ctx context.Context, req *blobspb.GlobRequest,
) (*blobspb.GlobResponse, error)

List implements the gRPC service.

func (*Service) PutStream Uses

func (s *Service) PutStream(stream blobspb.Blob_PutStreamServer) error

PutStream implements the gRPC service.

func (*Service) Stat Uses

func (s *Service) Stat(ctx context.Context, req *blobspb.StatRequest) (*blobspb.BlobStat, error)

Stat implements the gRPC service.



Package blobs imports 15 packages (graph) and is imported by 17 packages. Updated 2020-08-12. Refresh now. Tools for package owners.