zedUpload

package
v0.0.0-...-1c9e867 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 19, 2024 License: Apache-2.0 Imports: 26 Imported by: 2

README

zedUpload

This directory contains functions for uploading and downloading to various remote file storage services, specifically:

  • AWS S3
  • Azure Blob
  • http/s
  • sftp

Testing

In order to run full tests, you need to have a remote account. To run the tests, set an environment variable for the credentials you need. Those are listed below. If a credential is missing for a test, it simply will not run.

The required environment variables are as follows:

  • AWS: TEST_AWS_BUCKET, TEST_AWS_KEY, TEST_AWS_SECRET, TEST_AWS_REGION
  • Azure: TEST_AZURE_CONTAINER, TEST_AZURE_ACCOUNT_NAME, TEST_AZURE_ACCOUNT_KEY
  • http: none are needed, as tests use the public ptsv2 for post testing, and Cirros Cloud and Ubuntu Images for download testing
  • sftp: TEST_SFTP_DIR, TEST_SFTP_USER, TEST_SFTP_PASS, TEST_SFTP_REGION

Documentation

Index

Constants

View Source 
const (
	SyncOpUnknown               = 0
	SyncOpUpload                = 1
	SyncOpDownload              = 2
	SyncOpDelete                = 3
	SyncOpDownloadWithSignature = 4
	SyncOpList                  = 5
	SyncOpGetObjectMetaData     = 6
	SyncOpGetURI                = 7
	SysOpPutPart                = 8
	SysOpCompleteParts          = 9
	SysOpDownloadByChunks       = 10
	DefaultNumberOfHandlers     = 11

	StatsUpdateTicker = 1 * time.Second // timer for updating client for stats
	FailPostTimeout   = 2 * time.Minute
)

Operation types supported

View Source 
const (
	//SingleMB represents data size of 1MB
	SingleMB int64 = 1024 * 1024
)

Variables

View Source 
var (
	SyncUnsupport = errors.New("Unsupported function")
	SyncerRetry   = errors.New("Retry - syncer full")
)

Functions

This section is empty.

Types

type AuthInput 

type AuthInput struct {
	// type of auth
	AuthType string

	// required, auth for whom
	Uname string

	// optional, password
	Password string

	// optional, keytabs
	Keys []string
}

type AwsTransportMethod 

type AwsTransportMethod struct {
	// contains filtered or unexported fields
}

func (*AwsTransportMethod) Action 

func (ep *AwsTransportMethod) Action(req *DronaRequest) error

Action : execute selected action targeting AWS datastore.

func (*AwsTransportMethod) Close 

func (ep *AwsTransportMethod) Close() error

func (*AwsTransportMethod) GetNetTrace 

func (ep *AwsTransportMethod) GetNetTrace(description string) (
	nettrace.AnyNetTrace, []nettrace.PacketCapture, error)

GetNetTrace returns collected network trace and packet captures.

func (*AwsTransportMethod) NewRequest 

func (ep *AwsTransportMethod) NewRequest(opType SyncOpType, objname, objloc string, sizelimit int64, ackback bool, reply chan *DronaRequest) *DronaRequest

func (*AwsTransportMethod) Open 

func (ep *AwsTransportMethod) Open() error

func (*AwsTransportMethod) WithBindIntf 

func (ep *AwsTransportMethod) WithBindIntf(intf string) error

WithBindIntf : bind to specific interface for this connection

func (*AwsTransportMethod) WithLogging 

func (ep *AwsTransportMethod) WithLogging(onoff bool) error

WithLogging enables or disables logging.

func (*AwsTransportMethod) WithNetTracing 

func (ep *AwsTransportMethod) WithNetTracing(opts ...nettrace.TraceOpt) error

WithNetTracing enables network tracing.

func (*AwsTransportMethod) WithProxy 

func (ep *AwsTransportMethod) WithProxy(proxy *url.URL) error

WithProxy : connect via the provided proxy URL.

func (*AwsTransportMethod) WithSrcIP 

func (ep *AwsTransportMethod) WithSrcIP(localAddr net.IP) error

WithSrcIP : use the specific IP as source address for this connection.

func (*AwsTransportMethod) WithTrustedCerts 

func (ep *AwsTransportMethod) WithTrustedCerts(certs [][]byte) error

WithTrustedCerts : run requests with these certificates added as trusted.

type AzureTransportMethod 

type AzureTransportMethod struct {
	// contains filtered or unexported fields
}

func (*AzureTransportMethod) Action 

func (ep *AzureTransportMethod) Action(req *DronaRequest) error

Action : execute selected action targeting Azure datastore.

func (*AzureTransportMethod) Close 

func (ep *AzureTransportMethod) Close() error

func (*AzureTransportMethod) GetNetTrace 

func (ep *AzureTransportMethod) GetNetTrace(description string) (
	nettrace.AnyNetTrace, []nettrace.PacketCapture, error)

GetNetTrace returns collected network trace and packet captures.

func (*AzureTransportMethod) NewRequest 

func (ep *AzureTransportMethod) NewRequest(opType SyncOpType, objname, objloc string, sizelimit int64, ackback bool, reply chan *DronaRequest) *DronaRequest

func (*AzureTransportMethod) Open 

func (ep *AzureTransportMethod) Open() error

func (*AzureTransportMethod) WithBindIntf 

func (ep *AzureTransportMethod) WithBindIntf(intf string) error

WithBindIntf : bind to specific interface for this connection

func (*AzureTransportMethod) WithLogging 

func (ep *AzureTransportMethod) WithLogging(onoff bool) error

WithLogging enables or disables logging.

func (*AzureTransportMethod) WithNetTracing 

func (ep *AzureTransportMethod) WithNetTracing(opts ...nettrace.TraceOpt) error

WithNetTracing enables network tracing.

func (*AzureTransportMethod) WithProxy 

func (ep *AzureTransportMethod) WithProxy(proxy *url.URL) error

WithProxy : connect via the provided proxy URL.

func (*AzureTransportMethod) WithSrcIP 

func (ep *AzureTransportMethod) WithSrcIP(localAddr net.IP) error

WithSrcIP : use the specific IP as source address for this connection.

func (*AzureTransportMethod) WithTrustedCerts 

func (ep *AzureTransportMethod) WithTrustedCerts(certs [][]byte) error

WithTrustedCerts : run requests with these certificates added as trusted.

type ChunkData 

type ChunkData struct {
	Size  int64  // complete size to upload/download
	Chunk []byte // chunk data being uploaded/downloaded
	EOF   bool   // denotes last chunk of the file
}

ChunkData contains the details of Chunks being downloaded

type DronaCtx 

type DronaCtx struct {
	// contains filtered or unexported fields
}

func NewDronaCtx 

func NewDronaCtx(name string, noHandlers int) (*DronaCtx, error)

NewDronaCtx

func (*DronaCtx) ListenAndServe 

func (ctx *DronaCtx) ListenAndServe()

Keep working till we are told otherwise

func (*DronaCtx) NewSyncerDest 

func (ctx *DronaCtx) NewSyncerDest(tr SyncTransportType, UrlOrRegion, PathOrBkt string, auth *AuthInput, opts ...SyncerDestOption) (DronaEndPoint, error)

NewSyncerDest add another location end point to syncer. The options are passed directly to the specific transport and should match its type.

type DronaEndPoint 

type DronaEndPoint interface {
	NewRequest(SyncOpType, string, string, int64, bool, chan *DronaRequest) *DronaRequest
	Open() error
	Action(req *DronaRequest) error
	Close() error
	WithSrcIP(localAddr net.IP) error
	WithTrustedCerts(certs [][]byte) error
	WithProxy(proxy *url.URL) error
	WithBindIntf(intf string) error
	WithLogging(onoff bool) error
	WithNetTracing(opts ...nettrace.TraceOpt) error
	// GetNetTrace : if network tracing is enabled (WithNetTracing() was called),
	// this method returns trace of all network operations performed up to this point,
	// possibly also accompanied by per-interface packet captures (only if enabled by
	// tracing options).
	GetNetTrace(description string) (
		nettrace.AnyNetTrace, []nettrace.PacketCapture, error)
	// contains filtered or unexported methods
}

Interface for various transport implementation

type DronaRequest 

type DronaRequest struct {
	sync.RWMutex

	// Image Sha256
	ImageSha256 string

	// used by Multipart upload
	Adata  []byte
	PartID int64
	SasURI string

	// used by azure and s3 while Multipart Upload
	Blocks []string
	// generated SasURI TTL
	Duration time.Duration
	// generated while creating multipart file
	UploadID string
	// generated after uploading the part to the multipart file
	EtagID string
	// contains filtered or unexported fields
}

func ReadMetaFile 

func ReadMetaFile(metaloc string) (error, *DronaRequest)

func (*DronaRequest) Cancel 

func (req *DronaRequest) Cancel() error

Cancel checks if WithCancel was used.

func (*DronaRequest) GetAsize 

func (req *DronaRequest) GetAsize() int64

Return object actual synced down size

func (*DronaRequest) GetChunkDetails 

func (req *DronaRequest) GetChunkDetails() (int64, []byte, bool)

GetChunkDetails Return the chunk details

func (*DronaRequest) GetContentLength 

func (req *DronaRequest) GetContentLength() int64

func (*DronaRequest) GetContentType 

func (req *DronaRequest) GetContentType() string

GetContentType return the content type if available. If not, return an empty string.

func (*DronaRequest) GetDnStatus 

func (req *DronaRequest) GetDnStatus() error

Return object error status string

func (*DronaRequest) GetDoneParts 

func (req *DronaRequest) GetDoneParts() types.DownloadedParts

GetDoneParts returns already downloaded parts indexes

func (*DronaRequest) GetImageList 

func (req *DronaRequest) GetImageList() []string

func (*DronaRequest) GetLocalName 

func (req *DronaRequest) GetLocalName() string

Return object local name

func (*DronaRequest) GetOsize 

func (req *DronaRequest) GetOsize() int64

Return object actual synced down size

func (*DronaRequest) GetRemoteFileMD5 

func (req *DronaRequest) GetRemoteFileMD5() string

func (*DronaRequest) GetSha256 

func (req *DronaRequest) GetSha256() string

GetSha256 returns an image sha256

func (*DronaRequest) GetStatus 

func (req *DronaRequest) GetStatus() string

Return the status

func (*DronaRequest) GetUpStatus 

func (req *DronaRequest) GetUpStatus() (string, error)

func (*DronaRequest) IsDnUpdate 

func (req *DronaRequest) IsDnUpdate() bool

Return is it update

func (*DronaRequest) IsError 

func (req *DronaRequest) IsError() bool

Return the if the object was downloaded with error

func (*DronaRequest) Post 

func (req *DronaRequest) Post() error

func (*DronaRequest) Progress 

func (req *DronaRequest) Progress() (int64, int64, uint)

Progress return download progress returns asize (bytes), osize (bytes), progress (percent out of 100)

func (*DronaRequest) WithCancel 

func (req *DronaRequest) WithCancel(ctx context.Context) *DronaRequest

WithCancel can be used to setup cancellation Caller should call req.Cancel when done even on success

func (*DronaRequest) WithDoneParts 

func (req *DronaRequest) WithDoneParts(doneParts types.DownloadedParts) *DronaRequest

WithDoneParts can be used to set already downloaded parts indexes

func (*DronaRequest) WithLogger 

func (req *DronaRequest) WithLogger(logger types.Logger) *DronaRequest

WithLogger sets logger

func (*DronaRequest) WriteMetaFile 

func (req *DronaRequest) WriteMetaFile(metaloc string) error

helper function to write the metafile for the transcation that is defined by this request

type GsTransportMethod 

type GsTransportMethod struct {
	// contains filtered or unexported fields
}

GsTransportMethod stores data needed to communicate with Google Cloud Storage

func (*GsTransportMethod) Action 

func (ep *GsTransportMethod) Action(req *DronaRequest) error

Action do the operation with Google Storage datastore

func (*GsTransportMethod) Close 

func (ep *GsTransportMethod) Close() error

Close not implemented

func (*GsTransportMethod) GetNetTrace 

func (ep *GsTransportMethod) GetNetTrace(description string) (
	nettrace.AnyNetTrace, []nettrace.PacketCapture, error)

GetNetTrace returns collected network trace and packet captures.

func (*GsTransportMethod) NewRequest 

func (ep *GsTransportMethod) NewRequest(opType SyncOpType, objname, objloc string, sizelimit int64, ackback bool, reply chan *DronaRequest) *DronaRequest

NewRequest returns DronaRequest for provided options

func (*GsTransportMethod) Open 

func (ep *GsTransportMethod) Open() error

Open not implemented

func (*GsTransportMethod) WithBindIntf 

func (ep *GsTransportMethod) WithBindIntf(intf string) error

WithBindIntf : bind to specific interface for this connection

func (*GsTransportMethod) WithLogging 

func (ep *GsTransportMethod) WithLogging(onoff bool) error

WithLogging enables or disables logging.

func (*GsTransportMethod) WithNetTracing 

func (ep *GsTransportMethod) WithNetTracing(opts ...nettrace.TraceOpt) error

WithNetTracing enables network tracing.

func (*GsTransportMethod) WithProxy 

func (ep *GsTransportMethod) WithProxy(proxy *url.URL) error

WithProxy : connect via the provided proxy URL.

func (*GsTransportMethod) WithSrcIP 

func (ep *GsTransportMethod) WithSrcIP(localAddr net.IP) error

WithSrcIP : use the specific IP as source address for this connection.

func (*GsTransportMethod) WithTrustedCerts 

func (ep *GsTransportMethod) WithTrustedCerts(certs [][]byte) error

WithTrustedCerts : run requests with these certificates added as trusted.

type HttpTransportMethod 

type HttpTransportMethod struct {
	// contains filtered or unexported fields
}

func (*HttpTransportMethod) Action 

func (ep *HttpTransportMethod) Action(req *DronaRequest) error

Action : execute selected action targeting HTTP datastore.

func (*HttpTransportMethod) Close 

func (ep *HttpTransportMethod) Close() error

func (*HttpTransportMethod) GetNetTrace 

func (ep *HttpTransportMethod) GetNetTrace(description string) (
	nettrace.AnyNetTrace, []nettrace.PacketCapture, error)

GetNetTrace returns collected network trace and packet captures.

func (*HttpTransportMethod) NewRequest 

func (ep *HttpTransportMethod) NewRequest(opType SyncOpType, objname, objloc string,
	sizelimit int64, ackback bool, reply chan *DronaRequest) *DronaRequest

func (*HttpTransportMethod) Open 

func (ep *HttpTransportMethod) Open() error

func (*HttpTransportMethod) WithBindIntf 

func (ep *HttpTransportMethod) WithBindIntf(intf string) error

WithBindIntf : bind to specific interface for this connection

func (*HttpTransportMethod) WithLogging 

func (ep *HttpTransportMethod) WithLogging(onoff bool) error

WithLogging enables or disables logging.

func (*HttpTransportMethod) WithNetTracing 

func (ep *HttpTransportMethod) WithNetTracing(opts ...nettrace.TraceOpt) error

WithNetTracing enables network tracing.

func (*HttpTransportMethod) WithProxy 

func (ep *HttpTransportMethod) WithProxy(proxy *url.URL) error

WithProxy : connect via the provided proxy URL.

func (*HttpTransportMethod) WithSrcIP 

func (ep *HttpTransportMethod) WithSrcIP(localAddr net.IP) error

WithSrcIP : use the specific IP as source address for this connection.

func (*HttpTransportMethod) WithTrustedCerts 

func (ep *HttpTransportMethod) WithTrustedCerts(certs [][]byte) error

WithTrustedCerts : run requests with these certificates added as trusted.

type OCITransportMethod 

type OCITransportMethod struct {
	// contains filtered or unexported fields
}

OCITransportMethod transport method to send images from OCI distribution registries

func (*OCITransportMethod) Action 

func (ep *OCITransportMethod) Action(req *DronaRequest) error

Action perform an action using this method, one of Download/Upload/Delete/List/GetObjectMetaData

func (*OCITransportMethod) Close 

func (ep *OCITransportMethod) Close() error

Close unsupported

func (*OCITransportMethod) GetNetTrace 

func (ep *OCITransportMethod) GetNetTrace(description string) (
	nettrace.AnyNetTrace, []nettrace.PacketCapture, error)

GetNetTrace returns collected network trace and packet captures.

func (*OCITransportMethod) NewRequest 

func (ep *OCITransportMethod) NewRequest(opType SyncOpType, objname, objloc string,
	sizelimit int64, ackback bool, reply chan *DronaRequest) *DronaRequest

NewRequest create a new DronaRequest with this OCITransportMethod as the endpoint

func (*OCITransportMethod) Open 

func (ep *OCITransportMethod) Open() error

Open unsupported

func (*OCITransportMethod) WithBindIntf 

func (ep *OCITransportMethod) WithBindIntf(intf string) error

WithBindIntf : bind to specific interface for this connection

func (*OCITransportMethod) WithLogging 

func (ep *OCITransportMethod) WithLogging(onoff bool) error

WithLogging enables or disables logging.

func (*OCITransportMethod) WithNetTracing 

func (ep *OCITransportMethod) WithNetTracing(opts ...nettrace.TraceOpt) error

WithNetTracing enables network tracing.

func (*OCITransportMethod) WithProxy 

func (ep *OCITransportMethod) WithProxy(proxy *url.URL) error

WithProxy : connect via the provided proxy URL.

func (*OCITransportMethod) WithSrcIP 

func (ep *OCITransportMethod) WithSrcIP(localAddr net.IP) error

WithSrcIP : use the specific IP as source address for this connection.

func (*OCITransportMethod) WithTrustedCerts 

func (ep *OCITransportMethod) WithTrustedCerts(certs [][]byte) error

WithTrustedCerts : run requests with these certificates added as trusted.

type SftpTransportMethod 

type SftpTransportMethod struct {
	// contains filtered or unexported fields
}

func (*SftpTransportMethod) Action 

func (ep *SftpTransportMethod) Action(req *DronaRequest) error

Action : execute selected action targeting SFTP datastore.

func (*SftpTransportMethod) Close 

func (ep *SftpTransportMethod) Close() error

func (*SftpTransportMethod) GetNetTrace 

func (ep *SftpTransportMethod) GetNetTrace(description string) (
	nettrace.AnyNetTrace, []nettrace.PacketCapture, error)

GetNetTrace is not supported.

func (*SftpTransportMethod) NewRequest 

func (ep *SftpTransportMethod) NewRequest(opType SyncOpType, objname, objloc string, sizelimit int64, ackback bool, reply chan *DronaRequest) *DronaRequest

func (*SftpTransportMethod) Open 

func (ep *SftpTransportMethod) Open() error

func (*SftpTransportMethod) WithBindIntf 

func (ep *SftpTransportMethod) WithBindIntf(intf string) error

WithBindIntf is not supported.

func (*SftpTransportMethod) WithLogging 

func (ep *SftpTransportMethod) WithLogging(onoff bool) error

WithLogging enables or disables logging.

func (*SftpTransportMethod) WithNetTracing 

func (ep *SftpTransportMethod) WithNetTracing(opts ...nettrace.TraceOpt) error

WithNetTracing is not supported.

func (*SftpTransportMethod) WithProxy 

func (ep *SftpTransportMethod) WithProxy(proxy *url.URL) error

WithProxy is not supported.

func (*SftpTransportMethod) WithSrcIP 

func (ep *SftpTransportMethod) WithSrcIP(localAddr net.IP) error

WithSrcIP is not supported.

func (*SftpTransportMethod) WithTrustedCerts 

func (ep *SftpTransportMethod) WithTrustedCerts(certs [][]byte) error

WithTrustedCerts is not supported.

type SyncMetaFile 

type SyncMetaFile struct {
	Operation SyncOpType

	// Object that needs to be downloaded
	Name string

	// Filled by Syncer, actual size
	Asize int64

	// Status of Download, we convert here to string because this
	// field is going to be json marshalled
	Status string

	EndTime   string
	StartTime string
}

type SyncOpType 

type SyncOpType int

Sync Operation type

type SyncTransportType 

type SyncTransportType string

Sync Transport Type 

const (
	SyncAwsTr         SyncTransportType = "s3"
	SyncAzureTr       SyncTransportType = "azure"
	SyncGSTr          SyncTransportType = "google"
	SyncHttpTr        SyncTransportType = "http"
	SyncSftpTr        SyncTransportType = "sftp"
	SyncOCIRegistryTr SyncTransportType = "oci"
)

type SyncerDestOption 

type SyncerDestOption func(endpoint DronaEndPoint) error

SyncerDestOption is a function that configures a DronaEndpoint. It is expected to check that the passed DronaEndpoint is of the correct type for it and, if not, return an error.

func WithHTTPInactivityTimeout 

func WithHTTPInactivityTimeout(timeout time.Duration) SyncerDestOption

WithHTTPInactivityTimeout set the inactivity timeout for HTTP datastore. Default if not set is 5 minutes. This is different than the timeouts for individual sections available in http.RoundTripper or the entire-transaction client.Timeout. This is reset every time data is received, so it is an inactivity timeout.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.