api

package
v1.3.22 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 25, 2024 License: MIT Imports: 28 Imported by: 4

Documentation

Overview

Package api provides Go based AIStore API/SDK over HTTP(S)

  • Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.

Package api provides Go based AIStore API/SDK over HTTP(S)

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package api provides Go based AIStore API/SDK over HTTP(S)

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package api provides Go based AIStore API/SDK over HTTP(S)

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package api provides Go based AIStore API/SDK over HTTP(S)

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package api provides Go based AIStore API/SDK over HTTP(S)

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package api provides Go based AIStore API/SDK over HTTP(S)

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package api provides Go based AIStore API/SDK over HTTP(S)

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package api provides Go based AIStore API/SDK over HTTP(S)

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package api provides Go based AIStore API/SDK over HTTP(S)

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package api provides Go based AIStore API/SDK over HTTP(S)

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package api provides Go based AIStore API/SDK over HTTP(S)

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package api provides Go based AIStore API/SDK over HTTP(S)

  • Copyright (c) 2018-2023, NVIDIA CORPORATION. All rights reserved.

Package api provides Go based AIStore API/SDK over HTTP(S)

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func AbortDownload 

func AbortDownload(bp BaseParams, id string) error

func AbortDsort added in v1.3.21 

func AbortDsort(bp BaseParams, managerUUID string) error

func AbortXaction 

func AbortXaction(bp BaseParams, args *xact.ArgsMsg) (err error)

Abort ("stop") xactions

func AppendObject 

func AppendObject(args *AppendArgs) (string, error)

AppendObject adds a reader (`args.Reader` - e.g., an open file) to an object. The API can be called multiple times - each call returns a handle that may be used for subsequent append requests. Once all the "appending" is done, the caller must call `api.FlushObject` to finalize the object. NOTE: object becomes visible and accessible only _after_ the call to `api.FlushObject`.

func ArchiveMultiObj added in v1.3.18 

func ArchiveMultiObj(bp BaseParams, bckFrom cmn.Bck, msg *cmn.ArchiveBckMsg) (string, error)

Archive multiple objects from the specified source bucket. The option to append multiple objects to an existing archive is also supported. The source and the destination buckets are defined as `bckFrom` and `bckTo`, respectively (not necessarily distinct) For supported archiving formats, see `archive.FileExtensions`. See also: api.PutApndArch

func AttachMountpath 

func AttachMountpath(bp BaseParams, node *meta.Snode, mountpath string, force bool) error

TODO: rewrite tests that come here with `force`

func AttachRemoteAIS 

func AttachRemoteAIS(bp BaseParams, alias, u string) error

func BlobDownload added in v1.3.22 

func BlobDownload(bp BaseParams, bck cmn.Bck, objName string, msg *apc.BlobMsg) (xid string, err error)

func CopyBucket 

func CopyBucket(bp BaseParams, bckFrom, bckTo cmn.Bck, msg *apc.CopyBckMsg, fltPresence ...int) (xid string, err error)

CopyBucket copies existing `bckFrom` bucket to the destination `bckTo` thus, effectively, creating a copy of the `bckFrom`.

  • AIS will create `bckTo` on the fly but only if the destination bucket does not exist and _is_ provided by AIStore; 3rd party backend destination must exist - otherwise the copy operation won't be successful.
  • There are no limitations on copying buckets across Backend providers: you can copy AIS bucket to (or from) AWS bucket, and the latter to Google or Azure bucket, etc.
  • Copying multiple buckets to the same destination bucket is also permitted.

`fltPresence` applies exclusively to remote `bckFrom` and is ignored if the source is ais:// The value is enum { apc.FltExists, apc.FltPresent, ... } - for complete enum, see api/apc/query.go Namely: * apc.FltExists - copy all objects, including those that are not (present) in AIS * apc.FltPresent - copy the current `bckFrom` content in the cluster (default) * apc.FltExistsOutside - copy only those remote objects that are not (present) in AIS

msg.Prefix, if specified, applies always and regardless.

Returns xaction ID if successful, an error otherwise. See also closely related api.ETLBucket

func CopyMultiObj 

func CopyMultiObj(bp BaseParams, bckFrom cmn.Bck, msg *cmn.TCObjsMsg, fltPresence ...int) (xid string, err error)

func CreateBucket 

func CreateBucket(bp BaseParams, bck cmn.Bck, props *cmn.BpropsToSet, dontHeadRemote ...bool) error

CreateBucket sends request to create an AIS bucket with the given name and, optionally, specific non-default properties (via cmn.BpropsToSet).

See also:

  • github.com/NVIDIA/aistore/blob/main/docs/bucket.md#default-bucket-properties
  • cmn.BpropsToSet (cmn/api.go)

Bucket properties can be also changed at any time via SetBucketProps (above).

func DecommissionCluster 

func DecommissionCluster(bp BaseParams, rmUserData bool) error

DecommissionCluster permanently decommissions entire cluster

func DecommissionNode 

func DecommissionNode(bp BaseParams, actValue *apc.ActValRmNode) (xid string, err error)

func DeleteMultiObj added in v1.3.22 

func DeleteMultiObj(bp BaseParams, bck cmn.Bck, objNames []string, template string) (string, error)

func DeleteObject 

func DeleteObject(bp BaseParams, bck cmn.Bck, objName string) error

func DestroyBucket 

func DestroyBucket(bp BaseParams, bck cmn.Bck) error

DestroyBucket sends request to remove an AIS bucket with the given name.

func DetachMountpath 

func DetachMountpath(bp BaseParams, node *meta.Snode, mountpath string, dontResilver bool) error

func DetachRemoteAIS 

func DetachRemoteAIS(bp BaseParams, alias string) error

func DisableMountpath 

func DisableMountpath(bp BaseParams, node *meta.Snode, mountpath string, dontResilver bool) error

func DoWithRetry 

func DoWithRetry(client *http.Client, cb NewRequestCB, reqArgs *cmn.HreqArgs) (resp *http.Response, err error)

DoWithRetry executes `http-client.Do` and retries *retriable connection errors*, such as "broken pipe" and "connection refused". This function always closes the `reqArgs.BodR`, even in case of error. Usage: PUT and simlar requests that transfer payload from the user side. NOTE: always closes request body reader (reqArgs.BodyR) - explicitly or via Do() TODO: refactor

func DownloadBackend 

func DownloadBackend(bp BaseParams, descr string, bck cmn.Bck, prefix, suffix string, ivals ...time.Duration) (string, error)

func DownloadGetList 

func DownloadGetList(bp BaseParams, regex string, onlyActive bool) (dlList dload.JobInfos, err error)

func DownloadMulti 

func DownloadMulti(bp BaseParams, description string, bck cmn.Bck, msg any, intervals ...time.Duration) (string, error)

func DownloadRange 

func DownloadRange(bp BaseParams, description string, bck cmn.Bck, template string, intervals ...time.Duration) (string, error)

func DownloadSingle 

func DownloadSingle(bp BaseParams, description string,
	bck cmn.Bck, objName, link string, intervals ...time.Duration) (string, error)

func DownloadStatus 

func DownloadStatus(bp BaseParams, id string, onlyActive bool) (dlStatus *dload.StatusResp, err error)

func DownloadWithParam 

func DownloadWithParam(bp BaseParams, dlt dload.Type, body any) (id string, err error)

func ECEncodeBucket 

func ECEncodeBucket(bp BaseParams, bck cmn.Bck, data, parity int) (xid string, err error)

Erasure-code entire `bck` bucket at a given `data`:`parity` redundancy. The operation requires at least (`data + `parity` + 1) storage targets in the cluster. Returns xaction ID if successful, an error otherwise.

func ETLBucket 

func ETLBucket(bp BaseParams, bckFrom, bckTo cmn.Bck, msg *apc.TCBMsg, fltPresence ...int) (xid string, err error)

Transform src bucket => dst bucket, i.e.: - visit all (matching) source objects; for each object: - read it, transform using the specified (ID-ed) ETL, and write the result to dst bucket

`fltPresence` applies exclusively to remote `bckFrom` (is ignored otherwise) and is one of: { apc.FltExists, apc.FltPresent, ... } - for complete enum, see api/apc/query.go Namely: * apc.FltExists - copy all objects, including those that are not (present) in AIS * apc.FltPresent - copy the current `bckFrom` content in the cluster (default) * apc.FltExistsOutside - copy only those remote objects that are not (present) in AIS

msg.Prefix, if specified, applies always and regardless.

Returns xaction ID if successful, an error otherwise. See also: api.CopyBucket

func ETLDelete 

func ETLDelete(bp BaseParams, etlName string) (err error)

func ETLGetInitMsg 

func ETLGetInitMsg(params BaseParams, etlName string) (etl.InitMsg, error)

func ETLHealth 

func ETLHealth(params BaseParams, etlName string) (healths etl.HealthByTarget, err error)

func ETLInit 

func ETLInit(bp BaseParams, msg etl.InitMsg) (xid string, err error)

Initiate custom ETL workload by executing one of the documented `etl.InitMsg` message types. The API call results in deploying multiple ETL containers (K8s pods): one container per storage target. Returns xaction ID if successful, an error otherwise.

func ETLList 

func ETLList(bp BaseParams) (list []etl.Info, err error)

func ETLLogs 

func ETLLogs(bp BaseParams, etlName string, targetID ...string) (logs etl.LogsByTarget, err error)

func ETLMetrics added in v1.3.16 

func ETLMetrics(params BaseParams, etlName string) (healths etl.CPUMemByTarget, err error)

func ETLMultiObj 

func ETLMultiObj(bp BaseParams, bckFrom cmn.Bck, msg *cmn.TCObjsMsg, fltPresence ...int) (xid string, err error)

func ETLObject 

func ETLObject(bp BaseParams, etlName string, bck cmn.Bck, objName string, w io.Writer) (err error)

TODO: add ETL-specific query param and change the examples/docs (!4455)

func ETLStart 

func ETLStart(bp BaseParams, etlName string) (err error)

func ETLStop 

func ETLStop(bp BaseParams, etlName string) (err error)

func EnableMountpath 

func EnableMountpath(bp BaseParams, node *meta.Snode, mountpath string) error

func EvictMultiObj added in v1.3.22 

func EvictMultiObj(bp BaseParams, bck cmn.Bck, objNames []string, template string) (string, error)

func EvictObject 

func EvictObject(bp BaseParams, bck cmn.Bck, objName string) error

func EvictRemoteBucket 

func EvictRemoteBucket(bp BaseParams, bck cmn.Bck, keepMD bool) error

EvictRemoteBucket sends request to evict an entire remote bucket from the AIStore - keepMD: evict objects but keep bucket metadata

func FlushObject 

func FlushObject(args *FlushArgs) error

FlushObject must be called after all the appends (via `api.AppendObject`). To "flush", it uses the handle returned by `api.AppendObject`. This call will create a fully operational and accessible object.

func FreeRp 

func FreeRp(reqParams *ReqParams)

func GetAllRunningXactions added in v1.3.16 

func GetAllRunningXactions(bp BaseParams, kindOrName string) (out []string, err error)

returns a slice of canonical xaction names, as in: `xact.Cname()` e.g.: put-copies[D-ViE6HEL_j] list[H96Y7bhR2s] copy-bck[matRQMRes] put-copies[pOibtHExY] TODO: return idle xactions separately

func GetAllXactionStatus added in v1.3.16 

func GetAllXactionStatus(bp BaseParams, args *xact.ArgsMsg) (matching nl.StatusVec, err error)

same as above, except that it returns _all_ matching xactions

func GetBMD 

func GetBMD(bp BaseParams) (bmd *meta.BMD, err error)

get bucket metadata (BMD) from a BaseParams-referenced node

func GetBucketInfo 

func GetBucketInfo(bp BaseParams, bck cmn.Bck, args BinfoArgs) (string, *cmn.Bprops, *cmn.BsummResult, error)

Bucket information - a runtime addendum to `BucketProps`. In addition to `cmn.Bprops` properties (which are user configurable), bucket runtime info: - includes usage, capacity, other statistics - is obtained via GetBucketInfo() API - and delivered via apc.HdrBucketInfo header (compare with GetBucketSummary) The API uses http.MethodHead and can be considered an extension of HeadBucket (above)

func GetBucketSummary 

func GetBucketSummary(bp BaseParams, qbck cmn.QueryBcks, msg *apc.BsummCtrlMsg, args BsummArgs) (xid string,
	res cmn.AllBsummResults, err error)

GetBucketSummary returns bucket capacity ulitization percentages, sizes (total and min/max/average), and the numbers of objects, both _in_ the cluster and remote GetBucketSummary supports a single specified bucket or multiple buckets, as per `cmn.QueryBcks` query. (e.g., GetBucketSummary with an empty bucket query will return "summary" info for all buckets)

func GetClusterConfig 

func GetClusterConfig(bp BaseParams) (*cmn.ClusterConfig, error)

GetClusterConfig returns cluster-wide configuration (compare with `api.GetDaemonConfig`)

func GetClusterMap 

func GetClusterMap(bp BaseParams) (smap *meta.Smap, err error)

get cluster map from a BaseParams-referenced node

func GetClusterStats 

func GetClusterStats(bp BaseParams) (res stats.Cluster, err error)

How to compute throughputs:

- AIS supports several enumerated metric "kinds", including `KindThroughput` (for complete enumeration, see stats/api.go) - By convention, metrics that have `KindThroughput` kind are named with ".bps" ("bytes per second") suffix. - ".bps" metrics reported by api.GetClusterStats and api.GetDaemonStats are, in fact, cumulative byte numbers. - It is the client's responsibility to compute the actual throughputs as only the client knows _when_ exactly the same ".bps" metric was queried the previous time.

- See also: `api.GetDaemonStats`, stats/api.go

func GetClusterSysInfo 

func GetClusterSysInfo(bp BaseParams) (info apc.ClusterSysInfo, err error)

GetClusterSysInfo retrieves cluster's system information

func GetDaemonConfig 

func GetDaemonConfig(bp BaseParams, node *meta.Snode) (config *cmn.Config, err error)

GetDaemonConfig returns the configuration of a specific daemon in a cluster. (compare with `api.GetClusterConfig`)

func GetDaemonLog 

func GetDaemonLog(bp BaseParams, node *meta.Snode, args GetLogInput) (int64, error)

Returns log of a specific node in a cluster.

func GetDaemonStats 

func GetDaemonStats(bp BaseParams, node *meta.Snode) (ds *stats.Node, err error)

How to compute throughputs:

- AIS supports several enumerated metric "kinds", including `KindThroughput` (for complete enumeration, see stats/api.go) - By convention, metrics that have `KindThroughput` kind are named with ".bps" ("bytes per second") suffix. - ".bps" metrics reported by the API are, in fact, cumulative byte numbers. - It is the client's responsibility to compute the actual throughputs as only the client knows _when_ exactly the same ".bps" metric was queried the previous time.

See also: - api.GetClusterStats - api.GetStatsAndStatus (below) - stats/api.go

func GetDiskStats added in v1.3.16 

func GetDiskStats(bp BaseParams, tid string) (res ios.AllDiskStats, err error)

func GetMetricNames added in v1.3.16 

func GetMetricNames(bp BaseParams, node *meta.Snode) (kvs cos.StrKVs, err error)

names _and_ kinds, i.e. (name, kind) pairs

func GetMountpaths 

func GetMountpaths(bp BaseParams, node *meta.Snode) (mpl *apc.MountpathList, err error)

GetMountpaths given the direct public URL of the target, returns the target's mountpaths or error.

func GetNodeClusterMap 

func GetNodeClusterMap(bp BaseParams, sid string) (smap *meta.Smap, err error)

GetNodeClusterMap retrieves cluster map from the specified node.

func GetNodeMeta added in v1.3.19 

func GetNodeMeta(bp BaseParams, sid, what string) (out any, err error)

- get (smap, bmd, config) *cluster-level* metadata from the spec-ed node - compare with GetClusterMap, GetNodeClusterMap, GetClusterConfig et al. - TODO: etl meta

func GetObjectReader 

func GetObjectReader(bp BaseParams, bck cmn.Bck, objName string, args *GetArgs) (r io.ReadCloser, size int64, err error)

GetObjectReader returns reader of the requested object. It does not read body bytes, nor validates a checksum. Caller is responsible for closing the reader.

func GetObjectS3 

func GetObjectS3(bp BaseParams, bck cmn.Bck, objectName string, args ...GetArgs) (int64, error)

s3/<bucket-name>/<object-name>

func GetOneXactionStatus added in v1.3.16 

func GetOneXactionStatus(bp BaseParams, args *xact.ArgsMsg) (status *nl.Status, err error)

GetOneXactionStatus queries one of the IC (proxy) members for status of the `args`-identified xaction. NOTE: - is used internally by the WaitForXactionIC() helper function (to wait on xaction) - returns a single matching xaction or none; - when the `args` filter "covers" multiple xactions the returned status corresponds to any matching xaction that's currently running, or - if nothing's running - the one that's finished most recently, if exists

func GetProxyReadiness 

func GetProxyReadiness(bp BaseParams) error

to be used by external watchdogs (Kubernetes, etc.) (compare with api.Health below)

func GetRemoteAIS 

func GetRemoteAIS(bp BaseParams) (remais meta.RemAisVec, err error)

func GetStatsAndStatus added in v1.3.16 

func GetStatsAndStatus(bp BaseParams, node *meta.Snode) (daeStatus *stats.NodeStatus, err error)

Returns both node's stats and extended status

func GetWhatRawQuery 

func GetWhatRawQuery(getWhat, getProps string) string

func HTTPStatus 

func HTTPStatus(err error) int

HTTPStatus returns HTTP status or (-1) for non-HTTP error.

func HeadBucket 

func HeadBucket(bp BaseParams, bck cmn.Bck, dontAddRemote bool) (p *cmn.Bprops, err error)

HEAD(bucket): apc.HdrBucketProps => cmn.Bprops{} and apc.HdrBucketInfo => BucketInfo{}

Converts the string type fields returned from the HEAD request to their corresponding counterparts in the cmn.Bprops struct.

By default, AIStore adds remote buckets to the cluster metadata on the fly. Remote bucket that was never accessed before just "shows up" when user performs HEAD, PUT, GET, SET-PROPS, and a variety of other operations. This is done only once (and after confirming the bucket's existence and accessibility) and doesn't require any action from the user. Use `dontAddRemote` to override the default behavior: as the name implies, setting `dontAddRemote = true` prevents AIS from adding remote bucket to the cluster's metadata.

func HeadObject 

func HeadObject(bp BaseParams, bck cmn.Bck, objName string, fltPresence int, silent bool) (*cmn.ObjectProps, error)

HeadObject returns object properties; can be conventionally used to establish in-cluster presence. - fltPresence: as per QparamFltPresence enum (for values and comments, see api/apc/query.go) - silent==true: not to log (not-found) error

func Health 

func Health(bp BaseParams, readyToRebalance ...bool) error

func HealthUptime 

func HealthUptime(bp BaseParams, readyToRebalance ...bool) (string, string, error)

func JoinCluster 

func JoinCluster(bp BaseParams, nodeInfo *meta.Snode) (rebID, sid string, err error)

JoinCluster add a node to a cluster.

func ListBuckets 

func ListBuckets(bp BaseParams, qbck cmn.QueryBcks, fltPresence int) (cmn.Bcks, error)

ListBuckets returns buckets for provided query, where - `fltPresence` is one of { apc.FltExists, apc.FltPresent, ... } - see api/apc/query.go - ListBuckets utilizes `cmn.QueryBcks` - control structure that's practically identical to `cmn.Bck`, except for the fact that some or all its fields can be empty (to facilitate the corresponding query). See also: QueryBuckets, ListObjects

func ListDsort added in v1.3.21 

func ListDsort(bp BaseParams, regex string, onlyActive bool) (jobInfos []*dsort.JobInfo, err error)

func ListObjects 

func ListObjects(bp BaseParams, bck cmn.Bck, lsmsg *apc.LsoMsg, args ListArgs) (*cmn.LsoResult, error)

ListObjects returns a list of objects in a bucket - a slice of structures in the `cmn.LsoResult` that look like `cmn.LsoEntry`.

The `numObjects` argument is the maximum number of objects to be returned (where 0 (zero) means returning all objects in the bucket).

This API supports numerous options and flags. In particular, `apc.LsoMsg` structure supports "opening" objects formatted as one of the supported archival types and include contents of archived directories in generated result sets.

In addition, `lsmsg` (`apc.LsoMsg`) provides options (flags) to optimize the request's latency, to list anonymous public-access Cloud buckets, and more. Further details at `api/apc/lsmsg.go` source.

AIS supports listing buckets that have millions of objects. For large and very large buckets, it is strongly recommended to use the `ListObjectsPage` API - effectively, an iterator returning _next_ listed page along with associated _continuation token_.

See also: - `ListObjectsPage` - usage examples in CLI docs under docs/cli.

func ListObjectsInvalidateCache 

func ListObjectsInvalidateCache(bp BaseParams, bck cmn.Bck) error

TODO: obsolete this function after introducing mechanism to detect remote bucket changes.

func ListObjectsPage 

func ListObjectsPage(bp BaseParams, bck cmn.Bck, lsmsg *apc.LsoMsg) (*cmn.LsoResult, error)

ListObjectsPage returns the first page of bucket objects. On success the function updates `lsmsg.ContinuationToken` which client then can reuse to fetch the next page. See also: CLI and CLI usage examples See also: `apc.LsoMsg` See also: `api.ListObjectsInvalidateCache` See also: `api.ListObjects`

func MakeNCopies 

func MakeNCopies(bp BaseParams, bck cmn.Bck, copies int) (xid string, err error)

MakeNCopies starts an extended action (xaction) to bring a given bucket to a certain redundancy level (num copies). Returns xaction ID if successful, an error otherwise.

func MetricsDsort added in v1.3.21 

func MetricsDsort(bp BaseParams, managerUUID string) (metrics map[string]*dsort.JobInfo, err error)

func Prefetch added in v1.3.22 

func Prefetch(bp BaseParams, bck cmn.Bck, msg apc.PrefetchMsg) (string, error)

func PrefetchObject added in v1.3.22 

func PrefetchObject(bp BaseParams, bck cmn.Bck, objName string) (string, error)

prefetch object - a convenience method added for "symmetry" with the evict (above) - compare with api.PrefetchList and api.PrefetchRange

func Promote 

func Promote(bp BaseParams, bck cmn.Bck, args *apc.PromoteArgs) (xid string, err error)

promote files and directories to ais objects

func PutApndArch added in v1.3.18 

func PutApndArch(args *PutApndArchArgs) (err error)

Archive the content of a reader (`args.Reader` - e.g., an open file). Destination, depending on the options, can be an existing (.tar, .tgz or .tar.gz, .zip, .tar.lz4) formatted object (aka "shard") or a new one (or, a new version). --- For the updated list of supported archival formats -- aka MIME types -- see cmn/cos/archive.go. -- See also: - api.ArchiveMultiObj(msg.AppendIfExists = true) - api.AppendObject

func QueryBuckets 

func QueryBuckets(bp BaseParams, qbck cmn.QueryBcks, fltPresence int) (bool, error)

QueryBuckets is a little convenience helper. It returns true if the selection contains at least one bucket that satisfies the (qbck) criteria. - `fltPresence` - as per QparamFltPresence enum (see api/apc/query.go)

func QueryXactionSnaps 

func QueryXactionSnaps(bp BaseParams, args *xact.ArgsMsg) (xs xact.MultiSnap, err error)

QueryXactionSnaps gets all xaction snaps based on the specified selection. NOTE: args.Kind can be either xaction kind or name - here and elsewhere

func RemoveDownload 

func RemoveDownload(bp BaseParams, id string) error

func RemoveDsort added in v1.3.21 

func RemoveDsort(bp BaseParams, managerUUID string) error

func RemoveNodeUnsafe added in v1.3.18 

func RemoveNodeUnsafe(bp BaseParams, sid string) error

Remove node node from the cluster immediately. - NOTE: potential data loss, advanced usage only! - NOTE: the node remains running (compare w/ shutdown) and can be re-joined at a later time (see api.JoinCluster).

func RenameBucket 

func RenameBucket(bp BaseParams, bckFrom, bckTo cmn.Bck) (xid string, err error)

RenameBucket renames bckFrom as bckTo. Returns xaction ID if successful, an error otherwise.

func RenameObject 

func RenameObject(bp BaseParams, bck cmn.Bck, oldName, newName string) error

RenameObject renames object name from `oldName` to `newName`. Works only across single, specified bucket.

func ResetBucketProps 

func ResetBucketProps(bp BaseParams, bck cmn.Bck) (string, error)

ResetBucketProps resets the properties of a bucket to the global configuration.

func ResetClusterConfig 

func ResetClusterConfig(bp BaseParams) error

all nodes: reset configuration to cluster defaults

func ResetClusterStats added in v1.3.16 

func ResetClusterStats(bp BaseParams, errorsOnly bool) (err error)

zero out: all metrics _or_ only error counters

func ResetDaemonConfig 

func ResetDaemonConfig(bp BaseParams, nodeID string) error

reset node's configuration to cluster defaults

func ResetDaemonStats added in v1.3.16 

func ResetDaemonStats(bp BaseParams, node *meta.Snode, errorsOnly bool) error

see also: ResetClusterStats

func RotateClusterLogs added in v1.3.22 

func RotateClusterLogs(bp BaseParams) error

func RotateLogs added in v1.3.22 

func RotateLogs(bp BaseParams, nodeID string) error

func SetAuxHeaders 

func SetAuxHeaders(r *http.Request, bp *BaseParams)

func SetBucketProps 

func SetBucketProps(bp BaseParams, bck cmn.Bck, props *cmn.BpropsToSet) (string, error)

SetBucketProps sets the properties of a bucket. Validation of the properties passed in is performed by AIStore Proxy.

func SetClusterConfig 

func SetClusterConfig(bp BaseParams, nvs cos.StrKVs, transient bool) error

SetClusterConfig given key-value pairs of cluster configuration parameters, sets the cluster-wide configuration accordingly. Setting cluster-wide configuration requires sending the request to a proxy.

func SetClusterConfigUsingMsg 

func SetClusterConfigUsingMsg(bp BaseParams, configToUpdate *cmn.ConfigToSet, transient bool) error

SetClusterConfigUsingMsg sets the cluster-wide configuration using the `cmn.ConfigToSet` parameter provided.

func SetDaemonConfig 

func SetDaemonConfig(bp BaseParams, nodeID string, nvs cos.StrKVs, transient ...bool) error

SetDaemonConfig, given key value pairs, sets the configuration accordingly for a specific node.

func SetObjectCustomProps 

func SetObjectCustomProps(bp BaseParams, bck cmn.Bck, objName string, custom cos.StrKVs, setNew bool) error

Given cos.StrKVs (map[string]string) keys and values, sets object's custom properties. By default, adds new or updates existing custom keys. Use `setNewCustomMDFlag` to _replace_ all existing keys with the specified (new) ones. See also: HeadObject() and apc.HdrObjCustomMD

func SetPrimaryProxy 

func SetPrimaryProxy(bp BaseParams, newPrimaryID string, force bool) error

SetPrimaryProxy given a daemonID sets that corresponding proxy as the primary proxy of the cluster.

func ShutdownCluster 

func ShutdownCluster(bp BaseParams) error

ShutdownCluster shuts down the whole cluster

func ShutdownNode 

func ShutdownNode(bp BaseParams, actValue *apc.ActValRmNode) (id string, err error)

ShutdownNode shuts down a specific node

func StartDsort added in v1.3.21 

func StartDsort(bp BaseParams, rs *dsort.RequestSpec) (id string, err error)

func StartMaintenance 

func StartMaintenance(bp BaseParams, actValue *apc.ActValRmNode) (xid string, err error)

func StartXaction 

func StartXaction(bp BaseParams, args *xact.ArgsMsg, extra string) (xid string, err error)

Start xaction

func StopMaintenance 

func StopMaintenance(bp BaseParams, actValue *apc.ActValRmNode) (xid string, err error)

func WaitForXactionIC 

func WaitForXactionIC(bp BaseParams, args *xact.ArgsMsg) (status *nl.Status, err error)

WaitForXactionIC waits for a given xaction to complete. Use it only for global xactions (those that execute on all targets and report their status to IC, e.g. rebalance).

func WaitForXactionIdle 

func WaitForXactionIdle(bp BaseParams, args *xact.ArgsMsg) (err error)

WaitForXactionIdle waits for a given on-demand xaction to be idle.

func WaitForXactionNode 

func WaitForXactionNode(bp BaseParams, args *xact.ArgsMsg, fn func(xact.MultiSnap) (bool, bool)) error

WaitForXactionNode waits for a given xaction to complete. Use for xactions that do _not_ report their status to IC members, namely: - xact.IdlesBeforeFinishing() - x-resilver (as it usually runs on a single node)

Types

type AppendArgs 

type AppendArgs struct {
	Reader     cos.ReadOpenCloser
	BaseParams BaseParams
	Bck        cmn.Bck
	Object     string
	Handle     string
	Size       int64
}

(see also: api.PutApndArchArgs)

type BaseParams 

type BaseParams struct {
	Client *http.Client
	URL    string
	Method string
	Token  string
	UA     string
}

type BinfoArgs added in v1.3.21 

type BinfoArgs struct {
	BsummArgs
	UUID          string
	FltPresence   int
	Summarize     bool
	WithRemote    bool
	DontAddRemote bool
}

type BsummArgs added in v1.3.21 

type BsummArgs struct {
	Callback  BsummCB
	CallAfter time.Duration
	DontWait  bool
}

type BsummCB added in v1.3.21 

type BsummCB func(*cmn.AllBsummResults, bool)

type FlushArgs 

type FlushArgs struct {
	Cksum      *cos.Cksum
	BaseParams BaseParams
	Bck        cmn.Bck
	Object     string
	Handle     string
}

PUT, APPEND, PROMOTE (object)

type GetArgs added in v1.3.16 

type GetArgs struct {
	// If not specified (or same: if `nil`), Writer defaults to `io.Discard`
	// (in other words, with no writer the object that is being read will be discarded)
	Writer io.Writer

	// Currently, this (optional) Query field can optionally carry:
	// 1. `apc.QparamETLName`: named ETL to transform the object (i.e., perform "inline transformation")
	// 2. `apc.QparamOrigURL`: GET from a vanilla http(s) location (`ht://` bucket with the corresponding `OrigURLBck`)
	// 3. `apc.QparamSilent`: do not log errors
	// 4. `apc.QparamLatestVer`: get latest version from the associated Cloud bucket; see also: `ValidateWarmGet`
	Query url.Values

	// The field is used to facilitate a) range read, and b) blob download
	// E.g. range:
	// * Header.Set(cos.HdrRange, fmt.Sprintf("bytes=%d-%d", fromOffset, toOffset))
	//   For range formatting, see https://www.rfc-editor.org/rfc/rfc7233#section-2.1
	// E.g. blob download:
	// * Header.Set(apc.HdrBlobDownload, "true")
	Header http.Header
}

GET (object)

type GetLogInput 

type GetLogInput struct {
	Writer   io.Writer
	Severity string // one of: {cmn.LogInfo, ...}
	Offset   int64
	All      bool
}

type ListArgs added in v1.3.18 

type ListArgs struct {
	Callback  LsoCB
	CallAfter time.Duration
	Limit     uint
}

additional and optional list-objects args (compare with: GetArgs, PutArgs)

type LsoCB added in v1.3.21 

type LsoCB func(*LsoCounter)

type LsoCounter added in v1.3.21 

type LsoCounter struct {
	// contains filtered or unexported fields
}

func (*LsoCounter) Count added in v1.3.21 

func (ctx *LsoCounter) Count() int

func (*LsoCounter) Elapsed added in v1.3.21 

func (ctx *LsoCounter) Elapsed() time.Duration

func (*LsoCounter) IsFinished added in v1.3.21 

func (ctx *LsoCounter) IsFinished() bool

type NewRequestCB 

type NewRequestCB func(args *cmn.HreqArgs) (*http.Request, error)

PUT, APPEND, PROMOTE (object)

type ObjAttrs added in v1.3.16 

type ObjAttrs struct {
	// contains filtered or unexported fields
}

`ObjAttrs` represents object attributes and can be further used to retrieve the object's size, checksum, version, and other metadata.

Note that while `GetObject()` and related GET APIs return `ObjAttrs`, `HeadObject()` API returns `cmn.ObjectProps` - a superset.

func GetObject 

func GetObject(bp BaseParams, bck cmn.Bck, objName string, args *GetArgs) (oah ObjAttrs, err error)

If GetArgs.Writer is specified GetObject will use it to write the response body; otherwise, it'll `io.Discard` the latter.

`io.Copy` is used internally to copy response bytes from the request to the writer.

Returns `ObjAttrs` that can be further used to get the size and other object metadata.

func GetObjectWithValidation 

func GetObjectWithValidation(bp BaseParams, bck cmn.Bck, objName string, args *GetArgs) (oah ObjAttrs, err error)

Same as above with checksum validation.

Returns `cmn.ErrInvalidCksum` when the expected and actual checksum values are different.

func PutObject 

func PutObject(args *PutArgs) (oah ObjAttrs, err error)

PutObject PUTs the specified reader (`args.Reader`) as a new object (or a new version of the object) it in the specified bucket.

Assumes that `args.Reader` is already opened and ready for usage. Returns `ObjAttrs` that can be further used to get the size and other object metadata.

func (*ObjAttrs) Attrs added in v1.3.16 

func (oah *ObjAttrs) Attrs() (out cmn.ObjAttrs)

func (*ObjAttrs) RespHeader added in v1.3.16 

func (oah *ObjAttrs) RespHeader() http.Header

e.g. usage: range read response

func (*ObjAttrs) Size added in v1.3.16 

func (oah *ObjAttrs) Size() int64

most often used (convenience) method

type PutApndArchArgs added in v1.3.18 

type PutApndArchArgs struct {
	ArchPath string // filename _in_ archive
	Mime     string // user-specified mime type (NOTE: takes precedence if defined)
	Flags    int64  // apc.ArchAppend and apc.ArchAppendIfExist (the former requires destination shard to exist)
	PutArgs
}

Archive files and directories (see related: cmn.ArchiveBckMsg)

type PutArgs added in v1.3.16 

type PutArgs struct {
	Reader cos.ReadOpenCloser

	// optional; if provided:
	// - if object exists: load the object's metadata, compare checksums - skip writing if equal
	// - otherwise, compare the two checksums upon writing (aka, "end-to-end protection")
	Cksum *cos.Cksum

	BaseParams BaseParams

	Bck     cmn.Bck
	ObjName string

	Size uint64 // optional

	// Skip loading existing object's metadata in order to
	// compare its Checksum and update its existing Version (if exists);
	// can be used to reduce PUT latency when:
	// - we massively write a new content into a bucket, and/or
	// - we simply don't care.
	SkipVC bool
}

PUT, APPEND, PROMOTE (object)

type ReqParams 

type ReqParams struct {
	Query      url.Values
	Header     http.Header
	BaseParams BaseParams
	Path       string

	// Authentication
	User     string
	Password string

	// amsg, lsmsg etc.
	Body []byte
	// contains filtered or unexported fields
}

ReqParams is used in constructing client-side API requests to aistore. Stores Query and Headers for providing arguments that are not used commonly in API requests 

See also: cmn.HreqArgs

func AllocRp 

func AllocRp() *ReqParams

func (*ReqParams) DoReqAny added in v1.3.16 

func (reqParams *ReqParams) DoReqAny(out any) (int, error)

Makes request via do(), decodes `resp.Body` into the `out` structure, closes the former, and returns the entire wrapped response (as well as `out`)

Returns an error if the response status >= 400.

func (*ReqParams) DoRequest 

func (reqParams *ReqParams) DoRequest() error

uses do() to make the request; if successful, checks, drains, and closes the response body

Source Files

View all Source files

Directories

Path Synopsis
Package apc: API control messages and constants
 Package apc: API control messages and constants
Package authn provides AuthN API over HTTP(S)
 Package authn provides AuthN API over HTTP(S)
Package env contains environment variables
 Package env contains environment variables

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.