Documentation
¶
Index ¶
Constants ¶
const ( // ModeReadWrite is to allow both read and write ModeReadWrite = Mode("RW") // ModeReadOnly is to allow only read operations ModeReadOnly = Mode("RO") // ModeWriteOnly is to allow only write operations ModeWriteOnly = Mode("WO") // FilesystemObjectStoreType holds the periodic config type for the filesystem store FilesystemObjectStoreType = "filesystem" // UploadInterval defines interval for when we check if there are new index files to upload. // It's also used to snapshot the currently written index tables so the snapshots can be used for reads. UploadInterval = 1 * time.Minute )
Variables ¶
This section is empty.
Functions ¶
func NewTableClient ¶
func NewTableClient(objectClient client.ObjectClient, storageKeyPrefix string) index.TableClient
NewTableClient creates a client for managing tables in object storage based index store. It is typically used when running a table manager.
Types ¶
type Config ¶
type Config struct {
ActiveIndexDirectory string `yaml:"active_index_directory"`
CacheLocation string `yaml:"cache_location"`
CacheTTL time.Duration `yaml:"cache_ttl"`
ResyncInterval time.Duration `yaml:"resync_interval"`
QueryReadyNumDays int `yaml:"query_ready_num_days"`
IndexGatewayClientConfig gatewayclient.IndexGatewayClientConfig `yaml:"index_gateway_client"`
IngesterName string
Mode Mode
IngesterDBRetainPeriod time.Duration
}
func (*Config) GetUniqueUploaderName ¶
GetUniqueUploaderName builds a unique uploader name using IngesterName + `-` + <nanosecond-timestamp>. The name is persisted in the configured ActiveIndexDirectory and reused when already exists.
func (*Config) RegisterFlags ¶
func (*Config) RegisterFlagsWithPrefix ¶
RegisterFlagsWithPrefix registers flags.
type IndexShipper ¶
type IndexShipper interface {
// AddIndex adds an immutable index to a logical table which would eventually get uploaded to the object store.
AddIndex(tableName, userID string, index index.Index) error
// ForEach lets us iterates through each index file in a table for a specific user.
// On the write path, it would iterate on the files given to the shipper for uploading, until they eventually get dropped from local disk.
// On the read path, it would iterate through the files if already downloaded else it would download and iterate through them.
ForEach(ctx context.Context, tableName, userID string, callback index.ForEachIndexCallback) error
ForEachConcurrent(ctx context.Context, tableName, userID string, callback index.ForEachIndexCallback) error
Stop()
}
func NewIndexShipper ¶
func NewIndexShipper(prefix string, cfg Config, storageClient client.ObjectClient, limits downloads.Limits, tenantFilter downloads.TenantFilter, open index.OpenIndexFileFunc, tableRangeToHandle config.TableRange, reg prometheus.Registerer, logger log.Logger) (IndexShipper, error)
NewIndexShipper creates a shipper for providing index store functionality using index files and object storage. It manages the whole life cycle of uploading the index and downloading the index at query time.
Since IndexShipper is generic, which means it can be used to manage various index types under the same object storage and/or local disk path, it accepts ranges of table numbers(config.TableRanges) to be managed by the shipper. This is mostly useful on the read path to sync and manage specific index tables within the given table number ranges.
Source Files
Directories