druid

package module
v0.0.10 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 27, 2024 License: Apache-2.0 Imports: 16 Imported by: 0

README

Join us

go-druid

A Golang client for Druid. Now supports Query API and Common API.

Development
Testing

go-druid uses mage to run tests locally. Install Mage:

    git clone https://github.com/magefile/mage
    cd mage
    go run bootstrap.go

mage -l provides a list of targets that can be run. Default is Check

Targets:
  build            runs go mod download and then installs the binary.
  check*           run linters and tests
  fmt              run gofmt linter
  lint             run golint linter https://github.com/golang/lint
  testCoverHTML    generates test coverage report
  testRace         run tests with race detector
  vet              run go vet linter

* default target

Documentation

Index

Constants

View Source 
const (
	StatusEndpoint         = "status"
	HealthEndpoint         = "status/health"
	PropertiesEndpoint     = "status/properties"
	SelfDiscoveredEndpoint = "status/selfDiscovered/status"
)
View Source 
const (
	NativeQueryEndpoint = "druid/v2"
	SQLQueryEndpoint    = "druid/v2/sql"
)

Variables

This section is empty.

Functions

This section is empty.

Types

type AutoScalerConfig 

type AutoScalerConfig struct {
	EnableTaskAutoScaler                 bool    `json:"enableTaskAutoScaler"`
	LagCollectionIntervalMillis          int     `json:"lagCollectionIntervalMillis"`
	LagCollectionRangeMillis             int     `json:"lagCollectionRangeMillis"`
	ScaleOutThreshold                    int     `json:"scaleOutThreshold"`
	TriggerScaleOutFractionThreshold     float64 `json:"triggerScaleOutFractionThreshold"`
	ScaleInThreshold                     int     `json:"scaleInThreshold"`
	TriggerScaleInFractionThreshold      float64 `json:"triggerScaleInFractionThreshold"`
	ScaleActionStartDelayMillis          int     `json:"scaleActionStartDelayMillis"`
	ScaleActionPeriodMillis              int     `json:"scaleActionPeriodMillis"`
	TaskCountMax                         int     `json:"taskCountMax"`
	TaskCountMin                         int     `json:"taskCountMin"`
	ScaleInStep                          int     `json:"scaleInStep"`
	ScaleOutStep                         int     `json:"scaleOutStep"`
	MinTriggerScaleActionFrequencyMillis int     `json:"minTriggerScaleActionFrequencyMillis"`
}

AutoScalerConfig is part of IOConfig that controls ingestion auto-scaling.

type BitmapFactory 

type BitmapFactory struct {
	Type string `json:"type"`
}

BitmapFactory is a field of IndexSpec.

type Client 

type Client struct {
	// contains filtered or unexported fields
}

func NewClient 

func NewClient(baseURL string, options ...ClientOption) (*Client, error)

func (*Client) Close 

func (c *Client) Close() error

func (*Client) Common 

func (c *Client) Common() *CommonService

func (*Client) Do 

func (c *Client) Do(r *retryablehttp.Request, result interface{}) (*Response, error)

func (*Client) ExecuteRequest 

func (c *Client) ExecuteRequest(method, path string, opt, result interface{}) (*Response, error)

func (*Client) NewRequest 

func (c *Client) NewRequest(method, path string, opt interface{}) (*retryablehttp.Request, error)

func (*Client) Query 

func (c *Client) Query() *QueryService

func (*Client) Supervisor 

func (c *Client) Supervisor() *SupervisorService

func (*Client) Tasks 

func (c *Client) Tasks() *TasksService

type ClientOption 

type ClientOption func(*clientOptions)

func WithBasicAuth 

func WithBasicAuth(username, password string) ClientOption

func WithCustomBackoff 

func WithCustomBackoff(backoff retryablehttp.Backoff) ClientOption

func WithCustomErrorHandler 

func WithCustomErrorHandler(h retryablehttp.ErrorHandler) ClientOption

func WithCustomRetry 

func WithCustomRetry(retry retryablehttp.CheckRetry) ClientOption

func WithHTTPClient 

func WithHTTPClient(httpClient *http.Client) ClientOption

func WithRetryMax 

func WithRetryMax(retryMax int) ClientOption

func WithRetryWaitMax 

func WithRetryWaitMax(retryWaitMax time.Duration) ClientOption

func WithRetryWaitMin 

func WithRetryWaitMin(retryWaitMin time.Duration) ClientOption

func WithSkipTLSVerify 

func WithSkipTLSVerify() ClientOption

type CommonService 

type CommonService struct {
	// contains filtered or unexported fields
}

func (*CommonService) Health 

func (c *CommonService) Health() (*Health, *Response, error)

func (*CommonService) Properties 

func (c *CommonService) Properties() (*Properties, *Response, error)

func (*CommonService) SelfDiscovered 

func (c *CommonService) SelfDiscovered() (*SelfDiscovered, *Response, error)

func (*CommonService) Status 

func (c *CommonService) Status() (*Status, *Response, error)

type CompactionInputSpec 

type CompactionInputSpec struct {
	Type string `json:"type"`
	// CompactionIntervalSpec fields.
	Interval                 string `json:"interval,omitempty"`
	Sha256OfSortedSegmentIds string `json:"sha256OfSortedSegmentIds,omitempty"`
	// SpecificSegmentsSpec fields.
	Segments []string `json:"segments,omitempty"`
}

CompactionInputSpec is a specification for compaction task.

type ConnectorConfig 

type ConnectorConfig struct {
	ConnectURI string `json:"connectURI"`
	User       string `json:"user"`
	Password   string `json:"password"`
}

ConnectorConfig is connection configuration for Database.

type ConsumerProperties 

type ConsumerProperties struct {
	BootstrapServers           string                 `json:"bootstrap.servers,omitempty"`
	DruidDynamicConfigProvider *DynamicConfigProvider `json:"druid.dynamic.config.provider,omitempty"`
}

ConsumerProperties is a set of properties that is passed to a specific consumer, i.e. Kafka consumer.

type CreateSupervisorResponse 

type CreateSupervisorResponse struct {
	SupervisorID string `json:"id"`
}

CreateSupervisorResponse is a response object of Druid SupervisorService's SubmitTask method.

type DataSchema 

type DataSchema struct {
	DataSource      string           `json:"dataSource"`
	Parser          string           `json:"parser,omitempty"`
	TimeStampSpec   *TimestampSpec   `json:"timestampSpec,omitempty"`
	TransformSpec   *TransformSpec   `json:"transformSpec,omitempty"`
	DimensionsSpec  *DimensionsSpec  `json:"dimensionsSpec,omitempty"`
	GranularitySpec *GranularitySpec `json:"granularitySpec,omitempty"`
	MetricSpec      []Metric         `json:"metricSpec,omitempty"`
}

DataSchema represents the Druid dataSchema spec.

type Database 

type Database struct {
	Type            string           `json:"type"`
	ConnectorConfig *ConnectorConfig `json:"connectorConfig"`
}

Database configuration for InputSource "sql".

type Dimension 

type Dimension struct {
	Type string `json:"type"`
	Name string `json:"name"`
}

Dimension is a typed definition of a datasource dimension.

type DimensionExclusionsSet 

type DimensionExclusionsSet []string

DimensionExclusionsSet represents set of excluded dimensions.

type DimensionSet 

type DimensionSet []DimensionSpec

DimensionSet is a unique set of druid datasource dimensions(labels).

type DimensionSpec 

type DimensionSpec struct {
	Value any
}

DimensionSpec is a single dataset dimension that can be represented by a typed Dimension or a string value.

func (*DimensionSpec) MarshalJSON 

func (g *DimensionSpec) MarshalJSON() ([]byte, error)

func (*DimensionSpec) UnmarshalJSON 

func (g *DimensionSpec) UnmarshalJSON(b []byte) error

type DimensionsSpec 

type DimensionsSpec struct {
	Dimensions           DimensionSet           `json:"dimensions,omitempty"`
	DimensionExclusions  DimensionExclusionsSet `json:"dimensionExclusions,omitempty"`
	SpatialDimensions    SpatialDimensionSet    `json:"spatialDimensions,omitempty"`
	IncludeAllDimensions bool                   `json:"includeAllDimensions,omitempty"`
	UseSchemaDiscovery   bool                   `json:"useSchemaDiscovery,omitempty"`
}

DimensionsSpec is responsible for configuring Druid's dimensions. They're a set of columns in Druid's data model that can be used for grouping, filtering or applying aggregations. https://druid.apache.org/docs/latest/ingestion/ingestion-spec#dimensionsspec

type DynamicConfigProvider added in v0.0.10 

type DynamicConfigProvider struct {
	Value any
}

DynamicConfigProvider is an umbrella type for different DynamicConfigProviders.

func (*DynamicConfigProvider) MarshalJSON added in v0.0.10 

func (p *DynamicConfigProvider) MarshalJSON() ([]byte, error)

func (*DynamicConfigProvider) UnmarshalJSON added in v0.0.10 

func (p *DynamicConfigProvider) UnmarshalJSON(b []byte) error

type EnvironmentVariableDynamicConfigProvider added in v0.0.10 

type EnvironmentVariableDynamicConfigProvider struct {
	Type      string            `json:"type"`
	Variables map[string]string `json:"variables"`
}

EnvironmentVariableDynamicConfigProvider provides configuration values via environment variables.

type Field 

type Field struct {
	Type string `json:"type"`
	Name string `json:"name"`
	Expr string `json:"expr"`
}

Field defines a single filed configuration of the FlattenSpec.

type FieldList 

type FieldList []Field

FieldList is a list of Fields for ingestion FlattenSpec.

type Firehose 

type Firehose struct {
	Type string `json:"type,omitempty"`

	// EventReceiverFirehoseFactory fields.
	ServiceName string `json:"serviceName,omitempty"`
	BufferSize  int    `json:"bufferSize,omitempty"`
	MaxIdleTime int64  `json:"maxIdleTime,omitempty"`

	// FixedCountFirehoseFactory / ClippedFirehoseFactory / TimedShutoffFirehoseFactory fields.
	Delegate    []Firehose `json:"delegate,omitempty"`
	Count       int        `json:"count,omitempty"`
	Interval    string     `json:"interval,omitempty"`
	ShutoffTime string     `json:"shutoffTime,omitempty"`
}

Firehose is an IOConfig firehose configuration.

type FlattenSpec 

type FlattenSpec struct {
	UseFieldDiscovery bool      `json:"useFieldDiscovery,omitempty"`
	Fields            FieldList `json:"fields"`
}

FlattenSpec is responsible for flattening nested input JSON data into Druid's flat data model.

type GranularitySpec 

type GranularitySpec struct {
	Type               string                `json:"type"`
	SegmentGranularity string                `json:"segmentGranularity,omitempty"`
	QueryGranularity   *QueryGranularitySpec `json:"queryGranularity,omitempty"`
	Rollup             bool                  `json:"rollup"`
	Intervals          []string              `json:"intervals,omitempty"`
}

GranularitySpec allows for configuring operations such as data segment partitioning, truncating timestamps, time chunk segmentation or roll-up. https://druid.apache.org/docs/latest/ingestion/ingestion-spec#granularityspec

type Health 

type Health bool

type HttpInputSourceConfig 

type HttpInputSourceConfig struct {
	AllowedProtocols []string `json:" allowedProtocols,omitempty"`
}

HttpInputSourceConfig is a field of HttpInputSource specification.

type IOConfig 

type IOConfig struct {
	Type string `json:"type,omitempty"`

	// IndexIOConfig / RealtimeIOConfig shared field
	Firehose *Firehose `json:"firehose,omitempty"`
	// IndexIOConfig field
	InputSource      *InputSource `json:"inputSource,omitempty"`
	AppendToExisting bool         `json:"appendToExisting,omitempty"`
	// IndexIOConfig / CompactionIOConfig shared fields.
	DropExisting bool `json:"dropExisting,omitempty"`

	// CompactionIOConfig / HadoopIOConfig fields.
	InputSpec map[string]any `json:"inputSpec,omitempty"`

	// CompactionIOConfig fields.
	AllowNonAlignedInterval bool `json:"allowNonAlignedInterval,omitempty"`

	// HadoopIOConfig fields.
	MetadataUpdateSpec *MetadataStorageUpdaterJobSpec `json:"metadataUpdateSpec,omitempty"`
	SegmentOutputPath  string                         `json:"segmentOutputPath,omitempty"`

	// KafkaIndexTaskIOConfig / KinesisIndexTaskIOConfig fields.
	Topic                     string              `json:"topic,omitempty"`
	ConsumerProperties        *ConsumerProperties `json:"consumerProperties,omitempty"`
	TaskDuration              string              `json:"taskDuration,omitempty"`
	Replicas                  int                 `json:"replicas,omitempty"`
	TaskCount                 int                 `json:"taskCount,omitempty"`
	UseEarliestOffset         bool                `json:"useEarliestOffset"`
	AutoScalerConfig          *AutoScalerConfig   `json:"autoScalerConfig,omitempty"`
	TaskGroupID               int                 `json:"taskGroupID,omitempty"`
	BaseSequenceName          string              `json:"baseSequenceName,omitempty"`
	CompletionTimeout         string              `json:"completionTimeout,omitempty"`
	PollTimeout               int                 `json:"pollTimeout,omitempty"`
	StartDelay                string              `json:"startDelay,omitempty"`
	Period                    string              `json:"period,omitempty"`
	Stream                    string              `json:"stream,omitempty"`
	UseEarliestSequenceNumber bool                `json:"useEarliestSequenceNumber,omitempty"`

	// Common fields.
	FlattenSpec *FlattenSpec `json:"flattenSpec,omitempty"`
	InputFormat *InputFormat `json:"inputFormat,omitempty"`
	IdleConfig  *IdleConfig  `json:"idleConfig,omitempty"`
}

IOConfig influences how data is read into Druid from a source system. https://druid.apache.org/docs/latest/ingestion/ingestion-spec/#ioconfig

type IdleConfig 

type IdleConfig struct {
	Enabled             bool  `json:"enabled"`
	InactiveAfterMillis int64 `json:"inactiveAfterMillis"`
}

IdleConfig defines if and when stream Supervisor can become idle.

type IndexSpec 

type IndexSpec struct {
	Bitmap                 *BitmapFactory          `json:"bitmap,omitempty"`
	DimensionCompression   string                  `json:"dimensionCompression"`
	StringEncodingStrategy *StringEncodingStrategy `json:"stringEncodingStrategy,omitempty"`
	MetricCompression      string                  `json:"metricCompression,omitempty"`
	LongEncoding           string                  `json:"longEncoding,omitempty"`
	JsonCompression        string                  `json:"jsonCompression,omitempty"`
	SegmentLoader          string                  `json:"segmentLoader,omitempty"`
}

IndexSpec defines segment storage format options to be used at indexing time. https://druid.apache.org/docs/latest/ingestion/ingestion-spec#indexspec

type IngestionSpecData 

type IngestionSpecData struct {
	DataSchema   *DataSchema   `json:"dataSchema,omitempty"`
	IOConfig     *IOConfig     `json:"ioConfig,omitempty"`
	TuningConfig *TuningConfig `json:"tuningConfig,omitempty"`
}

IngestionSpecData is the core supervisor specification data returned by druid supervisor APIs. It is a part of OutputIngestionSpec.

type IngestionSpecOptions 

type IngestionSpecOptions func(*InputIngestionSpec)

IngestionSpecOptions allows for configuring a InputIngestionSpec.

func SetBrokers 

func SetBrokers(brokers string) IngestionSpecOptions

SetBrokers sets the addresses of Kafka brokers. in the list form: 'kafka01:9092, kafka02:9092,kafka03:9092' or as a cluster DNS: kafka.default.svc.cluster.local:9092”.

func SetDataSource 

func SetDataSource(ds string) IngestionSpecOptions

SetDataSource sets the name of the dataSource used in Druid.

func SetDimensions 

func SetDimensions(dimensions DimensionSet) IngestionSpecOptions

SetDimensions sets druid datasource dimensions.

func SetDimensionsAutodiscovery 

func SetDimensionsAutodiscovery(discover bool) IngestionSpecOptions

SetDimensionsAutodiscovery sets druid autodiscovery for datasource dimensions.

func SetEnvironmentDynamicConfigProvider added in v0.0.10 

func SetEnvironmentDynamicConfigProvider(dynamicConfig EnvironmentVariableDynamicConfigProvider) IngestionSpecOptions

func SetGranularitySpec 

func SetGranularitySpec(segmentGranularity string, queryGranularity *QueryGranularitySpec, rollup bool) IngestionSpecOptions

SetGranularitySpec sets granularity spec settings that are applied at druid ingestion partitioning stage.

func SetIOConfigType 

func SetIOConfigType(ioctype string) IngestionSpecOptions

SetIOConfigType sets the type of the supervisor IOConfig.

func SetInputFormat 

func SetInputFormat(format string) IngestionSpecOptions

SetInputFormat sets the input format type, i.e. json, protobuf etc.

func SetMapStringDynamicConfigProvider added in v0.0.10 

func SetMapStringDynamicConfigProvider(dynamicConfig MapStringDynamicConfigProvider) IngestionSpecOptions

func SetSQLInputSource 

func SetSQLInputSource(dbType, connectURI, user, password string, sqls []string) IngestionSpecOptions

SetSQLInputSource configures sql input source.

func SetTaskDuration 

func SetTaskDuration(duration string) IngestionSpecOptions

SetTaskDuration sets the upper limit for druid ingestion task.

func SetTimestampColumn 

func SetTimestampColumn(column string) IngestionSpecOptions

SetTimestampColumn sets timestamp column for the druid datasource.

func SetTopic 

func SetTopic(topic string) IngestionSpecOptions

SetTopic sets the Kafka topic to consume data from.

func SetType 

func SetType(stype string) IngestionSpecOptions

SetType sets the type of the supervisor (IOConfig).

func SetUseEarliestOffset 

func SetUseEarliestOffset(useEarliestOffset bool) IngestionSpecOptions

SetUseEarliestOffset configures kafka druid ingestion supervisor to start reading from the earliest or latest offsets in Kafka.

type InputFormat 

type InputFormat struct {
	Type string `json:"type"`

	// FlatTextInputFormat / DelimitedInputFormat fields.
	Delimiter             string   `json:"delimiter,omitempty"`
	ListDelimiter         string   `json:"listDelimiter,omitempty"`
	FindColumnsFromHeader string   `json:"findColumnsFromHeader,omitempty"`
	SkipHeaderRows        int      `json:"skipHeaderRows,omitempty"`
	Columns               []string `json:"columns,omitempty"`

	// JsonInputFormat fields.
	FlattenSpec *FlattenSpec    `json:"flattenSpec,omitempty"`
	FeatureSpec map[string]bool `json:"featureSpec,omitempty"`

	// Common CsvInputFormat / JsonInputFormat fields.
	KeepNullColumns        bool `json:"keepNullColumns,omitempty"`
	AssumeNewlineDelimited bool `json:"assumeNewlineDelimited,omitempty"`
	UseJsonNodeReader      bool `json:"useJsonNodeReader,omitempty"`
}

InputFormat specifies kafka messages format type and describes any conversions applied to the input data while parsing. Type can take values 'json', 'protobuf' or 'kafka'.

type InputIngestionSpec 

type InputIngestionSpec struct {
	Type         string        `json:"type"`
	DataSchema   *DataSchema   `json:"dataSchema,omitempty"`
	IOConfig     *IOConfig     `json:"ioConfig,omitempty"`
	TuningConfig *TuningConfig `json:"tuningConfig,omitempty"`
}

InputIngestionSpec is the root-level type defining an ingestion spec used by Apache Druid.

func NewIngestionSpec 

func NewIngestionSpec(options ...IngestionSpecOptions) *InputIngestionSpec

NewIngestionSpec returns a default InputIngestionSpec and applies any options passed to it.

type InputSource 

type InputSource struct {
	Type string `json:"type"`

	// LocalInputSource fields.
	BaseDir string   `json:"baseDir,omitempty"`
	Filter  string   `json:"filter,omitempty"`
	Files   []string `json:"files,omitempty"`

	// HttpInputSource fields.
	URIs                       []string               `json:"uris,omitempty"`
	HttpAuthenticationUsername string                 `json:"httpAuthenticationUsername,omitempty"`
	HttpAuthenticationPassword string                 `json:"httpAuthenticationPassword,omitempty"`
	HttpSourceConfig           *HttpInputSourceConfig `json:"config,omitempty"`

	// InlineInputSource fields.
	Data string `json:"data,omitempty"`

	// CombiningInputSource fields.
	Delegates []InputSource `json:"delegates,omitempty"`

	// SqlInputSource.
	SQLs     []string  `json:"sqls,omitempty"`
	Database *Database `json:"database,omitempty"`

	// Druid input source.
	Datasource string `json:"dataSource,omitempty"`
	// ISO-8601 interval, which defines the time range to fetch the data over.
	Interval string   `json:"interval,omitempty"`
	Metrics  []string `json:"metrics,omitempty"`
}

InputSource is a specification of the storage system where input data is stored.

type MapStringDynamicConfigProvider added in v0.0.10 

type MapStringDynamicConfigProvider struct {
	Config map[string]string `json:"config"`
}

MapStringDynamicConfigProvider passes config values as a <key, value> map.

type MetadataStorageUpdaterJobSpec 

type MetadataStorageUpdaterJobSpec struct {
	Type           string         `json:"type"`
	ConnectURI     string         `json:"connectURI"`
	User           string         `json:"user"`
	Password       string         `json:"password"`
	SegmentTable   string         `json:"segmentTable"`
	CreteTable     bool           `json:"creteTable"`
	Host           string         `json:"host"`
	Port           string         `json:"port"`
	DBCPProperties map[string]any `json:"dbcp"`
}

MetadataStorageUpdaterJobSpec is a specification of endpoint for HadoopIOConfig.

type Metric 

type Metric struct {
	Name      string `json:"name"`
	Type      string `json:"type"`
	FieldName string `json:"fieldName"`
}

Metric is a Druid aggregator that is applied at ingestion time. https://druid.apache.org/docs/latest/ingestion/ingestion-spec#metricsspec

type OutputIngestionSpec 

type OutputIngestionSpec struct {
	Type      string             `json:"type"`
	Context   string             `json:"context"`
	Suspended bool               `json:"suspended"`
	Spec      *IngestionSpecData `json:"spec"`
}

OutputIngestionSpec is full supervisor specification format returned by druid supervisor APIs.

type Properties 

type Properties map[string]string

type QueryGranularity 

type QueryGranularity struct {
	Type string `json:"type,omitempty"`
}

QueryGranularity is a typed representation of query granularity.

type QueryGranularitySpec 

type QueryGranularitySpec struct {
	Value any
}

QueryGranularitySpec is an umbrella type for different representations of query granularity, can be string or QueryGranularity value.

func (*QueryGranularitySpec) MarshalJSON 

func (g *QueryGranularitySpec) MarshalJSON() ([]byte, error)

func (*QueryGranularitySpec) UnmarshalJSON 

func (g *QueryGranularitySpec) UnmarshalJSON(b []byte) error

type QueryService 

type QueryService struct {
	// contains filtered or unexported fields
}

func (*QueryService) Execute 

func (q *QueryService) Execute(qry builder.Query, result interface{}, headers ...http.Header) (*Response, error)

func (*QueryService) Load 

func (q *QueryService) Load(data []byte) (builder.Query, error)

type Response 

type Response struct {
	*http.Response
}

func (*Response) ExtractError 

func (r *Response) ExtractError() error

type RunningTask 

type RunningTask struct {
	ID         string `json:"id"`
	Type       string `json:"type"`
	Status     string `json:"status"`
	Datasource string `json:"dataSource"`
}

RunningTask defines running task returned by GetRunningTasks method. https://druid.apache.org/docs/latest/api-reference/tasks-api#sample-response-2

type RunningTasksOptions 

type RunningTasksOptions struct {
	Datasource string `url:"datasource"`
	Type       string `url:"type"`
}

RunningTasksOptions defines supported options which can be passed to GetRunningTasks method. https://druid.apache.org/docs/latest/api-reference/tasks-api#query-parameters-2

type SelfDiscovered 

type SelfDiscovered struct {
	SelfDiscovered bool `json:"selfDiscovered"`
}

type ShutdownTaskResponse 

type ShutdownTaskResponse struct {
	Task string `json:"task"`
}

ShutdownTaskResponse is a response object of Druid SupervisorService's Terminate method.

type SpatialDimension 

type SpatialDimension struct {
	DimensionName string   `json:"dimName"`
	Dimensions    []string `json:"dims,omitempty"`
}

SpatialDimension represents single spatial dimension datum. https://druid.apache.org/docs/latest/querying/geo/#spatial-indexing

type SpatialDimensionSet 

type SpatialDimensionSet []SpatialDimension

SpatialDimensionSet is a unique set of druid datasource spatial dimensions.

type Status 

type Status struct {
	Version string `json:"version"`
	Modules []struct {
		Name     string `json:"name"`
		Artifact string `json:"artifact"`
		Version  string `json:"version"`
	} `json:"modules"`
	Memory struct {
		MaxMemory    int `json:"maxMemory"`
		TotalMemory  int `json:"totalMemory"`
		FreeMemory   int `json:"freeMemory"`
		UsedMemory   int `json:"usedMemory"`
		DirectMemory int `json:"directMemory"`
	} `json:"memory"`
}

type StringEncodingStrategy 

type StringEncodingStrategy struct {
	Type string `json:"type"`
	// FrontCoded fields.
	BucketSize    int `json:"bucketSize,omitempty"`
	FormatVersion int `json:"formatVersion,omitempty"`
}

StringEncodingStrategy type for specifying string encoding at indexing stage.

type SubmitTaskResponse 

type SubmitTaskResponse struct {
	Task string `json:"task"`
}

SubmitTaskResponse is a response object of Druid Task API Submit task method.

type SupervisorAuditHistory 

type SupervisorAuditHistory struct {
	Spec    OutputIngestionSpec `json:"spec"`
	Version string              `json:"version"`
}

SupervisorAuditHistory is audit data for supervisor reurned by supervisor audit history APIs.

type SupervisorService 

type SupervisorService struct {
	// contains filtered or unexported fields
}

SupervisorService is a service that submits ingestion tasks to druid supervisor API.

func (*SupervisorService) CreateOrUpdate 

func (s *SupervisorService) CreateOrUpdate(spec *InputIngestionSpec) (string, error)

CreateOrUpdate submits an ingestion specification to druid Supervisor API with a pre-configured druid client. https://druid.apache.org/docs/latest/api-reference/supervisor-api/#create-or-update-a-supervisor

func (*SupervisorService) GetActiveIDs 

func (s *SupervisorService) GetActiveIDs() ([]string, error)

GetActiveIDs returns array of active supervisor IDs. https://druid.apache.org/docs/latest/api-reference/supervisor-api/#get-an-array-of-active-supervisor-ids

func (*SupervisorService) GetActiveStates 

func (s *SupervisorService) GetActiveStates() ([]SupervisorState, error)

GetAllActiveStates returns an array of supervisor states. https://druid.apache.org/docs/latest/api-reference/supervisor-api/#get-an-array-of-supervisor-states

func (*SupervisorService) GetAllActiveSupervisors 

func (s *SupervisorService) GetAllActiveSupervisors() ([]SupervisorStateWithSpec, error)

GetAllActiveSupervisors returns array of active supervisor objects. https://druid.apache.org/docs/latest/api-reference/supervisor-api/#get-an-array-of-active-supervisor-objects

func (*SupervisorService) GetAuditHistory 

func (s *SupervisorService) GetAuditHistory(string) (SupervisorAuditHistory, error)

GetAuditHistory calls druid Supervisor service's AuditHistory API. https://druid.apache.org/docs/latest/api-reference/supervisor-api/#get-audit-history-for-a-specific-supervisor

func (*SupervisorService) GetAuditHistoryAll 

func (s *SupervisorService) GetAuditHistoryAll() (map[string]SupervisorAuditHistory, error)

GetAuditHistoryAll calls druid Supervisor service's AuditHistoryAll API. https://druid.apache.org/docs/latest/api-reference/supervisor-api/#get-audit-history-for-all-supervisors

func (*SupervisorService) GetSpec 

func (s *SupervisorService) GetSpec(supervisorId string) (OutputIngestionSpec, error)

GetSpec calls druid Supervisor Status API. https://druid.apache.org/docs/latest/api-reference/supervisor-api/#get-supervisor-specification

func (*SupervisorService) GetStatus 

func (s *SupervisorService) GetStatus(supervisorId string) (SupervisorStatus, error)

GetStatus calls druid Supervisor service's Get status API. https://druid.apache.org/docs/latest/api-reference/supervisor-api/#get-supervisor-status

func (*SupervisorService) Reset 

func (s *SupervisorService) Reset(string) (string, error)

Reset calls druid Supervisor service's Reset API. https://druid.apache.org/docs/latest/api-reference/supervisor-api/#reset-a-supervisor

func (*SupervisorService) Resume 

func (s *SupervisorService) Resume(supervisorID string) (OutputIngestionSpec, error)

Resume calls druid Supervisor service's Resume API. https://druid.apache.org/docs/latest/api-reference/supervisor-api/#resume-a-supervisor

func (*SupervisorService) ResumeAll 

func (s *SupervisorService) ResumeAll() (string, error)

ResumeAll calls druid Supervisor service's ResumeAll API. https://druid.apache.org/docs/latest/api-reference/supervisor-api/#resume-all-supervisors

func (*SupervisorService) Shutdown 

func (s *SupervisorService) Shutdown(string) (string, error)

Shutdown calls druid Supervisor service's Shutdown API. https://druid.apache.org/docs/latest/api-reference/supervisor-api/#shut-down-a-supervisor

func (*SupervisorService) Suspend 

func (s *SupervisorService) Suspend(supervisorID string) (OutputIngestionSpec, error)

Suspend calls druid Supervisor service's Suspend API. https://druid.apache.org/docs/latest/api-reference/supervisor-api/#suspend-a-running-supervisor

func (*SupervisorService) SuspendAll 

func (s *SupervisorService) SuspendAll() (string, error)

SuspendAll calls druid Supervisor service's SuspendAll API. https://druid.apache.org/docs/latest/api-reference/supervisor-api/#suspend-all-supervisors

func (*SupervisorService) Terminate 

func (s *SupervisorService) Terminate(supervisorId string) (string, error)

Terminate calls druid Supervisor service's Terminate API. https://druid.apache.org/docs/latest/api-reference/supervisor-api/#terminate-a-supervisor

func (*SupervisorService) TerminateAll 

func (s *SupervisorService) TerminateAll() (string, error)

TerminateAll calls druid Supervisor service's TerminateAll API. https://druid.apache.org/docs/latest/api-reference/supervisor-api/#terminate-all-supervisors

type SupervisorState 

type SupervisorState struct {
	ID            string `json:"id"`
	State         string `json:"state"`
	DetailedState string `json:"detailedState"`
	Healthy       bool   `json:"healthy"`
	Suspended     bool   `json:"suspended"`
}

SupervisorState is a short form of supervisor state returned by druid APIs.

type SupervisorStateWithSpec 

type SupervisorStateWithSpec struct {
	SupervisorState
	Spec *InputIngestionSpec `json:"spec"`
}

type SupervisorStatus 

type SupervisorStatus struct {
	SupervisorId   string                   `json:"id"`
	GenerationTime string                   `json:"generationTime"`
	Payload        *SupervisorStatusPayload `json:"payload"`
}

SupervisorStatus is a response object containing status of a supervisor alongside with the response metadata.

type SupervisorStatusPayload 

type SupervisorStatusPayload struct {
	Datasource      string `json:"dataSource"`
	Stream          string `json:"stream"`
	State           string `json:"state"`
	Partitions      int    `json:"partitions"`
	Replicas        int    `json:"replicas"`
	DurationSeconds int    `json:"durationSeconds"`
	Suspended       bool   `json:"suspended"`
}

SupervisorStatusPayload is an object representing the status of supervisor.

type TaskIngestionSpec 

type TaskIngestionSpec struct {
	Type string             `json:"type"`
	Spec *IngestionSpecData `json:"spec"`
}

TaskIngestionSpec is a specification for a druid task execution.

func NewTaskIngestionSpec 

func NewTaskIngestionSpec(options ...TaskIngestionSpecOptions) *TaskIngestionSpec

NewTaskIngestionSpec returns a default TaskIngestionSpec and applies any options passed to it.

type TaskIngestionSpecOptions 

type TaskIngestionSpecOptions func(*TaskIngestionSpec)

TaskIngestionSpecOptions allows for configuring a TaskIngestionSpec.

func SetForceGuaranteedRollup 

func SetForceGuaranteedRollup(rollup bool) TaskIngestionSpecOptions

SetForceGuaranteedRollup sets guaranteed rollup setting for ingestion spec. https://druid.apache.org/docs/latest/ingestion/rollup#perfect-rollup-vs-best-effort-rollup

func SetTaskDataDimensions 

func SetTaskDataDimensions(dimensions DimensionSet) TaskIngestionSpecOptions

SetTaskDataDimensions sets druid datasource dimensions.

func SetTaskDataSource 

func SetTaskDataSource(datasource string) TaskIngestionSpecOptions

SetTaskDataSource sets the destination datasource of the task IOConfig.

func SetTaskDruidInputSource 

func SetTaskDruidInputSource(datasource string, startTime time.Time, endTime time.Time) TaskIngestionSpecOptions

SetTaskDruidInputSource configures druid reindex input source for the task based ingestion.

func SetTaskGranularitySpec 

func SetTaskGranularitySpec(segmentGranularity string, queryGranularity *QueryGranularitySpec, rollup bool) TaskIngestionSpecOptions

SetTaskGranularitySpec sets granularity spec settings that are applied at druid ingestion partitioning stage.

func SetTaskIOConfigType 

func SetTaskIOConfigType(typ string) TaskIngestionSpecOptions

SetTaskIOConfigType sets the type of the task IOConfig.

func SetTaskInlineInputData 

func SetTaskInlineInputData(data string) TaskIngestionSpecOptions

SetTaskInlineInputData configures inline data for the task based ingestion.

func SetTaskInputFormat 

func SetTaskInputFormat(typ string, findColumnsHeader string, columns []string) TaskIngestionSpecOptions

SetTaskInputFormat configures input format for the task based ingestion.

func SetTaskSQLInputSource 

func SetTaskSQLInputSource(typ, connectURI, user, password string, sqls []string) TaskIngestionSpecOptions

SetTaskSQLInputSource configures sql input source for the task based ingestion.

func SetTaskSchemaDiscovery 

func SetTaskSchemaDiscovery(discovery bool) TaskIngestionSpecOptions

SetTaskSchemaDiscovery sets auto discovery of dimensions.

func SetTaskTimestampColumn 

func SetTaskTimestampColumn(column string) TaskIngestionSpecOptions

SetTaskTimestampColumn sets the type of the task IOConfig.

func SetTaskTuningConfig 

func SetTaskTuningConfig(typ string, maxRowsInMemory, maxRowsPerSegment int) TaskIngestionSpecOptions

SetTaskTuningConfig sets the tuning configuration the task IOConfig.

func SetTaskType 

func SetTaskType(stype string) TaskIngestionSpecOptions

SetTaskType sets the type of the task IOConfig.

type TaskLocation 

type TaskLocation struct {
	Host    string `json:"host"`
	Port    int    `json:"port"`
	TlsPort int    `json:"tlsPort"`
}

TaskLocation holds location of the task execution.

type TaskStatus 

type TaskStatus struct {
	ID                 string        `json:"id"`
	Type               string        `json:"type"`
	CreatedTime        string        `json:"createdTime"`
	QueueInsertionTime string        `json:"queueInsertionTime"`
	StatusCode         string        `json:"statusCode"`
	Status             string        `json:"status"`
	RunnerStatusCode   string        `json:"runnerStatusCode"`
	Duration           int           `json:"duration"`
	GroupId            string        `json:"groupId"`
	Location           *TaskLocation `json:"location|omitempty"`
	Datasource         string        `json:"datasource"`
	ErrorMessage       string        `json:"errorMessage"`
}

TaskStatus is an object representing status of a druid task.

type TaskStatusResponse 

type TaskStatusResponse struct {
	Task   string     `json:"task"`
	Status TaskStatus `json:"status"`
}

TaskStatusResponse is a response object containing status of a task.

type TasksService 

type TasksService struct {
	// contains filtered or unexported fields
}

TasksService is a service that runs requests to druid tasks API.

func (*TasksService) GetRunningTasks 

func (s *TasksService) GetRunningTasks(options RunningTasksOptions) ([]*RunningTask, error)

GetRunningTasks calls druid task service's running tasks API. https://druid.apache.org/docs/latest/api-reference/tasks-api#get-an-array-of-running-tasks

func (*TasksService) GetStatus 

func (s *TasksService) GetStatus(taskId string) (TaskStatusResponse, error)

GetStatus calls druid tasks service's Get status API. https://druid.apache.org/docs/latest/api-reference/tasks-api/#get-task-status

func (*TasksService) Shutdown 

func (s *TasksService) Shutdown(taskId string) (string, error)

Shutdown calls druid task service's shutdown task API. https://druid.apache.org/docs/latest/api-reference/tasks-api/#shut-down-a-task

func (*TasksService) SubmitTask 

func (s *TasksService) SubmitTask(spec *TaskIngestionSpec) (string, error)

SubmitTask submits an ingestion specification to druid tasks API with a pre-configured druid client. https://druid.apache.org/docs/latest/api-reference/tasks-api/#submit-a-task

type TerminateSupervisorResponse 

type TerminateSupervisorResponse struct {
	SupervisorID string `json:"id"`
}

TerminateSupervisorResponse is a response object of Druid SupervisorService's Terminate method.

type TimestampSpec 

type TimestampSpec struct {
	Column string `json:"column"`
	Format string `json:"format"`
}

TimestampSpec is responsible for configuring the primary timestamp.

type Transform 

type Transform struct {
	Type string `json:"type"`
	Name string `json:"name"`
	Expr string `json:"expression"`
}

Transform defines a single filed transformation of the TransformSpec.

type TransformSet 

type TransformSet []Transform

TransformSet is a unique set of transforms applied to the input.

type TransformSpec 

type TransformSpec struct {
	Transforms TransformSet `json:"transforms"`
}

TransformSpec is responsible for transforming druid input data after it was read from kafka and after flattenSpec was applied. https://druid.apache.org/docs/latest/ingestion/ingestion-spec#transformspec

type TuningConfig 

type TuningConfig struct {
	Type                             string     `json:"type"`
	IntermediatePersistPeriod        string     `json:"intermediatePersistPeriod,omitempty"`
	MaxRowsPerSegment                int        `json:"maxRowsPerSegment,omitempty"`
	MaxRowsInMemory                  int        `json:"maxRowsInMemory,omitempty"`
	IndexSpecForIntermediatePersists *IndexSpec `json:"indexSpecForIntermediatePersists,omitempty"`
	ForceGuaranteedRollup            bool       `json:"forceGuaranteedRollup,omitempty"`
}

TuningConfig controls various tuning parameters specific to each ingestion method. https://druid.apache.org/docs/latest/ingestion/ingestion-spec#tuningconfig

Source Files

View all Source files

Directories

Path Synopsis
aggregation
bound
datasource
dimension
extractionfn
filter
granularity
havingspec
intervals
limitspec
lookup
postaggregation
query
searchqueryspec
testutil
toinclude
topnmetric
types
virtualcolumn

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.