Documentation
¶
Overview ¶
Package kusto provides a Kusto client for accessing Kusto storage. Author: jdoak@microsoft.com
Example ¶
const (
// endpoint is the name of our Kusto endpoint.
endpoint = `https://clustername.kusto.windows.net`
// db is the name of the Kusto database.
db = "Reports"
// query is just an example query to use.
query = "userIDs | limit 100"
)
// Client ids and client secrets can be set in multiple places.
// For apps, usually these can be found in AD for the user.
// Tenant ids can be found in the subscription info.
const (
clientID = "clientID"
clientSecret = "clientSecret"
tenantID = "tenantID"
)
// UserID represents a user ID record stored in Kusto DB "Reports", table "userIDs".
type UserID struct {
// ID is a unique integer identifier for a user.
ID int64 `kusto:"Id"` // The tag creates a mapping from Kusto column "Id" to our field name "ID".
// UserName is the user's user name.
UserName string
// LastSeen represents the last time the system saw this user.
LastSeen time.Time
}
authConfig := auth.NewClientCredentialsConfig(clientID, clientSecret, tenantID)
// This creates a Reader, the most common way to interact with Kusto.
reader, err := New(endpoint, Authorization{Config: authConfig})
if err != nil {
panic(err)
}
ctx := context.Background()
// Queries the service and asks the service to use progressive mode, which provides a set of frame
// fragments instead of one large DataTable holding the results.
iter, err := reader.Query(ctx, db, query)
if err != nil {
panic(err)
}
defer iter.Stop()
// Loop through the iterated results, read them into our UserID structs and append them
// to our list of recs.
var recs []UserID
for {
row, err := iter.Next()
if err != nil {
// This indicates we are done.
if err == io.EOF {
break
}
// We ran into an error during the stream.
panic(err)
}
rec := UserID{}
if err := row.ToStruct(&rec); err != nil {
panic(err)
}
recs = append(recs, rec)
}
fmt.Println("100 User Records:")
fmt.Println(pretty.Sprint(recs))
Output:
Index ¶
- Constants
- type Authorization
- type Client
- type Column
- type Columns
- type DataScope
- type Option
- type QueryOption
- func AdminSuperSlackerMode() QueryOption
- func BlockSplitting() QueryOption
- func CustomQueryOption(paramName string, i interface{}) QueryOption
- func DatabasePattern(s string) QueryOption
- func DebugQueryExternalDataProjectionFusionDisabled() QueryOption
- func DebugQueryFanoutThreadsPercentExternalData(i int) QueryOption
- func DeferPartialQueryFailures() QueryOption
- func MaxMemoryConsumptionPerIterator(i uint64) QueryOption
- func MaxMemoryConsumptionPerQueryPerNode(i uint64) QueryOption
- func MaxOutputColumns(i int) QueryOption
- func NoRequestTimeout() QueryOption
- func NoTruncation() QueryOption
- func PushSelectionThroughAggregation() QueryOption
- func QueryCursorAfterDefault(s string) QueryOption
- func QueryCursorAllowReferencingStreamingIngestionTables() QueryOption
- func QueryCursorBeforeOrAtDefault(s string) QueryOption
- func QueryCursorCurrent(s string) QueryOption
- func QueryCursorScopedTables(l []string) QueryOption
- func QueryDataScope(ds DataScope) QueryOption
- func QueryDateTimeScopeColumn(s string) QueryOption
- func QueryDateTimeScopeFrom(t time.Time) QueryOption
- func QueryDateTimeScopeTo(t time.Time) QueryOption
- func ResultsProgressiveDisable() QueryOption
- func ServerTimeout(d time.Duration) QueryOption
- type Row
- type RowIterator
- func (r *RowIterator) Do(f func(r *Row) error) error
- func (r *RowIterator) Next() (*Row, error)
- func (r *RowIterator) NonPrimary(ctx context.Context, tableKind string, tableName string) (dataTable, error)
- func (r *RowIterator) Progress() float64
- func (r *RowIterator) Progressive() bool
- func (r *RowIterator) Stop()
- type Rows
Examples ¶
Constants ¶
const ( // CTBool indicates that a Column stores a Kusto boolean value. CTBool = "bool" // CTDateTime indicates that a Column stores a Kusto datetime value. CTDateTime = "datetime" // CTDynamic indicates that a Column stores a Kusto dynamic value. CTDynamic = "dynamic" // CTGUID indicates that a Column stores a Kusto guid value. CTGUID = "guid" // CTInt indicates that a Column stores a Kusto int value. CTInt = "int" // CTLong indicates that a Column stores a Kusto long value. CTLong = "long" // CTReal indicates that a Column stores a Kusto real value. CTReal = "real" // CTString indicates that a Column stores a Kusto string value. CTString = "string" // CTTimespan indicates that a Column stores a Kusto timespan value. CTTimespan = "timespan" // We have NOT written a conversion // CTDecimal indicates that a Column stores a Kusto decimal value. CTDecimal = "decimal" // We have NOT written a conversion )
These constants represent the value type stored in a Colunmn.
const ( TFDataAppend = "DataAppend" TFDataReplace = "DataReplace" )
const ( // DSDefault is used to set a query's datascope to default. DSDefault dataScope = "default" // DSAll is used to set a query's datascope to all. DSAll dataScope = "all" // DSHotCache is used to set a query's datascope to hotcache. DSHotCache dataScope = "hotcache" )
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Authorization ¶
type Authorization struct {
// Authorizer provides an authorizer to use when talking to Kusto. If this is set, the
// Authorizer must have its Resource (also called Resource ID) set to the endpoint passed
// to the New() constructor. This will be something like "https://somename.westus.kusto.windows.net".
Authorizer autorest.Authorizer
// Config provides the authorizer's config that can create the authorizer. We recommending setting
// this instead of Authorizer, as we will automatically set the Resource ID with the endpoint passed.
Config auth.AuthorizerConfig
}
Authorization provides the ADAL authorizer needed to access the resource. You can set Authorizer or Config, but not both.
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client is a client to a Kusto instance.
func New ¶
func New(endpoint string, auth Authorization, options ...Option) (*Client, error)
New returns a new Client.
func (*Client) Mgmt ¶
func (c *Client) Mgmt(ctx context.Context, db, query string, options ...QueryOption) (chan frame, error)
Mgmt is used to do management queries to Kusto.
func (*Client) Query ¶
func (c *Client) Query(ctx context.Context, db, query string, options ...QueryOption) (*RowIterator, error)
Query makes a query for the purpose of extracting data from Kusto. Context can be used to set a timeout or cancel the query. Queries cannot take longer than 5 minutes.
type Column ¶
type Column struct {
// ColumnName is the name of the column.
ColumnName string
// ColumnType is the type of value stored in this column. These are described
// via constants starting with CT<type>.
ColumnType string
}
Column describes a column descriptor.
type DataScope ¶
type DataScope interface {
// contains filtered or unexported methods
}
DataScope is used with QueryDataScope to control a query's datascope.
type Option ¶
type Option func(c *Client)
Option is an optional argument type for New().
func Timeout ¶
Timeout adjusts the maximum time any query can take from the client side. This defaults to 5 minutes. Note that the server has a timeout of 4 minutes for a query, 10 minutes for a management action. You will need to use the ServerTimeout() QueryOption in order to allow the server to take larger queries.
type QueryOption ¶
type QueryOption func(q *queryOptions)
QueryOption provides options for Query().
func AdminSuperSlackerMode ¶
func AdminSuperSlackerMode() QueryOption
AdminSuperSlackerMode delegate execution of the query to another node.
func BlockSplitting ¶
func BlockSplitting() QueryOption
BlockSplitting enables splitting of sequence blocks after aggregation operator
func CustomQueryOption ¶
func CustomQueryOption(paramName string, i interface{}) QueryOption
CustomQueryOption exists to allow a QueryOption that is not defined in the Go SDK, as all options are not defined. Please Note: you should always use the type safe options provided below when available. Also note that Kusto does not error on non-existent paramater names or bad values, it simply doesn't work as expected.
func DatabasePattern ¶
func DatabasePattern(s string) QueryOption
DatabasePattern overrides database name and picks the 1st database that matches the pattern. '*' means any database that user has access to.
func DebugQueryExternalDataProjectionFusionDisabled ¶
func DebugQueryExternalDataProjectionFusionDisabled() QueryOption
DebugQueryExternalDataProjectionFusionDisabled prevents fusing projection into ExternalData operator.
func DebugQueryFanoutThreadsPercentExternalData ¶
func DebugQueryFanoutThreadsPercentExternalData(i int) QueryOption
DebugQueryFanoutThreadsPercentExternalData sets percentage of threads to fanout execution to for external data nodes.
func DeferPartialQueryFailures ¶
func DeferPartialQueryFailures() QueryOption
DeferPartialQueryFailures disables reporting partial query failures as part of the result set.
func MaxMemoryConsumptionPerIterator ¶
func MaxMemoryConsumptionPerIterator(i uint64) QueryOption
MaxMemoryConsumptionPerIterator overrides the default maximum amount of memory a query operator may allocate.
func MaxMemoryConsumptionPerQueryPerNode ¶
func MaxMemoryConsumptionPerQueryPerNode(i uint64) QueryOption
MaxMemoryConsumptionPerQueryPerNode overrides the default maximum amount of memory a whole query may allocate per node.
func MaxOutputColumns ¶
func MaxOutputColumns(i int) QueryOption
MaxOutputColumns overrides the default maximum number of columns a query is allowed to produce.
func NoRequestTimeout ¶
func NoRequestTimeout() QueryOption
NoRequestTimeout enables setting the request timeout to its maximum value.
func NoTruncation ¶
func NoTruncation() QueryOption
NoTruncation enables suppressing truncation of the query results returned to the caller.
func PushSelectionThroughAggregation ¶
func PushSelectionThroughAggregation() QueryOption
PushSelectionThroughAggregation will push simple selection through aggregation .
func QueryCursorAfterDefault ¶
func QueryCursorAfterDefault(s string) QueryOption
QueryCursorAfterDefault sets the default parameter value of the cursor_after() function when called without parameters.
func QueryCursorAllowReferencingStreamingIngestionTables ¶
func QueryCursorAllowReferencingStreamingIngestionTables() QueryOption
QueryCursorAllowReferencingStreamingIngestionTables enable usage of cursor functions over databases which have streaming ingestion enabled.
func QueryCursorBeforeOrAtDefault ¶
func QueryCursorBeforeOrAtDefault(s string) QueryOption
QueryCursorBeforeOrAtDefault sets the default parameter value of the cursor_before_or_at() function when called without parameters.
func QueryCursorCurrent ¶
func QueryCursorCurrent(s string) QueryOption
QueryCursorCurrent overrides the cursor value returned by the cursor_current() or current_cursor() functions.
func QueryCursorScopedTables ¶
func QueryCursorScopedTables(l []string) QueryOption
QueryCursorScopedTables is a list of table names that should be scoped to cursor_after_default .. cursor_before_or_at_default (upper bound is optional).
func QueryDataScope ¶
func QueryDataScope(ds DataScope) QueryOption
QueryDataScope controls the query's datascope -- whether the query applies to all data or just part of it. ['default', 'all', or 'hotcache']
func QueryDateTimeScopeColumn ¶
func QueryDateTimeScopeColumn(s string) QueryOption
QueryDateTimeScopeColumn controls the column name for the query's datetime scope (query_datetimescope_to / query_datetimescope_from)
func QueryDateTimeScopeFrom ¶
func QueryDateTimeScopeFrom(t time.Time) QueryOption
QueryDateTimeScopeFrom controls the query's datetime scope (earliest) -- used as auto-applied filter on query_datetimescope_column only (if defined).
func QueryDateTimeScopeTo ¶
func QueryDateTimeScopeTo(t time.Time) QueryOption
QueryDateTimeScopeTo controls the query's datetime scope (latest) -- used as auto-applied filter on query_datetimescope_column only (if defined).
func ResultsProgressiveDisable ¶
func ResultsProgressiveDisable() QueryOption
ResultsProgressiveDisable disables the progressive query stream.
func ServerTimeout ¶
func ServerTimeout(d time.Duration) QueryOption
ServerTimeout is the amount of time the server will allow a query to take. Note that Kusto uses TimeSpans for durations, which are measured in ticks (100 nanonseconds). Sub tick durations (which are in nanoseconds) will get rounded.
type Row ¶
type Row struct {
// contains filtered or unexported fields
}
Row represents a row of Kusto data. Methods are not thread-safe.
func (*Row) ColumnNames ¶
ColumnNames returns a list of all column names.
func (*Row) Columns ¶
Columns fetches all the columns in the row at once. The value of the kth column will be decoded into the kth argument to Columns. The number of arguments must be equal to the number of columns. Pass nil to specify that a column should be ignored. ptrs may be either the *string or *types.Column type. An error in decoding may leave some ptrs set and others not.
func (*Row) ToStruct ¶
ToStruct fetches the columns in a row into the fields of a struct. p must be a pointer to struct. The rules for mapping a row's columns into a struct's exported fields are:
If a field has a `kusto: "column_name"` tag, then decode column 'column_name' into the field. A special case is the `column_name: "-"` tag, which instructs ToStruct to ignore the field during decoding.
Otherwise, if the name of a field matches the name of a column (ignoring case), decode the column into the field.
Slice and pointer fields will be set to nil if the source column is a null value, and a non-nil value if the column is not NULL. To decode NULL values of other types, use one of the kusto types (Int, Long, Dynamic, ...) as the type of the destination field. You can check the .Valid field of those types to see if the value was set.
func (*Row) Values ¶
func (r *Row) Values() types.KustoValues
Values returns a list of KustoValues that represent the row.
type RowIterator ¶
type RowIterator struct {
// contains filtered or unexported fields
}
RowIterator is used to iterate over the returned Row objects returned by Kusto.
func (*RowIterator) Do ¶
func (r *RowIterator) Do(f func(r *Row) error) error
Do calls f for every row returned by the query. If f returns a non-nil error, iteration stops.
func (*RowIterator) Next ¶
func (r *RowIterator) Next() (*Row, error)
Next gets the next Row from the query. io.EOF is returned if there are no more entries in the output. Once Next() returns an error, all subsequent calls will return the same error.
func (*RowIterator) NonPrimary ¶
func (r *RowIterator) NonPrimary(ctx context.Context, tableKind string, tableName string) (dataTable, error)
NonPrimary will return a non-primary dataTable if it exists from the last query. The non-primary table kinds are defined as constants starting with TK<name>. Returns io.ErrUnexpectedEOF if not found. May not have all tables until RowIterator has reached io.EOF.
func (*RowIterator) Progress ¶
func (r *RowIterator) Progress() float64
Progress returns the progress of the query, 0-100%. This is only valid on Progressive data returns.
func (*RowIterator) Progressive ¶
func (r *RowIterator) Progressive() bool
Progressive indicates if the RowIterator is unpacking progressive (streaming) frames.
func (*RowIterator) Stop ¶
func (r *RowIterator) Stop()
Stop is called to stop any further iteration. Always defer a Stop() call after receiving a RowIterator.
Source Files
Directories