devstatscode

package module
v0.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 13, 2019 License: Apache-2.0 Imports: 29 Imported by: 1

README

Build Status CII Best Practices

DevStats code

This is a code reposotory for DevStats used to display CNCF projects dashboards, CDF projects dashboards, GraphQL projects dashboards and example Kubernetes/helm deployment.

Authors: Łukasz Gryglicki lgryglicki@cncf.io, Justyna Gryglicka lgryglicka@cncf.io, Josh Berkus jberkus@redhat.com.

Building and installing

  • Follow this guide to see how to deploy on Kubernetes using Helm.
  • Follow this guide to see GraphQL deployment using Kubernetes & Helm.
  • Follow this guide for installing on bare metal.
  • Follow this guide to deploy your oiwn project on bare metal (this example deployes Homebrew statistics).
  • Fetch dependency libraries.
  • make then make test finally make install.

Documentation

Index

Constants

View Source 
const Abuse string = "abuse"

Abuse - common constant string

View Source 
const All string = "all"

All - common constant string

View Source 
const DefaultDataDir string = "/etc/gha2db/"

DefaultDataDir - common constant string

View Source 
const Devstats string = "devstats"

Devstats - common constant string

View Source 
const DevstatsCode string = "devstatscode"

DevstatsCode - common constant string

View Source 
const EngineIsClosedError string = "engine is closed"

EngineIsClosedError - common constant string

View Source 
const GHA string = "gha"

GHA - common constant string

View Source 
const GHAAdmin string = "gha_admin"

GHAAdmin - common constant string

View Source 
const HideCfgFile string = "hide/hide.csv"

HideCfgFile - common constant string

View Source 
const InvalidCatalogName string = "invalid_catalog_name"

InvalidCatalogName - common constant string

View Source 
const Kubernetes string = "kubernetes"

Kubernetes - common constant string

View Source 
const LocalGitScripts string = "./git/"

LocalGitScripts - common constant string

View Source 
const Localhost string = "localhost"

Localhost - common constant string

View Source 
const Merged string = "merged"

Merged - common constant string

View Source 
const Metrics string = "metrics/"

Metrics - common constant string

View Source 
const Nil string = "(nil)"

Nil - common constant string

View Source 
const NotFound string = "not_found"

NotFound - common constant string

View Source 
const Now string = "now"

Now - common constant string

View Source 
const Null string = "null"

Null - common constant string

View Source 
const Password string = "password"

Password - common constant string

View Source 
const Quarter string = "quarter"

Quarter - common constant string

View Source 
const Retry string = "retry"

Retry - common constant string

View Source 
const TimeCol string = "time"

TimeCol - common constant string

View Source 
const TimeoutError string = "{\"error\":\"timeout\"}\n"

TimeoutError - common constant string

View Source 
const Today string = "today"

Today - common constant string

View Source 
const Unset string = "{{unset}}"

Unset - common constant string

Variables

This section is empty.

Functions

func ActorHit 

func ActorHit(ctx *Ctx, actorName string) bool

ActorHit - are we intereste din this actor?

func ActorIDOrNil 

func ActorIDOrNil(actPtr *Actor) interface{}

ActorIDOrNil - return Actor ID from pointer or nil

func ActorLoginOrNil 

func ActorLoginOrNil(actPtr *Actor, maybeHide func(string) string) interface{}

ActorLoginOrNil - return Actor Login from pointer or nil

func AddNIntervals 

func AddNIntervals(dt time.Time, n int, nextIntervalStart, prevIntervalStart func(time.Time) time.Time) time.Time

AddNIntervals adds (using nextIntervalStart) or subtracts (using prevIntervalStart) N itervals to the given date Functions Next/Prev can use Hour, Day, Week, Month, Quarter, Year functions (defined in this module) or other custom defined functions With `func(time.Time) time.Time` signature

func AddTSPoint 

func AddTSPoint(ctx *Ctx, pts *TSPoints, pt TSPoint)

AddTSPoint add single point to the batch

func ArtificialEvent 

func ArtificialEvent(c *sql.DB, ctx *Ctx, cfg *IssueConfig) (err error)

ArtificialEvent - create artificial API event (but from the past)

func ArtificialPREvent 

func ArtificialPREvent(c *sql.DB, ctx *Ctx, cfg *IssueConfig, pr *github.PullRequest) (err error)

ArtificialPREvent - create artificial API event (PR state for now())

func BoolOrNil 

func BoolOrNil(boolPtr *bool) interface{}

BoolOrNil - return either nil or value of boolPtr

func CleanUTF8 

func CleanUTF8(str string) string

CleanUTF8 - clean UTF8 string to containg only Pq allowed runes

func ClearDBLogs 

func ClearDBLogs()

ClearDBLogs clears logs older by defined period (in context.go) It clears logs on `devstats` database

func CommentIDOrNil 

func CommentIDOrNil(commPtr *Comment) interface{}

CommentIDOrNil - return Comment ID from pointer or nil

func CompareFloat64Ptr 

func CompareFloat64Ptr(p1, p2 *float64) bool

CompareFloat64Ptr - compares two string pointers returns true if both pointers are null or both are non null and point to floats with the same value (with epsilon)

func CompareIntPtr 

func CompareIntPtr(p1, p2 *int) bool

CompareIntPtr - compares two int pointers returns true if both pointers are null or both are non null and point to integers with the same value

func CompareStringPtr 

func CompareStringPtr(p1, p2 *string) bool

CompareStringPtr - compares two string pointers returns true if both pointers are null or both are non null and point to strings with the same content

func ComputePeriodAtThisDate 

func ComputePeriodAtThisDate(ctx *Ctx, period string, idt time.Time, hist bool) bool

ComputePeriodAtThisDate - for some longer periods, only recalculate them on specific dates/times see: time_test.go

func CreateDatabaseIfNeeded 

func CreateDatabaseIfNeeded(ctx *Ctx) bool

CreateDatabaseIfNeeded - creates requested database if not exists Returns true if database was not existing existed and created dropped

func CreateDatabaseIfNeededExtended 

func CreateDatabaseIfNeededExtended(ctx *Ctx, extraParams string) bool

CreateDatabaseIfNeededExtended - creates requested database if not exists Returns true if database was not existing existed and created dropped Allows specifying additional database parameters

func CreateTable 

func CreateTable(tdef string) string

CreateTable is used to replace DB specific parts of Create Table SQL statement

func DatabaseExists 

func DatabaseExists(ctx *Ctx, closeConn bool) (exists bool, c *sql.DB)

DatabaseExists - checks if database stored in context exists If closeConn is true - then it closes connection after checking if database exists If closeConn is false, then it returns open connection to default database "postgres"

func DayStart 

func DayStart(dt time.Time) time.Time

DayStart - return time rounded to current day start

func DeleteArtificialEvent 

func DeleteArtificialEvent(c *sql.DB, ctx *Ctx, cfg *IssueConfig) (err error)

DeleteArtificialEvent - create artificial API event (but from the past)

func DeleteArtificialPREvent 

func DeleteArtificialPREvent(c *sql.DB, ctx *Ctx, cfg *IssueConfig) (err error)

DeleteArtificialPREvent - create artificial API event (but from the past)

func DescriblePeriodInHours 

func DescriblePeriodInHours(hrs float64) (desc string)

DescriblePeriodInHours - return string description of a time period given in hours

func DropDatabaseIfExists 

func DropDatabaseIfExists(ctx *Ctx) bool

DropDatabaseIfExists - drops requested database if exists Returns true if database existed and was dropped

func EnvReplace 

func EnvReplace(prefix, suffix string) map[string]string

EnvReplace - replace all environment variables starting with "prefix" with contents of variables with "suffix" added - if defined If prefix is "DB_" and suffix is "_SRC" then: So if there is "DB_HOST_SRC" variable defined - it will replace "DB_HOST" and so on

func EnvRestore 

func EnvRestore(env map[string]string)

EnvRestore - restores all environment variables given in the map

func ExecCommand 

func ExecCommand(ctx *Ctx, cmdAndArgs []string, env map[string]string) (string, error)

ExecCommand - execute command given by array of strings with eventual environment map

func ExecSQL 

func ExecSQL(con *sql.DB, ctx *Ctx, query string, args ...interface{}) (sql.Result, error)

ExecSQL executes given SQL on Postgres DB (and return single state result, that doesn't need to be closed)

func ExecSQLTx 

func ExecSQLTx(con *sql.Tx, ctx *Ctx, query string, args ...interface{}) (sql.Result, error)

ExecSQLTx executes given SQL on Postgres DB (and return single state result, that doesn't need to be closed) It is for running inside transaction

func ExecSQLTxWithErr 

func ExecSQLTxWithErr(con *sql.Tx, ctx *Ctx, query string, args ...interface{}) sql.Result

ExecSQLTxWithErr wrapper to ExecSQLTx that exists on error It is for running inside transaction

func ExecSQLWithErr 

func ExecSQLWithErr(con *sql.DB, ctx *Ctx, query string, args ...interface{}) sql.Result

ExecSQLWithErr wrapper to ExecSQL that exists on error

func FatalNoLog 

func FatalNoLog(err error) string

FatalNoLog displays error message (if error present) and exits program, should be used for very early init state

func FatalOnError 

func FatalOnError(err error) string

FatalOnError displays error message (if error present) and exits program

func Fatalf 

func Fatalf(f string, a ...interface{})

Fatalf - it will call FatalOnError using fmt.Errorf with args provided

func FirstIntOrNil 

func FirstIntOrNil(intPtrs []*int) interface{}

FirstIntOrNil - return either nil or value of intPtr

func ForkeeIDOrNil 

func ForkeeIDOrNil(forkPtr *Forkee) interface{}

ForkeeIDOrNil - return Forkee ID from pointer or nil

func ForkeeNameOrNil 

func ForkeeNameOrNil(forkPtr *Forkee) interface{}

ForkeeNameOrNil - return Forkee Name from pointer or nil

func ForkeeOldIDOrNil 

func ForkeeOldIDOrNil(forkPtr *ForkeeOld) interface{}

ForkeeOldIDOrNil - return ForkeeOld ID from pointer or nil

func GHClient 

func GHClient(ctx *Ctx) (ghCtx context.Context, clients []*github.Client)

GHClient - get GitHub client

func GetDateAgo 

func GetDateAgo(con *sql.DB, ctx *Ctx, from time.Time, ago string) (tm time.Time)

GetDateAgo returns date: 'from' - 'n hours/days' etc.

func GetFloatFromInterface 

func GetFloatFromInterface(i interface{}) (float64, bool)

GetFloatFromInterface if an interface is of numeric type, return its value as float64

func GetHidden 

func GetHidden(configFile string) map[string]string

GetHidden - return list of shas to replace

func GetIntervalFunctions 

func GetIntervalFunctions(intervalAbbr string, allowUnknown bool) (interval string, n int, intervalStart, nextIntervalStart, prevIntervalStart func(time.Time) time.Time)

GetIntervalFunctions - return interval name, interval number, interval start, next, prev function from interval abbr: h|d2|w3|m4|q|y w3 = 3 weeks, q2 = 2 quarters, y = year (1), d7 = 7 days (not the same as w), m3 = 3 months (not the same as q)

func GetRateLimits 

func GetRateLimits(gctx context.Context, ctx *Ctx, gcs []*github.Client, core bool) (int, []int, []int, []time.Duration)

GetRateLimits - returns all and remaining API points and duration to wait for reset when core=true - returns Core limits, when core=false returns Search limits

func GetRecentRepos 

func GetRecentRepos(c *sql.DB, ctx *Ctx, dtFrom time.Time) (repos []string, rids []int64)

GetRecentRepos - get list of repos active last day

func GetTagValues 

func GetTagValues(con *sql.DB, ctx *Ctx, name, key string) (ret []string)

GetTagValues returns tag values for a given key

func GetThreadsNum 

func GetThreadsNum(ctx *Ctx) int

GetThreadsNum returns the number of available CPUs If environment variable GHA2DB_ST is set it retuns 1 It can be used to debug single threaded verion It runs on 95% CPU power by default

func HandlePossibleError 

func HandlePossibleError(err error, cfg, info string) string

HandlePossibleError - display error specific message, detect rate limit and abuse

func HashArray 

func HashArray(ia []interface{}) string

HashArray takes []interface{} and returns hash string from given keys from map

func HashObject 

func HashObject(iv map[string]interface{}, keys []string) string

HashObject takes map[string]interface{} and keys from []string and returns hash string from given keys from map

func HashStrings 

func HashStrings(strs []string) int

HashStrings - returns unique Hash for strings array This value is supposed to be used as ID (negative) to mark it was artificially generated

func HourStart 

func HourStart(dt time.Time) time.Time

HourStart - return time rounded to current hour start

func InsertIgnore 

func InsertIgnore(query string) string

InsertIgnore - will return insert statement with ignore option specific for DB

func IntOrNil 

func IntOrNil(intPtr *int) interface{}

IntOrNil - return either nil or value of intPtr

func IsProjectDisabled 

func IsProjectDisabled(ctx *Ctx, proj string, yamlDisabled bool) bool

IsProjectDisabled - checks if project is disabled or not: fullName comes from makeOldRepoName for pre-2015 data! yamlDisabled (this is from projects.yaml - can be true or false) it also checks context (which can override `disabled: true` from projects.yaml) +pro1,-pro2 creates map {"pro1":true, "pro2":false}

func IssueIDOrNil 

func IssueIDOrNil(issuePtr *Issue) interface{}

IssueIDOrNil - return Issue ID from pointer or nil

func MakeOldRepoName 

func MakeOldRepoName(repo *ForkeeOld) string

MakeOldRepoName - before 2015 repository name should be Organization/Name (if Organization present) or just Name

func MakeUniqueSort 

func MakeUniqueSort(ary []string) (outAry []string)

MakeUniqueSort - make string array unique & sorted

func MapFromString 

func MapFromString(str string) map[string]string

MapFromString - returns maps from string formatted as map[k1:v1 k2:v2 k3:v3 ...]

func MaybeHideFunc 

func MaybeHideFunc(shas map[string]string) func(string) string

MaybeHideFunc - use closure as a data storage

func Mgetc 

func Mgetc(ctx *Ctx) string

Mgetc waits for single key press and return character pressed

func MilestoneIDOrNil 

func MilestoneIDOrNil(milPtr *Milestone) interface{}

MilestoneIDOrNil - return Milestone ID from pointer or nil

func MonthStart 

func MonthStart(dt time.Time) time.Time

MonthStart - return time rounded to current month start

func NValue 

func NValue(index int) string

NValue will return $n

func NValues 

func NValues(n int) string

NValues will return values($1, $2, .., $n)

func NegatedBoolOrNil 

func NegatedBoolOrNil(boolPtr *bool) interface{}

NegatedBoolOrNil - return either nil or negated value of boolPtr

func NextDayStart 

func NextDayStart(dt time.Time) time.Time

NextDayStart - return time rounded to next day start

func NextHourStart 

func NextHourStart(dt time.Time) time.Time

NextHourStart - return time rounded to next hour start

func NextMonthStart 

func NextMonthStart(dt time.Time) time.Time

NextMonthStart - return time rounded to next month start

func NextQuarterStart 

func NextQuarterStart(dt time.Time) time.Time

NextQuarterStart - return time rounded to next quarter start

func NextWeekStart 

func NextWeekStart(dt time.Time) time.Time

NextWeekStart - return time rounded to next week start

func NextYearStart 

func NextYearStart(dt time.Time) time.Time

NextYearStart - return time rounded to next year start

func NormalizeName 

func NormalizeName(str string) string

NormalizeName - clean DB string from ", ', -, /, ., " ",), (, ], [ trim leading and trailing space, lowercase Normalize Unicode characters

func ObjectToJSON 

func ObjectToJSON(obj interface{}, fn string)

ObjectToJSON - serialize given object as JSON

func ObjectToYAML 

func ObjectToYAML(obj interface{}, fn string)

ObjectToYAML - serialize given object as YAML

func OrgIDOrNil 

func OrgIDOrNil(orgPtr *Org) interface{}

OrgIDOrNil - return Org ID from pointer or nil

func OrgLoginOrNil 

func OrgLoginOrNil(orgPtr *Org) interface{}

OrgLoginOrNil - return Org ID from pointer or nil

func PeriodParse 

func PeriodParse(perStr string) (dur time.Duration, ok bool)

PeriodParse - tries to parse period

func PgConn 

func PgConn(ctx *Ctx) *sql.DB

PgConn Connects to Postgres database

func PgConnDB 

func PgConnDB(ctx *Ctx, dbName string) *sql.DB

PgConnDB Connects to Postgres database (with specific DB name) uses database 'dbname' instead of 'PgDB'

func PrepareQuickRangeQuery 

func PrepareQuickRangeQuery(sql, period, from, to string) string

PrepareQuickRangeQuery Perpares query using either ready `period` string or using `from` and `to` strings Values to replace are specially encoded {{period:alias.column}} Can either replace with: (alias.column >= now() - 'period'::interval) Or (alias.column >= 'from' and alias.column < 'to')

func PrettyPrintJSON 

func PrettyPrintJSON(jsonBytes []byte) []byte

PrettyPrintJSON - pretty formats raw JSON bytes

func PrevDayStart 

func PrevDayStart(dt time.Time) time.Time

PrevDayStart - return time rounded to prev day start

func PrevHourStart 

func PrevHourStart(dt time.Time) time.Time

PrevHourStart - return time rounded to prev hour start

func PrevMonthStart 

func PrevMonthStart(dt time.Time) time.Time

PrevMonthStart - return time rounded to prev month start

func PrevQuarterStart 

func PrevQuarterStart(dt time.Time) time.Time

PrevQuarterStart - return time rounded to prev quarter start

func PrevWeekStart 

func PrevWeekStart(dt time.Time) time.Time

PrevWeekStart - return time rounded to prev week start

func PrevYearStart 

func PrevYearStart(dt time.Time) time.Time

PrevYearStart - return time rounded to prev year start

func Printf 

func Printf(format string, args ...interface{}) (n int, err error)

Printf is a wrapper around Printf(...) that supports logging.

func ProcessAnnotations 

func ProcessAnnotations(ctx *Ctx, annotations *Annotations, startDate, joinDate *time.Time)

ProcessAnnotations Creates IfluxDB annotations and quick_series

func ProcessTag 

func ProcessTag(con *sql.DB, es *ES, ctx *Ctx, tg *Tag, replaces [][]string)

ProcessTag - insert given Tag into Postgres TSDB

func ProgressInfo 

func ProgressInfo(i, n int, start time.Time, last *time.Time, period time.Duration, msg string)

ProgressInfo display info about progress: i/n if current time >= last + period If displayed info, update last

func PullRequestIDOrNil 

func PullRequestIDOrNil(prPtr *PullRequest) interface{}

PullRequestIDOrNil - return PullRequest ID from pointer or nil

func QuarterStart 

func QuarterStart(dt time.Time) time.Time

QuarterStart - return time rounded to current month start

func QueryRowSQL 

func QueryRowSQL(con *sql.DB, ctx *Ctx, query string, args ...interface{}) *sql.Row

QueryRowSQL executes given SQL on Postgres DB (and returns single row)

func QueryRowSQLTx 

func QueryRowSQLTx(tx *sql.Tx, ctx *Ctx, query string, args ...interface{}) *sql.Row

QueryRowSQLTx executes given SQL on Postgres DB (and returns single row)

func QuerySQL 

func QuerySQL(con *sql.DB, ctx *Ctx, query string, args ...interface{}) (*sql.Rows, error)

QuerySQL executes given SQL on Postgres DB (and returns rowset that needs to be closed)

func QuerySQLTx 

func QuerySQLTx(con *sql.Tx, ctx *Ctx, query string, args ...interface{}) (*sql.Rows, error)

QuerySQLTx executes given SQL on Postgres DB (and returns rowset that needs to be closed) It is for running inside transaction

func QuerySQLTxWithErr 

func QuerySQLTxWithErr(con *sql.Tx, ctx *Ctx, query string, args ...interface{}) *sql.Rows

QuerySQLTxWithErr wrapper to QuerySQLTx that exists on error It is for running inside transaction

func QuerySQLWithErr 

func QuerySQLWithErr(con *sql.DB, ctx *Ctx, query string, args ...interface{}) *sql.Rows

QuerySQLWithErr wrapper to QuerySQL that exists on error

func ReadFile 

func ReadFile(ctx *Ctx, path string) ([]byte, error)

ReadFile tries to read any filename, but have a fallback it attempts to replace current project name with shared: /proj/ -> /shared/ This is to allow reading files that can be shared between projects

func ReleaseIDOrNil 

func ReleaseIDOrNil(relPtr *Release) interface{}

ReleaseIDOrNil - return Release ID from pointer or nil

func RepoHit 

func RepoHit(ctx *Ctx, fullName string, forg, frepo map[string]struct{}) bool

RepoHit - are we interested in this org/repo ?

func RepoIDOrNil 

func RepoIDOrNil(repoPtr *Repo) interface{}

RepoIDOrNil - return Repo ID from pointer or nil

func RepoNameOrNil 

func RepoNameOrNil(repoPtr *Repo) interface{}

RepoNameOrNil - return Repo Name from pointer or nil

func SkipEmpty 

func SkipEmpty(arr []string) []string

SkipEmpty - skip one element arrays contining only empty string This is what strings.Split() returns for empty input We expect empty array or empty map returned in such cases

func Slugify 

func Slugify(arg string) string

Slugify replace all whitespace with "-", remove all non-word letters downcase

func StringOrNil 

func StringOrNil(strPtr *string) interface{}

StringOrNil - return either nil or value of strPtr

func StringsMapToArray 

func StringsMapToArray(f func(string) string, strArr []string) []string

StringsMapToArray this is a function that calls given function for all array items and returns array of items processed by this func Example call: lib.StringsMapToArray(func(x string) string { return strings.TrimSpace(x) }, []string{" a", " b ", "c "})

func StringsMapToSet 

func StringsMapToSet(f func(string) string, strArr []string) map[string]struct{}

StringsMapToSet this is a function that calls given function for all array items and returns set of items processed by this func Example call: lib.StringsMapToSet(func(x string) string { return strings.TrimSpace(x) }, []string{" a", " b ", "c "})

func StringsSetKeys 

func StringsSetKeys(set map[string]struct{}) []string

StringsSetKeys - returns all keys from string map

func StripUnicode 

func StripUnicode(str string) string

StripUnicode strip non-unicode and control characters from a string From: https://rosettacode.org/wiki/Strip_control_codes_and_extended_characters_from_a_string#Go

func Structure 

func Structure(ctx *Ctx)

Structure creates full database structure, indexes, views/summary tables etc

func SyncIssuesState 

func SyncIssuesState(gctx context.Context, gc []*github.Client, ctx *Ctx, c *sql.DB, issues map[int64]IssueConfigAry, prs map[int64]github.PullRequest, manual bool)

SyncIssuesState synchonizes issues states manual: 

false: normal devstats sync cron mode using 'ghapi2db' tool
true: manual sync using 'sync_issues' tool

func TableColumnExists 

func TableColumnExists(con *sql.DB, ctx *Ctx, tableName, columnName string) bool

TableColumnExists - checks if a given table's has a given column

func TableExists 

func TableExists(con *sql.DB, ctx *Ctx, tableName string) bool

TableExists - checks if a given table exists

func TimeOrNil 

func TimeOrNil(timePtr *time.Time) interface{}

TimeOrNil - return either nil or value of timePtr

func TimeParseAny 

func TimeParseAny(dtStr string) time.Time

TimeParseAny - attempts to parse time from string YYYY-MM-DD HH:MI:SS Skipping parts from right until only YYYY id left

func ToGHADate 

func ToGHADate(dt time.Time) string

ToGHADate - return time formatted as YYYY-MM-DD-H

func ToYMDDate 

func ToYMDDate(dt time.Time) string

ToYMDDate - return time formatted as YYYY-MM-DD

func ToYMDHDate 

func ToYMDHDate(dt time.Time) string

ToYMDHDate - return time formatted as YYYY-MM-DD HH

func ToYMDHMSDate 

func ToYMDHMSDate(dt time.Time) string

ToYMDHMSDate - return time formatted as YYYY-MM-DD HH:MI:SS

func TruncStringOrNil 

func TruncStringOrNil(strPtr *string, maxLen int) interface{}

TruncStringOrNil - return either nil or value of strPtr truncated to maxLen chars

func TruncToBytes 

func TruncToBytes(str string, size int) string

TruncToBytes - truncates text to <= size bytes (note that this can be a lot less UTF-8 runes)

func WeekStart 

func WeekStart(dt time.Time) time.Time

WeekStart - return time rounded to current week start Assumes first week day is Sunday

func WriteTSPoints 

func WriteTSPoints(ctx *Ctx, con *sql.DB, pts *TSPoints, mergeSeries string, mut *sync.Mutex)

WriteTSPoints write batch of points to postgresql use mergeSeries = "name" to put all series in "name" table, and create "series" column that conatins all point names. 

without merge, alee names will create separate tables.

use non-null mut when you are using this function from multiple threads that write to the same series name at the same time 

use non-null mut only then.

No more giant lock approach here, but it is up to user to spcify call context, especially 2 last parameters!

func YearStart 

func YearStart(dt time.Time) time.Time

YearStart - return time rounded to current month start

Types

type Actor 

type Actor struct {
	ID    int    `json:"id"`
	Login string `json:"login"`
	Name  string `json:"-"`
}

Actor - GHA Actor structure Name is unexported and not used by JSON load/save But is used when importing affiliations from cncf/gitdm:github_users.json

type AllProjects 

type AllProjects struct {
	Projects map[string]Project `yaml:"projects"`
}

AllProjects contain all projects data

type Annotation 

type Annotation struct {
	Name        string
	Description string
	Date        time.Time
}

Annotation contain each annotation data

type Annotations 

type Annotations struct {
	Annotations []Annotation
}

Annotations contain list of annotations

func GetAnnotations 

func GetAnnotations(ctx *Ctx, orgRepo, annoRegexp string) (annotations Annotations)

GetAnnotations queries uses `git` to get `orgRepo` all tags list for all tags and returns those matching `annoRegexp`

func GetFakeAnnotations 

func GetFakeAnnotations(startDate, joinDate time.Time) (annotations Annotations)

GetFakeAnnotations - returns 'startDate - joinDate' and 'joinDate - now' annotations

type AnnotationsByDate 

type AnnotationsByDate []Annotation

AnnotationsByDate annotations Sort interface

func (AnnotationsByDate) Len 

func (a AnnotationsByDate) Len() int

func (AnnotationsByDate) Less 

func (a AnnotationsByDate) Less(i, j int) bool

func (AnnotationsByDate) Swap 

func (a AnnotationsByDate) Swap(i, j int)

type AnyArray 

type AnyArray []interface{}

AnyArray - holds array of interface{} - just a shortcut

type Asset 

type Asset struct {
	ID            int       `json:"id"`
	CreatedAt     time.Time `json:"created_at"`
	UpdatedAt     time.Time `json:"updated_at"`
	Name          string    `json:"name"`
	Label         *string   `json:"label"`
	Uploader      Actor     `json:"uploader"`
	ContentType   string    `json:"content_type"`
	State         string    `json:"state"`
	Size          int       `json:"size"`
	DownloadCount int       `json:"download_count"`
}

Asset - GHA Asset structure

type Author 

type Author struct {
	Name  string `json:"name"`
	Email string `json:"email"`
}

Author - GHA Commit Author structure

type Branch 

type Branch struct {
	SHA   string  `json:"sha"`
	User  *Actor  `json:"user"`
	Repo  *Forkee `json:"repo"` // This is confusing, but actually GHA has "repo" fields that holds "forkee" structure
	Label string  `json:"label"`
	Ref   string  `json:"ref"`
}

Branch - GHA Branch structure

type Comment 

type Comment struct {
	ID                  int       `json:"id"`
	Body                string    `json:"body"`
	CreatedAt           time.Time `json:"created_at"`
	UpdatedAt           time.Time `json:"updated_at"`
	User                Actor     `json:"user"`
	CommitID            *string   `json:"commit_id"`
	OriginalCommitID    *string   `json:"original_commit_id"`
	DiffHunk            *string   `json:"diff_hunk"`
	Position            *int      `json:"position"`
	OriginalPosition    *int      `json:"original_position"`
	Path                *string   `json:"path"`
	PullRequestReviewID *int      `json:"pull_request_review_id"`
	Line                *int      `json:"line"`
}

Comment - GHA Comment structure

type Commit 

type Commit struct {
	SHA      string `json:"sha"`
	Author   Author `json:"author"`
	Message  string `json:"message"`
	Distinct bool   `json:"distinct"`
}

Commit - GHA Commit structure

type Ctx 

type Ctx struct {
	DataDir             string                       // From GHA2DB_DATADIR, default /etc/gha2db/
	Debug               int                          // From GHA2DB_DEBUG Debug level: 0-no, 1-info, 2-verbose, including SQLs, default 0
	CmdDebug            int                          // From GHA2DB_CMDDEBUG Commands execution Debug level: 0-no, 1-only output commands, 2-output commands and their output, 3-output full environment as well, default 0
	GitHubDebug         int                          // From GHA2DB_GITHUB_DEBUG debug GitHub rate limits
	DryRun              bool                         // From GHA2DB_DRY_RUN, import_affs tool - stop before doing any updates
	JSONOut             bool                         // From GHA2DB_JSON gha2db: write JSON files? default false
	DBOut               bool                         // From GHA2DB_NODB gha2db: write to SQL database, default true
	ST                  bool                         // From GHA2DB_ST true: use single threaded version, false: use multi threaded version, default false
	NCPUs               int                          // From GHA2DB_NCPUS, set to override number of CPUs to run, this overwrites GHA2DB_ST, default 0 (which means do not use it)
	PgHost              string                       // From PG_HOST, default "localhost"
	PgPort              string                       // From PG_PORT, default "5432"
	PgDB                string                       // From PG_DB, default "gha"
	PgUser              string                       // From PG_USER, default "gha_admin"
	PgPass              string                       // From PG_PASS, default "password"
	PgSSL               string                       // From PG_SSL, default "disable"
	Index               bool                         // From GHA2DB_INDEX Create DB index? default false
	Table               bool                         // From GHA2DB_SKIPTABLE Create table structure? default true
	Tools               bool                         // From GHA2DB_SKIPTOOLS Create DB tools (like views, summary tables, materialized views etc)? default true
	Mgetc               string                       // From GHA2DB_MGETC Character returned by mgetc (if non empty), default ""
	QOut                bool                         // From GHA2DB_QOUT output all SQL queries?, default false
	CtxOut              bool                         // From GHA2DB_CTXOUT output all context data (this struct), default false
	LogTime             bool                         // From GHA2DB_SKIPTIME, output time with all lib.Printf(...) calls, default true, use GHA2DB_SKIPTIME to disable
	DefaultStartDate    time.Time                    // From GHA2DB_STARTDT, default `2012-07-01 00:00 UTC`, expects format "YYYY-MM-DD HH:MI:SS", can be set in `projects.yaml` via `start_date:`, value from projects.yaml (if set) has the highest priority.
	ForceStartDate      bool                         // From GHA2DB_STARTDT_FORCE, default false
	LastSeries          string                       // From GHA2DB_LASTSERIES, use this TSDB series to determine last timestamp date, default "events_h"
	SkipTSDB            bool                         // From GHA2DB_SKIPTSDB gha2db_sync tool, skip TS DB processing? for calc_metric it skips final series write, default false
	SkipPDB             bool                         // From GHA2DB_SKIPPDB gha2db_sync tool, skip Postgres DB processing (gha2db part) default false
	ResetTSDB           bool                         // From GHA2DB_RESETTSDB sync tool, regenerate all TS points? default false
	ResetRanges         bool                         // From GHA2DB_RESETRANGES sync tool, regenerate all past quick ranges? default false
	Explain             bool                         // From GHA2DB_EXPLAIN runq tool, prefix query with "explain " - it will display query plan instead of executing real query, default false
	OldFormat           bool                         // From GHA2DB_OLDFMT gha2db tool, if set then use pre 2015 GHA JSONs format
	Exact               bool                         // From GHA2DB_EXACT gha2db tool, if set then orgs list provided from commandline is used as a list of exact repository full names, like "a/b,c/d,e", if not only full names "a/b,x/y" can be treated like this, names without "/" are either orgs or repos.
	LogToDB             bool                         // From GHA2DB_SKIPLOG all tools, if set, DB logging into Postgres table `gha_logs` in `devstats` database will be disabled
	Local               bool                         // From GHA2DB_LOCAL many tools, if set it will use data files prefixed with "./" to use local ones. Otherwise it will search for data files in /etc/gha2db.
	LocalCmd            bool                         // From GHA2DB_LOCAL_CMD many tools, if set it will call other tools prefixed with "./" to use locally compiled ones. Otherwise it will call binaries without prefix (so it will use those in $PATH).
	MetricsYaml         string                       // From GHA2DB_METRICS_YAML gha2db_sync tool, set other metrics.yaml file, default is "metrics/{{project}}metrics.yaml"
	TagsYaml            string                       // From GHA2DB_TAGS_YAML tags tool, set other tags.yaml file, default is "metrics/{{project}}/tags.yaml"
	ColumnsYaml         string                       // From GHA2DB_COLUMNS_YAML tags tool, set other columns.yaml file, default is "metrics/{{project}}/columns.yaml"
	VarsYaml            string                       // From GHA2DB_VARS_YAML db_vars tool, set other vars.yaml file (full path), default is "metrics/{{project}}/vars.yaml"
	VarsFnYaml          string                       // From GHA2DB_VARS_FN_YAML db_vars tool, set other vars.yaml file (final file name without path), default is "vars.yaml"
	SkipDatesYaml       string                       // From GHA2DB_SKIP_DATES_YAML gha2db tool, set other skip_dates.yaml file, default is "skip_dates.yaml"
	GitHubOAuth         string                       // From GHA2DB_GITHUB_OAUTH ghapi2db tool, if not set reads from /etc/github/oauth file, set to "-" to force public access.
	ClearDBPeriod       string                       // From GHA2DB_MAXLOGAGE gha2db_sync tool, maximum age of devstats.gha_logs entries, default "1 week"
	Trials              []int                        // From GHA2DB_TRIALS, all Postgres related tools, retry periods for "too many connections open" error
	WebHookRoot         string                       // From GHA2DB_WHROOT, webhook tool, default "/hook", must match .travis.yml notifications webhooks
	WebHookPort         string                       // From GHA2DB_WHPORT, webhook tool, default ":1982", note that webhook listens using http:1982, but we use apache on https:2982 (to enable https protocol and proxy requests to http:1982)
	WebHookHost         string                       // From GHA2DB_WHHOST, webhook tool, default "127.0.0.1" (this can be localhost to disable access by IP, we use Apache proxy to enable https and then apache only need 127.0.0.1)
	CheckPayload        bool                         // From GHA2DB_SKIP_VERIFY_PAYLOAD, webhook tool, default true, use GHA2DB_SKIP_VERIFY_PAYLOAD=1 to manually test payloads
	FullDeploy          bool                         // From GHA2DB_SKIP_FULL_DEPLOY, webhook tool, default true, use GHA2DB_SKIP_FULL_DEPLOY=1 to ignore "[deploy]" requests that call `./devel/deploy_all.sh`.
	DeployBranches      []string                     // From GHA2DB_DEPLOY_BRANCHES, webhook tool, default "master" - comma separated list
	DeployStatuses      []string                     // From GHA2DB_DEPLOY_STATUSES, webhook tool, default "Passed,Fixed", - comma separated list
	DeployResults       []int                        // From GHA2DB_DEPLOY_RESULTS, webhook tool, default "0", - comma separated list
	DeployTypes         []string                     // From GHA2DB_DEPLOY_TYPES, webhook tool, default "push", - comma separated list
	ProjectRoot         string                       // From GHA2DB_PROJECT_ROOT, webhook tool, no default, must be specified to run webhook tool
	ExecFatal           bool                         // default true, set this manually to false to avoid lib.ExecCommand calling os.Exit() on failure and return error instead
	ExecQuiet           bool                         // default false, set this manually to true to have quite exec failures (for example `get_repos` git-clones or git-pulls on errors).
	ExecOutput          bool                         // default false, set to true to capture commands STDOUT
	Project             string                       // From GHA2DB_PROJECT, gha2db_sync default "", You should set it to something like "kubernetes", "prometheus" etc.
	TestsYaml           string                       // From GHA2DB_TESTS_YAML ./dbtest.sh tool, set other tests.yaml file, default is "tests.yaml"
	ReposDir            string                       // From GHA2DB_REPOS_DIR get_repos tool, default "~/devstats_repos/"
	ProcessRepos        bool                         // From GHA2DB_PROCESS_REPOS get_repos tool, enable processing (cloning/pulling) all devstats repos, default false
	ProcessCommits      bool                         // From GHA2DB_PROCESS_COMMITS get_repos tool, enable update/create mapping table: commit - list of file that commit refers to, default false
	ExternalInfo        bool                         // From GHA2DB_EXTERNAL_INFO get_repos tool, enable outputing data needed by external tools (cncf/gitdm), default false
	ProjectsCommits     string                       // From GHA2DB_PROJECTS_COMMITS get_repos tool, set list of projects for commits analysis instead of analysing all, default "" - means all
	PropagateOnlyVar    bool                         // From GHA2DB_PROPAGATE_ONLY_VAR, if set the it will check ONLY="a b c" env variable and propagate it into other project filter variables if they're not set, for example GHA2DB_PROJECTS_COMMITS
	ProjectsYaml        string                       // From GHA2DB_PROJECTS_YAML, many tools - set main projects file, default "projects.yaml"
	CompanyAcqYaml      string                       // From GHA2DB_COMPANY_ACQ_YAML, import_affs tool, set non-standard "companies.yaml" file
	ProjectsOverride    map[string]bool              // From GHA2DB_PROJECTS_OVERRIDE, get_repos and ./devstats tools - for example "-pro1,+pro2" means never sync pro1 and always sync pro2 (even if disabled in `projects.yaml`).
	AffiliationsJSON    string                       // From GHA2DB_AFFILIATIONS_JSON, import_affs tool - set main affiliations file, default "github_users.json"
	ExcludeRepos        map[string]bool              // From GHA2DB_EXCLUDE_REPOS, gha2db tool, default "" - comma separated list of repos to exclude, example: "theupdateframework/notary,theupdateframework/other"
	InputDBs            []string                     // From GHA2DB_INPUT_DBS, merge_dbs tool - list of input databases to merge, order matters - first one will insert on a clean DB, next will do insert ignore (to avoid constraints failure due to common data)
	OutputDB            string                       // From GHA2DB_OUTPUT_DB, merge_dbs tool - output database to merge into
	TmOffset            int                          // From GHA2DB_TMOFFSET, gha2db_sync tool - uses time offset to decide when to calculate various metrics, default offset is 0 which means UTC, good offset for USA is -6, and for Poland is 1 or 2
	DefaultHostname     string                       // "devstats.cncf.io"
	RecentRange         string                       // From GHA2DB_RECENT_RANGE, ghapi2db tool, default '2 hours'. This is a recent period to check open issues/PR to fix their labels and milestones.
	RecentReposRange    string                       // From GHA2DB_RECENT_REPOS_RANGE, ghapi2db tool, default '1 day'. This is a recent period to check modified repositories.
	MinGHAPIPoints      int                          // From GHA2DB_MIN_GHAPI_POINTS, ghapi2db tool, minimum GitHub API points, before waiting for reset.
	MaxGHAPIWaitSeconds int                          // From GHA2DB_MAX_GHAPI_WAIT, ghapi2db tool, maximum wait time for GitHub API points reset (in seconds).
	MaxGHAPIRetry       int                          // From GHA2DB_MAX_GHAPI_RETRY, ghapi2db tool, maximum wait retries
	GHAPIErrorIsFatal   bool                         // From GHA2DB_GHAPI_ERROR_FATAL, ghapi2db tool, make any GH API error fatal, default false
	SkipGHAPI           bool                         // From GHA2DB_GHAPISKIP, ghapi2db tool, if set then tool is skipping GH API calls (all: events (artificial events to make sure we are in sync with GH) and commits (enriches obfuscated GHA commits data)
	SkipAPIEvents       bool                         // From GHA2DB_GHAPISKIPEVENTS, ghapi2db tool, if set then tool is skipping GH API events sync
	SkipAPICommits      bool                         // From GHA2DB_GHAPISKIPCOMMITS, ghapi2db tool, if set then tool is skipping GH API commits enrichment
	SkipGetRepos        bool                         // From GHA2DB_GETREPOSSKIP, get_repos tool, if set then tool does nothing
	CSVFile             string                       // From GHA2DB_CSVOUT, runq tool, if set, saves result in this file
	ComputeAll          bool                         // From GHA2DB_COMPUTE_ALL, all tools, if set then no period decisions are taken based on time, but all possible periods are recalculated
	ActorsFilter        bool                         // From GHA2DB_ACTORS_FILTER gha2db tool, if enabled then actor filterning will be added, default false
	ActorsAllow         *regexp.Regexp               // From GHA2DB_ACTORS_ALLOW, gha2db tool, process JSON if actor matches this regexp, default "" which means skip this check
	ActorsForbid        *regexp.Regexp               // From GHA2DB_ACTORS_FORBID, gha2db tool, process JSON if actor doesn't match this regexp, default "" which means skip this check
	SkipMetrics         map[string]bool              // From GHA2DB_SKIP_METRICS, gha2db_sync tool, default "" - comma separated list of metrics to skip, as given by "sql: name" in the "metrics.yaml" file. Those metrics will be skipped.
	OnlyMetrics         map[string]bool              // From GHA2DB_ONLY_METRICS, gha2db_sync tool, default "" - comma separated list of metrics to process, as given by "sql: name" in the "metrics.yaml" file. Only those metrics will be calculated.
	AllowBrokenJSON     bool                         // From GHA2DB_ALLOW_BROKEN_JSON, gha2db tool, default false. If set then gha2db skips broken jsons and saves them as jsons/error_YYYY-MM-DD-h-n-m.json (n is the JSON number (1-m) of m JSONS array)
	JSONsDir            string                       // From GHA2DB_JSONS_DIR, website_data tool, default "./jsons/"
	WebsiteData         bool                         // From GHA2DB_WEBSITEDATA, devstats tool, run website_data just after sync is complete, default false.
	SkipUpdateEvents    bool                         // From GHA2DB_SKIP_UPDATE_EVENTS, ghapi2db tool, drop and recreate artificial events if their state differs, default false
	ComputePeriods      map[string]map[bool]struct{} // From GHA2DB_FORCE_PERIODS, gha2db_sync tool, force recompute only given periods, "y10:t,m:f,...", default ""
	AutoFetchCommits    bool                         // From GHA2DB_NO_AUTOFETCHCOMMITS, ghapi2db, disable fetching from last enriched commit data, it will fetch from 'RecentRange instead, AutoFetchCommits is enabled by default
	SkipTags            bool                         // From GHA2DB_SKIP_TAGS, gha2db_sync tool, skip calling tags tool, default false
	SkipAnnotations     bool                         // From GHA2DB_SKIP_ANNOTATIONS, gha2db_sync tool, skip calling annotations tool, default false
	SkipColumns         bool                         // From GHA2DB_SKIP_COLUMNS, gha2db_sync tool, skip calling columns tool, default false
	RunColumns          bool                         // From GHA2DB_RUN_COLUMNS, gha2db_sync tool, force calling columns tool, default false
	SkipVars            bool                         // From GHA2DB_SKIP_VARS, gha2db_sync tool, skip calling vars tool, default false
	ExcludeVars         map[string]bool              // From GHA2DB_EXCLUDE_VARS, vars tool, default "" - comma separated list of variable names to exclude, example: "hostname,projects_health_partial_html"
	OnlyVars            map[string]bool              // From GHA2DB_ONLY_VARS, vars tool, default "" - comma separated list of variable names to write (and skip all others): "hostname,projects_health_partial_html", not used if empty
	ElasticURL          string                       // From GHA2DB_ES_URL, calc_metric, tags, annotations tools - ElasticSearch URL (if used), default http://127.0.0.1:9200
	UseES               bool                         // From GHA2DB_USE_ES, calc_metric, tags, annotations tools - enable ElasticSearch, default false
	UseESOnly           bool                         // From GHA2DB_USE_ES_ONLY, calc_metric, annotations tools - enable ElasticSearch and do not write PSQL TSDB, default false
	UseESRaw            bool                         // From GHA2DB_USE_ES_RAW, gha2es, gha2db_sync tools - enable generating RAW ElasticSearch data (directly from gha_tables instead of aggregated data from TSDB)
	ResetESRaw          bool                         // From GHA2DB_RESET_ES_RAW, gha2db_sync tools - generate RAW ES index from project start date
	SkipSharedDB        bool                         // From GHA2DB_SKIP_SHAREDDB, annotations tool, default false, will skip writing to shared_db (from projects.yaml) if set
	SkipPIDFile         bool                         // From GHA2DB_SKIP_PIDFILE, devstats tool, skip creating, checking and removing PID file
	SkipCompanyAcq      bool                         // From GHA2DB_SKIP_COMPANY_ACQ, import_affs tool, skip processing company acquisitions from companies.yaml file
	CheckProvisionFlag  bool                         // From GHA2DB_CHECK_PROVISION_FLAG, devstats tool - check if there is a 'provision' metric saved in 'gha_computed' table - if not, abort
	CheckRunningFlag    bool                         // From GHA2DB_CHECK_RUNNING_FLAG, devstats tool - check if there is a 'devstats_running' metric saved in 'gha_computed' table - if yes, abort
	SetRunningFlag      bool                         // From GHA2DB_SET_RUNNING_FLAG, devstats tool - set 'devstats_running' flag on 'gha_computed' table while devstats cronjob is running
	ESBulkSize          int                          // From GHA2DB_ES_BULK_SIZE, calc_metric and gha2es tools, default 10000
	HTTPTimeout         int                          // From GHA2DB_HTTP_TIMEOUT, gha2db - data.gharchive.org timeout value in minutes, default 2
	PidFileRoot         string                       // From GHA2DB_PID_FILE_ROOT, devstats tool, use '/tmp/PidFileRoot.pid' as PID file, default 'devstats' -> '/tmp/devstats.pid'
	SharedDB            string                       // Currently annotations tool read this from projects.yaml:shared_db and if set, outputs annotations data to the sharded DB in addition to the current DB
	ProjectMainRepo     string                       // Used by annotations tool to store project's main repo name
	TestMode            bool                         // True when running tests
}

Ctx - environment context packed in structure

func (*Ctx) Init 

func (ctx *Ctx) Init()

Init - get context from environment variables

func (*Ctx) Print 

func (ctx *Ctx) Print()

Print context contents

type Dummy 

type Dummy struct{}

Dummy - structure with no data pointer to this struct is used to test if such field was present in JSON or not

type ES 

type ES struct {
	// contains filtered or unexported fields
}

ES - ElasticSearch connection client, context and default mapping

func ESConn 

func ESConn(ctx *Ctx, prefix string) *ES

ESConn Connects to ElasticSearch

func (*ES) AddBulksItems 

func (es *ES) AddBulksItems(ctx *Ctx, b *ESBulks, doc map[string]interface{}, keys []string)

AddBulksItems adds items to the Bulk Request

func (*ES) AddBulksItemsI 

func (es *ES) AddBulksItemsI(ctx *Ctx, b *ESBulks, doc interface{}, docHash string)

AddBulksItemsI adds items to the Bulk Request

func (*ES) Bulks 

func (es *ES) Bulks() (*elastic.BulkService, *elastic.BulkService)

Bulks returns Delete and Add requests

func (*ES) CreateIndex 

func (es *ES) CreateIndex(ctx *Ctx, raw bool)

CreateIndex creates index

func (*ES) DeleteByQuery 

func (es *ES) DeleteByQuery(ctx *Ctx, propNames []string, propValues []interface{})

DeleteByQuery deletes data from given index & type by simple bool query

func (*ES) DeleteByWildcardQuery 

func (es *ES) DeleteByWildcardQuery(ctx *Ctx, propName, propQuery string)

DeleteByWildcardQuery deletes data from given index & type by using wildcard query

func (*ES) ESEscapeFieldName 

func (es *ES) ESEscapeFieldName(fieldName string) string

ESEscapeFieldName escape characters non allowed in ES field names

func (*ES) ESIndexName 

func (es *ES) ESIndexName(ctx *Ctx) string

ESIndexName returns ES index name "d_{{project}}" --> "d_kubernetes"

func (*ES) ExecuteBulkAdd 

func (es *ES) ExecuteBulkAdd(ctx *Ctx, bulkAdd *elastic.BulkService)

ExecuteBulkAdd executes scheduled commands (delete and then inserts)

func (*ES) ExecuteBulkDel 

func (es *ES) ExecuteBulkDel(ctx *Ctx, bulkDel *elastic.BulkService)

ExecuteBulkDel executes scheduled commands (delete and then inserts)

func (*ES) ExecuteBulks 

func (es *ES) ExecuteBulks(ctx *Ctx, b *ESBulks)

ExecuteBulks executes scheduled commands (delete and then inserts)

func (*ES) GetElasticClient 

func (es *ES) GetElasticClient() *elastic.Client

GetElasticClient - returns embedded ES client

func (*ES) IndexExists 

func (es *ES) IndexExists(ctx *Ctx) bool

IndexExists checks if index exists

func (*ES) WriteESPoints 

func (es *ES) WriteESPoints(ctx *Ctx, pts *TSPoints, mergeS string, outputs [3]bool)

WriteESPoints write batch of points to postgresql outputs[0] - output using variable column name (1 doc) [used by annotations, tags and vars] outputs[1] - output using data[] array containing {name,ivalue,svalue,svalue2,svalue3,dtvalue} (1 doc), any of those keys is optional [not used currently] outputs[2] - output using N separate docs, each containing {name,ivalue,svalue,svalue2,svalue3,dtvalue} (N docs) (but trying to keep both int and string value in the same record) [used by metrics/time-series]

type ESBulks 

type ESBulks struct {
	// contains filtered or unexported fields
}

ESBulks keeps array of bulk services to add/delete each delete/add but can hold 10 items

func (*ESBulks) CurrentAdd 

func (b *ESBulks) CurrentAdd() *elastic.BulkService

CurrentAdd returns current bulk add object

func (*ESBulks) CurrentDel 

func (b *ESBulks) CurrentDel() *elastic.BulkService

CurrentDel returns current bulk delete object

func (*ESBulks) Init 

func (b *ESBulks) Init(ec *elastic.Client, ctx *Ctx)

Init creates structure to hanle bulk inserts/deletes

func (*ESBulks) Next 

func (b *ESBulks) Next(ec *elastic.Client)

Next will increase objects count and possibly switch to another bulk objects

func (*ESBulks) String 

func (b *ESBulks) String() string

String - output bulks config

type ESDataObject 

type ESDataObject struct {
	Name    string    `json:"name"`
	IValue  float64   `json:"ivalue"`
	SValue  string    `json:"svalue"`
	SValue2 string    `json:"svalue2"`
	SValue3 string    `json:"svalue3"`
	DtValue time.Time `json:"dtvalue"`
}

ESDataObject internal JSON data for stored documents

type Event 

type Event struct {
	ID        string    `json:"id"`
	Type      string    `json:"type"`
	Public    bool      `json:"public"`
	CreatedAt time.Time `json:"created_at"`
	Actor     Actor     `json:"actor"`
	Repo      Repo      `json:"repo"`
	Org       *Org      `json:"org"`
	Payload   Payload   `json:"payload"`
}

Event - full GHA (GitHub Archive) event structure

type EventOld 

type EventOld struct {
	ID         string      `json:"-"`
	Type       string      `json:"type"`
	Public     bool        `json:"public"`
	CreatedAt  time.Time   `json:"created_at"`
	Actor      string      `json:"actor"`
	Repository ForkeeOld   `json:"repository"`
	Payload    *PayloadOld `json:"payload"`
}

EventOld - full GHA (GitHub Archive) event structure, before 2015

type Forkee 

type Forkee struct {
	ID              int        `json:"id"`
	Name            string     `json:"name"`
	FullName        string     `json:"full_name"`
	Owner           Actor      `json:"owner"`
	Description     *string    `json:"description"`
	Public          *bool      `json:"public"`
	Fork            bool       `json:"fork"`
	CreatedAt       time.Time  `json:"created_at"`
	UpdatedAt       time.Time  `json:"updated_at"`
	PushedAt        *time.Time `json:"pushed_at"`
	Homepage        *string    `json:"homepage"`
	Size            int        `json:"size"`
	StargazersCount int        `json:"stargazers_count"`
	HasIssues       bool       `json:"has_issues"`
	HasProjects     *bool      `json:"has_projects"`
	HasDownloads    bool       `json:"has_downloads"`
	HasWiki         bool       `json:"has_wiki"`
	HasPages        *bool      `json:"has_pages"`
	Forks           int        `json:"forks"`
	OpenIssues      int        `json:"open_issues"`
	Watchers        int        `json:"watchers"`
	DefaultBranch   string     `json:"default_branch"`
}

Forkee - GHA Forkee structure

type ForkeeOld 

type ForkeeOld struct {
	ID            int        `json:"id"`
	CreatedAt     time.Time  `json:"created_at"`
	Description   *string    `json:"description"`
	Fork          bool       `json:"fork"`
	Forks         int        `json:"forks"`
	HasDownloads  bool       `json:"has_downloads"`
	HasIssues     bool       `json:"has_issues"`
	HasWiki       bool       `json:"has_wiki"`
	Homepage      *string    `json:"homepage"`
	Language      *string    `json:"language"`
	DefaultBranch string     `json:"master_branch"`
	Name          string     `json:"name"`
	OpenIssues    int        `json:"open_issues"`
	Organization  *string    `json:"organization"`
	Owner         string     `json:"owner"`
	Private       *bool      `json:"private"`
	PushedAt      *time.Time `json:"pushed_at"`
	Size          int        `json:"size"`
	Stargazers    int        `json:"stargazers"`
	Watchers      int        `json:"watchers"`
}

ForkeeOld - GHA Forkee structure (from before 2015) Handle missing 4 last properties (including two non-nulls!)

type Int64Ary 

type Int64Ary []int64

Int64Ary - sortable Int64 array

func (Int64Ary) Len 

func (a Int64Ary) Len() int

func (Int64Ary) Less 

func (a Int64Ary) Less(i, j int) bool

func (Int64Ary) Swap 

func (a Int64Ary) Swap(i, j int)

type Issue 

type Issue struct {
	ID          int        `json:"id"`
	Number      int        `json:"number"`
	Comments    int        `json:"comments"`
	Title       string     `json:"title"`
	State       string     `json:"state"`
	Locked      bool       `json:"locked"`
	Body        *string    `json:"body"`
	User        Actor      `json:"user"`
	Assignee    *Actor     `json:"assignee"`
	Labels      []Label    `json:"labels"`
	Assignees   []Actor    `json:"assignees"`
	Milestone   *Milestone `json:"milestone"`
	CreatedAt   time.Time  `json:"created_at"`
	UpdatedAt   time.Time  `json:"updated_at"`
	ClosedAt    *time.Time `json:"closed_at"`
	PullRequest *Dummy     `json:"pull_request"`
}

Issue - GHA Issue structure

type IssueConfig 

type IssueConfig struct {
	Repo         string
	Number       int
	IssueID      int64
	Pr           bool
	MilestoneID  *int64
	Labels       string
	LabelsMap    map[int64]string
	GhIssue      *github.Issue
	CreatedAt    time.Time
	EventID      int64
	EventType    string
	GhEvent      *github.IssueEvent
	AssigneeID   *int64
	Assignees    string
	AssigneesMap map[int64]string
}

IssueConfig - holds issue data

func (IssueConfig) String 

func (ic IssueConfig) String() string

type IssueConfigAry 

type IssueConfigAry []IssueConfig

IssueConfigAry - allows sorting IssueConfig array by IssueID annd then event creation date

func (IssueConfigAry) Len 

func (ic IssueConfigAry) Len() int

func (IssueConfigAry) Less 

func (ic IssueConfigAry) Less(i, j int) bool

func (IssueConfigAry) Swap 

func (ic IssueConfigAry) Swap(i, j int)

type Label 

type Label struct {
	ID      *int   `json:"id"`
	Name    string `json:"name"`
	Color   string `json:"color"`
	Default *bool  `json:"default"`
}

Label - GHA Label structure

type Milestone 

type Milestone struct {
	ID           int        `json:"id"`
	Name         string     `json:"name"`
	Number       int        `json:"number"`
	Title        string     `json:"title"`
	Description  *string    `json:"description"`
	Creator      *Actor     `json:"creator"`
	OpenIssues   int        `json:"open_issues"`
	ClosedIssues int        `json:"closed_issues"`
	State        string     `json:"state"`
	CreatedAt    time.Time  `json:"created_at"`
	UpdatedAt    time.Time  `json:"updated_at"`
	ClosedAt     *time.Time `json:"closed_at"`
	DueOn        *time.Time `json:"due_on"`
}

Milestone - GHA Milestone structure

type Org 

type Org struct {
	ID    int    `json:"id"`
	Login string `json:"login"`
}

Org - GHA Org structure

type Page 

type Page struct {
	SHA    string `json:"sha"`
	Action string `json:"action"`
	Title  string `json:"title"`
}

Page - GHA Page structure

type Payload 

type Payload struct {
	PushID       *int         `json:"push_id"`
	Size         *int         `json:"size"`
	Ref          *string      `json:"ref"`
	Head         *string      `json:"head"`
	Before       *string      `json:"before"`
	Action       *string      `json:"action"`
	RefType      *string      `json:"ref_type"`
	MasterBranch *string      `json:"master_branch"`
	Description  *string      `json:"description"`
	Number       *int         `json:"number"`
	Forkee       *Forkee      `json:"forkee"`
	Release      *Release     `json:"release"`
	Member       *Actor       `json:"member"`
	Issue        *Issue       `json:"issue"`
	Comment      *Comment     `json:"comment"`
	Commits      *[]Commit    `json:"commits"`
	Pages        *[]Page      `json:"pages"`
	PullRequest  *PullRequest `json:"pull_request"`
}

Payload - GHA Payload structure

type PayloadOld 

type PayloadOld struct {
	Issue        *int           `json:"issue"`
	IssueID      *int           `json:"issue_id"`
	Comment      *Comment       `json:"comment"`
	CommentID    *int           `json:"comment_id"`
	Description  *string        `json:"description"`
	MasterBranch *string        `json:"master_branch"`
	Ref          *string        `json:"ref"`
	Action       *string        `json:"action"`
	RefType      *string        `json:"ref_type"`
	Head         *string        `json:"head"`
	Size         *int           `json:"size"`
	Number       *int           `json:"number"`
	PullRequest  *PullRequest   `json:"pull_request"`
	Member       *Actor         `json:"member"`
	Release      *Release       `json:"release"`
	Pages        *[]Page        `json:"pages"`
	Commit       *string        `json:"commit"`
	SHAs         *[]interface{} `json:"shas"`
	Repository   *Forkee        `json:"repository"`
	Team         *Team          `json:"team"`
}

PayloadOld - GHA Payload structure (from before 2015)

type Project 

type Project struct {
	CommandLine      []string          `yaml:"command_line"`
	StartDate        *time.Time        `yaml:"start_date"`
	PDB              string            `yaml:"psql_db"`
	Disabled         bool              `yaml:"disabled"`
	MainRepo         string            `yaml:"main_repo"`
	AnnotationRegexp string            `yaml:"annotation_regexp"`
	Order            int               `yaml:"order"`
	JoinDate         *time.Time        `yaml:"join_date"`
	FilesSkipPattern string            `yaml:"files_skip_pattern"`
	Env              map[string]string `yaml:"env"`
	FullName         string            `yaml:"name"`
	Status           string            `yaml:"status"`
	SharedDB         string            `yaml:"shared_db"`
}

Project contain mapping from project name to its command line used to sync it

func GetProjectsList 

func GetProjectsList(ctx *Ctx, projects *AllProjects) (names []string, projs []Project)

GetProjectsList return list of projects to process It sorts them according to projects.yaml order Handles disabled/enabled projects Handles ONLY="project1 project2 ... projectN" env

type PullRequest 

type PullRequest struct {
	ID                  int        `json:"id"`
	Base                Branch     `json:"base"`
	Head                Branch     `json:"head"`
	User                Actor      `json:"user"`
	Number              int        `json:"number"`
	State               string     `json:"state"`
	Locked              *bool      `json:"locked"`
	Title               string     `json:"title"`
	Body                *string    `json:"body"`
	CreatedAt           time.Time  `json:"created_at"`
	UpdatedAt           time.Time  `json:"updated_at"`
	ClosedAt            *time.Time `json:"closed_at"`
	MergedAt            *time.Time `json:"merged_at"`
	MergeCommitSHA      *string    `json:"merge_commit_sha"`
	Assignee            *Actor     `json:"assignee"`
	Assignees           *[]Actor   `json:"assignees"`
	RequestedReviewers  *[]Actor   `json:"requested_reviewers"`
	Milestone           *Milestone `json:"milestone"`
	Merged              *bool      `json:"merged"`
	Mergeable           *bool      `json:"mergeable"`
	MergedBy            *Actor     `json:"merged_by"`
	MergeableState      *string    `json:"mergeable_state"`
	Rebaseable          *bool      `json:"rebaseable"`
	Comments            *int       `json:"comments"`
	ReviewComments      *int       `json:"review_comments"`
	MaintainerCanModify *bool      `json:"maintainer_can_modify"`
	Commits             *int       `json:"commits"`
	Additions           *int       `json:"additions"`
	Deletions           *int       `json:"deletions"`
	ChangedFiles        *int       `json:"changed_files"`
}

PullRequest - GHA Pull Request structure

type Release 

type Release struct {
	ID              int        `json:"id"`
	TagName         string     `json:"tag_name"`
	TargetCommitish string     `json:"target_commitish"`
	Name            *string    `json:"name"`
	Draft           bool       `json:"draft"`
	Author          Actor      `json:"author"`
	Prerelease      bool       `json:"prerelease"`
	CreatedAt       time.Time  `json:"created_at"`
	PublishedAt     *time.Time `json:"published_at"`
	Body            *string    `json:"body"`
	Assets          []Asset    `json:"assets"`
}

Release - GHA Release structure

type Repo 

type Repo struct {
	ID   int    `json:"id"`
	Name string `json:"name"`
}

Repo - GHA Repo structure

type SkipDatesList 

type SkipDatesList struct {
	Dates []time.Time `yaml:"skip_dates"`
}

SkipDatesList holds a list of GHA hours to skip

type TSPoint 

type TSPoint struct {
	// contains filtered or unexported fields
}

TSPoint keeps single time series point

func NewTSPoint 

func NewTSPoint(ctx *Ctx, name, period string, tags map[string]string, fields map[string]interface{}, t time.Time, exact bool) TSPoint

NewTSPoint returns new point as specified by args

func (*TSPoint) Str 

func (p *TSPoint) Str() string

Str - string pretty print

type TSPoints 

type TSPoints []TSPoint

TSPoints keeps batch of TSPoint values to write

func (*TSPoints) Str 

func (ps *TSPoints) Str() string

Str - string pretty print

type Tag 

type Tag struct {
	Name       string               `yaml:"name"`
	SQLFile    string               `yaml:"sql"`
	SeriesName string               `yaml:"series_name"`
	NameTag    string               `yaml:"name_tag"`
	ValueTag   string               `yaml:"value_tag"`
	OtherTags  map[string][2]string `yaml:"other_tags"`
	Limit      int                  `yaml:"limit"`
	Disabled   bool                 `yaml:"disabled"`
}

Tag contain each TSDB tag data

type Tags 

type Tags struct {
	Tags []Tag `yaml:"tags"`
}

Tags contain list of TSDB tags

type Team 

type Team struct {
	ID         int    `json:"id"`
	Name       string `json:"name"`
	Slug       string `json:"slug"`
	Permission string `json:"permission"`
}

Team - GHA Team structure (only used before 2015)

Source Files

View all Source files

Directories

Path Synopsis
cmd
annotations
calc_metric
columns
devstats
get_repos
gha2db
gha2db_sync
gha2es
ghapi2db
hide_data
import_affs
merge_dbs
replacer
runq
sqlitedb
structure
sync_issues
tags
vars
webhook
website_data

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.