utils

package
v0.0.0-...-2646333 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Feb 20, 2020 License: Apache-2.0 Imports: 16 Imported by: 1

Documentation

Overview

Package utils contains various common utils separate by utility types

Package utils contains various common utils separate by utility types

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func IsWebsocketURL

func IsWebsocketURL(rawurl string) bool

IsWebsocketURL return true if the given URL is a websocket URL

func SetupHTTPEthClient

func SetupHTTPEthClient(ethURL string) (*ethclient.Client, error)

SetupHTTPEthClient returns an HTTP ethclient if URL is valid

func SetupWebsocketEthClient

func SetupWebsocketEthClient(ethURL string, killChan <-chan struct{}, pingDelay int) (*ethclient.Client, error)

SetupWebsocketEthClient returns an websocket ethclient if URL is valid. Sets up a "command" ping if killChan is not nil and pingDelay > 0. If message is sent to killChan, stops the ping.

Types

type CrawlerConfig

type CrawlerConfig struct {
	EthAPIURL     string `envconfig:"eth_api_url" required:"true" desc:"Ethereum HTTP API address"`
	EthWsAPIURL   string `envconfig:"eth_ws_api_url" desc:"Ethereum Websocket API address (optional, disables watchers if empty)"`
	EthStartBlock uint64 `envconfig:"eth_start_block" desc:"Sets the start Eth block (default 0)" default:"0"`

	// Enables polling mode, disables websockets
	PollingEnabled      bool `envconfig:"polling_enabled" desc:"Enable polling only mode (true disables listeners)"`
	PollingIntervalSecs int  `envconfig:"polling_int_secs" desc:"Sets the polling interval"`

	// CivilListingsGraphqlURL enables call to retrieve newsroom listings from Civil.
	// Should pass in the URL to the GraphQL endpoint to enable.
	CivilListingsGraphqlURL string `envconfig:"civil_listing_graphql_url" desc:"URL of the Civil Listings GraphQL endpoint"`

	// ContractAddresses map a contract type to a string of contract addresses.  If there are more than 1
	// contract to be tracked for a particular type, delimit the addresses with '|'.
	ContractAddresses   map[string]string           `` /* 131-byte string literal not displayed */
	ContractAddressObjs map[string][]common.Address `ignored:"true"`

	PersisterType             cconfig.PersisterType `ignored:"true"`
	PersisterTypeName         string                `split_words:"true" required:"true" desc:"Sets the persister type to use"`
	PersisterPostgresAddress  string                `split_words:"true" desc:"If persister type is Postgresql, sets the address"`
	PersisterPostgresPort     int                   `split_words:"true" desc:"If persister type is Postgresql, sets the port"`
	PersisterPostgresDbname   string                `split_words:"true" desc:"If persister type is Postgresql, sets the database name"`
	PersisterPostgresUser     string                `split_words:"true" desc:"If persister type is Postgresql, sets the database user"`
	PersisterPostgresPw       string                `split_words:"true" desc:"If persister type is Postgresql, sets the database password"`
	PersisterPostgresMaxConns *int                  `split_words:"true" desc:"If persister type is Postgresql, sets the max conns in pool"`
	PersisterPostgresMaxIdle  *int                  `split_words:"true" desc:"If persister type is Postgresql, sets the max idle conns in pool"`
	PersisterPostgresConnLife *int                  `split_words:"true" desc:"If persister type is Postgresql, sets the max conn lifetime in secs"`

	PubSubProjectID string `split_words:"true" desc:"Sets the Google Cloud PubSub Project ID"`
	PubSubTopicName string `split_words:"true" desc:"Sets the Google Cloud PubSub Topic name"`

	// VersionNumber is the version of DB for postgres persistence
	VersionNumber string `split_words:"true" desc:"Sets the version for table"`

	// Configuration for supported error reporting (Stackdriver, Sentry)
	StackdriverProjectID      string `split_words:"true" desc:"Sets the Stackdriver Google Cloud project ID. If empty, will disable logging"`
	StackdriverServiceName    string `split_words:"true" desc:"Sets the Stackdriver service name"`
	StackdriverServiceVersion string `split_words:"true" desc:"Sets the Stackdriver service version"`

	SentryDsn        string `split_words:"true" desc:"Sets the Sentry DSN"`
	SentryEnv        string `split_words:"true" desc:"Sets the Sentry environment"`
	SentryLoggerName string `split_words:"true" desc:"Sets the Sentry logger name"`
	SentryRelease    string `split_words:"true" desc:"Sets the Sentry release value"`

	// Configuration for pprof profiling
	PprofEnable                bool   `split_words:"true" desc:"Enables the local pprof endpoints for debugging and profiling"`
	CloudProfileProjectID      string `split_words:"true" desc:"Sets the cloud profiler Google Cloud project ID. If empty, will disable cloud profiler agent"`
	CloudProfileServiceName    string `split_words:"true" desc:"Sets the service name to with in the Google Cloud profiler"`
	CloudProfileServiceVersion string `split_words:"true" desc:"Sets the Google Cloud profiler service version"`

	PreemptSecs *int `split_words:"true" desc:"Sets the secs delay before listener prempt restart. 0 if no preempt"`
}

CrawlerConfig is the master config for the crawler derived from environment variables.

func (*CrawlerConfig) Address

func (c *CrawlerConfig) Address() string

PostgresAddress returns the postgres persister address, implements PersisterConfig

func (*CrawlerConfig) DataVersion

func (c *CrawlerConfig) DataVersion() string

DataVersion returns the version number from the config

func (*CrawlerConfig) Dbname

func (c *CrawlerConfig) Dbname() string

PostgresDbname returns the postgres persister db name, implements PersisterConfig

func (*CrawlerConfig) FetchListingAddresses

func (c *CrawlerConfig) FetchListingAddresses() error

FetchListingAddresses retrieves the list of Civil newsroom addresses if given the endpoint URL

func (*CrawlerConfig) FetchMultiSigAddresses

func (c *CrawlerConfig) FetchMultiSigAddresses() error

FetchMultiSigAddresses retrieves the list of Civil multi sig addresses if given the endpoint URL

func (*CrawlerConfig) OutputUsage

func (c *CrawlerConfig) OutputUsage()

OutputUsage prints the usage string to os.Stdout

func (*CrawlerConfig) Password

func (c *CrawlerConfig) Password() string

PostgresPw returns the postgres persister password, implements PersisterConfig

func (*CrawlerConfig) PersistType

func (c *CrawlerConfig) PersistType() cconfig.PersisterType

PersistType returns the persister type, implements PersisterConfig

func (*CrawlerConfig) PoolConnLifetimeSecs

func (c *CrawlerConfig) PoolConnLifetimeSecs() *int

PoolConnLifetimeSecs returns the conn lifetime for a pool, if configured, implements PersisterConfig

func (*CrawlerConfig) PoolMaxConns

func (c *CrawlerConfig) PoolMaxConns() *int

PoolMaxConns returns the max conns for a pool, if configured, implements PersisterConfig

func (*CrawlerConfig) PoolMaxIdleConns

func (c *CrawlerConfig) PoolMaxIdleConns() *int

PoolMaxIdleConns returns the max idleconns for a pool, if configured, implements PersisterConfig

func (*CrawlerConfig) PopulateFromEnv

func (c *CrawlerConfig) PopulateFromEnv() error

PopulateFromEnv processes the environment vars, populates CrawlerConfig with the respective values, and validates the values.

func (*CrawlerConfig) Port

func (c *CrawlerConfig) Port() int

PostgresPort returns the postgres persister port, implements PersisterConfig

func (*CrawlerConfig) User

func (c *CrawlerConfig) User() string

PostgresUser returns the postgres persister user, implements PersisterConfig

type Edge

type Edge struct {
	Node struct {
		Name            graphql.String
		ContractAddress graphql.String
		Whitelisted     graphql.Boolean
		ApprovalDate    graphql.Int
		LastGovState    graphql.String
	}
}

Edge represents an edge field in the query

type PageInfo

type PageInfo struct {
	EndCursor   graphql.String
	HasNextPage graphql.Boolean
}

PageInfo represents a pageinfo field in the query

type RebuildConfig

type RebuildConfig struct {
	PersisterType            cconfig.PersisterType `ignored:"true"`
	PersisterPostgresAddress string                `split_words:"true" required:"true" desc:"If persister type is Postgresql, sets the address"`
	PersisterPostgresPort    int                   `split_words:"true" required:"true" desc:"If persister type is Postgresql, sets the port"`
	PersisterPostgresDbname  string                `split_words:"true" required:"true" desc:"If persister type is Postgresql, sets the database name"`
	PersisterPostgresUser    string                `split_words:"true" required:"true" desc:"If persister type is Postgresql, sets the database user"`
	PersisterPostgresPw      string                `split_words:"true" required:"true" desc:"If persister type is Postgresql, sets the database password"`
}

RebuildConfig is the master config for the rebuild script derived from environment variables.

func (*RebuildConfig) OutputUsage

func (r *RebuildConfig) OutputUsage()

OutputUsage prints the usage string to os.Stdout

func (*RebuildConfig) PopulateFromEnv

func (r *RebuildConfig) PopulateFromEnv() error

PopulateFromEnv processes the environment vars, populates RebuildConfig with the respective values, and validates the values.

type WatcherSubscription

type WatcherSubscription interface {
	Err() <-chan error
	Unsubscribe()
}

WatcherSubscription is represents a subscription to a stream of watcher events. This is a modification/custom version of the Subscription struct in the go-ethereum package, so much credit to the authors there. https://github.com/ethereum/go-ethereum/blob/master/event/subscription.go

func NewWatcherSubscription

func NewWatcherSubscription(watcherName string, producer func(<-chan struct{}) error) WatcherSubscription

NewWatcherSubscription runs a producer function as a watcher subscription in a new goroutine. The channel given to the producer is closed when Unsubscribe is called. Returns error returned by producer to the subscription error channel. This is a modification/custom version of the NewSubscription struct in the go-ethereum package, so much credit to the authors there. https://github.com/ethereum/go-ethereum/blob/master/event/subscription.go Differs in how it handles completion of the producer function with separate channel to prevent deadlocks.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL