oasishub-indexer

Description

This is a prototype of an app that is responsible for syncing, indexing Oasis blockchain data. It also exposes a JSON API for the clients to access this data. Processing pipeline is used to sync and index data. Currently we have 3 stages in the pipeline:

• Syncer - it is responsible for getting raw data from the node
• Sequencer - it is responsible for indexing data that we want to store for every height. This data later on can be used to get data for specific time intervals ie. get average block-time every hour.
• Aggregator - it is responsible for indexing only current height data ie. Current balances of accounts.

Syncables

Syncables are collections of data retrieved from Oasis node for given height. Above syncables are created during Syncer stage of the processing pipeline and stored to database.

Sequences

This is data that is stored in the database for every height. Sequences are used for data that changes frequently and we want to know about those changes. This data is perfect for displaying change over time using graphs on the front-end. Currently we store below sequences:

• Block
• Staking (disabled)
• Transactions (disabled)
• Validators
• Delegations (disabled)
• Debonding delegations (disabled)

Aggregates

This is data that is sored in the database for the most current "entity". Aggregates are used for data that does not change frequently or we don't care much about previous values. Currently we have below aggregates:

• Account (disabled)
• Validator

Internal dependencies:

This package connects via gRPC to a oasishub-proxy which in turn connects to Oasis node. This is required because for now the only way to connect to Oasis node is via unix socket. oasis-rpc-proxy also servers as a anti-corruption layer which is responsible for translating raw node data to the format indexer understands.

External Packages:

• oasis-rpc-proxy - Go proxy to Oasis node
• indexing-engine - A backbone for indexing process
• gin - Http server
• gorm - ORM with PostgreSQL interface
• cron - Cron jobs runner
• zap - logging

Environmental variables:

• APP_ENV - application environment (development | production)
• PROXY_URL - url to oasis-rpc-proxy
• SERVER_ADDR - address to use for API
• SERVER_PORT - port to use for API
• FIRST_BLOCK_HEIGHT - height of first block in chain
• INDEX_WORKER_INTERVAL - index interval for worker
• SUMMARIZE_WORKER_INTERVAL - summary interval for worker
• PURGE_WORKER_INTERVAL - purge interval for worker
• DEFAULT_BATCH_SIZE - syncing batch size. Setting this value to 0 means no batch size
• DATABASE_DSN - PostgreSQL database URL
• DEBUG - turn on db debugging mode
• LOG_LEVEL - level of log
• LOG_OUTPUT - log output (ie. stdout or /tmp/logs.json)
• ROLLBAR_ACCESS_TOKEN - Rollbar access token for error reporting
• INDEXER_METRIC_ADDR - Prometheus server address for indexer metrics
• METRIC_SERVER_URL - Url at which metrics will be accessible (for both indexer and server)
• PURGE_SEQUENCES_INTERVAL - Sequence older than given interval will be purged [DEFAULT: 24h]
• PURGE_SYSTE_EVENTS_INTERVAL - System events older than given interval will be purged [DEFAULT: 24h]
• PURGE_HOURLY_SUMMARY_INTERVAL - Hourly summaries records older than given interval will be purged [DEFAULT: 24h]
• INDEXER_CONFIG_FILE - JSON file with indexer configuration

Available endpoints:

Running app

Once you have created a database and specified all configuration options, you need to migrate the database. You can do that by running the command below:

oasishub-indexer -config path/to/config.json -cmd=migrate

Start the data indexer:

oasishub-indexer -config path/to/config.json -cmd=worker

Start the API server:

oasishub-indexer -config path/to/config.json -cmd=server

IMPORTANT!!! Make sure that you have oasishub-proxy running and connected to Oasis node.

Running one-off commands

Start indexing process:

oasishub-indexer -config path/to/config.json -cmd=indexer:index

Start backfill process:

oasishub-indexer -config path/to/config.json -cmd=indexer:backfill

Create summary tables for sequences:

oasishub-indexer -config path/to/config.json -cmd=indexer:summarize

Purge old data:

oasishub-indexer -config path/to/config.json -cmd=indexer:purge

Decorate validator aggregates:

oasishub-indexer -config path/to/config.json -cmd=validators:decorate -file=/file/to/csv

Running tests

To run tests with coverage you can use test Makefile target:

make test

Using indexer configuration file

Indexing process is configured using JSON file. A typical indexer config file looks similar to:

{
  "versions": [
    {
      "id": 1,
      "parallel": false,
      "targets": [1, 2]
    },
    {
      "id": 2,
      "parallel": false,
      "targets": [3]
    }
  ],
  "shared_tasks": [
    "HeightMetaRetriever",
    "MainSyncer"
  ],
  "available_targets": [
    {
      "id": 1,
      "name": "index_block_sequences",
      "desc": "Creates and persists block sequences",
      "tasks": [
        "BlockFetcher",
        "ValidatorFetcher",
        "TransactionFetcher",
        "BlockParser",
        "BlockSeqCreator",
        "SyncerPersistor",
        "BlockSeqPersistor"
      ]
    },
    {
      "id": 2,
      "name": "index_validator_sequences",
      "desc": "Creates and persists validator sequences",
      "tasks": [
        "BlockFetcher",
        "StakingStateFetcher",
        "ValidatorFetcher",
        "ValidatorsParser",
        "ValidatorSeqCreator",
        "SyncerPersistor",
        "ValidatorSeqPersistor"
      ]
    },
    {
      "id": 3,
      "name": "index_validator_aggregates",
      "desc": "Creates and persists validator aggregates",
      "tasks": [
        "BlockFetcher",
        "StakingStateFetcher",
        "ValidatorFetcher",
        "ValidatorsParser",
        "ValidatorAggCreator",
        "SyncerPersistor",
        "ValidatorAggPersistor"
      ]
    }
  ]
}

In this file we have 3 main sections:

• versions - describes available index versions and what targets to run for every version. parallel option specifies if the new index version can be run in parallel.
• shared_tasks - tasks that are shared between targets. These tasks will be run for every target in targets section.
• targets - array of available targets along with their tasks. Target represents some specific outcome of indexing process (ie. index validators) and it tells the system what tasks have to be run to satisfy this outcome.

Updating indexer version flow

• Add task to a stage in the indexer package
• If necessary, create migration file(s)
• Create a new target in the targets section of the indexer_config.json
• Add a new version in versions section. Inside of it, specify targets that need to be run to satisfy this version. If you provide parallel=true option it will mean that backfill and indexing can be run in parallel.