chunksPkg

package
v0.0.0-...-3f8eaf4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 19, 2024 License: GPL-3.0 Imports: 42 Imported by: 0

README

chifra chunks

The chifra chunks routine provides tools for interacting with, checking the validity of, cleaning up, and analyzing the Unchained Index. It provides options to list pins, the Manifest, summary data on the index, Bloom filters, addresses, and appearances. While still in its early stages, this tool will eventually allow users to clean their local index, clean their remote index, study the indexes, etc. Stay tuned.

Purpose:
  Manage, investigate, and display the Unchained Index.

Usage:
  chifra chunks <mode> [flags] [blocks...] [address...]

Arguments:
  mode - the type of data to process (required)
	One of [ manifest | index | blooms | pins | addresses | appearances | stats ]
  blocks - an optional list of blocks to intersect with chunk ranges

Flags:
  -c, --check              check the manifest, index, or blooms for internal consistency
  -i, --pin                pin the manifest or each index chunk and bloom
  -p, --publish            publish the manifest to the Unchained Index smart contract
  -r, --remote             prior to processing, retrieve the manifest from the Unchained Index smart contract
  -b, --belongs strings    in index mode only, checks the address(es) for inclusion in the given index chunk
  -F, --first_block uint   first block to process (inclusive)
  -L, --last_block uint    last block to process (inclusive)
  -m, --max_addrs uint     the max number of addresses to process in a given chunk
  -d, --deep               if true, dig more deeply during checking (manifest only)
  -e, --rewrite            for the --pin --deep mode only, writes the manifest back to the index folder (see notes)
  -U, --count              for the pins mode only, display only the count of records
  -s, --sleep float        for --remote pinning only, seconds to sleep between API calls
  -x, --fmt string         export format, one of [none|json*|txt|csv]
  -v, --verbose            enable verbose output
  -h, --help               display this help screen

Notes:
  - Mode determines which type of data to display or process.
  - Certain options are only available in certain modes.
  - If blocks are provided, only chunks intersecting with those blocks are displayed.
  - The --truncate option updates the manifest and removes local data, but does not alter remote pins.
  - The --belongs option is only available in the index mode.
  - The --first_block and --last_block options apply only to addresses, appearances, and index --belongs mode.
  - The --pin option requires a locally running IPFS node or a pinning service API key.
  - The --publish option requires a private key.
  - The --publisher option is ignored with the --publish option since the sender of the transaction is recorded as the publisher.
  - Without --rewrite, the manifest is written to the temporary cache. With it, the manifest is rewritten to the index folder.

Data models produced by this tool:

Other Options

All tools accept the following additional flags, although in some cases, they have no meaning.

  -v, --version         display the current version of the tool
      --output string   write the results to file 'fn' and return the filename
      --append          for --output command only append to instead of replace contents of file
      --file string     specify multiple sets of command line options in a file

Note: For the --file string option, you may place a series of valid command lines in a file using any valid flags. In some cases, this may significantly improve performance. A semi-colon at the start of any line makes it a comment.

Note: If you use --output --append option and at the same time the --file option, you may not switch export formats in the command file. For example, a command file with two different commands, one with --fmt csv and the other with --fmt json will produce both invalid CSV and invalid JSON.

Documentation

Overview

Package chunksPkg handles the chifra chunks command. It The routine provides tools for interacting with, checking the validity of, cleaning up, and analyzing the Unchained Index. It provides options to list pins, the Manifest, summary data on the index, Bloom filters, addresses, and appearances. While still in its early stages, this tool will eventually allow users to clean their local index, clean their remote index, study the indexes, etc. Stay tuned.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func FormattedTag

func FormattedTag(verbose bool, hash base.Hash) string

func GetChunkStats

func GetChunkStats(chain, path string) (s simpleChunkStats, err error)

EXISTING_CODE

func ResetOptions

func ResetOptions(testMode bool)

func RunChunks

func RunChunks(cmd *cobra.Command, args []string) error

RunChunks handles the chunks command for the command line. Returns error only as per cobra.

func ServeChunks

func ServeChunks(w http.ResponseWriter, r *http.Request) error

ServeChunks handles the chunks command for the API. Returns an error.

Types

type ChunksOptions

type ChunksOptions struct {
	Mode       string                   `json:"mode,omitempty"`       // The type of data to process
	Blocks     []string                 `json:"blocks,omitempty"`     // An optional list of blocks to intersect with chunk ranges
	BlockIds   []identifiers.Identifier `json:"blockIds,omitempty"`   // Block identifiers
	Check      bool                     `json:"check,omitempty"`      // Check the manifest, index, or blooms for internal consistency
	Pin        bool                     `json:"pin,omitempty"`        // Pin the manifest or each index chunk and bloom
	Publish    bool                     `json:"publish,omitempty"`    // Publish the manifest to the Unchained Index smart contract
	Publisher  string                   `json:"publisher,omitempty"`  // For some query options, the publisher of the index
	Truncate   uint64                   `json:"truncate,omitempty"`   // Truncate the entire index at this block (requires a block identifier)
	Remote     bool                     `json:"remote,omitempty"`     // Prior to processing, retrieve the manifest from the Unchained Index smart contract
	Belongs    []string                 `json:"belongs,omitempty"`    // In index mode only, checks the address(es) for inclusion in the given index chunk
	Diff       bool                     `json:"diff,omitempty"`       // Compare two index portions (see notes)
	FirstBlock uint64                   `json:"firstBlock,omitempty"` // First block to process (inclusive)
	LastBlock  uint64                   `json:"lastBlock,omitempty"`  // Last block to process (inclusive)
	MaxAddrs   uint64                   `json:"maxAddrs,omitempty"`   // The max number of addresses to process in a given chunk
	Deep       bool                     `json:"deep,omitempty"`       // If true, dig more deeply during checking (manifest only)
	Rewrite    bool                     `json:"rewrite,omitempty"`    // For the --pin --deep mode only, writes the manifest back to the index folder (see notes)
	List       bool                     `json:"list,omitempty"`       // For the pins mode only, list the remote pins
	Unpin      bool                     `json:"unpin,omitempty"`      // For the pins mode only, if true reads local ./unpins file for valid CIDs and remotely unpins each (skips non-CIDs)
	Count      bool                     `json:"count,omitempty"`      // For the pins mode only, display only the count of records
	Tag        string                   `json:"tag,omitempty"`        // Visits each chunk and updates the headers with the supplied version string (vX.Y.Z-str)
	Sleep      float64                  `json:"sleep,omitempty"`      // For --remote pinning only, seconds to sleep between API calls
	Globals    globals.GlobalOptions    `json:"globals,omitempty"`    // The global options
	Conn       *rpc.Connection          `json:"conn,omitempty"`       // The connection to the RPC server
	BadFlag    error                    `json:"badFlag,omitempty"`    // An error flag if needed
	// EXISTING_CODE
	PublisherAddr base.Address `json:"-"`
}

ChunksOptions provides all command options for the chifra chunks command.

func GetChunksOptions

func GetChunksOptions(args []string, g *globals.GlobalOptions) *ChunksOptions

GetChunksOptions returns the options for this tool so other tools may use it.

func GetOptions

func GetOptions() *ChunksOptions

func (*ChunksOptions) CheckDeep

func (opts *ChunksOptions) CheckDeep(cacheMan *manifest.Manifest, report *simpleReportCheck) error

CheckDeep digs deep into the data. In `index` mode, it opens each index and checks that all addresses in the index return true when checked against its corresponding Bloom filter. In `manifest` mode, it checks that each IPFS hash in the manifest is actually pinned. The later requires a locally running IPFS node.

func (*ChunksOptions) CheckHashes

func (opts *ChunksOptions) CheckHashes(cacheMan *manifest.Manifest, contractMan *manifest.Manifest, report *simpleReportCheck) error

CheckHashes looks at all the hashes in both the locally cached manifest and the manifest retrieved from the smart contract and simply checks the lengths of the bloom and index IPFS hashes.

func (*ChunksOptions) CheckInternal

func (opts *ChunksOptions) CheckInternal(fileNames []string, blockNums []uint64, report *simpleReportCheck) error

CheckInternal reads the header of each chunk on disc looking for the Magic number and the hash of the spec version for expected values.

func (*ChunksOptions) CheckManifest

func (opts *ChunksOptions) CheckManifest(arrayA, arrayB []string, report *simpleReportCheck) error

CheckManifest takes two arrays (either onDisc vs. LocalManifest, onDisc vs. RemoteManifest, or LocalManifest vs. RemoteManifest) and compares them for equality. If everything is up to date, all three arrays should be identical. Only the block ranges are in the arrays.

func (*ChunksOptions) CheckSequential

func (opts *ChunksOptions) CheckSequential(fnArray, cacheArray, remoteArray []string, allowMissing bool, report *simpleReportCheck) error

CheckSequential checks that the range of blocks in each of three arrays (onDisc, LocalManifest, RemoteManifest) "cover" the range without gaps. (There may be a bug for allow_missing chains where gaps are allowed.) It also makes sure than snap-to-grids happen where they should and that non-snaps have at least appsPerChunks records and snaps have exactly appsPerChunks records or less.

func (*ChunksOptions) CheckSizes

func (opts *ChunksOptions) CheckSizes(fileNames []string, blockNums []uint64, cacheManifest *manifest.Manifest, remoteManifest *manifest.Manifest, report *simpleReportCheck) error

CheckSizes compares the file on disc to the file size suggested in the manifest

func (*ChunksOptions) CheckStaging

func (opts *ChunksOptions) CheckStaging(lastBlock uint64, allow_missing bool, report *simpleReportCheck) error

CheckStaging checks the staging file which should be names first-second.txt

  1. Makes sure that finalized + 1 == first
  2. Makes sure the first < last
  3. Makes sure that the first block inside is == first if allow_missing == false, > otherwise
  4. Makes sure that the last block inside is == last if allow_missing == false, < otherwise

func (*ChunksOptions) CheckVersion

func (opts *ChunksOptions) CheckVersion(fileNames []string, blockNums []uint64, report *simpleReportCheck) error

CheckInternal reads the header of each chunk on disc looking for the Magic number and the hash of the spec version for expected values.

func (*ChunksOptions) ChunksInternal

func (opts *ChunksOptions) ChunksInternal() error

ChunksInternal handles the internal workings of the chunks command. Returns an error.

func (*ChunksOptions) HandleAddresses

func (opts *ChunksOptions) HandleAddresses(blockNums []uint64) error

func (*ChunksOptions) HandleAppearances

func (opts *ChunksOptions) HandleAppearances(blockNums []uint64) error

func (*ChunksOptions) HandleBlooms

func (opts *ChunksOptions) HandleBlooms(blockNums []uint64) error

func (*ChunksOptions) HandleCheck

func (opts *ChunksOptions) HandleCheck(blockNums []uint64) error

HandleCheck looks at three different arrays: index files on disc, manifest on disc, and manifest in the smart contract. It tries to check these three sources for cosnsistency. Smart contract rules, so it is checked more thoroughly.

func (*ChunksOptions) HandleDiff

func (opts *ChunksOptions) HandleDiff(blockNums []uint64) error

func (*ChunksOptions) HandleDump

func (opts *ChunksOptions) HandleDump(blockNums []uint64) error

func (*ChunksOptions) HandleIndex

func (opts *ChunksOptions) HandleIndex(blockNums []uint64) error

func (*ChunksOptions) HandleIndexBelongs

func (opts *ChunksOptions) HandleIndexBelongs(blockNums []uint64) error

HandleIndexBelongs displays the resolved records in a chunk given a single address

func (*ChunksOptions) HandleList

func (opts *ChunksOptions) HandleList(unused []uint64) error

func (*ChunksOptions) HandleManifest

func (opts *ChunksOptions) HandleManifest(blockNums []uint64) error

func (*ChunksOptions) HandlePin

func (opts *ChunksOptions) HandlePin(blockNums []uint64) error

func (*ChunksOptions) HandlePublish

func (opts *ChunksOptions) HandlePublish(blockNums []uint64) error

func (*ChunksOptions) HandleStats

func (opts *ChunksOptions) HandleStats(blockNums []uint64) error

func (*ChunksOptions) HandleTag

func (opts *ChunksOptions) HandleTag(blockNums []uint64) error

func (*ChunksOptions) HandleTruncate

func (opts *ChunksOptions) HandleTruncate(blockNums []uint64) error

func (*ChunksOptions) HandleUnpin

func (opts *ChunksOptions) HandleUnpin(unused []uint64) error

func (*ChunksOptions) String

func (opts *ChunksOptions) String() string

String implements the Stringer interface

type CompareState

type CompareState struct {
	// contains filtered or unexported fields
}

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL