Documentation ¶
Overview ¶
Package crawler provides helper methods and defines an interface for lauching source repository crawlers that retrieve files from a source and forwards to a channel for indexing and retrieval.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func CRunner ¶
func CRunner(ctx context.Context, output chan<- CrawledDocument, crawlers []Crawler) []error
CRunner is a blocking function and only returns once all of the crawlers are finished with execution.
This function uses the output channel to forward kustomization documents from a list of crawlers. The output is to be consumed by a database/search indexer for later retrieval.
The return value is an array of errors in which each index represents the index of the crawler that emitted the error. Although the errors themselves can be nil, the array will always be exactly the size of the crawlers array.
CRunner takes in a seed, which represents the documents stored in an index somewhere. The document data is not required to be populated. If there are many documents, this is preferable. The order of iteration over the seed is not guaranteed, but the CRunner does guarantee that every element from the seed will be processed before any other documents from the crawlers.
Types ¶
type CrawledDocument ¶
type Crawler ¶
type Crawler interface { // Crawl returns when it is done processing. This method does not take // ownership of the channel. The channel is write only, and it // designates where the crawler should forward the documents. Crawl(ctx context.Context, output chan<- CrawledDocument) error // Get the document data given the FilePath, Repo, and Ref/Tag/Branch. FetchDocument(context.Context, *doc.Document) error // Write to the document what the created time is. SetCreated(context.Context, *doc.Document) error Match(*doc.Document) bool }
Crawler forwards documents from source repositories to index and store them for searching. Each crawler is responsible for querying it's source of information, and forwarding files that have not been seen before or that need updating.
type IndexFunc ¶
type IndexFunc func(CrawledDocument, Crawler) error