Documentation
¶
Overview ¶
Copyright 2020-2021 William Perron. All rights reserved. MIT License.
Copyright 2020-2021 William Perron. All rights reserved. MIT License.
Copyright 2020-2021 William Perron. All rights reserved. MIT License.
Copyright 2020-2021 William Perron. All rights reserved. MIT License.
Index ¶
Constants ¶
const API_HOST = "api.deno.land"
const CDN_HOST = "cdn.deno.land"
const PREFIX_LENGTH = len("https://deno.land/x/")
Variables ¶
This section is empty.
Functions ¶
func Enqueue ¶
Enqueue serves as a passthrough for channels of deno.Module. It puts the incoming messages in a Queue and consumes it to send messages down the output channel as well. It serves as an intermediary steps where an implementation of a Queue that uses a persistent back end like SQS or Kafka can be used. This is necessary to be able to start and stop the crawler arbitrarily and pick up where it left off
Types ¶
type ChanQueue ¶
type ChanQueue struct {
// contains filtered or unexported fields
}
ChanQueue is an in-memory queue that uses channels under the hood. If the channel is unbuffered, Put and Get are blocking operations
func NewChanQueue ¶
NewChanQueue returns a new ChanQueue instance
type Client ¶
Client interface defines the basic functions of an HTTP crawler
func DefaultClient ¶
func DefaultClient() Client
DefaultClient returns an instance of a crawler that uses the default http client
func NewInstrumentedClient ¶
func NewInstrumentedClient() Client
NewInstrumentedClient returns an instance of a crawler that uses an http client intstrumented with Prometheus
type DenoInfo ¶
type DenoInfo struct {
TotalSize int `json:"totalSize"`
Module string `json:"module"`
Map *string `json:"map"`
Compiled *string `json:"compiled"`
DepCount int `json:"depCount"`
FileType string `json:"fileType"`
Files map[string]FileEntry `json:"files"`
}
DenoInfo is the in-memory representation of the output of `deno info --json`
type Module ¶
Module contains the name of the volume and a map of all its versions to all the files contained in the module
type Queue ¶
type Queue interface {
Put(Module) error
Get() (Module, error)
// contains filtered or unexported methods
}
Queue interface for putting and getting messages. The interface doesn make any guarantees about message ordering, this concern must be managed by the interface implementation.
type SQSQueue ¶
type SQSQueue struct {
// contains filtered or unexported fields
}
SQSQueue is a simple abstraction over the standard sqs.Client struct that implements the Queue interface
func NewSQSQueue ¶
NewSQSQueue instantiates a new SQS Client with the given config
func (*SQSQueue) Approx ¶
Approx returns the approximate total number of messages in the queue, visible, delayed or not visible.
func (*SQSQueue) Delete ¶
Delete uses the message's receipt handle to delete the message from the queue
type XQueuedCrawler ¶
XQueuedCrawler is a composite type composed of both a Queue and a Crawler
func NewXQueuedCrawler ¶
func NewXQueuedCrawler(q Queue) *XQueuedCrawler
NewXQueuedCrawler returns an instance of a crawler for https://deno.land with a Queue
func (*XQueuedCrawler) Crawl ¶
func (x *XQueuedCrawler) Crawl(ctx context.Context) chan error
Crawl asynchronously crawls https://deno.land and puts each Module in the queue to be processed later
func (*XQueuedCrawler) Done ¶
func (x *XQueuedCrawler) Done() <-chan bool
Done returns the done channel of the crawler
func (*XQueuedCrawler) IterateModules ¶
func (x *XQueuedCrawler) IterateModules(ctx context.Context) (chan Module, chan error)
IterateModules asynchronously consumes the queue and sends each Module to a channel
Source Files