Documentation
¶
Index ¶
- type Connection
- type Info
- type MapConn
- func (c *MapConn) CreateJob(jobUUID string) error
- func (c *MapConn) Done(jobUUID string) error
- func (c *MapConn) Processing(jobUUID string) error
- func (c *MapConn) Results(jobUUID string) ([][]byte, error)
- func (c *MapConn) Save(jobUUID string, src string) error
- func (c *MapConn) Status(jobUUID string) (*Info, error)
- func (c *MapConn) ViewPage(jobUUID string, url string) (bool, error)
- type Page
- type Pages
- type RiakConn
- func (d RiakConn) CreateJob(jobUUID string) error
- func (d RiakConn) Done(jobUUID string) error
- func (d RiakConn) Processing(jobUUID string) error
- func (d RiakConn) Results(jobUUID string) ([][]byte, error)
- func (d RiakConn) Save(jobUUID, src string) error
- func (d RiakConn) Status(jobUUID string) (*Info, error)
- func (d RiakConn) ViewPage(jobUUID string, url string) (bool, error)
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Connection ¶
type Connection interface {
// Create the job in the database.
CreateJob(string) error
// Processing increments the counter of currently processing urls for a given job.
Processing(string) error
// Done increments the counter of done urls
// and decrements the counter of processing urls for a given job.
Done(string) error
// Save adds an image source to the set of images for a given job.
Save(string, string) error
// Status returns the processing and done counters of a given job.
Status(string) (*Info, error)
// Results returns the processed images for a given job.
Results(string) ([][]byte, error)
// ViewPage decides whether a page needs to be crawled or not.
// One url should only be crawled once by a given job,
// but it depends on the guarantees that the storage provides.
ViewPage(string, string) (bool, error)
}
Connection is an interface that defines how data is saved and retrieved from a storage.
func NewRiakConn ¶
func NewRiakConn(conn *riak.Client) (Connection, error)
NewRiakConn creates a new new instance of the database to talk with Riak. It assumes that the client has already been initialized by the application context.
type MapConn ¶
type MapConn struct {
// contains filtered or unexported fields
}
MapConn implements the Connection interface using memory maps as backends. This interface is only suitable for testing. It offers no guarantees about the elements saved in it and it is not thread safe.
func (*MapConn) Done ¶
Done increments the counter of urls processed and decrements the counter of urls currently processing.
func (*MapConn) Processing ¶
Processing increments the counter of current urls processing.
type RiakConn ¶
type RiakConn struct {
// contains filtered or unexported fields
}
RiakConn implements the Connection interface using Riak as a backend. This is the prefered interface to use when running in a distributed environment.
func (RiakConn) CreateJob ¶
CreateJob initializes the job map in the Riak cluster. This operation must be performed before any crawling starts to guarantee that the process stores the data properly.
func (RiakConn) Done ¶
Done increments one element the counter of done urls and decrements the counter of processing urls for a given job.
func (RiakConn) Processing ¶
Processing increments the counter of currently processing urls for a given job.
Source Files