README
¶
This package is a fork of Obelisk.
It adds some needed features such as:
- A specific flag to fetch only images, and not all media
- Callbacks for URL and content processing.
- Ability to use your own HTTP Client
- Ability to use any logger
Obelisk is originally written by RadhiFadlillah and released under an MIT License.
Documentation
¶
Overview ¶
Package archiver provides functions to archive the content of a full HTML page.
Index ¶
- func DefaultImageProcessor(_ context.Context, _ *Archiver, input io.Reader, contentType string, ...) ([]byte, string, error)
- func DefaultURLProcessor(_ string, content []byte, contentType string) string
- func GetContextNode(ctx context.Context) (node *html.Node, ok bool)
- type ArchiveFlag
- type Archiver
- type Asset
- type Event
- type EventError
- type EventFetchURL
- type EventInfo
- type EventStartHTML
- type Request
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func DefaultImageProcessor ¶
func DefaultImageProcessor(_ context.Context, _ *Archiver, input io.Reader, contentType string, _ *url.URL, ) ([]byte, string, error)
DefaultImageProcessor is the default image processor. It simply reads and return the content.
func DefaultURLProcessor ¶
DefaultURLProcessor is the default URL processor. It returns the base64 encoded URL.
Types ¶
type ArchiveFlag ¶
type ArchiveFlag uint8
ArchiveFlag is an archiver feature to enable.
const ( // EnableCSS enables extraction of CSS files and tags. EnableCSS ArchiveFlag = 1 << iota // EnableEmbeds enables extraction of Embedes contents. EnableEmbeds // EnableJS enables extraction of JavaScript contents. EnableJS // EnableMedia enables extraction of media contents // other than image. EnableMedia // EnableImages enables extraction of images. EnableImages )
type Archiver ¶
type Archiver struct {
sync.RWMutex
Cache map[string]Asset
Request *Request
Result []byte
Flags ArchiveFlag
ImageProcessor imageProcessor
URLProcessor urlProcessor
EventHandler eventHandler
RequestTimeout time.Duration
SkipTLSVerification bool
MaxConcurrentDownload int64
// contains filtered or unexported fields
}
Archiver is the core of obelisk, which used to download a web page then embeds its assets.
type Event ¶
type Event interface {
Fields() map[string]interface{}
}
Event is the interface for events emitted by the archiver.
type EventError ¶
EventError is the event emitted when errors occur.
func (*EventError) Fields ¶
func (e *EventError) Fields() map[string]interface{}
Fields returns the field map.
type EventFetchURL ¶
type EventFetchURL struct {
// contains filtered or unexported fields
}
EventFetchURL is the event emitted when the archiver loads a remote resource.
func (*EventFetchURL) Fields ¶
func (e *EventFetchURL) Fields() map[string]interface{}
Fields returns the field map.
type EventInfo ¶
type EventInfo map[string]interface{}
EventInfo is a simple event for any type of data.
type EventStartHTML ¶
type EventStartHTML string
EventStartHTML is the event emitted at the beginning of the archiving process.
func (EventStartHTML) Fields ¶
func (e EventStartHTML) Fields() map[string]interface{}
Fields returns the field map.
Source Files