downloader

package
v0.0.0-...-0328e99 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 25, 2018 License: MPL-2.0 Imports: 15 Imported by: 0

Documentation

Overview

Package downloader is the main module of GO_SPIDER for download page.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Downloader

type Downloader interface {
	Download(req *request.Request) *page.Page
}

The Downloader interface. You can implement the interface by implement function Download. Function Download need to return Page instance pointer that has request result downloaded from Request.

type HttpDownloader

type HttpDownloader struct {
}

The HttpDownloader download page by package net/http. The "html" content is contained in dom parser of package goquery. The "json" content is saved. The "jsonp" content is modified to json. The "text" content will save body plain text only. The page result is saved in Page.

func NewHttpDownloader

func NewHttpDownloader() *HttpDownloader

func (*HttpDownloader) Download

func (this *HttpDownloader) Download(req *request.Request) *page.Page

func (*HttpDownloader) DownloadCore

func (this *HttpDownloader) DownloadCore(req *request.Request) *page.Page

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL