crawler

package

v1.1.0 Latest Latest Go to latest Published: May 7, 2024 License: AGPL-3.0 Imports: 12 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/yhy0/Jie

Links

Open Source Insights

Documentation ¶

Index ¶

Variables
func Katana(target string, headless bool, show bool, out func(result output.Result))
func NewBrowser(noHeadless bool)
func NewCrawlergo(noHeadless bool)
type Request
type Result

Constants ¶

This section is empty.

Variables ¶

View Source

var (
	TaskConfig crawlergo.TaskConfig
	Browser    *engine.Browser
)

View Source

var ExtensionFilter = []string{
	".png", ".gif", ".jpg", ".mp4", ".mp3", ".mng", ".pct", ".bmp", ".jpeg", ".pst", ".psp", ".ttf",
	".tif", ".tiff", ".ai", ".drw", ".wma", ".ogg", ".wav", ".ra", ".aac", ".mid", ".au", ".aiff",
	".dxf", ".eps", ".ps", ".svg", ".3gp", ".asf", ".asx", ".avi", ".mov", ".mpg", ".qt", ".rm",
	".wmv", ".m4a", ".bin", ".xls", ".xlsx", ".ppt", ".pptx", ".doc", ".docx", ".odt", ".ods", ".odg",
	".odp", ".exe", ".zip", ".rar", ".tar", ".gz", ".iso", ".rss", ".pdf", ".dll", ".ico",
	".gz2", ".apk", ".crt", ".woff", ".map", ".woff2", ".webp", ".less", ".dmg", ".bz2", ".otf", ".swf",
	".flv", ".mpeg", ".dat", ".xsl", ".csv", ".cab", ".exif", ".wps", ".m4v", ".rmvb",
}

ExtensionFilter 默认过滤的后缀名

Functions ¶

func Katana ¶ added in v0.2.10

func Katana(target string, headless bool, show bool, out func(result output.Result))

func NewBrowser ¶ added in v1.0.0

func NewBrowser(noHeadless bool)

func NewCrawlergo ¶ added in v0.3.1

func NewCrawlergo(noHeadless bool)

Types ¶

type Request ¶ added in v0.3.1

type Request struct {
	Url     string                 `json:"url"`
	Method  string                 `json:"method"`
	Headers map[string]interface{} `json:"headers"`
	Data    string                 `json:"data"`
	Source  string                 `json:"source"`
}

type Result ¶ added in v0.3.1

type Result struct {
	ReqList       []Request `json:"req_list"`
	AllReqList    []Request `json:"all_req_list"`
	AllDomainList []string  `json:"all_domain_list"`
	SubDomainList []string  `json:"sub_domain_list"`
}

Source Files ¶

View all Source files

Directories ¶

Path	Synopsis
crawlergo
config
engine
filter
js
model
tools
tools/requests
xss

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL