cmd

package

v0.0.0-...-28f2d27 Latest Latest Go to latest Published: Apr 4, 2021 License: Apache-2.0 Imports: 15 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/linuxing3/gospider

Links

Open Source Insights

Documentation ¶

Index ¶

Variables
func CreateDefaultInitConfig()
func ExampleScrape()
func Execute()
func GetArticleDetail(url string)
func InitPrismaClient() *db.PrismaClient
func SaveArticle(articles []GoogleNewsArticle)
func SaveMovies(movies []DoubanMovie)
type Any
type DoubanMovie
- func ParseMovies(doc *goquery.Document) (movies []DoubanMovie)
type GoogleNewsArticle
- func GetArticles(url string, keyword string) (articles []GoogleNewsArticle)
type Page
- func GetPages(url string) (pages []Page)

Constants ¶

This section is empty.

Variables ¶

View Source

var (
	DoubanBaseUrl         = "https://movie.douban.com/top250"
	DoubanTopPageSelector = "#content > div > div.article > div.paginator > a"
)

View Source

var (
	VenBaseUrl         = "https://news.google.com/search?q=venezuela&hl=es-419&gl=VE&ceid=VE%3Aes-419"
	VenAritcleSelector = "article > h3 > a"
)

Functions ¶

func CreateDefaultInitConfig ¶

func CreateDefaultInitConfig()

CreateDefaultInitConfig 创建默认的配置文件

func ExampleScrape ¶

func ExampleScrape()

ExampleScrape 测试抓取网页

func Execute ¶

func Execute()

Execute 执行rootCmd

func GetArticleDetail ¶

func GetArticleDetail(url string)

func InitPrismaClient ¶

func InitPrismaClient() *db.PrismaClient

TODO: InitPrismaClient 初始化PrismaClient, 不能成功调用,因为context不同

func SaveArticle ¶

func SaveArticle(articles []GoogleNewsArticle)

func SaveMovies ¶

func SaveMovies(movies []DoubanMovie)

SaveMovies 保存电影记录到数据库

Types ¶

type Any ¶

type Any map[string]interface{}

Any 别名

type DoubanMovie ¶

type DoubanMovie struct {
	Title    string
	Subtitle string
	Other    string
	Desc     string
	Year     string
	Area     string
	Tag      string
	Star     string
	Comment  string
	Quote    string
}

func ParseMovies ¶

func ParseMovies(doc *goquery.Document) (movies []DoubanMovie)

ParseMovies 在每一个页面上分析电影数据

type GoogleNewsArticle ¶

type GoogleNewsArticle struct {
	Title string
	Url   string
}

func GetArticles ¶

func GetArticles(url string, keyword string) (articles []GoogleNewsArticle)

GetPages 获取分页

type Page ¶

type Page struct {
	Page int
	Url  string
}

func GetPages ¶

func GetPages(url string) (pages []Page)

GetPages 获取分页

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL