fictionpress

package
v0.0.0-...-9b45353 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 30, 2018 License: MIT Imports: 13 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source
var Plugin = &dc.Plugin{
	Name:          "fictionpress",
	ConfigCreator: configCreator,
	Entrypoints: []string{
		`https:\/\/www.(fictionpress.com|fanfiction.net)\/s\/(.*)\/(\d+)(.*)`,
	},
	Scheduler: func(sr *dc.ScheduleRequest) ([]*dc.ScrapeSchedule, error) {
		if len(sr.LatestScrapes) == 0 {
			return nil, errors.New("discollect: cannot schedule a scrape without an initial scrape")
		}

		lastPosts, ok := sr.LatestDatums.([]*hydrocarbon.Post)
		if !ok {
			return nil, errors.New("fictionpress: did not get a hydrocarbon post back")
		}

		base := time.Now()

		if len(lastPosts) == 0 {
			conf := sr.LatestScrapes[0].Config

			return []*dc.ScrapeSchedule{{
				ScheduledStartAt: base.Add(time.Hour * 72),
				Config:           conf,
			}}, nil
		}

		fmt.Printf("%+v: \n %+v \n", lastPosts, lastPosts[0])

		return []*dc.ScrapeSchedule{{
			ScheduledStartAt: base.Add(time.Hour),
			Config: &dc.Config{
				Type:        dc.DeltaScrape,
				Entrypoints: []string{lastPosts[0].URL},
			},
		}}, nil
	},
	Routes: map[string]dc.Handler{
		`https:\/\/www.(fictionpress.com|fanfiction.net)\/s\/(.*)\/(\d+)(.*)`: storyPage,
	},
}

Plugin is a plugin that can scrape fictionpress

Functions

This section is empty.

Types

This section is empty.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL