ambler

package module

v0.0.0-...-2e03a14 Latest Latest Go to latest Published: Jun 23, 2017 License: Apache-2.0 Imports: 8 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/justanotherorganization/ambler

Links

Open Source Insights

README ¶

ambler

Something I started awhile ago but forgot about
- IngCr3at1on 2017

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use these files except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Documentation ¶

Index ¶

Variables
func FilterDuplicates(in []string) (out []string)
func FilterPrefix(in []string, filterBy []string) (out []string)
func FilterURLs(in []string) (out []string)
func FindLinks(r io.Reader) (links []string)
func GetDocumentFromWeb(_url string, timeout time.Duration) (io.Reader, error)
type Page

Constants ¶

This section is empty.

Variables ¶

View Source

var (
	// FilterByHTTP recommended default filter set.
	FilterByHTTP = []string{"http://", "https://"}
)

Functions ¶

func FilterDuplicates ¶

func FilterDuplicates(in []string) (out []string)

FilterDuplicates takes a slice of strings and filters out any duplicates.

func FilterPrefix ¶

func FilterPrefix(in []string, filterBy []string) (out []string)

FilterPrefix takes an in slice of strings and a slice of prefix strings to filter by, returning only the strings which have a filter prefix.

func FilterURLs ¶

func FilterURLs(in []string) (out []string)

FilterURLs takes an in slice of strings returning only valid URL strings.

func FindLinks ¶

func FindLinks(r io.Reader) (links []string)

FindLinks takes a reader for an HTML document and returns all the links in string slice.

func GetDocumentFromWeb ¶

func GetDocumentFromWeb(_url string, timeout time.Duration) (io.Reader, error)

GetDocumentFromWeb fetches a document from the web uses a URL string, it returns the document in the form of an FIXME:

Types ¶

type Page ¶

type Page struct {
	Doc io.Reader
	URL string
}

Page is a reader containing a document and URL string of where the document is located.

Source Files ¶

View all Source files

Directories ¶

Path	Synopsis
cmd
basic_crawler
link_scraper

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL