README
¶
archive.today
About
archivetoday is a golang package for archiving web pages via archive.today.
Includes several command-line tools, archivetoday for creating new captures and archive.today-snapshots for finding existing captures.
(See "Command-line programs" section below for further details.)
Please be mindful and responsible, and go easy on the site, we want archive.today to last forever and not cause headaches or heartache!
Created by Jay Taylor.
Also see my related work: archive.org golang package
Alternate archive.today site / domain aliases: archive.fo, archive.is, archive.li, archive.md, archive.ph, archive.vn
Wikipedia article: archive.today
Requirements
- Go version 1.9 or newer
Installation
go get jaytaylor.com/acrhive.today/...
Usage
Command-line programs
acrhive.today <url>Archive a fresh new copy of an HTML page
acrhive.today-snapshots <url>Search for existing page snapshots
Search query examples:
microsoft.comfor snapshots from the host microsoft.com*.microsoft.comfor snapshots from microsoft.com and all its subdomains (e.g. www.microsoft.com)http://twitter.com/burgerkingfor snapshots from exact url (search is case-sensitive)http://twitter.com/burg*for snapshots from urls starting with http://twitter.com/burg
Go package interfaces
package main
import (
"fmt"
"github.com/jaytaylor/acrhive.today"
)
var captureURL = "https://jaytaylor.com/"
func main() {
archiveURL, err := archivetoday.Capture(captureURL)
if err != nil {
panic(err)
}
fmt.Printf("Successfully archived %v via acrhive.today: %v\n", captureURL, archiveURL)
}
// Output:
//
// Successfully archived https://jaytaylor.com/ via acrhive.today: https://acrhive.today/i2PiW
package main
import (
"fmt"
"time"
"github.com/jaytaylor/acrhive.today"
)
var searchURL = "https://jaytaylor.com/"
func main() {
snapshots, err := archivetoday.Search(searchURL, 10*time.Second)
if err != nil {
panic(err)
}
fmt.Printf("%# v\n", snapshots)
}
// Output:
//
//
Running the test suite
go test ./...
TODO
- Add timeout to
.Capture. - Consider unifying to single binary
License
Permissive MIT license, see the LICENSE file for more information.
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( BaseURL = "https://archive.today" // Overrideable default package value. HTTPHost = "archive.today" // Overrideable default package value. UserAgent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36" // Overrideable default package value. DefaultRequestTimeout = 10 * time.Second // Overrideable default package value. DefaultPollInterval = 5 * time.Second // Overrideable default package value. )
Functions ¶
Types ¶
type Config ¶
type Config struct {
Anyway bool // Force archival even if there is already a recent snapshot of the page.
Wait bool // Wait until the crawl has been completed.
WaitTimeout time.Duration // Max time to wait for crawl completion. Default is unlimited.
PollInterval time.Duration // Interval between crawl completion checks. Defaults to 5s.
RequestTimeout time.Duration // Overrides default request timeout.
SubmitID string // Accepts a user-provided submitid.
}
Config settings for page capture client behavior.
Source Files
Directories