Index | Files | Directories

package importer

import ""

Package importer imports content from third-party websites.


Package Files

attrs.go html.go importer.go noop.go oauth.go


const (

    // AcctAttrUserID is the account's internal representation, and often an account number.
    // It is usually required as an argument in API calls to the site we import from.
    // Not found on
    // Example: "3179713".
    AcctAttrUserID = "userID"
    // AcctAttrUserName is the public identifier of the account. Commonly referred to as
    // "username", or "screen name", or "account name". Often a one word string.
    // Not found on
    // Example: "johnSmith" from Twitter's "@johnSmith".
    AcctAttrUserName = "userName"

    // AcctAttrCompletedVersion records the importer-specific
    // "version number" that last ran to completion, doing a full
    // importer. When importers are updated with new behavior,
    // they update their version number and that triggers a full
    // import, rather than incremental imports.
    AcctAttrCompletedVersion = "completedVersion"

    // AcctAttrName is a longer or alternate public representation of the account's name.
    // It is often the full name of the person's account (family name and given name), thus
    // sometimes redundant with the combination of acctAttrFamilyName and acctAttrGivenName.
    // Found at
    // Example: "John Smith".
    AcctAttrName = "name"
    // Example: "John".
    AcctAttrGivenName = "givenName"
    // Example: "Smith".
    AcctAttrFamilyName = "familyName"

    // ItemAttrID is the generic identifier of an item when nothing suitable and more specific
    // was found on Usually a number.
    AttrID = "ID"
    AttrName = "name"
    // Free-flowing text definition of a location or place, such
    // as a city name, or a full postal address.
    AttrLocationText = "locationText"
    // AttrURL is the item's original or origin URL.
    AttrURL = "url"

    // AttrStartDate is The start
    // date and time of the event or item (in ISO 8601 date
    // format)
    AttrStartDate = "startDate"

TODO(mpl): use these on all the importers.

const (
    AcctAttrTempToken         = "oauthTempToken"
    AcctAttrTempSecret        = "oauthTempSecret"
    AcctAttrAccessToken       = "oauthAccessToken"
    AcctAttrAccessTokenSecret = "oauthAccessTokenSecret"

func All Uses

func All() map[string]Importer

All returns the map of importer implementation name to implementation. This map should not be mutated.

func Register Uses

func Register(name string, im Importer)

Register registers a site-specific importer. It should only be called from init, and not from concurrent goroutines.

func RegisterTODO Uses

func RegisterTODO(name string, p Properties)

type Host Uses

type Host struct {
    // contains filtered or unexported fields

Host is the HTTP handler and state for managing all the importers linked into the binary, even if they're not configured.

func NewHost Uses

func NewHost(hc HostConfig) (*Host, error)

func (*Host) AccountsStatus Uses

func (h *Host) AccountsStatus() (interface{}, []camtypes.StatusError)

AccountsStatus returns the currently configured accounts and their status for inclusion in the status.json document, as rendered by the web UI.

func (*Host) BaseURL Uses

func (h *Host) BaseURL() string

BaseURL returns the root of the whole server, without trailing slash.

func (*Host) BlobSource Uses

func (h *Host) BlobSource() blob.Fetcher

func (*Host) HTTPClient Uses

func (h *Host) HTTPClient() *http.Client

HTTPClient returns the HTTP client to use.

func (*Host) HTTPTransport Uses

func (h *Host) HTTPTransport() http.RoundTripper

HTTPTransport returns the HTTP transport to use.

func (*Host) ImporterBaseURL Uses

func (h *Host) ImporterBaseURL() string

ImporterBaseURL returns the URL base of the importer handler, including trailing slash.

func (*Host) InitHandler Uses

func (h *Host) InitHandler(hl blobserver.FindHandlerByTyper) error

func (*Host) NewObject Uses

func (h *Host) NewObject() (*Object, error)

NewObject creates a new permanode and returns its Object wrapper.

func (*Host) ObjectFromRef Uses

func (h *Host) ObjectFromRef(permanodeRef blob.Ref) (*Object, error)

ObjectFromRef returns the object given by the named permanode

func (*Host) RunImporterAccount Uses

func (h *Host) RunImporterAccount(importerType string, accountNode blob.Ref) error

RunImporterAccount runs the importerType importer on the account described in accountNode.

func (*Host) Searcher Uses

func (h *Host) Searcher() search.QueryDescriber

func (*Host) ServeHTTP Uses

func (h *Host) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP serves:

http://host/importer/twitter/sha1-abcabcabcabcabc (single account)

func (*Host) Target Uses

func (h *Host) Target() blobserver.StatReceiver

type HostConfig Uses

type HostConfig struct {
    BaseURL      string
    Prefix       string                  // URL prefix for the importer handler
    Target       blobserver.StatReceiver // storage for the imported object blobs
    BlobSource   blob.Fetcher            // for additional resources, such as twitter zip file
    Signer       *schema.Signer
    Search       search.QueryDescriber
    ClientId     map[string]string // optionally maps importer impl name to a clientId credential
    ClientSecret map[string]string // optionally maps importer impl name to a clientSecret credential

    // HTTPClient optionally specifies how to fetch external network
    // resources. The Host will use http.DefaultClient otherwise.
    HTTPClient *http.Client

HostConfig holds the parameters to set up a Host.

type Importer Uses

type Importer interface {
    // Run runs a full or incremental import.
    // The importer should continually or periodically monitor the
    // RunContext.Context()'s Done channel to exit early if
    // requested. The return value should be ctx.Err() if the
    // importer exits for that reason.
    Run(*RunContext) error

    // Properties returns properties of this importer type.
    Properties() Properties

    // IsAccountReady reports whether the provided account node
    // is configured.
    IsAccountReady(acctNode *Object) (ok bool, err error)
    SummarizeAccount(acctNode *Object) string

    ServeSetup(w http.ResponseWriter, r *http.Request, ctx *SetupContext) error
    ServeCallback(w http.ResponseWriter, r *http.Request, ctx *SetupContext)

    // CallbackRequestAccount extracts the blobref of the importer account from
    // the callback URL parameters of r. For example, it will be encoded as:
    // For Twitter (OAuth1), in its own URL parameter: "acct=sha1-f2b0b7da718b97ce8c31591d8ed4645c777f3ef4"
    // For Picasa: (OAuth2), in the OAuth2 "state" parameter: "state=acct:sha1-97911b1a5887eb5862d1c81666ba839fc1363ea1"
    CallbackRequestAccount(r *http.Request) (acctRef blob.Ref, err error)

    // CallbackURLParameters uses the input importer account blobRef to build
    // and return the URL parameters, that will be appended to the callback URL.
    CallbackURLParameters(acctRef blob.Ref) url.Values

An Importer imports from a third-party site.

type ImporterSetupHTMLer Uses

type ImporterSetupHTMLer interface {
    AccountSetupHTML(*Host) string

ImporterSetupHTMLer is an optional interface that may be implemented by Importers to return some HTML to be included on the importer setup page.

type LongPoller Uses

type LongPoller interface {

    // LongPoll waits and returns nil when there's new content.
    // It does not fetch the content itself.
    // It returns a non-nil error if it failed to long poll.
    LongPoll(*RunContext) error

LongPoller is optionally implemented by importers which can long poll efficiently to wait for new content. For example, Twitter uses this to subscribe to the user's stream.

type OAuth1 Uses

type OAuth1 struct{}

OAuth1 provides methods that the importer implementations can use to help with OAuth authentication.

func (OAuth1) CallbackRequestAccount Uses

func (OAuth1) CallbackRequestAccount(r *http.Request) (blob.Ref, error)

func (OAuth1) CallbackURLParameters Uses

func (OAuth1) CallbackURLParameters(acctRef blob.Ref) url.Values

type OAuth2 Uses

type OAuth2 struct{}

OAuth2 provides methods that the importer implementations can use to help with OAuth2 authentication.

func (OAuth2) CallbackRequestAccount Uses

func (OAuth2) CallbackRequestAccount(r *http.Request) (blob.Ref, error)

func (OAuth2) CallbackURLParameters Uses

func (OAuth2) CallbackURLParameters(acctRef blob.Ref) url.Values

func (OAuth2) IsAccountReady Uses

func (OAuth2) IsAccountReady(acctNode *Object) (ok bool, err error)

IsAccountReady returns whether the account has been properly configured - whether the user ID and access token has been stored in the given account node.

func (OAuth2) RedirectState Uses

func (OAuth2) RedirectState(imp Importer, ctx *SetupContext) (state string, err error)

RedirectState returns the "state" query parameter that should be used for the authorization phase of OAuth2 authentication. This parameter contains the query component of the redirection URI. See

func (OAuth2) RedirectURL Uses

func (OAuth2) RedirectURL(imp Importer, ctx *SetupContext) string

RedirectURL returns the redirect URI that imp should set in an oauth.Config for the authorization phase of OAuth2 authentication.

func (OAuth2) SummarizeAccount Uses

func (im OAuth2) SummarizeAccount(acct *Object) string

SummarizeAccount returns a summary for the account if it is configured, or an error string otherwise.

type OAuthContext Uses

type OAuthContext struct {
    Ctx    context.Context
    Client *oauth.Client
    Creds  *oauth.Credentials

OAuthContext wraps the OAuth1 state needed to perform API calls.

It is used as a value type.

func (OAuthContext) Get Uses

func (octx OAuthContext) Get(url string, form url.Values) (*http.Response, error)

func (OAuthContext) POST Uses

func (octx OAuthContext) POST(url string, form url.Values) (*http.Response, error)

func (OAuthContext) PopulateJSONFromURL Uses

func (octx OAuthContext) PopulateJSONFromURL(result interface{}, method string, apiURL string, keyval ...string) error

PopulateJSONFromURL makes a POST or GET call at apiURL, using keyval as parameters of the associated form. The JSON response is decoded into result.

type OAuthURIs Uses

type OAuthURIs struct {
    TemporaryCredentialRequestURI string
    ResourceOwnerAuthorizationURI string
    TokenRequestURI               string

OAuthURIs holds the URIs needed to initialize an OAuth 1 client.

type Object Uses

type Object struct {
    // contains filtered or unexported fields

An Object is wrapper around a permanode that the importer uses to synchronize.

func (*Object) Attr Uses

func (o *Object) Attr(attr string) string

Attr returns the object's attribute value for the provided attr, or the empty string if unset. To distinguish between unset, an empty string, or multiple attribute values, use Attrs.

func (*Object) Attrs Uses

func (o *Object) Attrs(attr string) []string

Attrs returns the attribute values for the provided attr.

func (*Object) ChildPathObject Uses

func (o *Object) ChildPathObject(path string) (*Object, error)

ChildPathObject returns (creating if necessary) the child object from the permanode o, given by the "camliPath:xxxx" attribute, where xxx is the provided path.

func (*Object) ChildPathObjectOrFunc Uses

func (o *Object) ChildPathObjectOrFunc(path string, fn func() (*Object, error)) (*Object, error)

ChildPathObjectOrFunc returns the child object from the permanode o, given by the "camliPath:xxxx" attribute, where xxx is the provided path. If the path doesn't exist, the provided func should return an appropriate object. If the func fails, the return error is returned directly without any attempt to make a permanode.

func (*Object) DelAttr Uses

func (o *Object) DelAttr(key, value string) error

DelAttr removes value from the values set for the attribute attr of permaNode. If value is empty then all the values for attribute are cleared.

func (*Object) ForeachAttr Uses

func (o *Object) ForeachAttr(fn func(key, value string))

ForeachAttr runs fn for each of the object's attributes & values. There might be multiple values for the same attribute. The internal lock is held while running, so no mutations should be made or it will deadlock.

func (*Object) PermanodeRef Uses

func (o *Object) PermanodeRef() blob.Ref

PermanodeRef returns the permanode that this object wraps.

func (*Object) SetAttr Uses

func (o *Object) SetAttr(key, value string) error

SetAttr sets the attribute key to value.

func (*Object) SetAttrValues Uses

func (o *Object) SetAttrValues(key string, attrs []string) error

SetAttrValues sets multi-valued attribute.

func (*Object) SetAttrs Uses

func (o *Object) SetAttrs(keyval ...string) error

SetAttrs sets multiple attributes. The provided keyval should be an even number of alternating key/value pairs to set.

func (*Object) SetAttrs2 Uses

func (o *Object) SetAttrs2(keyval ...string) (changes bool, err error)

SetAttrs2 sets multiple attributes and returns whether there were any changes. The provided keyval should be an even number of alternating key/value pairs to set.

type ProgressMessage Uses

type ProgressMessage struct {
    ItemsDone, ItemsTotal int
    BytesDone, BytesTotal int64

type Properties Uses

type Properties struct {
    Title       string
    Description string

    // TODOIssue, if non-zero, marks the importer as invalid, but the UI
    // will link to a tracking bug for implementing it.
    TODOIssue int

    // NeedsAPIKey reports whether this importer requires an API key
    // (OAuth2 client_id & client_secret, or equivalent).
    // If the API only requires a username & password, or a flow to get
    // an auth token per-account without an overall API key, importers
    // can return false here.
    NeedsAPIKey bool

    // SupportsIncremental reports whether this importer has been optimized
    // to run efficiently in regular incremental runs. (e.g. every 5 minutes
    // or half hour). Eventually all importers might support this and we'll
    // make it required, in which case we might delete this option.
    // For now, some importers (e.g. Flickr) don't yet support this.
    SupportsIncremental bool

    // PermanodeImporterType optionally specifies the "importerType"
    // permanode attribute value that should be stored for
    // accounts of this type. By default, it is the string that it
    // was registered with. This should only be specified for
    // products that have been rebranded, and then this should be
    // the old branding, to not break people who have been
    // importing the account since before the rebranding.
    // For example, this is "foursquare" for "swarm", so "swarm" shows
    // in the UI and URLs, but it's "foursquare" in permanodes.
    PermanodeImporterType string

Properties contains the properties of an importer type.

type RunContext Uses

type RunContext struct {
    Host *Host
    // contains filtered or unexported fields

RunContext is the context provided for a given Run of an importer, importing a certain account on a certain importer.

func CreateAccount Uses

func CreateAccount(h *Host, impl string) (*RunContext, error)

CreateAccount creates a new importer account for the Host h, and the importer implementation named impl. It returns a RunContext setup with that account.

func (*RunContext) AccountNode Uses

func (rc *RunContext) AccountNode() *Object

AccountNode returns the permanode storing account information for this permanode. It will contain the attributes:

* camliNodeType = "importerAccount"
* importerType = "registered-type"

You must not change the camliNodeType or importerType.

You should use this permanode to store state about where your importer left off, if it can efficiently resume later (without missing anything).

func (*RunContext) Context Uses

func (rc *RunContext) Context() context.Context

Context returns the run's context. It is always non-nil.

func (*RunContext) Credentials Uses

func (rc *RunContext) Credentials() (clientID, clientSecret string, err error)

Credentials returns the credentials for the importer. This is typically the OAuth1, OAuth2, or equivalent client ID (api token) and client secret (api secret).

func (*RunContext) RootNode Uses

func (rc *RunContext) RootNode() *Object

RootNode returns the initially-empty permanode storing the root of this account's data. You can change anything at will. This will typically be modeled as a dynamic directory (with camliPath:xxxx attributes), where each path element is either a file, object, or another dynamic directory.

type SetupContext Uses

type SetupContext struct {
    Host        *Host
    AccountNode *Object
    // contains filtered or unexported fields

func (*SetupContext) AccountURL Uses

func (sc *SetupContext) AccountURL() string

AccountURL returns the URL to an account of an importer (http://host/importer/TYPE/sha1-sd8fsd7f8sdf7).

func (*SetupContext) CallbackURL Uses

func (sc *SetupContext) CallbackURL() string

func (*SetupContext) Credentials Uses

func (sc *SetupContext) Credentials() (clientID, clientSecret string, err error)

func (*SetupContext) NewOAuthClient Uses

func (ctx *SetupContext) NewOAuthClient(uris OAuthURIs) (*oauth.Client, error)

NewOAuthClient returns an oauth Client configured with uris and the credentials obtained from ctx.

type TestDataMaker Uses

type TestDataMaker interface {
    MakeTestData() http.RoundTripper
    // SetTestAccount allows an importer to set some needed attributes on the importer
    // account node before a run is started.
    SetTestAccount(acctNode *Object) error

TestDataMaker is an optional interface that may be implemented by Importers to generate test data locally. The returned Roundtripper will be used as the transport of the HTTPClient, in the RunContext that will be passed to Run during tests and devcam server --makethings. (See


allimportersPackage allimporters registers all the importer implementations.
dummyPackage dummy is an example importer for development purposes.
feedPackage feed implements an importer for RSS, Atom, and RDF feeds.
feed/atomPackage atom defines XML data structures for an Atom feed.
feed/rdfPackage rdf defines XML data structures for an RDF feed.
feed/rssPackage rss defines XML data structures for an RSS feed.
flickrPackage flickr implements an importer for accounts.
gphotosPackage gphotos implements a Google Photos importer, using the Google Drive API to access the Google Photos folder.
instapaperPackage instapaper implements a importer.
mastodonPackage mastodon provides an importer for servers using the Mastodon API.
picasaPackage picasa implements an importer for accounts.
pinboardPackage pinboard imports posts.
plaidPackage plaid implements an importer for financial transactions from
swarmPackage swarm implements an importer for Foursquare Swarm check-ins.
testPackage test provides common functionality for importer tests.
twitterPackage twitter implements a importer.

Package importer imports 26 packages (graph) and is imported by 15 packages. Updated 2019-06-14. Refresh now. Tools for package owners.