pkgsite: golang.org/x/pkgsite/internal Index | Files | Directories

package internal

import "golang.org/x/pkgsite/internal"

Index

Package Files

datasource.go discovery.go experiment.go

Constants

const (
    // LatestVersion signifies the latest available version in requests to the
    // proxy client.
    LatestVersion = "latest"

    // MasterVersion signifies the version at master.
    MasterVersion = "master"

    // UnknownModulePath signifies that the module path for a given package
    // path is ambiguous or not known. This is because requests to the
    // frontend can come in the form of <import-path>[@<version>], and it is
    // not clear which part of the import-path is the module path.
    UnknownModulePath = "unknownModulePath"
)
const (
    ExperimentAutocomplete          = "autocomplete"
    ExperimentFrontendFetch         = "frontend-fetch"
    ExperimentInsertPlaygroundLinks = "insert-playground-links"
    ExperimentMasterVersion         = "master-version"
    ExperimentSidenav               = "sidenav"
    ExperimentTranslateHTML         = "translate-html"
    ExperimentUseDirectories        = "use-directories"
    ExperimentUsePackageImports     = "use-package-imports"
    ExperimentUsePathInfo           = "use-path-info"
)
const StringFieldMissing = "!MISSING"

StringFieldMissing is the value for string fields that are not present in a struct. We use it to distinguish a (possibly valid) empty string from a field that was never populated.

Variables

var Experiments = map[string]string{
    ExperimentAutocomplete:          "Enable autocomplete with search.",
    ExperimentFrontendFetch:         "Enable ability to fetch a package that doesn't exist on pkg.go.dev.",
    ExperimentInsertPlaygroundLinks: "Insert Go playground links for examples.",
    ExperimentMasterVersion:         "Enable viewing path@master.",
    ExperimentSidenav:               "Display documentation index on the left sidenav.",
    ExperimentTranslateHTML:         "Parse HTML text in READMEs, to properly display images.",
    ExperimentUseDirectories:        "Read from paths, documentation, readmes, and package_imports tables.",
    ExperimentUsePathInfo:           "Check the paths table if a path exists, as opposed to the packages or modules table.",
    ExperimentUsePackageImports:     "Read imports from the package_imports table.",
}

Experiments represents all of the active experiments in the codebase and a description of each experiment.

func SeriesPathForModule Uses

func SeriesPathForModule(modulePath string) string

SeriesPathForModule returns the series path for the provided modulePath.

func V1Path Uses

func V1Path(modulePath, suffix string) string

V1Path returns the path for version 1 of the package whose path is modulePath + "/" + suffix. If modulePath is the standard library, then V1Path returns suffix.

type DataSource Uses

type DataSource interface {

    // GetDirectory returns information about a directory, which may also be a module and/or package.
    // The module and version must both be known.
    GetDirectory(ctx context.Context, dirPath, modulePath, version string) (_ *VersionedDirectory, err error)
    // GetImports returns a slice of import paths imported by the package
    // specified by path and version.
    GetImports(ctx context.Context, pkgPath, modulePath, version string) ([]string, error)
    // GetLicenses returns licenses at the given path for given modulePath and version.
    GetLicenses(ctx context.Context, fullPath, modulePath, resolvedVersion string) ([]*licenses.License, error)
    // GetModuleInfo returns the ModuleInfo corresponding to modulePath and
    // version.
    GetModuleInfo(ctx context.Context, modulePath, version string) (*ModuleInfo, error)
    // GetPathInfo returns information about a path.
    GetPathInfo(ctx context.Context, path, inModulePath, inVersion string) (outModulePath, outVersion string, isPackage bool, err error)

    // TODO(golang/go#39629): Deprecate these methods.
    //
    // LegacyGetDirectory returns packages whose import path is in a (possibly
    // nested) subdirectory of the given directory path. When multiple
    // package paths satisfy this query, it should prefer the module with
    // the longest path.
    LegacyGetDirectory(ctx context.Context, dirPath, modulePath, version string, fields FieldSet) (_ *LegacyDirectory, err error)
    // LegacyGetModuleInfo returns the LegacyModuleInfo corresponding to modulePath and
    // version.
    LegacyGetModuleInfo(ctx context.Context, modulePath, version string) (*LegacyModuleInfo, error)
    // LegacyGetModuleLicenses returns all top-level Licenses for the given modulePath
    // and version. (i.e., Licenses contained in the module root directory)
    LegacyGetModuleLicenses(ctx context.Context, modulePath, version string) ([]*licenses.License, error)
    // LegacyGetPackage returns the LegacyVersionedPackage corresponding to the given package
    // pkgPath, modulePath, and version. When multiple package paths satisfy this query, it
    // should prefer the module with the longest path.
    LegacyGetPackage(ctx context.Context, pkgPath, modulePath, version string) (*LegacyVersionedPackage, error)
    // LegacyGetPackagesInModule returns LegacyPackages contained in the module version
    // specified by modulePath and version.
    LegacyGetPackagesInModule(ctx context.Context, modulePath, version string) ([]*LegacyPackage, error)
    // LegacyGetPackageLicenses returns all Licenses that apply to pkgPath, within the
    // module version specified by modulePath and version.
    LegacyGetPackageLicenses(ctx context.Context, pkgPath, modulePath, version string) ([]*licenses.License, error)
    // LegacyGetPsuedoVersionsForModule returns ModuleInfo for all known
    // pseudo-versions for the module corresponding to modulePath.
    LegacyGetPsuedoVersionsForModule(ctx context.Context, modulePath string) ([]*ModuleInfo, error)
    // LegacyGetPsuedoVersionsForModule returns ModuleInfo for all known
    // pseudo-versions for any module containing a package with the given import
    // path.
    LegacyGetPsuedoVersionsForPackageSeries(ctx context.Context, pkgPath string) ([]*ModuleInfo, error)
    // LegacyGetTaggedVersionsForModule returns ModuleInfo for all known tagged
    // versions for the module corresponding to modulePath.
    LegacyGetTaggedVersionsForModule(ctx context.Context, modulePath string) ([]*ModuleInfo, error)
    // LegacyGetTaggedVersionsForModule returns ModuleInfo for all known tagged
    // versions for any module containing a package with the given import path.
    LegacyGetTaggedVersionsForPackageSeries(ctx context.Context, pkgPath string) ([]*ModuleInfo, error)
}

DataSource is the interface used by the frontend to interact with module data.

type Directory Uses

type Directory struct {
    DirectoryMeta
    Readme  *Readme
    Package *Package
}

Directory represents a directory in a module version, and the contents of that directory. It will replace LegacyDirectory once everything has been migrated.

type DirectoryMeta Uses

type DirectoryMeta struct {
    Path              string
    V1Path            string
    IsRedistributable bool
    Licenses          []*licenses.Metadata // metadata of applicable licenses
}

DirectoryMeta represents the metadata of a directory in a module version.

type Documentation Uses

type Documentation struct {
    // The values of the GOOS and GOARCH environment variables used to parse the
    // package.
    GOOS     string
    GOARCH   string
    Synopsis string
    HTML     safehtml.HTML
}

Documentation is the rendered documentation for a given package for a specific GOOS and GOARCH.

type Experiment Uses

type Experiment struct {
    // Name is the name of the feature.
    Name string

    // Rollout is the percentage of requests enrolled in the experiment.
    Rollout uint

    // Description provides a description of the experiment.
    Description string
}

Experiment holds data associated with an experimental feature for frontend or worker.

type ExperimentSource Uses

type ExperimentSource interface {
    // GetExperiments fetches active experiments from the
    // ExperimentSource.
    GetExperiments(ctx context.Context) ([]*Experiment, error)
}

ExperimentSource is the interface used by the middleware to interact with experiments data.

type FieldSet Uses

type FieldSet int64

A FieldSet is a bit set of struct fields. It is used to avoid reading large struct fields from the data store. FieldSet is also the type of the individual bit values. (Think of them as singleton sets.)

NoFields (the zero value) is the empty set. AllFields is the set containing every field.

FieldSet bits are unique across the entire project, because some types are concatenations (via embedding) of others. For example, a LegacyVersionedPackage contains the fields of both a LegacyModuleInfo and a LegacyPackage, so we can't use the same bits for both LegacyModuleInfo's LegacyReadmeContents field and LegacyPackage's DocumentationHTML field.

const (
    WithReadmeContents FieldSet = 1 << iota
    WithDocumentationHTML
)

FieldSet bits for fields that can be conditionally read from the data store.

const AllFields FieldSet = -1

AllFields is the FieldSet that contains all fields.

const MinimalFields FieldSet = 0

MinimalFields is the empty FieldSet.

type IndexVersion Uses

type IndexVersion struct {
    Path      string
    Version   string
    Timestamp time.Time
}

IndexVersion holds the version information returned by the module index.

type LegacyDirectory Uses

type LegacyDirectory struct {
    LegacyModuleInfo
    Path     string
    Packages []*LegacyPackage
}

LegacyDirectory represents a directory in a module version, and all of the packages inside that directory.

type LegacyModuleInfo Uses

type LegacyModuleInfo struct {
    ModuleInfo
    LegacyReadmeFilePath string
    LegacyReadmeContents string
}

LegacyModuleInfo holds metadata associated with a module.

type LegacyPackage Uses

type LegacyPackage struct {
    Path              string
    Name              string
    Synopsis          string
    IsRedistributable bool
    Licenses          []*licenses.Metadata // metadata of applicable licenses
    Imports           []string
    DocumentationHTML safehtml.HTML
    // The values of the GOOS and GOARCH environment variables used to parse the
    // package.
    GOOS   string
    GOARCH string

    // V1Path is the package path of a package with major version 1 in a given
    // series.
    V1Path string
}

A LegacyPackage is a group of one or more Go source files with the same package header. LegacyPackages are part of a module.

type LegacyVersionedPackage Uses

type LegacyVersionedPackage struct {
    LegacyPackage
    LegacyModuleInfo
}

LegacyVersionedPackage is a LegacyPackage along with its corresponding module information.

type LocalExperimentSource Uses

type LocalExperimentSource struct {
    // contains filtered or unexported fields
}

LocalExperimentSource is used when developing locally using the direct proxy mode.

func NewLocalExperimentSource Uses

func NewLocalExperimentSource(experiments []*Experiment) *LocalExperimentSource

NewLocalExperimentSource returns a LocalExperimentSource with the provided experiments.

func (*LocalExperimentSource) GetExperiments Uses

func (e *LocalExperimentSource) GetExperiments(ctx context.Context) ([]*Experiment, error)

GetExperiments returns the experiments for the given LocalExperimentSource.

type Module Uses

type Module struct {
    LegacyModuleInfo
    // Licenses holds all licenses within this module version, including those
    // that may be contained in nested subdirectories.
    Licenses    []*licenses.License
    Directories []*Directory

    LegacyPackages []*LegacyPackage
}

A Module is a specific, reproducible build of a module.

type ModuleInfo Uses

type ModuleInfo struct {
    ModulePath        string
    Version           string
    CommitTime        time.Time
    VersionType       version.Type
    IsRedistributable bool
    HasGoMod          bool // whether the module zip has a go.mod file
    SourceInfo        *source.Info
}

ModuleInfo holds metadata associated with a module.

func (*ModuleInfo) SeriesPath Uses

func (v *ModuleInfo) SeriesPath() string

SeriesPath returns the series path for the module.

A series is a group of modules that share the same base path and are assumed to be major-version variants.

The series path is the module path without the version. For most modules, this will be the module path for all module versions with major version 0 or 1. For gopkg.in modules, the series path does not correspond to any module version.

Examples: The module paths "a/b" and "a/b/v2" both have series path "a/b". The module paths "gopkg.in/yaml.v1" and "gopkg.in/yaml.v2" both have series path "gopkg.in/yaml".

type ModuleVersionState Uses

type ModuleVersionState struct {
    ModulePath string
    Version    string

    // IndexTimestamp is the timestamp received from the Index for this version,
    // which should correspond to the time this version was committed to the
    // Index.
    IndexTimestamp time.Time
    // CreatedAt is the time this version was originally inserted into the
    // module version state table.
    CreatedAt time.Time

    // Status is the most recent HTTP status code received from the Fetch service
    // for this version, or nil if no request to the fetch service has been made.
    Status int
    // Error is the most recent HTTP response body received from the Fetch
    // service, for a response with an unsuccessful status code. It is used for
    // debugging only, and has no semantic significance.
    Error string
    // TryCount is the number of times a fetch of this version has been
    // attempted.
    TryCount int
    // LastProcessedAt is the last time this version was updated with a result
    // from the fetch service.
    LastProcessedAt *time.Time
    // NextProcessedAfter is the next time a fetch for this version should be
    // attempted.
    NextProcessedAfter time.Time

    // AppVersion is the value of the GAE_VERSION environment variable, which is
    // set by app engine. It is a timestamp in the format 20190709t112655 that
    // is close to, but not the same as, the deployment time. For example, the
    // deployment time for the above timestamp might be Jul 9, 2019, 11:29:59 AM.
    AppVersion string

    // GoModPath is the path declared in the go.mod file.
    GoModPath string

    // NumPackages it the number of packages that were processed as part of the
    // module (regardless of whether the processing was successful).
    NumPackages *int
}

ModuleVersionState holds a worker module version state.

type Package Uses

type Package struct {
    Name          string
    Path          string
    Documentation *Documentation
    Imports       []string
}

Package is a group of one or more Go source files with the same package header. A Package is part of a directory. It will replace LegacyPackage once everything has been migrated.

type PackageMeta Uses

type PackageMeta struct {
    DirectoryMeta
    Name     string
    Synopsis string
}

PackageMeta represents the metadata of a package in a module version.

func PackageMetaFromLegacyPackage Uses

func PackageMetaFromLegacyPackage(pkg *LegacyPackage) *PackageMeta

PackageMetaFromLegacyPackage returns a PackageMeta based on data from a LegacyPackage.

type PackageVersionState Uses

type PackageVersionState struct {
    PackagePath string
    ModulePath  string
    Version     string
    Status      int
    Error       string
}

PackageVersionState holds a worker package version state. It is associated with a given module version state.

type Readme Uses

type Readme struct {
    Filepath string
    Contents string
}

Readme is a README at a given directory.

type SearchResult Uses

type SearchResult struct {
    Name        string
    PackagePath string
    ModulePath  string
    Version     string
    Synopsis    string
    Licenses    []string

    CommitTime time.Time
    // Score is used to sort items in an array of SearchResult.
    Score float64

    // NumImportedBy is the number of packages that import PackagePath.
    NumImportedBy uint64

    // NumResults is the total number of packages that were returned for this
    // search.
    NumResults uint64
    // Approximate reports whether NumResults is an approximate count. NumResults
    // can be approximate if search scanned only a subset of documents, and
    // result count is estimated using the hyperloglog algorithm.
    Approximate bool
}

SearchResult represents a single search result from SearchDocuments.

type VersionMap Uses

type VersionMap struct {
    ModulePath       string
    RequestedVersion string
    ResolvedVersion  string
    GoModPath        string
    Status           int
    Error            string
}

VersionMap holds metadata associated with module queries for a version.

type VersionedDirectory Uses

type VersionedDirectory struct {
    Directory
    ModuleInfo
}

VersionedDirectory is a Directory along with its corresponding module information.

Directories

PathSynopsis
breakerPackage breaker implements the circuit breaker pattern.
completePackage complete defines a Completion type that is used in auto-completion, along with Encode and Decode methods that can be used for storing this type in redis.
configPackage config resolves shared configuration for Go Discovery services, and provides functions to access this configuration.
databasePackage database adds some useful functionality to a sql.DB.
dcensusPackage dcensus provides functionality for debug instrumentation.
derrorsPackage derrors defines internal error values to categorize the different types error semantics we support.
experiment
fetch
fetch/dochtmlPackage dochtml renders Go package documentation into HTML.
fetch/internal/docPackage doc extracts source code documentation from a Go AST.
frontend
index
licensesPackage licenses detects licenses and determines whether they are redistributable.
logPackage log supports structured and unstructured logging with levels.
middlewarePackage middleware implements a simple middleware pattern for http handlers, along with implementations for some common middlewares.
postgres
proxy
proxydatasourcePackage proxydatasource implements an internal.DataSource backed solely by a proxy instance.
queuePackage queue provides queue implementations that can be used for asynchronous scheduling of fetch actions.
secrets
sourcePackage source constructs public URLs that link to the source files in a module.
stdlibPackage stdlib supports special handling of the Go standard library.
teeproxy
testing/dbtestPackage dbtest supports testing with a database.
testing/htmlcheckPackage htmlcheck provides a set of functions that check for properties of a parsed HTML document.
testing/pagecheckPackage pagecheck implements HTML checkers for discovery site pages.
testing/samplePackage sample provides functionality for generating sample values of the types contained in the internal package.
testing/testhelperPackage testhelper provides shared functionality and constants to be used in Discovery tests.
versionPackage version handles version types.
worker
xcontextPackage xcontext is a package to offer the extra functionality we need from contexts that is not available from the standard context package.

Package internal imports 9 packages (graph) and is imported by 15 packages. Updated 2020-07-15. Refresh now. Tools for package owners.