dbt

package
v0.0.0-...-0481eb9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 13, 2023 License: MIT Imports: 37 Imported by: 1

Documentation

Index

Constants

View Source 
const AUTH_BASIC_HTPASSWD = "basic-htpasswd"

AUTH_BASIC_HTPASSWD config flag for basic auth

View Source 
const AUTH_BASIC_LDAP = "basic-ldap"

AUTH_BASIC_LDAP config flag for user/password auth off an LDAP directory server

View Source 
const AUTH_SSH_AGENT_FILE = "ssh-agent-file"

AUTH_SSH_AGENT_FILE config setting for file based ssh-agent auth (file mapping principals to public keys similer to .htaccess files)

View Source 
const AUTH_SSH_AGENT_FUNC = "ssh-agent-func"

AUTH_SSH_AGENT_FUNC config setting for using a shell function to retrieve the public key for a principal

View Source 
const AUTH_SSH_AGENT_LDAP = "ssh-agent-ldap"

AUTH_SSH_AGENT_LDAP flag for configuring ssh-agent auth pulling public key from an LDAP directory

View Source 
const AWS_ID_ENV_VAR = "AWS_ACCESS_KEY_ID"

AWS_ID_ENV_VAR Default env var for AWS access key

View Source 
const AWS_REGION_ENV_VAR = "AWS_DEFAULT_REGION"

AWS_REGION_ENV_VAR Default env var for AWS region

View Source 
const AWS_SECRET_ENV_VAR = "AWS_SECRET_ACCESS_KEY"

AWS_SECRET_ENV_VAR Default env var for AWS secret key

View Source 
const ConfigDir = DbtDir + "/conf"

ConfigDir is the directory where Dbt expects to find configuration info

View Source 
const ConfigFilePath = ConfigDir + "/dbt.json"

ConfigFilePath is the actual dbt config file path

View Source 
const DbtDir = ".dbt"

DbtDir is the standard dbt directory. Usually ~/.dbt

View Source 
const ToolDir = DbtDir + "/tools"

ToolDir is the directory where tools get downloaded to

View Source 
const TrustDir = DbtDir + "/trust"

TrustDir is the directory under the dbt dir where the trust store is downloaded to

View Source 
const TruststorePath = TrustDir + "/truststore"

TruststorePath is the actual file path to the downloaded trust store

View Source 
const VERSION = "3.6.0"

VERSION DBT's version

Variables

View Source 
var NOPROGRESS = false

NOPROGRESS turns off the progress bar on file fetches. Primarily used for testing to avoid cluttering up the output and confusing the test harness.

Functions

func CheckPubkeyAuth 

func CheckPubkeyAuth(w http.ResponseWriter, r *http.Request, pubkeyRetrievalFunc func(subject string) (pubkeys string, err error)) (username string)

CheckPubkeyAuth Function that actually checks the Token sent by the client in the headers.

func DefaultSession 

func DefaultSession(s3meta *S3Meta) (awssession *session.Session, err error)

DefaultSession creates a default AWS session from local config path. Hooks directly into credentials if present, or Credentials Provider if configured.

func DirCopy 

func DirCopy(src string, dst string) error

DirCopy copies a whole directory recursively

func DirsForURL 

func DirsForURL(uri string) (dirs []string, err error)

DirsForURL given a URL, return a list of path elements suitable for creating directories/ folders

func FileCopy 

func FileCopy(src, dst string) error

FileCopy copies a single file from src to dst

func FileSha1 

func FileSha1(fileName string) (checksum string, err error)

FileSha1 returns the hex encoded Sha1 checksum for the given file

func FileSha256 

func FileSha256(fileName string) (checksum string, err error)

FileSha256 returns the hex encoded Sha256 checksum for the given file

func GenerateDbtDir 

func GenerateDbtDir(homedir string, verbose bool) (err error)

GenerateDbtDir generates the necessary dbt dirs in the user's homedir if they don't already exist. If they do exist, it does nothing.

func GetFunc 

func GetFunc(shellCommand string) (result string, err error)

GetFunc runs a shell command that is a getter function. This could certainly be dangerous, so be careful how you use it.

func GetFuncUsername 

func GetFuncUsername(shellCommand string, username string) (result string, err error)

GetFuncUsername runs a shell command that is a getter function for the username. This could certainly be dangerous, so be careful how you use it.

func GetHomeDir 

func GetHomeDir() (dir string, err error)

GetHomeDir get's the current user's homedir

func LatestVersion 

func LatestVersion(versions []string) (latest string)

LatestVersion returns the latest version teased out of a list of semantic version strings.

func SemverParse 

func SemverParse(version string) (parts []int, err error)

SemverParse breaks apart a semantic version strings and returns a slice of int's holding the parts

func Spaceship 

func Spaceship(a int, b int) int

Spaceship A very simple implementation of a useful operator that go seems not to have. returns 1 if a > b, -1 if a < b, and 0 if a == b

func StringInSlice 

func StringInSlice(a string, list []string) bool

StringInSlice returns true if the given string is in the given slice

func VersionAIsNewerThanB 

func VersionAIsNewerThanB(a string, b string) (result bool)

VersionAIsNewerThanB returns true if Semantic Version string v1 is newer (higher numbers) than Semantic Version string v2

func Wrap 

func Wrap(wrapped AuthenticatedHandlerFunc, pubkeyRetrievalFunc func(subject string) (pubkeys string, err error)) http.HandlerFunc

Wrap returns an http.HandlerFunc which wraps AuthenticatedHandlerFunc

Types

type AuthOpts 

type AuthOpts struct {
	IdpFile string `json:"idpFile"`
	IdpFunc string `json:"idpFunc,omitempty"`
}

AuthOpts Struct for holding Auth options

type AuthenticatedHandlerFunc 

type AuthenticatedHandlerFunc func(http.ResponseWriter, *AuthenticatedRequest)

AuthenticatedHandlerFunc is like http.HandlerFunc, but takes AuthenticatedRequest instead of http.Request

type AuthenticatedRequest 

type AuthenticatedRequest struct {
	http.Request
	/*
	 Authenticated user name. Current API implies that Username is
	 never empty, which means that authentication is always done
	 before calling the request handler.
	*/
	Username string
}

AuthenticatedRequest Basically an http.Request with an added Username field. The Username should never be empty.

type Config 

type Config struct {
	Dbt          DbtConfig   `json:"dbt"`
	Tools        ToolsConfig `json:"tools"`
	Username     string      `json:"username,omitempty"`
	Password     string      `json:"password,omitempty"`
	UsernameFunc string      `json:"usernamefunc,omitempty"`
	PasswordFunc string      `json:"passwordfunc,omitempty"`
	Pubkey       string      `json:"pubkey,omitempty"`
	PubkeyPath   string      `json:"pubkeypath,omitempty"`
	PubkeyFunc   string      `json:"pubkeyfunc,omitempty"`
}

Config configuration of the dbt object

func LoadDbtConfig 

func LoadDbtConfig(homedir string, verbose bool) (config Config, err error)

LoadDbtConfig loads the dbt config from the expected location on the filesystem

type DBT 

type DBT struct {
	Config    Config
	Verbose   bool
	Logger    *log.Logger
	S3Session *session.Session
}

DBT the dbt object itself

func NewDbt 

func NewDbt(homedir string) (dbt *DBT, err error)

NewDbt creates a new dbt object

func (*DBT) AuthHeaders 

func (dbt *DBT) AuthHeaders(r *http.Request) (err error)

AuthHeaders Convenience function to add auth headers - basic or token for non-s3 requests. Depending on how client is configured, could result in both Basic Auth and Token headers. Reposerver will, however only pay attention to one or the other.

func (*DBT) FetchCatalog 

func (dbt *DBT) FetchCatalog(showVersions bool) (err error)

FetchCatalog shows you what tools are available in your trusted repo. Repo is figured out from the config in ~/.dbt/conf/dbt.json

func (*DBT) FetchFile 

func (dbt *DBT) FetchFile(fileUrl string, destPath string) (err error)

FetchFile Fetches a file and places it on the filesystem. Does not validate the signature. That's a different step.

func (*DBT) FetchToolDescription 

func (dbt *DBT) FetchToolDescription(tool string, version string) (description string, err error)

FetchToolDescription fetches the tool description from the repository.

func (*DBT) FetchToolNames 

func (dbt *DBT) FetchToolNames() (tools []Tool, err error)

FetchToolNames returns a list of tool names found in the trusted repo

func (*DBT) FetchToolVersions 

func (dbt *DBT) FetchToolVersions(toolName string) (versions []string, err error)

FetchToolVersions Given the name of a tool, returns the available versions, and possibly an error if things didn't go well. If tool name is "", fetches versions of dbt itself.

func (*DBT) FetchTrustStore 

func (dbt *DBT) FetchTrustStore(homedir string) (err error)

FetchTrustStore writes the downloaded trusted signing public keys to disk.

func (*DBT) FindLatestVersion 

func (dbt *DBT) FindLatestVersion(toolName string) (latest string, err error)

FindLatestVersion finds the latest version of the tool available in the tool repo. If the tool name is "", it is expecting to parse versions of dbt itself.

func (*DBT) IsCurrent 

func (dbt *DBT) IsCurrent(binaryPath string) (ok bool, err error)

IsCurrent returns whether the currently running version is the latest version, and possibly an error if the version check fails

func (*DBT) ParseVersionResponse 

func (dbt *DBT) ParseVersionResponse(resp *http.Response) (versions []string)

ParseVersionResponse does an http get of an url and returns a list of semantic version links found at that place

func (*DBT) RunTool 

func (dbt *DBT) RunTool(version string, args []string, homedir string, offline bool) (err error)

RunTool runs the dbt tool indicated by the args

func (*DBT) S3FetchDescription 

func (dbt *DBT) S3FetchDescription(meta S3Meta) (description string, err error)

S3FetchDescription fetches the tool description from S3

func (*DBT) S3FetchFile 

func (dbt *DBT) S3FetchFile(fileUrl string, meta S3Meta, outFile *os.File) (err error)

S3FetchFile fetches a file out of S3 instead of using a normal HTTP GET

func (*DBT) S3FetchToolNames 

func (dbt *DBT) S3FetchToolNames(meta S3Meta) (tools []Tool, err error)

S3FetchToolNames fetches the list of available tools from S3

func (*DBT) S3FetchToolVersions 

func (dbt *DBT) S3FetchToolVersions(meta S3Meta) (versions []string, err error)

S3FetchToolVersions fetches available versions for a tool from S3

func (*DBT) S3FetchTruststore 

func (dbt *DBT) S3FetchTruststore(homedir string, meta S3Meta) (err error)

S3FetchTruststore fetches the truststore out of S3 writing it into the dbt dir on the local disk

func (*DBT) S3ToolExists 

func (dbt *DBT) S3ToolExists(meta S3Meta) (found bool, err error)

S3ToolExists detects whether a tool exists in S3 by looking at the top level folder for the tool

func (*DBT) S3ToolVersionExists 

func (dbt *DBT) S3ToolVersionExists(meta S3Meta) (ok bool, err error)

S3ToolVersionExists returns true if the tool version exists

func (*DBT) S3VerifyFileVersion 

func (dbt *DBT) S3VerifyFileVersion(filePath string, meta S3Meta) (success bool, err error)

S3VerifyFileVersion verifies the version of a file on the filesystem matches the sha256 hash stored in the s3 bucket for that file

func (*DBT) SetVerbose 

func (dbt *DBT) SetVerbose(verbose bool)

SetVerbose Sets the verbose option on the dbt object

func (*DBT) ToolExists 

func (dbt *DBT) ToolExists(toolName string) (found bool, err error)

ToolExists Returns true if a tool of the name input exists in the repository given.

func (*DBT) ToolVersionExists 

func (dbt *DBT) ToolVersionExists(tool string, version string) (ok bool, err error)

ToolVersionExists returns true if the specified version of a tool is in the repo

func (*DBT) UpgradeInPlace 

func (dbt *DBT) UpgradeInPlace(binaryPath string) (err error)

UpgradeInPlace upgraded dbt in place

func (*DBT) VerboseOutput 

func (dbt *DBT) VerboseOutput(message string, args ...interface{})

VerboseOutput Convenience function so I don't have to write 'if verbose {...}' all the time.

func (*DBT) VerifyFileChecksum 

func (dbt *DBT) VerifyFileChecksum(filePath string, expected string) (success bool, err error)

VerifyFileChecksum Verifies the sha256 checksum of a given file against an expected value

func (*DBT) VerifyFileSignature 

func (dbt *DBT) VerifyFileSignature(homedir string, filePath string) (success bool, err error)

VerifyFileSignature verifies the signature on the given file

func (*DBT) VerifyFileVersion 

func (dbt *DBT) VerifyFileVersion(fileUrl string, filePath string) (success bool, err error)

VerifyFileVersion verifies the version by matching it's Sha256 checksum against what the repo says it should be

type DBTRepoServer 

type DBTRepoServer struct {
	Address     string   `json:"address"`
	Port        int      `json:"port"`
	ServerRoot  string   `json:"serverRoot"`
	AuthTypeGet string   `json:"authTypeGet"`
	AuthTypePut string   `json:"authTypePut"`
	AuthGets    bool     `json:"authGets"`
	AuthOptsGet AuthOpts `json:"authOptsGet"`
	AuthOptsPut AuthOpts `json:"authOptsPut"`
}

DBTRepoServer The reference 'trusted repository' server for dbt.

func NewRepoServer 

func NewRepoServer(configFilePath string) (server *DBTRepoServer, err error)

NewRepoServer creates a new DBTRepoServer object from the config file provided.

func (*DBTRepoServer) CheckPubkeysGetFile 

func (d *DBTRepoServer) CheckPubkeysGetFile(wrapped http.HandlerFunc) http.HandlerFunc

CheckPubkeysGetFile Checks the pubkey signature in the JWT token against a public key found in a htpasswd like file and if things check out, passes things along to the provided handler.

func (*DBTRepoServer) CheckPubkeysGetFunc 

func (d *DBTRepoServer) CheckPubkeysGetFunc(wrapped http.HandlerFunc) http.HandlerFunc

CheckPubkeysGetFunc Checks the pubkey signature in the JWT token against a public key produced from a function and if things check out, passes things along to the provided handler.

func (*DBTRepoServer) HandlePut 

func (d *DBTRepoServer) HandlePut(path string, body io.ReadCloser, md5sum string, sha1sum string, sha256sum string) (err error)

func (*DBTRepoServer) PubkeyFromFileGet 

func (d *DBTRepoServer) PubkeyFromFileGet(subject string) (pubkeys string, err error)

PubkeyFromFileGet takes a subject name, and pulls the corresponding pubkey out of the identity provider file for puts

func (*DBTRepoServer) PubkeyFromFilePut 

func (d *DBTRepoServer) PubkeyFromFilePut(subject string) (pubkeys string, err error)

PubkeyFromFilePut takes a subject name, and pulls the corresponding pubkey out of the identity provider file for puts

func (*DBTRepoServer) PubkeysFromFuncGet 

func (d *DBTRepoServer) PubkeysFromFuncGet(subject string) (pubkey string, err error)

PubkeyFromFuncGet takes a subject name, and runs the configured function to return the corresponding public key

func (*DBTRepoServer) PubkeysFromFuncPut 

func (d *DBTRepoServer) PubkeysFromFuncPut(subject string) (pubkey string, err error)

PubkeyFromFuncPut takes a subject name, and runs the configured function to return the corresponding public key

func (*DBTRepoServer) PutHandlerHtpasswd 

func (d *DBTRepoServer) PutHandlerHtpasswd(w http.ResponseWriter, r *auth.AuthenticatedRequest)

PutHandlerHtpasswd Handles puts with htpasswd auth

func (*DBTRepoServer) PutHandlerPubkeyFile 

func (d *DBTRepoServer) PutHandlerPubkeyFile(w http.ResponseWriter, r *http.Request)

PutHandlerPubKeyFile

func (*DBTRepoServer) PutHandlerPubkeyFunc 

func (d *DBTRepoServer) PutHandlerPubkeyFunc(w http.ResponseWriter, r *http.Request)

PutHandlerPubkeyFunc

func (*DBTRepoServer) RunRepoServer 

func (d *DBTRepoServer) RunRepoServer() (err error)

RunRepoServer Run runs the test repository server.

type DbtConfig 

type DbtConfig struct {
	Repo       string `json:"repository"`
	TrustStore string `json:"truststore"`
}

DbtConfig internal config of dbt

type PubkeyIdpFile 

type PubkeyIdpFile struct {
	GetUsers []PubkeyUser `json:"getUsers"`
	PutUsers []PubkeyUser `json:"putUsers"`
}

PubkeyIdpFile A representation of a public key IDP (Identity Provider) file. Will have a list of users allowed to GET and a list of users authorized to PUT.

func LoadPubkeyIdpFile 

func LoadPubkeyIdpFile(filePath string) (pkidp PubkeyIdpFile, err error)

LoadPubkeyIdpFile Loads a public key IDP JSON file.

type PubkeyUser 

type PubkeyUser struct {
	Username      string `json:"username"`
	AuthorizedKey string `json:"publickey"`
}

PubkeyUser A representation of a user permitted to authenticate via public key. PubkeyUsers will have at minimum a Username, and a list of authorized public keys.

type S3Meta 

type S3Meta struct {
	Bucket string
	Region string
	Key    string
	Url    string
}

S3Meta a struct for holding metadata for S3 Objects. There's probably already a struct that holds this, but this is all I need.

func S3Url 

func S3Url(url string) (ok bool, meta S3Meta)

S3Url returns true, and a metadata struct if the url given appears to be in s3

type Tool 

type Tool struct {
	Name          string
	FormattedName string
	Version       string
	Description   string
}

Tool is a struct representing pertinent info on a dbt tool

type ToolsConfig 

type ToolsConfig struct {
	Repo string `json:"repository"`
}

ToolsConfig is the config information for the tools to be downloaded and run

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.