ls3

package module
v0.21.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 29, 2022 License: MIT Imports: 32 Imported by: 0

README

LS3

Lightweight read-only S3 compatible object storage interface for local filesystems.

  • Zero state
  • Works across filesystems
  • Multiple identities
  • Rule based access control
  • Automatic Content-Type detection

Authentication and Access Control

Access to LS3 resources are controlled through an identity and an optional global policy.

An identity that allows access to any action and resource on the server

{
  "Name": "example",
  "AccessKeyId": "EXAMPLE",
  "SecretAccessKey": "<securestring>",
  "Policy": [
    {
      "Action": "*",
      "Resource": "*"
    }
  ]
}
Identities

An identity is an access key ID and secret access key pair, along with a policy specific to that identity. An identity also has a name to identify it in logs, and in conditional policy evaluation.

Root Identity

The root identity has full access to any action and resource on the server. You can provide an access key id and secret access key through the command-line or environment, or ls3 will generate random keys on startup.

Public Identity

The public identity is a special identity used for when a request provides no authentication mechanism. A public identity is one that has an empty AccessKeyId.

By default, the public identity is denied access to everything unless you enable public access.

When public access is not enabled, the public identity is configured as such:

{
  "Name": "public",
  "AccessKeyId": "",
  "Policy": [
    {
      "Deny": true,
      "Action": "*",
      "Resource": "*"
    }
  ]
}

You can override this behaviour by configuring your own public identity. When you configure a public identity this overrides the behaviour set by allow public access, and is up to you to deny access if desired.

Policies

Policies control what an identity has access to. A policy consists of one or more actions, along with one or more resources of that action that describe what each policy applies to.

If there are no policies that match a given request then access is denied.

You can configure a global policy that applies to all identities.

Wildcards

You can use wildcard characters (* and ?) anywhere in an action or resource. A * character matches anything up to the character proceeding it, and a ? matches any single character.

Allow access to s3:GetObject in the example bucket on any file that ends with .txt or .html

{
  "Action": "s3:GetObject",
  "Resource": [
    "example/*.txt",
    "example/*.html"
  ]
}
Explicit Deny

If a policy is an explicit deny, then any requests that match that policy will be denied. Even if there is another policy that allows that access.

Deny any access to the secret bucket

{
  "Deny": true,
  "Action": [
    "s3:GetObject",
    "s3:ListBucket"
  ],
  "Resource": "secret/*"
}
Conditions

You can add one or more Condition to any policy that limits the scope of that policy to only requests that match those conditions.

{
  "Condition": {
    "<ConditionOperator>": {
      "<ContextKey>": [
        "<value>",
        "<value>"
      ]
    }
  }
}

Allow access from 127.0.0.1

{
  "Action": "*",
  "Resource": "*",
  "Condition": {
    "IpAddress": {
      "aws:SourceIp": "127.0.0.1"
    }
  }
}

Deny access if request is not secure (not using HTTPS)

{
  "Deny": true,
  "Action": "*",
  "Resource": "*",
  "Condition": {
    "Bool": {
      "aws:SecureTransport": "false"
    }
  }
}
Condition Operators
Operator Description
StringEquals True if any are exactly equal
StringNotEquals True if none are exactly equal
StringEqualsIgnoreCase True if any are exactly equal (case insensitive)
StringNotEqualsIgnoreCase True if none are exactly equal (case insensitive)
StringLike True if matches any wildcard pattern
StringNotLike True if not match all wildcard patterns
IpAddress True if matches any IP or CIDR range
NotIpAdress True if not matches all IP or CIDR range
Bool True if all boolean values are equal. False if not a boolean
Global Context Keys

These context keys apply to all requests

Key Type Description
aws:SourceIp IpAddress The IP address of the client
aws:SecureTransport Bool Was the request made over HTTPS
aws:username String The Name of the identity making the request. public if unauthorized
ls3:authenticated Bool Is the request made with an authenticated identity

Documentation

Index

Constants

View Source
const (
	// IdentityUnauthenticatedPublic is a special AccessKeyId for unauthenticated requests.
	IdentityUnauthenticatedPublic = ""
)

Variables

View Source
var (
	AccessDenied                 = ErrorCode{Code: "AccessDenied", StatusCode: 403}
	InvalidAccessKeyId           = ErrorCode{Code: "InvalidAccessKeyId", StatusCode: 403}
	SignatureDoesNotMatch        = ErrorCode{Code: "SignatureDoesNotMatch", StatusCode: 403}
	MethodNotAllowed             = ErrorCode{Code: "MethodNotAllowed", StatusCode: 405}
	InvalidRequest               = ErrorCode{Code: "InvalidRequest", StatusCode: 400}
	ExpiredToken                 = ErrorCode{Code: "ExpiredToken", StatusCode: 400}
	InvalidArgument              = ErrorCode{Code: "InvalidArgument", StatusCode: 400}
	BadDigest                    = ErrorCode{Code: "BadDigest", StatusCode: 400}
	NoSuchKey                    = ErrorCode{Code: "NoSuchKey", StatusCode: 404}
	MissingSecurityHeader        = ErrorCode{Code: "MissingSecurityHeader", StatusCode: 400}
	InvalidToken                 = ErrorCode{Code: "InvalidToken", StatusCode: 400}
	InvalidObjectState           = ErrorCode{Code: "InvalidObjectState", StatusCode: 403}
	InvalidRange                 = ErrorCode{Code: "InvalidRange", StatusCode: 416}
	NoSuchBucket                 = ErrorCode{Code: "NoSuchBucket", StatusCode: 404}
	InvalidBucketState           = ErrorCode{Code: "InvalidBucketState", StatusCode: 409}
	InternalError                = ErrorCode{Code: "InternalError", StatusCode: 500}
	MalformedXML                 = ErrorCode{Code: "MalformedXML", StatusCode: 400}
	AuthorizationHeaderMalformed = ErrorCode{Code: "AuthorizationHeaderMalformed", StatusCode: 400}
	InvalidSecurity              = ErrorCode{Code: "InvalidSecurity", StatusCode: 403}
	AccountProblem               = ErrorCode{Code: "AccountProblem", StatusCode: 403}
)
View Source
var ErrMissingAccessKeyId = &Error{
	ErrorCode: InvalidAccessKeyId,
	Message:   "The AWS access key ID that you provided does not exist in our records.",
}
View Source
var PreAuthenticationIdentity = &Identity{
	Name: "PreAuthentication",
	Policy: []*PolicyStatement{
		{
			Deny:     true,
			Resource: []Resource{"*"},
			Action:   []Action{"*"},
		},
	},
}

PreAuthenticationIdentity is used in logging as the initial identity given to a new request context. It always denies access to all resources

View Source
var (
	StatRegistry = prometheus.NewRegistry()
)

Functions

func JoinContext added in v0.20.0

func JoinContext(parent PolicyContextVars, this PolicyContextVars) *joinContext

func MatchesConditions added in v0.20.0

func MatchesConditions(conditions PolicyConditions, context PolicyContextVars) bool

func ParseAmzTime added in v0.21.0

func ParseAmzTime(text string) (t time.Time, err error)

func WildcardMatch added in v0.20.0

func WildcardMatch[T ~string](rule, obj T) bool

Types

type Action added in v0.20.0

type Action string
const (
	GetObject         Action = "s3:GetObject"
	ListAllMyBuckets  Action = "s3:ListAllMyBuckets"
	ListBucket        Action = "s3:ListBucket"
	GetBucketLocation Action = "s3:GetBucketLocation"
)

type Authorization

type Authorization struct {
	Credentials   Credential
	SignedHeaders []string
	Signature     []byte // raw decoded hex
}

func ParseAuthorizationHeader added in v0.9.0

func ParseAuthorizationHeader(hdr string) (*Authorization, error)

ParseAuthorizationHeader parses the contents of the given Authorization header. Returns a non-nil *ErrInvalidAuthorizationHeader when an invalid header is given.

func (Authorization) AppendFormat

func (a Authorization) AppendFormat(b []byte) []byte

type BucketFilesystemProvider added in v0.5.0

type BucketFilesystemProvider interface {
	// ListBuckets lists all available filesystemProvider in the provider.
	ListBuckets() ([]string, error)

	// Open returns a filesystem for a given bucket name.
	Open(bucket string) (fs.FS, error)
}

type BucketIterator added in v0.6.0

type BucketIterator struct {
	IsTruncated bool
	Continue    string
	// contains filtered or unexported fields
}

func NewBucketIterator added in v0.6.0

func NewBucketIterator(fs fs.FS) *BucketIterator

func (*BucketIterator) CommonPrefixes added in v0.6.0

func (it *BucketIterator) CommonPrefixes() (prefixes []CommonPrefixes)

func (*BucketIterator) PrefixScan added in v0.6.0

func (it *BucketIterator) PrefixScan(prefix string, delimiter string, objectKeyEncoding bool, maxKeys int) ([]Contents, error)

func (*BucketIterator) Seek added in v0.6.0

func (it *BucketIterator) Seek(after string)

Seek sets the starting object key to begin seeking during the next PrefixScan. It set, all objects will be discarded until the first occurrence of after.

type CommonPrefixes added in v0.6.0

type CommonPrefixes struct {
	Prefix string
}

type ConditionOperator added in v0.20.0

type ConditionOperator string
const (
	StringEquals              ConditionOperator = "StringEquals"
	StringNotEquals           ConditionOperator = "StringNotEquals"
	StringEqualsIgnoreCase    ConditionOperator = "StringEqualsIgnoreCase"
	StringNotEqualsIgnoreCase ConditionOperator = "StringNotEqualsIgnoreCase"
	StringLike                ConditionOperator = "StringLike"
	StringNotLike             ConditionOperator = "StringNotLike"
	IpAddress                 ConditionOperator = "IpAddress"
	NotIpAddress              ConditionOperator = "NotIpAddress"
	Bool                      ConditionOperator = "Bool"
)

type Contents added in v0.6.0

type Contents struct {
	ChecksumAlgorithm string
	ETag              string
	Key               string
	LastModified      time.Time
	Size              int
	StorageClass      string
}

type Credential

type Credential struct {
	AccessKeyID string
	Date        time.Time
	Region      string
	Service     string
	Type        string
}

func ParseCredential

func ParseCredential(value string) (*Credential, error)

func (Credential) AppendFormat

func (c Credential) AppendFormat(b []byte) []byte

type Error

type Error struct {
	ErrorCode
	Message string `xml:"Message"`
}

func ErrorFrom

func ErrorFrom(err error) *Error

func EvaluatePolicy added in v0.20.0

func EvaluatePolicy(action Action, resource Resource, policies []*PolicyStatement, context PolicyContextVars) *Error

EvaluatePolicy returns true if the given concrete action and resource applies to any of the given policies. The default action is to deny.

func (*Error) Error

func (e *Error) Error() string

func (*Error) Is added in v0.20.0

func (e *Error) Is(target error) bool

type ErrorCode

type ErrorCode struct {
	Code       string `xml:"Code"`
	StatusCode int    `xml:"-"`
}

type FileIdentityProvider added in v0.20.0

type FileIdentityProvider struct {
	// contains filtered or unexported fields
}

FileIdentityProvider implements ls3.IdentityProvider by reading from a single JSON file. The file is cached for up to the configured amount of time.

func NewFileIdentityProvider added in v0.20.0

func NewFileIdentityProvider(log *zap.Logger, path string, cache time.Duration) (*FileIdentityProvider, error)

func (*FileIdentityProvider) Get added in v0.20.0

func (fp *FileIdentityProvider) Get(keyId string) (*Identity, error)

type Identity added in v0.20.0

type Identity struct {
	Name            string
	AccessKeyId     string
	SecretAccessKey string
	Policy          []*PolicyStatement
}

type IdentityProvider added in v0.20.0

type IdentityProvider interface {
	// Get returns the identity associated with the provided access key ID.
	// It should return ErrMissingAccessKeyId if the given access key does not exist.
	Get(keyId string) (*Identity, error)
}

type Keyring added in v0.20.0

type Keyring map[string]*Identity

Keyring implements IdentityProvider for a static set of identities, where the key is the AccessKeyId.

func (Keyring) Get added in v0.20.0

func (k Keyring) Get(keyId string) (*Identity, error)

type ListBucketResult added in v0.20.0

type ListBucketResult struct {
	XMLName        xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ ListBucketResult"`
	IsTruncated    bool
	Marker         string
	NextMarker     string
	Name           string
	Prefix         string
	Delimiter      string
	MaxKeys        int
	EncodingType   string
	Contents       []Contents
	CommonPrefixes []CommonPrefixes
}

func (*ListBucketResult) Get added in v0.20.0

func (r *ListBucketResult) Get(k string) (string, bool)

Get implements PolicyContextVars based on parameters set for a list bucket request

type ListBucketResultV2 added in v0.20.0

type ListBucketResultV2 struct {
	XMLName               xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ ListBucketResult"`
	Name                  string
	Prefix                string
	Delimiter             string
	MaxKeys               int
	IsTruncated           bool
	EncodingType          string
	ContinuationToken     string
	NextContinuationToken string
	StartAfter            string
	Contents              []Contents
	CommonPrefixes        []CommonPrefixes
}

func (*ListBucketResultV2) Get added in v0.20.0

func (r *ListBucketResultV2) Get(k string) (string, bool)

Get implements PolicyContextVars based on parameters set for a list bucket objects V2 request

type MapContext added in v0.20.0

type MapContext map[string]string

MapContext implements PolicyContextVars for a map.

func (MapContext) Get added in v0.20.0

func (ctx MapContext) Get(k string) (string, bool)

type Method

type Method func(ctx *RequestContext) *Error

type MultiIdentityProvider added in v0.20.0

type MultiIdentityProvider []IdentityProvider

MultiIdentityProvider implements IdentityProvider by trying each provider in order from first to last. If the provider returns InvalidAccessKeyId then MultiIdentityProvider continues to the next identity. If there are no valid identities then ErrMissingAccessKeyId is returned.

func (MultiIdentityProvider) Get added in v0.20.0

func (mp MultiIdentityProvider) Get(keyId string) (*Identity, error)

type NullContext added in v0.20.0

type NullContext struct{}

NullContext implements PolicyContextVars but never returns a value

func (NullContext) Get added in v0.20.0

func (NullContext) Get(_ string) (string, bool)

type Object

type Object struct {
	io.ReadCloser

	Size         int64
	Range        *http_range.Range
	LastModified time.Time
	// ContentType contains the MIME type of the object data.
	// It is the best guess based on the filetype library.
	// This is always set, if unknown the MIME type becomes application/octet-stream
	ContentType string
	// ETag represents the ETag field of the Object.
	// It is always empty.
	ETag string
}

func (*Object) Get added in v0.20.0

func (obj *Object) Get(k string) (string, bool)

Get implements PolicyContextVars for this Object

type OptionalList added in v0.20.0

type OptionalList[T any] []T

OptionalList provides JSON unmarshalling for a slice of objects of type T. T may not be itself a list. If the JSON object is a list or 'null', then the contents is assumed to be a list of T, otherwise it is a single T.

func (*OptionalList[T]) UnmarshalJSON added in v0.20.0

func (l *OptionalList[T]) UnmarshalJSON(b []byte) error

type PolicyConditions added in v0.20.0

type PolicyConditions map[ConditionOperator]map[string]OptionalList[string]

type PolicyContextVars added in v0.20.0

type PolicyContextVars interface {
	Get(k string) (string, bool)
}

type PolicyStatement added in v0.20.0

type PolicyStatement struct {
	// Deny marks this policy as an explicit deny.
	Deny bool
	// Action one or more actions that this policy applies to.
	Action OptionalList[Action]
	// Resource is one or more resources that this policy applies to.
	Resource OptionalList[Resource]
	// Condition sets conditions on when this policy applies.
	Condition PolicyConditions `json:",omitempty"`
}

func (*PolicyStatement) AppliesTo added in v0.20.0

func (p *PolicyStatement) AppliesTo(action Action, resource Resource, context PolicyContextVars) bool

AppliesTo returns true if the given concrete action and resource matches this policy. resource may be empty, in which case this policy applies as long as the action matches.

type RequestContext

type RequestContext struct {
	*zap.Logger
	ID         uuid.UUID
	Bucket     string
	Filesystem fs.FS
	Request    *http.Request
	Identity   *Identity

	// The client IP address
	RemoteIP net.IP
	// Is this connection secure
	Secure bool
	// contains filtered or unexported fields
}

func (*RequestContext) CheckAccess added in v0.20.0

func (ctx *RequestContext) CheckAccess(action Action, resource Resource, vars PolicyContextVars) *Error

CheckAccess verifies that the current identity has the appropriate permissions to execute the given access for the given resource. vars are additional PolicyContextVars that will be used in the conditional policy evaluation. CheckAccess will first verify that the request meets the global policy, if that succeeds it will then check the identity specific PolicyStatement.

func (*RequestContext) Get added in v0.20.0

func (ctx *RequestContext) Get(k string) (string, bool)

Get implements PolicyContextVars for this request.

func (*RequestContext) Header

func (ctx *RequestContext) Header() http.Header

func (*RequestContext) SendKnownError

func (ctx *RequestContext) SendKnownError(err *Error)

SendKnownError replies to the caller with a concrete *Error type using the standard Amazon S3 XML error encoding.

func (*RequestContext) SendPlain

func (ctx *RequestContext) SendPlain(statusCode int) io.Writer

func (*RequestContext) SendXML

func (ctx *RequestContext) SendXML(statusCode int, payload any)

type Resource added in v0.20.0

type Resource string

type Server

type Server struct {
	// contains filtered or unexported fields
}

func NewServer

func NewServer(opts *ServerOptions) *Server

func (*Server) GetBucketLocation

func (s *Server) GetBucketLocation(ctx *RequestContext) *Error

func (*Server) GetObject

func (s *Server) GetObject(ctx *RequestContext) *Error

func (*Server) HeadBucket

func (s *Server) HeadBucket(ctx *RequestContext) *Error

func (*Server) HeadObject

func (s *Server) HeadObject(ctx *RequestContext) *Error

func (*Server) ListBuckets added in v0.5.0

func (s *Server) ListBuckets(ctx *RequestContext) *Error

func (*Server) ListObjects added in v0.6.0

func (s *Server) ListObjects(ctx *RequestContext) *Error

func (*Server) ListObjectsV2

func (s *Server) ListObjectsV2(ctx *RequestContext) *Error

func (*Server) ServeHTTP

func (s *Server) ServeHTTP(rw http.ResponseWriter, r *http.Request)

type ServerOptions added in v0.21.0

type ServerOptions struct {
	Log          *zap.Logger
	Signer       Signer
	Identity     IdentityProvider
	Filesystem   BucketFilesystemProvider
	Domain       string
	GlobalPolicy []*PolicyStatement
	ClientIP     security.ClientIP
	ClientTLS    security.ClientTLS
}

type SignAWSV4 added in v0.2.0

type SignAWSV4 struct {
	// contains filtered or unexported fields
}

func (SignAWSV4) Sign added in v0.2.0

func (s SignAWSV4) Sign(r *http.Request, identity *Identity, payload []byte) error

func (SignAWSV4) Verify added in v0.2.0

func (s SignAWSV4) Verify(r *http.Request, provider IdentityProvider) (*Identity, error)

func (SignAWSV4) VerifyHeaders added in v0.9.0

func (s SignAWSV4) VerifyHeaders(r *http.Request, provider IdentityProvider) (*Identity, error)

func (SignAWSV4) VerifyQuery added in v0.9.0

func (s SignAWSV4) VerifyQuery(r *http.Request, provider IdentityProvider) (*Identity, error)

type Signer

type Signer interface {
	// Sign computes and signs the given HTTP request using the given request payload and Identity.
	// payload should be the contents of r.Body.
	Sign(r *http.Request, identity *Identity, payload []byte) error

	// Verify verifies the authorization and request signature present in the HTTP request.
	// Verify should return a non-nil error on verification failure, ideally this should contain the underlying type *Error.
	// If Verify reads data from r.Body, it must ensure that the data can be re-read from r if Verify returns a nil error.
	// If the request is not signed using the Signer's algorithm then it shall return nil, nil.
	Verify(r *http.Request, provider IdentityProvider) (*Identity, error)
}

A Signer is a type capable of signing and verifying the authorization and request signature present in an HTTP request.

type SingleBucketFilesystem added in v0.2.0

type SingleBucketFilesystem struct {
	fs.FS
}

SingleBucketFilesystem implements BucketFilesystemProvider that always returns the same filesystem for any bucket name provided.

func (*SingleBucketFilesystem) ListBuckets added in v0.5.0

func (p *SingleBucketFilesystem) ListBuckets() ([]string, error)

ListBuckets always returns the same bucket name. The actual name doesn't matter, as the provider will always return the same filesystem.

func (*SingleBucketFilesystem) Open added in v0.5.0

func (p *SingleBucketFilesystem) Open(_ string) (fs.FS, error)

type SubdirBucketFilesystem added in v0.2.0

type SubdirBucketFilesystem struct {
	fs.FS
}

func (*SubdirBucketFilesystem) ListBuckets added in v0.5.0

func (p *SubdirBucketFilesystem) ListBuckets() ([]string, error)

ListBuckets returns all subdirectories of the base filesystem.

func (*SubdirBucketFilesystem) Open added in v0.5.0

func (p *SubdirBucketFilesystem) Open(bucket string) (fs.FS, error)

Open returns a subdirectory of the base filesystem for each bucket. The error NoSuchBucket is returned if fs.Stat of the bucket path returns an error. The error InvalidBucketState is returned if fs.Sub returns an error.

Directories

Path Synopsis
cmd
ls3

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL