tilenol

package module
v1.0.9 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 30, 2023 License: MIT Imports: 33 Imported by: 1

README

tilenol GoDoc Go Report Card Build Status Docker Pulls

Tilenol is a scalable web server for serving geospatial data stored in multiple supported backends as Mapbox Vector Tiles.

Installation

Navigate to the root tilenol/ directory (where the Makefile is located) and run:

make install

Usage

tilenol
usage: tilenol [<flags>] <command> [<args> ...]

Flags:
  --help  Show context-sensitive help (also try --help-long and --help-man).

Commands:
  help [<command>...]
    Show help.

  run [<flags>]
    Runs the Tilenol server

  version
    Prints out the version
tilenol run
usage: tilenol run [<flags>]

Runs the Tilenol server

Flags:
      --help                 Show context-sensitive help (also try --help-long and --help-man).
  -d, --debug                    Enable debug mode
  -f, --config-file=tilenol.yml  Server configuration file
  -p, --port=3000                Port to serve tiles on
  -i, --internal-port=3001       Port for internal metrics and healthchecks
  -x, --enable-cors              Enables cross-origin resource sharing (CORS)
  -s, --simplify-shapes          Simplifies geometries based on zoom level
  -n, --num-processes=0          Sets the number of processes to be used
Configuration
# Cache configuration (optional)
cache:
  redis:
    host: localhost
    port: 6379
    ttl: 24h
# Layer configuration
layers:
  - name: buildings
    minzoom: 14
    source:
      elasticsearch:
        host: localhost
        port: 9200
        index: buildings
        geometryField: geometry
        sourceFields:
          area_sqft: building.area_sqft
          height_ft: building.height_ft
Docker

Tilenol is also available as stationa/tilenol on DockerHub. This can be helpful when running in cloud environments or on Kubernetes:

# To run the Docker image on your machine, you can use the following:

docker run \
  --rm \
  -it \
  -p 3000:3000 \
  -v my-tilenol-conf.yml:/conf/tilenol.yml \
  stationa/tilenol \
  run -p 3000 -f /conf/tilenol.yml

# Note that this will bind tilenol to your local port 3000, and requires a configuration file to be
# mounted at the /conf/tilenol.yml path

Currently, the Docker image tags are published according to the following scheme:

  • vM.m.p is the image built for the exact SemVer major/minor/patch version M.m.p, e.g. v1.0.7
  • vM is the latest image available for the given SemVer major version number M, e.g. v1
  • latest is the latest stable image available
  • devel is the latest unstable image available based on the last commit made to the main branch

We recommend you use either vM or vM.m.p where possible, though your appetite for risk may vary by use case.

Supported backends

Currently, tilenol supports the following data backends:

QGIS support

Tilenol layers can also be viewed in GIS software such as QGIS.

Instructions for adding a Tilenol layer to QGIS 3.16:

  1. Navigate to Layer > Add Layer > Add Vector Tile Layer
  2. Click on New, and then Create a New Generic Connection
  3. Configure the Connection Details, for example:
  4. Name: Places
  5. URL: http://localhost:3000/places/{z}/{x}/{y}.mvt
  6. Min.Zoom Level: 0
  7. Max.Zoom Level: 25

Example screenshot, pointing QGIS to a locally running instance of Tilenol:

  1. Click OK and then Add
  2. You should be able to see the Tilenol places layer at appropriate zoom levels

Contributing

When contributing to this repository, please follow the steps below:

  1. Fork the repository
  2. Submit your patch in one commit, or a series of well-defined commits
  3. Submit your pull request and make sure you reference the issue you are addressing

Documentation

Index

Constants

View Source
const (

	// ScrollSize is the max number of documents per scroll page
	ScrollSize = 250
	// ScrollTimeout is the time.Duration to keep the scroll context alive
	ScrollTimeout = 10 * time.Second
)
View Source
const (
	TableAlias = "__tilenol__table"
	// TODO: Externalize this?
	QueryTimeout = 30 * time.Second
)
View Source
const (
	// MinZoom is the absolute minimum zoom for a layer
	//  tile requests with z-values lower than this will be rejected
	//  layer configurations with Minzoom values lower than this will be rejected
	MinZoom = 0
	// MaxZoom is the absolute maximum zoom for a layer
	//  tile requests with z-values higher than this will be rejected
	//  layer configurations with Maxzoom values lower than this will be rejected
	MaxZoom = 22
	// MinSimplify is the minimum simplification radius
	MinSimplify = 1.0
	// MaxSimplify is the maximum simplification radius
	MaxSimplify = 10.0
	// AllLayers is the special request parameter for returning all source layers
	AllLayers = "_all"
)

Variables

View Source
var (
	MultipleSourcesErr         = errors.New("Layers can only support a single backend source")
	NoSourcesErr               = errors.New("Layers must have a single backend source configured")
	LayerMinZoomOutOfBoundsErr = errors.New("Layer Min zoom is below absolute min zoom")
	LayerMaxZoomOutOfBoundsErr = errors.New("Layer Max Zoom is above absolute max zoom")
)
View Source
var (
	InvalidTableConfig = errors.New("Either \"tableExpression\" or \"table\" + \"schema\" can be set, not both.")
	MissingTableConfig = errors.New("Either \"tableExpression\" or \"table\" + \"schema\" must be set.")
)
View Source
var (
	// ErrNoValue occurs when trying to access a value that doesn't exist in the cache
	ErrNoValue = errors.New("No value exists in cache")
)
View Source
var (
	InvalidGeometryErr = errors.New("Column value was not a valid geometry")
)
View Source
var (
	// Logger is the global logging instance
	Logger = logrus.New()
)

Functions

func CheckPing added in v1.0.4

func CheckPing(db *sql.DB) error

CheckPing asserts that we can ping the connected database

func CheckReadOnly added in v1.0.4

func CheckReadOnly(db *sql.DB) error

CheckReadOnly warns if the current database connection is capable of read-write transactions

func EnableCORS

func EnableCORS(s *Server) error

EnableCORS configures the server for CORS (cross-origin resource sharing)

func GetNested

func GetNested(something interface{}, keyParts []string) (interface{}, bool)

GetNested is a utility function to traverse a path of keys in a nested JSON object

func RowsToMaps added in v1.0.4

func RowsToMaps(rows *sql.Rows, geomColumn string) ([]map[string]interface{}, error)

RowsToMaps converts Go's sql.Rows data structure into a list of maps where the key is a string column name, and the value is the raw SQL value

func SimplifyShapes

func SimplifyShapes(s *Server) error

SimplifyShapes enables geometry simplification based on the requested zoom level

Types

type Cache

type Cache interface {
	// Exists determines whether or not there is a cache value for the given key
	Exists(key string) bool
	// Get retrieves the cached data given a key
	Get(key string) ([]byte, error)
	// Put stores a new value in the cache at a given key
	Put(key string, val []byte) error
}

Cache is a generic interface for a tile server cache

func CreateCache

func CreateCache(config *CacheConfig) (Cache, error)

CreateCache creates a new generic Cache from a CacheConfig

func NewInMemoryCache

func NewInMemoryCache() Cache

NewInMemoryCache allocates a new InMemoryCache

func NewRedisCache

func NewRedisCache(config *RedisConfig) (Cache, error)

NewRedisCache creates a new RedisCache given a RedisConfig

type CacheConfig

type CacheConfig struct {
	// Redis is an optional YAML key for configuring a RedisCache
	Redis *RedisConfig `yaml:"redis"`
}

CacheConfig is a generic YAML cache configuration object

type Config

type Config struct {
	// Cache configures the tile server cache
	Cache *CacheConfig `yaml:"cache"`
	// Layers configures the tile server layers
	Layers []LayerConfig `yaml:"layers"`
}

Config is a YAML server configuration object

func LoadConfig

func LoadConfig(configFile *os.File) (*Config, error)

LoadConfig loads the configuration from disk, and decodes it into a Config object

type ConfigOption

type ConfigOption func(s *Server) error

ConfigOption is a function that changes a configuration setting of the server.Server

func ConfigFile

func ConfigFile(configFile *os.File) ConfigOption

ConfigFile loads a YAML configuration file from disk to set up the server

func InternalPort

func InternalPort(internalPort uint16) ConfigOption

InternalPort changes the port number used for administrative endpoints (e.g. healthcheck)

func Port

func Port(port uint16) ConfigOption

Port changes the port number used for serving tile data

type Dict

type Dict map[string]interface{}

Dict is a type alias for map[string]interface{} that cleans up literals and also adds a helper method to implement the elastic.Query interface

func (*Dict) Map

func (d *Dict) Map() map[string]interface{}

Map is a helper method to cleanly alias back to map[string]interface{}

func (*Dict) Source

func (d *Dict) Source() (interface{}, error)

Source implements the elastic.Query interface, by simply returning the raw Dict contents

type DumbScanner added in v1.0.4

type DumbScanner struct {
	Value interface{}
}

func (*DumbScanner) Scan added in v1.0.4

func (d *DumbScanner) Scan(src interface{}) error

type ElasticsearchConfig

type ElasticsearchConfig struct {
	// Host is the hostname part of the backend Elasticsearch cluster
	Host string `yaml:"host"`
	// Host is the port number of the backend Elasticsearch cluster
	Port int `yaml:"port"`
	// Index is the name of the Elasticsearch index used for retrieving feature data
	Index string `yaml:"index"`
	// GeometryField is the name of the document field that holds the feature geometry
	GeometryField string `yaml:"geometryField"`
	// SourceFields is a mapping from the feature property name to the source document
	// field name
	SourceFields map[string]string `yaml:"sourceFields"`
}

ElasticsearchConfig is the YAML configuration structure for configuring a new ElasticsearchSource

type ElasticsearchSource

type ElasticsearchSource struct {
	// ES is the internal Elasticsearch cluster client
	ES *elastic.Client
	// Index is the name of the Elasticsearch index used for retrieving feature data
	Index string
	// GeometryField is the name of the document field that holds the feature geometry
	GeometryField string
	// SourceFields is a mapping from the feature property name to the source document
	// field name
	SourceFields map[string]string
}

ElasticsearchSource is a Source implementation that retrieves feature data from an Elasticsearch cluster

func (*ElasticsearchSource) GetFeatures

GetFeatures implements the Source interface, to get feature data from an Elasticsearch cluster

func (*ElasticsearchSource) HitToFeature

func (e *ElasticsearchSource) HitToFeature(hit *elastic.SearchHit) (*geojson.Feature, error)

HitToFeature converts an Elasticsearch hit object into a GeoJSON feature, by using the hit's geometry as the feature geometry, and mapping all other requested source fields to feature properties

type Handler

type Handler func(context.Context, io.Writer, *http.Request) error

Handler is a type alias for a more functional HTTP request handler

type InMemoryCache

type InMemoryCache struct {
	// contains filtered or unexported fields
}

InMemoryCache implements the Cache interface backed by an in-memory map

func (*InMemoryCache) Exists

func (i *InMemoryCache) Exists(key string) bool

Exists checks the internal map for the existence of the key

func (*InMemoryCache) Get

func (i *InMemoryCache) Get(key string) ([]byte, error)

Get retrieves the value stored in the internal map

func (*InMemoryCache) Put

func (i *InMemoryCache) Put(key string, val []byte) error

Put stores a new value in the internal map at a given key

type InvalidRequestError added in v1.0.2

type InvalidRequestError struct {
	// contains filtered or unexported fields
}

Error type for HTTP Status code 400

func (InvalidRequestError) Error added in v1.0.2

func (f InvalidRequestError) Error() string

type Layer

type Layer struct {
	Name        string
	Description string
	Minzoom     int
	Maxzoom     int
	Cacheable   bool
	// contains filtered or unexported fields
}

Layer is a configured, hydrated tile server layer

func CreateLayer

func CreateLayer(layerConfig LayerConfig) (*Layer, error)

CreateLayer creates a new Layer given a LayerConfig

func (Layer) GetFeatures added in v1.0.5

func (l Layer) GetFeatures(ctx context.Context, r *TileRequest) (*geojson.FeatureCollection, error)

GetFeatures implements a passthrough interface to the layer's underlying source

func (Layer) Hash added in v1.0.5

func (l Layer) Hash() string

Hash computes a content-based SHA256 digest to diff layer "versions"

func (Layer) String added in v1.0.5

func (l Layer) String() string

type LayerConfig

type LayerConfig struct {
	// Name is the effective name of the layer
	Name string `yaml:"name"`
	// Description is an optional short descriptor of the layer
	Description string `yaml:"description"`
	// Minzoom specifies the minimum z value for the layer
	Minzoom int `yaml:"minzoom"`
	// Maxzoom specifies the maximum z value for the layer
	Maxzoom int `yaml:"maxzoom"`
	// NoCache indicates that this layer should not cache its source data
	NoCache bool `yaml:"nocache"`
	// Source configures the underlying Source for the layer
	Source SourceConfig `yaml:"source"`
}

LayerConfig represents a general YAML layer configuration object

type NilCache

type NilCache struct{}

NilCache implements the Cache interface as a no-op

func (*NilCache) Exists

func (n *NilCache) Exists(key string) bool

Exists always returns false, since the NilCache stores no values

func (*NilCache) Get

func (n *NilCache) Get(key string) ([]byte, error)

Get should not be called for the NilCache, as it stores no values

func (*NilCache) Put

func (n *NilCache) Put(key string, val []byte) error

Put is a no-op for the NilCache

type NilSource added in v1.0.5

type NilSource struct{}

NilSource implements the Source interface with no data

func (*NilSource) GetFeatures added in v1.0.5

func (n *NilSource) GetFeatures(ctx context.Context, req *TileRequest) (*geojson.FeatureCollection, error)

GetFeatures returns no data for all requests

type PostGISConfig added in v1.0.4

type PostGISConfig struct {
	// DSN is the "data source name" that specifies how to connect to the database server
	DSN string `yaml:"dsn"`
	// Schema is the table space to use for queries
	Schema string `yaml:"schema"`
	// Table is the name of the table to use for queries
	Table string `yaml:"table"`
	// TableExpression is a valid SQL query that is used as an alternative to Schema and Table
	TableExpression string `yaml:"tableExpression"`
	// GeometryField is the name of the column that holds the feature geometry
	GeometryField string `yaml:"geometryField"`
	// SourceFields is a mapping from the feature property name to the source row
	// column names
	SourceFields map[string]string `yaml:"sourceFields"`
}

PostGISConfig is the YAML configuration structure for configuring a new PostGISSource

func (*PostGISConfig) Dataset added in v1.0.4

func (c *PostGISConfig) Dataset() (*goqu.SelectDataset, error)

Dataset constructs a subquery to be used as the source table for all request-time queries

type PostGISSource added in v1.0.4

type PostGISSource struct {
	DB            *goqu.Database
	Dataset       *goqu.SelectDataset
	GeometryField string
	SourceFields  map[string]string
}

PostGISSource is a Source implementation that retrieves feature data from a PostGIS server

func (*PostGISSource) GetFeatures added in v1.0.4

func (p *PostGISSource) GetFeatures(ctx context.Context, req *TileRequest) (*geojson.FeatureCollection, error)

GetFeatures implements the Source interface, to get feature data from an PostGIS server

type RedisCache

type RedisCache struct {
	// Client is the backend Redis cluster client
	Client *redis.Client
	// TTL is how long each cache entry should remain before refresh
	TTL time.Duration
}

RedisCache is a Redis-backed Cache implementation

func (*RedisCache) Exists

func (r *RedisCache) Exists(key string) bool

Exists tries to get the cached value for a key, returning whether or not the key exists

func (*RedisCache) Get

func (r *RedisCache) Get(key string) ([]byte, error)

Get retrieves the cached value for a given key

func (*RedisCache) Put

func (r *RedisCache) Put(key string, val []byte) error

Put attempts to store a new value into Redis at a given key

type RedisConfig

type RedisConfig struct {
	// Host is the Redis cluster host name
	Host string `yaml:"host"`
	// Port is the Redis cluster port number
	Port int `yaml:"port"`
	// TTL is how long each cache entry should remain before refresh
	TTL time.Duration `yaml:"ttl"`
}

RedisConfig is the YAML configuration for a RedisCache

type Server

type Server struct {
	// Port is the port number to bind the tile server
	Port uint16
	// InternalPort is the port number to bind the internal metrics endpoints
	InternalPort uint16
	// EnableCORS configures whether or not the tile server responds with CORS headers
	EnableCORS bool
	// Simplify configures whether or not the tile server simplifies outgoing feature
	// geometries based on zoom level
	Simplify bool
	// Layers is the list of configured layers supported by the tile server
	Layers []Layer
	// Cache is an optional cache object that the server uses to cache responses
	Cache Cache
}

Server is a tilenol server instance

func NewServer

func NewServer(configOpts ...ConfigOption) (*Server, error)

NewServer creates a new server instance pre-configured with the given ConfigOption's

func (*Server) Start

func (s *Server) Start()

Start actually starts the server instance. Note that this blocks until an interrupting signal

type Source

type Source interface {
	// GetFeatures retrieves the GeoJSON FeatureCollection for the given request
	GetFeatures(context.Context, *TileRequest) (*geojson.FeatureCollection, error)
}

Source is a generic interface for all feature data sources

func NewElasticsearchSource

func NewElasticsearchSource(config *ElasticsearchConfig) (Source, error)

NewElasticsearchSource creates a new Source that retrieves feature data from an Elasticsearch cluster

func NewPostGISSource added in v1.0.4

func NewPostGISSource(config *PostGISConfig) (Source, error)

NewPostGISSource creates a new Source that retrieves feature data from a PostGIS server

type SourceConfig

type SourceConfig struct {
	// Elasticsearch is an optional YAML key for configuring an ElasticsearchConfig
	Elasticsearch *ElasticsearchConfig `yaml:"elasticsearch"`
	// PostGIS is an optional YAML key for configuring a PostGISConfig
	PostGIS *PostGISConfig `yaml:"postgis"`
}

SourceConfig represents a generic YAML source configuration object

type TileRequest

type TileRequest struct {
	X    int
	Y    int
	Z    int
	Args map[string][]string
}

TileRequest is an object containing the tile request context

func MakeTileRequest added in v1.0.2

func MakeTileRequest(req *http.Request, x int, y int, z int) (*TileRequest, error)

Sanitize TileRequest arguments and return an error if sanity checking fails.

func (*TileRequest) MapTile

func (t *TileRequest) MapTile() maptile.Tile

MapTile creates a maptile.Tile object from the TileRequest

func (*TileRequest) String added in v1.0.5

func (r *TileRequest) String() string

Directories

Path Synopsis
cmd

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL