README
¶
osm/osmpbf 
Package osmpbf provides a scanner for decoding large OSM PBF files. They are typically found at planet.osm.org or Geofabrik Download.
Example:
file, err := os.Open("./delaware-latest.osm.pbf")
if err != nil {
panic(err)
}
defer f.Close()
// The third parameter is the number of parallel decoders to use.
scanner := osmpbf.New(context.Background(), file, runtime.GOMAXPROCS(-1))
defer scanner.Close()
for scanner.Scan() {
switch o := scanner.Object().(type)
case *osm.Node:
case *osm.Way:
case *osm.Relation:
}
if err := scanner.Err(); err != nil {
panic(err)
}
Note: Scanners are not safe for parallel use. One should feed the objects into a channel and have workers read from that.
Skipping Types
Sometimes only ways or relations are needed. In this case reading and creating those objects can be skipped completely. After creating the Scanner set the appropriate attributes to true.
type Scanner struct {
// Skip element types that are not needed. The data is skipped
// at the encoded protobuf level, but each block still
// needs to be decompressed.
SkipNodes bool
SkipWays bool
SkipRelations bool
// contains filtered or unexported fields
}
Filtering Elements
The above skips all elements of a type. To filter based on the element's tags or other values, use the filter functions. These filter functions are called in parallel and not in a predefined order. This can be a performant way to filter for elements with a certain set of tags.
type Scanner struct {
// If the Filter function is false, the element well be skipped
// at the decoding level. The functions should be fast, they block the
// decoder, there are `procs` number of concurrent decoders.
// Elements can be stored if the function returns true. Memory is
// reused if the filter returns false.
FilterNode func(*osm.Node) bool
FilterWay func(*osm.Way) bool
FilterRelation func(*osm.Relation) bool
// contains filtered or unexported fields
}
OSM PBF files with node locations on ways
This package supports reading OSM PBF files where the ways have been annotated with the coordinates of each node. Such files can be generated using osmium, with the add-locations-to-ways subcommand. This feature makes it possible to work with the ways and their geometries without having to keep all node locations in some index (which takes work and memory resources).
Coordinates are stored in the Lat and Lon fields of each WayNode. There is no need to specify an explicit option; when the node locations are present on the ways, they are loaded automatically. For more info about the OSM PBF format extension, see the original blog post.
Using cgo/czlib for decompression
OSM PBF files are a set of blocks that are zlib compressed. When using the pure golang implementation this can account for about 1/3 of the read time. When cgo is enabled the package will used czlib.
$ CGO_ENABLED=0 go test -bench . > disabled.txt
$ CGO_ENABLED=1 go test -bench . > enabled.txt
$ benchcmp disabled.txt enabled.txt
benchmark old ns/op new ns/op delta
BenchmarkLondon-12 312294630 229927205 -26.37%
BenchmarkLondon_nodes-12 246562457 160021768 -35.10%
BenchmarkLondon_ways-12 216803544 134747327 -37.85%
BenchmarkLondon_relations-12 158722633 80560144 -49.24%
benchmark old allocs new allocs delta
BenchmarkLondon-12 2469128 2416804 -2.12%
BenchmarkLondon_nodes-12 1056166 1003850 -4.95%
BenchmarkLondon_ways-12 1845032 1792716 -2.84%
BenchmarkLondon_relations-12 509090 456772 -10.28%
benchmark old bytes new bytes delta
BenchmarkLondon-12 963734544 954877896 -0.92%
BenchmarkLondon_nodes-12 658337435 649482060 -1.35%
BenchmarkLondon_ways-12 441674734 432819378 -2.00%
BenchmarkLondon_relations-12 187941609 179086389 -4.71%
Documentation
¶
Overview ¶
Example (Stats) ¶
ExampleStats demonstrates how to read a full file and gather some stats. This is similar to `osmconvert --out-statistics`
package main
import (
"context"
"fmt"
"math"
"os"
"time"
"github.com/paulmach/osm"
"github.com/paulmach/osm/osmpbf"
)
// ExampleStats demonstrates how to read a full file and gather some stats.
// This is similar to `osmconvert --out-statistics`
func main() {
f, err := os.Open("../testdata/delaware-latest.osm.pbf")
if err != nil {
fmt.Printf("could not open file: %v", err)
os.Exit(1)
}
defer f.Close()
nodes, ways, relations := 0, 0, 0
stats := newElementStats()
minLat, maxLat := math.MaxFloat64, -math.MaxFloat64
minLon, maxLon := math.MaxFloat64, -math.MaxFloat64
minTS, maxTS := time.Date(2100, 1, 1, 0, 0, 0, 0, time.UTC), time.Time{}
var (
maxNodeRefs int
maxNodeRefsID osm.WayID
)
var (
maxRelRefs int
maxRelRefsID osm.RelationID
)
scanner := osmpbf.New(context.Background(), f, 3)
defer scanner.Close()
for scanner.Scan() {
var ts time.Time
switch e := scanner.Object().(type) {
case *osm.Node:
nodes++
ts = e.Timestamp
stats.Add(e.ElementID(), e.Tags)
if e.Lat > maxLat {
maxLat = e.Lat
}
if e.Lat < minLat {
minLat = e.Lat
}
if e.Lon > maxLon {
maxLon = e.Lon
}
if e.Lon < minLon {
minLon = e.Lon
}
case *osm.Way:
ways++
ts = e.Timestamp
stats.Add(e.ElementID(), e.Tags)
if l := len(e.Nodes); l > maxNodeRefs {
maxNodeRefs = l
maxNodeRefsID = e.ID
}
case *osm.Relation:
relations++
ts = e.Timestamp
stats.Add(e.ElementID(), e.Tags)
if l := len(e.Members); l > maxRelRefs {
maxRelRefs = l
maxRelRefsID = e.ID
}
}
if ts.After(maxTS) {
maxTS = ts
}
if ts.Before(minTS) {
minTS = ts
}
}
if err := scanner.Err(); err != nil {
fmt.Printf("scanner returned error: %v", err)
os.Exit(1)
}
fmt.Println("timestamp min:", minTS.Format(time.RFC3339))
fmt.Println("timestamp max:", maxTS.Format(time.RFC3339))
fmt.Printf("lon min: %0.7f\n", minLon)
fmt.Printf("lon max: %0.7f\n", maxLon)
fmt.Printf("lat min: %0.7f\n", minLat)
fmt.Printf("lat max: %0.7f\n", maxLat)
fmt.Println("nodes:", nodes)
fmt.Println("ways:", ways)
fmt.Println("relations:", relations)
fmt.Println("version max:", stats.MaxVersion)
fmt.Println("node id min:", stats.Ranges[osm.TypeNode].Min)
fmt.Println("node id max:", stats.Ranges[osm.TypeNode].Max)
fmt.Println("way id min:", stats.Ranges[osm.TypeWay].Min)
fmt.Println("way id max:", stats.Ranges[osm.TypeWay].Max)
fmt.Println("relation id min:", stats.Ranges[osm.TypeRelation].Min)
fmt.Println("relation id max:", stats.Ranges[osm.TypeRelation].Max)
fmt.Println("keyval pairs max:", stats.MaxTags)
fmt.Println("keyval pairs max object:", stats.MaxTagsID.Type(), stats.MaxTagsID.Ref())
fmt.Println("noderefs max:", maxNodeRefs)
fmt.Println("noderefs max object: way", maxNodeRefsID)
fmt.Println("relrefs max:", maxRelRefs)
fmt.Println("relrefs max object: relation", maxRelRefsID)
}
// Stats is a shared bit of code to accumulate stats from the element ids.
type elementStats struct {
Ranges map[osm.Type]*idRange
MaxVersion int
MaxTags int
MaxTagsID osm.ElementID
}
type idRange struct {
Min, Max int64
}
func newElementStats() *elementStats {
return &elementStats{
Ranges: map[osm.Type]*idRange{
osm.TypeNode: {Min: math.MaxInt64},
osm.TypeWay: {Min: math.MaxInt64},
osm.TypeRelation: {Min: math.MaxInt64},
},
}
}
func (s *elementStats) Add(id osm.ElementID, tags osm.Tags) {
s.Ranges[id.Type()].Add(id.Ref())
if v := id.Version(); v > s.MaxVersion {
s.MaxVersion = v
}
if l := len(tags); l > s.MaxTags {
s.MaxTags = l
s.MaxTagsID = id
}
}
func (r *idRange) Add(ref int64) {
if ref > r.Max {
r.Max = ref
}
if ref < r.Min {
r.Min = ref
}
}
Output: timestamp min: 2007-10-16T15:59:24Z timestamp max: 2016-08-10T17:32:02Z lon min: -76.1748935 lon max: -74.4929376 lat min: 38.0273717 lat max: 39.9688859 nodes: 723870 ways: 73144 relations: 1644 version max: 421 node id min: 75385503 node id max: 4343778904 way id min: 9650669 way id max: 436488690 relation id min: 82010 relation id max: 6462005 keyval pairs max: 276 keyval pairs max object: relation 148838 noderefs max: 1811 noderefs max object: way 318739264 relrefs max: 7177 relrefs max object: relation 4799100
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Header ¶
type Header struct {
Bounds *osm.Bounds
RequiredFeatures []string
OptionalFeatures []string
WritingProgram string
Source string
ReplicationTimestamp time.Time
ReplicationSeqNum uint64
ReplicationBaseURL string
}
Header contains the contents of the header in the pbf file.
type Scanner ¶
type Scanner struct {
// Skip element types that are not needed. The data is skipped
// at the encoded protobuf level, but each block still needs to be decompressed.
SkipNodes bool
SkipWays bool
SkipRelations bool
// If the Filter function is false, the element well be skipped
// at the decoding level. The functions should be fast, they block the
// decoder, there are `procs` number of concurrent decoders.
// Elements can be stored if the function returns true. Memory is
// reused if the filter returns false.
FilterNode func(*osm.Node) bool
FilterWay func(*osm.Way) bool
FilterRelation func(*osm.Relation) bool
// contains filtered or unexported fields
}
Scanner provides a convenient interface reading a stream of osm data from a file or url. Successive calls to the Scan method will step through the data.
Scanning stops unrecoverably at EOF, the first I/O error, the first xml error or the context being cancelled. When a scan stops, the reader may have advanced arbitrarily far past the last token.
The Scanner API is based on bufio.Scanner https://golang.org/pkg/bufio/#Scanner
func New ¶
New returns a new Scanner to read from r. procs indicates amount of paralellism, when reading blocks which will off load the unzipping/decoding to multiple cpus.
func (*Scanner) Close ¶
Close cleans up all the reading goroutines, it does not close the underlying reader.
func (*Scanner) FullyScannedBytes ¶
FullyScannedBytes returns the number of bytes that have been read and fully scanned. OSM protobuf files contain data blocks with 8000 nodes each. The returned value contains the bytes for the blocks that have been fully scanned.
A user can use this number of seek forward in a file and begin reading mid-data. Note that while elements are usually sorted by Type, ID, Version in OSM protobuf files, versions of given element may span blocks.
func (*Scanner) Header ¶
Header returns the pbf file header with interesting information about how it was created.
func (*Scanner) Object ¶
Object returns the most recent token generated by a call to Scan as a new osm.Object. Currently osm.pbf files only contain nodes, ways and relations. This method returns an object so match the osm.Scanner interface and allows this Scanner to share an interface with osmxml.Scanner.
func (*Scanner) PreviousFullyScannedBytes ¶ added in v0.1.0
PreviousFullyScannedBytes returns the previous value of FullyScannedBytes. This is interesting because it's not totally clear if a feature spans a block. For example, if one quits after finding the first relation, upon restarting there is no way of knowing if the first relation is complete, so skip it. But if this relation is the first relation in the file we'll skip a full relation.
func (*Scanner) Scan ¶
Scan advances the Scanner to the next element, which will then be available through the Element method. It returns false when the scan stops, either by reaching the end of the input, an io error, an xml error or the context being cancelled. After Scan returns false, the Err method will return any error that occurred during scanning, except that if it was io.EOF, Err will return nil.
Source Files
Directories