mmdbmeld

package module
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 6, 2023 License: GPL-3.0 Imports: 20 Imported by: 0

README

Build your own GeoIP .mmdb!

Step 0: Setup Go

Follow instructions here: https://go.dev/dl/

Step 1: Compile

$ go build -C cmd/mmdbmeld
$ go build -C cmd/mmdbcheck
$ go build -C cmd/mmdbquery

Step 2: Download geoip data sources

Default sources are CC0/PDDL and provided by https://github.com/sapics/ip-location-db

$ ./input/update.sh

[...]

Note:
If you start using mmdbmeld, it is highly recommended you choose geoip data sources suitable for your use case, in terms of size and quality. Be sure to also check the licenses of the data sources before using them.

Step 3: Build your MMDBs

$ ./cmd/mmdbmeld/mmdbmeld config-example.yml

==========
building My IPv4 GeoIP DB
database options set: IPVersion=4 RecordSize=24 (IncludeReservedNetworks=true DisableIPv4Aliasing=true)
database types: autonomous_system_number, autonomous_system_organization, country.iso_code
optimizations set: FloatDecimals=2 ForceIPVersion=true MaxPrefix=0
conditional resets: [{IfChanged:[country] Reset:[location]}]
---
processing input/iptoasn-asn-ipv4.csv...
inserted 100000 entries - batch in 356ms (4µs/op)
inserted 200000 entries - batch in 327ms (3µs/op)
inserted 300000 entries - batch in 322ms (3µs/op)
inserted 360970 entries - batch in 224ms (2µs/op)
---
processing input/geo-whois-asn-country-ipv4.csv...
inserted 100000 entries - batch in 1.012s (10µs/op)
inserted 200000 entries - batch in 889ms (9µs/op)
inserted 238756 entries - batch in 418ms (4µs/op)
---
My IPv4 GeoIP DB finished: inserted 599726 entries in 4s, resulting in 7.67 MB written to output/geoip-v4.mmdb

==========
building My IPv6 GeoIP DB
database options set: IPVersion=6 RecordSize=24 (IncludeReservedNetworks=true DisableIPv4Aliasing=true)
database types: autonomous_system_number, autonomous_system_organization, country.iso_code
optimizations set: FloatDecimals=2 ForceIPVersion=true MaxPrefix=0
conditional resets: [{IfChanged:[country] Reset:[location]}]
---
processing input/iptoasn-asn-ipv6.csv...
inserted 81939 entries - batch in 352ms (4µs/op)
---
processing input/geo-whois-asn-country-ipv6.csv...
inserted 82321 entries - batch in 1.609s (16µs/op)
---
My IPv6 GeoIP DB finished: inserted 164260 entries in 2s, resulting in 5.42 MB written to output/geoip-v6.mmdb

Step 4: Check your MMDBs

$ ./cmd/mmdbcheck/mmdbcheck all output/geoip-v4.mmdb
$ ./cmd/mmdbcheck/mmdbcheck all output/geoip-v6.mmdb

[...]

loading output/geoip-v4.mmdb with 7.67 MB

Running all checks:

Probing:
..............................................................................................................................................................................................................................................................::
analyzed with 0 lookup errors
Total=  14462461 Country=99.57% Coords=0.00% ASN=82.40% ASOrg=82.40% AC=0.00% SP=0.00% AP=0.00%

Network Mask Stats:
        Total=    800822 Country=99.93% Coords=0.00% ASN=85.24% ASOrg=85.24% AC=0.00% SP=0.00% AP=0.00%
CIDR=/  7 Total=         1 Country=100.00% Coords=0.00% ASN=100.00% ASOrg=100.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/  8 Total=         6 Country=100.00% Coords=0.00% ASN=100.00% ASOrg=100.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/  9 Total=         8 Country=100.00% Coords=0.00% ASN=62.50% ASOrg=62.50% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 10 Total=        35 Country=100.00% Coords=0.00% ASN=71.43% ASOrg=71.43% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 11 Total=        85 Country=100.00% Coords=0.00% ASN=78.82% ASOrg=78.82% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 12 Total=       278 Country=100.00% Coords=0.00% ASN=83.45% ASOrg=83.45% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 13 Total=       569 Country=100.00% Coords=0.00% ASN=87.17% ASOrg=87.17% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 14 Total=      1327 Country=100.00% Coords=0.00% ASN=84.78% ASOrg=84.78% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 15 Total=      2760 Country=100.00% Coords=0.00% ASN=82.14% ASOrg=82.14% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 16 Total=     10167 Country=99.95% Coords=0.00% ASN=80.14% ASOrg=80.14% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 17 Total=      7563 Country=100.00% Coords=0.00% ASN=83.49% ASOrg=83.49% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 18 Total=     13354 Country=99.97% Coords=0.00% ASN=83.91% ASOrg=83.91% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 19 Total=     25354 Country=99.98% Coords=0.00% ASN=84.98% ASOrg=84.98% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 20 Total=     39116 Country=99.96% Coords=0.00% ASN=84.53% ASOrg=84.53% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 21 Total=     54874 Country=99.95% Coords=0.00% ASN=82.60% ASOrg=82.60% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 22 Total=    117368 Country=99.94% Coords=0.00% ASN=83.46% ASOrg=83.46% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 23 Total=    124541 Country=99.94% Coords=0.00% ASN=79.46% ASOrg=79.46% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 24 Total=    240113 Country=99.92% Coords=0.00% ASN=81.46% ASOrg=81.46% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 25 Total=     10581 Country=99.94% Coords=0.00% ASN=95.53% ASOrg=95.53% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 26 Total=     15500 Country=99.95% Coords=0.00% ASN=96.35% ASOrg=96.35% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 27 Total=     22059 Country=99.91% Coords=0.00% ASN=96.78% ASOrg=96.78% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 28 Total=     30393 Country=99.91% Coords=0.00% ASN=98.38% ASOrg=98.38% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 29 Total=     34128 Country=99.89% Coords=0.00% ASN=98.76% ASOrg=98.76% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 30 Total=     24836 Country=99.81% Coords=0.00% ASN=99.44% ASOrg=99.44% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 31 Total=      7847 Country=99.86% Coords=0.00% ASN=98.89% ASOrg=98.89% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 32 Total=     17959 Country=99.96% Coords=0.00% ASN=99.20% ASOrg=99.20% AC=0.00% SP=0.00% AP=0.00%

This check queries the following fields:

  • Country: country.iso_code
  • Coords: location.latitude, location.longitude
  • ASN: autonomous_system_number
  • ASOrg: autonomous_system_organization
  • AC: is_anycast
  • SP: is_satellite_provider
  • AP: is_anonymous_proxy

Step 5: Query your MMDBs

$ ./cmd/mmdbquery/mmdbquery output/geoip-v4.mmdb 1.1.1.1

1.1.1.0/24:
  autonomous_system_number: 13335
  autonomous_system_organization: CLOUDFLARENET
  country:
    iso_code: AU
Customize your MMDBs.

Take a look at the config-example.yml to get an idea how to customize your MMDB.

There also several options to optimize your MMDB for smaller sizes (and reduced accuracy).

It is highly recommended you choose geoip data sources suitable for your use case, in terms of size and quality. Be sure to also check the licenses of the data sources before using them.

If you add more fields, it is a good idea to stick to the keys that MaxMind already uses. This keeps your MMDB compatible with existing systems. You can find the keys they use here: https://pkg.go.dev/github.com/oschwald/geoip2-golang#City

Supported Sources

Start by defining the fields and their data type of the output mmdb:

databases:
  - name: "My IPv4 GeoIP DB"
    types:
      "country.iso_code": string
      "location.latitude": float32
      "location.longitude": float32
      "autonomous_system_organization": string
      "autonomous_system_number": uint32
      "is_anycast": bool
      "is_satellite_provider": bool
      "is_anonymous_proxy": bool

There are three special fields which are not defined in the types:

  • from: The start address of an IP range.
  • to: The end address of an IP range.
  • net: An IP range in CIDR notation.

These are used to derive the IP ranges the data (row, entry) is applicable for.

CSV

Define columns with fields, which must match a field defined in the types:

databases:
  ...
    fields: ["from", "to", "autonomous_system_number", "autonomous_system_organization"]

All rows must have exactly the specified amount of columns. Use - to define a column you are not using, eg.:

databases:
  ...
    fields: ["from", "to", "country.iso_code", "-", "-", "-", "-", "location.latitude", "location.longitude", "-"]
IPFire

The IPFire Firewall maintains a geoip database in a custom format, which notably includes IP categorization, such as is-anycast.

Define fields with fieldMap, mapping IPFire database keys to types:

databases:
  ...
    fieldMap:
      "aut-num": "autonomous_system_number"
      "name": "autonomous_system_organization"
      "country": "country.iso_code"
      "is-anycast": "is_anycast"
      "is-satellite-provider": "is_satellite_provider"
      "is-anonymous-proxy": "is_anonymous_proxy"
Defaults

If you are building more than one mmdb file, you can use defaults to apply certain configuration to all databases. A common case is when you build an IPv4 and IPv6 database separately.

When using defaults, always check if they are applied correctly by checking the logs when building the databases.

defaults:
  types: # Entries are merged.
    # Databases inherit defaults types, if not yet set.
    # You can define a type with a "-" type string to ignore a default in a database config.
    "country.iso_code": string
    "location.latitude": float32
    "location.longitude": float32
    "autonomous_system_organization": string
    "autonomous_system_number": uint32
    "is_anycast": bool
    "is_satellite_provider": bool
    "is_anonymous_proxy": bool
  optimize: # Entries are used as default separately.
    floatDecimals: 2 # Default is used when database value is 0.
    forceIPVersion: true # Default is used when database value is not defined.
    maxPrefix: 24 # Default is used when database value is 0.
  merge: # Entries are used as default separately.
    conditionalResets: # Default is used when not defined or empty in database config.
    - ifChanged: ["country"]
        reset: ["location"]

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ConditionalResetTopLevelMergeWith added in v0.2.0 

func ConditionalResetTopLevelMergeWith(newValue mmdbtype.DataType, cfg []ConditionalResetConfig) inserter.Func

ConditionalResetTopLevelMergeWith is based on TopLevelMergeWith, but conditionally resets fields as defined in the conditional reset config. Both the new and existing value must be a Map. An error will be returned otherwise.

func WriteMMDB 

func WriteMMDB(dbConfig DatabaseConfig, sources []Source, updates chan string) error

WriteMMDB writes a mmdb file using given config and sources. Supply an updates channel to receive update messages about the progress.

Types

type CSVSource 

type CSVSource struct {
	// contains filtered or unexported fields
}

CSVSource reads geoip data in csv format.

func LoadCSVSource 

func LoadCSVSource(input DatabaseInput, types map[string]string) (*CSVSource, error)

LoadCSVSource returns a new CSVSource.

func (*CSVSource) Err 

func (csv *CSVSource) Err() error

Err returns the processing error encountered by the source.

func (*CSVSource) Name 

func (csv *CSVSource) Name() string

Name returns an identifying name for the source.

func (*CSVSource) NextEntry 

func (csv *CSVSource) NextEntry() (*SourceEntry, error)

NextEntry returns the next entry of the source. If nil, nil is returned, stop reading and check Err().

type ConditionalResetConfig added in v0.2.0 

type ConditionalResetConfig struct {
	IfChanged []string `yaml:"ifChanged"`
	Reset     []string `yaml:"reset"`
}

ConditionalResetConfig defines a conditional reset merge config.

type Config 

type Config struct {
	Databases []DatabaseConfig `yaml:"databases"`
	Defaults  DefaultConfig    `yaml:"defaults"`
}

Config is the geoip build config.

func LoadConfig 

func LoadConfig(filePath string) (*Config, error)

LoadConfig loads a configuration file.

type DatabaseConfig 

type DatabaseConfig struct {
	Name     string            `yaml:"name"`
	MMDB     MMDBConfig        `yaml:"mmdb"`
	Types    map[string]string `yaml:"types"`
	Inputs   []DatabaseInput   `yaml:"inputs"`
	Output   string            `yaml:"output"`
	Optimize Optimizations     `yaml:"optimize"`
	Merge    MergeConfig       `yaml:"merge"`
}

DatabaseConfig holds the config for building one database.

type DatabaseInput 

type DatabaseInput struct {
	File     string            `yaml:"file"`
	Fields   []string          `yaml:"fields"`
	FieldMap map[string]string `yaml:"fieldMap"`
}

DatabaseInput holds database input config.

type DefaultConfig added in v0.3.0 

type DefaultConfig struct {
	Types    map[string]string `yaml:"types"`
	Optimize Optimizations     `yaml:"optimize"`
	Merge    MergeConfig       `yaml:"merge"`
}

DefaultConfig holds a subset of DatabaseConfig fields to be used as a default config.

func (DefaultConfig) ApplyTo added in v0.3.0 

func (d DefaultConfig) ApplyTo(c *DatabaseConfig)

ApplyTo applies the default config to the given database config.

type IPFireSource 

type IPFireSource struct {
	// contains filtered or unexported fields
}

IPFireSource reads geoip data in the ipfire format.

func LoadIPFireSource 

func LoadIPFireSource(input DatabaseInput, types map[string]string) (*IPFireSource, error)

LoadIPFireSource returns a new IPFireSource.

func (*IPFireSource) Err 

func (ipf *IPFireSource) Err() error

Err returns the processing error encountered by the source.

func (*IPFireSource) Name 

func (ipf *IPFireSource) Name() string

Name returns an identifying name for the source.

func (*IPFireSource) NextEntry 

func (ipf *IPFireSource) NextEntry() (*SourceEntry, error)

NextEntry returns the next entry of the source. If nil, nil is returned, stop reading and check Err().

func (*IPFireSource) SourceEntryFromMimeHeader 

func (ipf *IPFireSource) SourceEntryFromMimeHeader(data textproto.MIMEHeader) (*SourceEntry, error)

SourceEntryFromMimeHeader parses an IPFire entry. The keys in the IPFire file are: - net - aut-num - name - country - is-anycast - is-satellite-provider - is-anonymous-proxy - drop .

type MMDBConfig 

type MMDBConfig struct {
	IPVersion  int `yaml:"ipVersion"`
	RecordSize int `yaml:"recordSize"`
}

MMDBConfig holds mmdb specific config.

type MergeConfig added in v0.2.0 

type MergeConfig struct {
	ConditionalResets []ConditionalResetConfig `yaml:"conditionalResets"`
}

MergeConfig holds merge configuration.

type Optimizations 

type Optimizations struct {
	FloatDecimals  int   `yaml:"floatDecimals"`
	ForceIPVersion *bool `yaml:"forceIPVersion"`
	MaxPrefix      int   `yaml:"maxPrefix"`
}

Optimizations holds optimization config.

func (Optimizations) ForceIPVersionEnabled added in v0.3.0 

func (o Optimizations) ForceIPVersionEnabled() bool

ForceIPVersionEnabled reports whether ForceIPVersion is set and true.

type Source 

type Source interface {
	Name() string
	NextEntry() (*SourceEntry, error)
	Err() error
}

Source describes a generic geoip data source.

func LoadSources 

func LoadSources(dbConfig DatabaseConfig) ([]Source, error)

LoadSources loads the given input files from the database config.

type SourceEntry 

type SourceEntry struct {
	Net    *net.IPNet
	From   net.IP
	To     net.IP
	Values map[string]SourceValue
}

SourceEntry describes a geoip data source entry. Either Net or both From and To must be set.

func (SourceEntry) ToMMDBMap 

func (se SourceEntry) ToMMDBMap(optim Optimizations) (mmdbtype.Map, error)

ToMMDBMap transforms the source entry to a mmdb map type.

type SourceValue 

type SourceValue struct {
	Type  string
	Value string
}

SourceValue holds an unprocessed source data value, including its type.

func (SourceValue) ToMMDBType 

func (sv SourceValue) ToMMDBType(optim Optimizations) (mmdbtype.DataType, error)

ToMMDBType transforms the source value to the correct mmdb type.

Source Files

View all Source files

Directories

Path Synopsis
cmd
mmdbcheck
mmdbmeld
mmdbquery

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.