README
¶
Logrange - streaming database
Logrange is highly performant streaming database for aggregating streams of records like application logs, system metrics, audit logs etc. from thousands of sources. Logrange provides an API to do operations from simple search to the data analysis and machine learning.
The product
- Size tolerant. Logrange is able to store as much data as written there. The only limitation is the storage space available for the database.
- Highly performant. Writing and reading millions of records per second (hundered megabytes or gigabytes of data).
- Write optimized. Logrange persists the raw data in realtime. All other data processing like indexing can be done later.
- Low latency. Data becomes available for read within milliseconds after it is written
- Scalable. Supporting tens of thousands different streams of records (terrabytes of the data)
- Highly available. In clustering solution, data could be replicated between logrange nodes. Logrange will support data and load distribution policies.
- Native for stream processing. Merging, filtering and search using LQL (Logrange Query Language)
- Open Source. Logrange is 100% open source. It can be used for building trustworthy data storages.
- Ready to use. Basic installation includes pre-configured log processing tools: collector, forwarder, CLI tool and Logrange database service.
- Easy installation either in a containerized (k8s or docker) or a custom environment.
Quick start
Logrange shipment includes 2 executables - logrange server and lr - the logrange client. In the quick start you can use precompiled binaries to try logrange out within 1 minute:
Step 1. Let's put everything into one dir
Make a directory and enter there:
mkdir lrquick
cd lrquick
Step 2. Install logrange server and run it
curl -s http://get.logrange.io/install | bash -s logrange -d ./bin
./bin/logrange start --base-dir=./data --daemon
Normally, you have to see something like Started. pid=12345
Step 3. Install logrange client and start collecting logs from the machine
curl -s http://get.logrange.io/install | bash -s lr -d ./bin
./bin/lr collect --storage-dir=./collector --daemon
The command above runs collector in background. It will send logs found in /var/log folder to the logrange server started in step 2.
Step 4. Connect to the server, using CLI tool.
./bin/lr shell
...
In the logrange shell, you can try select to retrieve collected data:
> select limit 10
Or try help command to find out what commands are available.
Quick stop
From the logrange folder (lrquick) type the following commands to stop collector and the logrange server:
./bin/lr stop-collect --storage-dir=./collector
./bin/logrange stop --base-dir=./data
Now, to clean up, just remove the lrquick folder:
cd ..
rm -rf ./lrquick/
Documentation
Getting Help
- Found a bug or thinking about a new feature? File an issue
- If you have any question or feedback regarding Logrange, send us e-mail to: mail@logrange.io
Contact us
Whether you have problems with log aggregating, processing or analysis, or wanting to build a secure data aggregating solution.
Reach out to mail@logrange.io
Join our gitter chat
License
Apache License 2.0, see LICENSE.
Acknowledgments
- GoLand IDE by JetBrains is used for the code development
Documentation
¶
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package api contains structures and data-type definitions that could be used for accessing the Logrange database using the api.Client interface.
|
Package api contains structures and data-type definitions that could be used for accessing the Logrange database using the api.Client interface. |
|
examples
|
|
|
pkg
|
|
|
backend
Package backend contains some code for serving API endpoints (refer to the logrange/api package)
|
Package backend contains some code for serving API endpoints (refer to the logrange/api package) |
|
lql
lql package contains parser's functions and structures for working with Logrange Query Language (LQL) The LQL supports the following constructions so far: SELECT [<format string>] [FROM ({<tags>}|<tags expression)] [RANGE <time range>] [WHERE <fields expression>] [POSITION (head|tail|<specific pos>)] [OFFSET <number>][LIMIT <number>] SHOW PARTITIONS [({<tags>}|<tags expression)][OFFSET <number>][LIMIT <number>] SHOW PIPES [OFFSET <number>][LIMIT <number>] DESCRIBE PARTITION {<tags>} DESCRIBE PIPE <pipe name> TRUNCATE [({<tags>}|<tags expression)][MINSIZE <size>][MAXSIZE <size>][BEFORE <timestamp>][MAXDBSIZE <size>] CREATE PIPE <pipe name> [FROM ({<tags>}|<tags expression)] [WHERE <fields expression>] DELETE PIPE <pipe name> This file is copied from github.com/alecthomas/participle/lexer and modified to provide choosing longest match regexp functionality
|
lql package contains parser's functions and structures for working with Logrange Query Language (LQL) The LQL supports the following constructions so far: SELECT [<format string>] [FROM ({<tags>}|<tags expression)] [RANGE <time range>] [WHERE <fields expression>] [POSITION (head|tail|<specific pos>)] [OFFSET <number>][LIMIT <number>] SHOW PARTITIONS [({<tags>}|<tags expression)][OFFSET <number>][LIMIT <number>] SHOW PIPES [OFFSET <number>][LIMIT <number>] DESCRIBE PARTITION {<tags>} DESCRIBE PIPE <pipe name> TRUNCATE [({<tags>}|<tags expression)][MINSIZE <size>][MAXSIZE <size>][BEFORE <timestamp>][MAXDBSIZE <size>] CREATE PIPE <pipe name> [FROM ({<tags>}|<tags expression)] [WHERE <fields expression>] DELETE PIPE <pipe name> This file is copied from github.com/alecthomas/participle/lexer and modified to provide choosing longest match regexp functionality |
|
model/tag
Package tag provides functions for working with tags.
|
Package tag provides functions for working with tags. |
|
utils/kvstring
kvstring package contains functions for representing string key:value pairs as one string and converting such strings to maps etc.
|
kvstring package contains functions for representing string key:value pairs as one string and converting such strings to maps etc. |
Click to show internal directories.
Click to hide internal directories.