logquery

LogQuery

LogQuery is a tool for reading logfmt formatted data, then running SQL queries on it using SQLite.

Demo:

$ cat sample.txt
event=error code=123 msg="something bad happened" timestamp="1970-01-01 15:23"
event=error code=123 msg="something bad happened" timestamp="1970-01-01 17:35"
event=error code=456 msg="something went wrong" timestamp="1970-01-01 18:56"
$ logquery "SELECT * FROM log WHERE code=123" < sample.txt
timestamp="1970-01-01 15:23" code=123 event="error" msg="something bad happened"
timestamp="1970-01-01 17:35" code=123 event="error" msg="something bad happened"
$ logquery -o csv "SELECT * FROM log WHERE code=123" < sample.txt
event,msg,timestamp,code
error,something bad happened,1970-01-01 15:23,123
error,something bad happened,1970-01-01 17:35,123

Installation

make install

Or

go get -u git.sr.ht/~charles/logquery

Usage

Usage: logquery <query> [<path>]

Arguments:
  <query>     SQLite query to run on the input log.
  [<path>]    Input path to read from instead of standard in.

Flags:
  -h, --help                      Show context-sensitive help.
  -f, --output-format="logfmt"    Specify the output format for results. Must be one of: logfmt, csv.
  -o, --output="-"                Specify the output file to use instead of standard out.
      --cpuprofile="-"            Run the program with CPU profiling and write the results to this file.
      --memprofile="-"            Run the program with memory profiling and write the results to this file.
      --version

LogFmt Parser

LogQuery also implements its own LogFmt parser via the parser module.

Performance

The initial version of LogQuery was written in Python, and achieved a performance of roughly 11k records/second. It has since been rewritten Go, and now runs at roughly 70k records/second. Both benchmarks were run on a Thinkpad T430 with an i7-3632QM, 16GB DDR3 memory, and an SSD with ZFS (with encryption and compression enabled).

Limitations

• LogQuery currently works by reading the entire input stream into an in-memory database. This means it can take a while for large input files.

Future Work

• Support more output formats
• Support CSV input