log-parser

kata-log-parser

Introduction

kata-log-parser is a tool that combines logfiles generated by the various system components, sorts them by timestamp, and re-displays the log entries. A time delta is added to show how much time has elapsed between each log entry.

The tool is also able to check the validity of all log records, can re-format the logs, and output them in a different format.

For more information on the kata-log-parser tool, use the help command:

$ kata-log-parser --help

Logfile requirements

The tool reads logfiles in the logfmt structured logging format. For example, a logfile created by the golang Logrus package.

By default the tool requires that the following fields are defined for each log record:

• Log level field (level): must be one of the Logrus LogLevel values in string format (e.g. debug, info, error).

• Name field (name): a single word that specifies the name of the application that generates the log record (e.g. kata-runtime).

• Process ID field (pid): the numeric process identifier for the process that generates the log record.

• Source field (source): a single word that specifies the name of a unique part of the system (e.g. runtime).

• Timestamp field (time): in RFC3339 format and including a nanosecond value.

Additional to the fields above, the tool also expects the following field:

• Message field (msg): a textual message allowing log records to be disambiguated.

Note: These requirements can be ignored by using the --ignore-missing-fields flag

Component logfiles

The primary logfiles the tool reads are:

• The runtime log.

  This log also includes virtcontainers log entries and agent best effort logs unpacking (unless --no-agent-unpack is specified).

Usage

To merge all logs:

1. Enable full debug.
2. Clear the systemd journal (optional):

   $ sudo systemctl stop systemd-journald
   $ sudo rm -f /var/log/journal/*/* /run/log/journal/*/*
   $ sudo systemctl start systemd-journald   
   

3. Create a Kata container.
4. Collect the logs (alternatively to journal clearing you may consider constraining collected logs by adding --since=<container creation time>).

   $ sudo journalctl -q -o cat -a -t kata > ./kata.log
   

5. Ensure the logs are readable:

   $ sudo chown $USER *.log
   

6. To install the program:

   $ go get -d github.com/kata-containers/kata-containers
   $ pushd $GOPATH/src/github.com/kata-containers/kata-containers/src/tools/log-parser && make install && popd
   

7. To run the program:

   $ kata-log-parser kata.log
   

Advanced processing using jq

jq is a command-line JSON processor which can be combined with kata-log-parser to filter and fetch specific log entries.

Examples

$ kata-log-parser --ignore-missing-fields --output-format json --no-agent-unpack kata.log | jq '.Entries[] | select(.Msg=="reading guest console") | .Data.vmconsole'

This example also demonstrates how to get logs from the journal directly to the parser.

$ journalctl -q -o cat -a -t kata | kata-log-parser --ignore-missing-fields --output-format json - | jq '.Entries[] | select(.Source=="agent")'

These logs sourced from containerd-kata-shim-v2 and being printed along with their Msg content, Time and Container ID.

$ kata-log-parser --ignore-missing-fields --output-format json kata.log  | jq '.Entries[] | select(.Source=="containerd-kata-shim-v2" and .Sandbox=="2fa50251ccc3b9a85350e8fe6836d1875023714153b503b548360946fcec3829") | "\(.Msg) \(.Time) \(.Container)"'