README
¶
Lumberjack
Lumberjack is not an official Google product.
Lumberjack helps your applications on Google Cloud to write audit logs similar to Cloud Audit Logs and provides building blocks to centrally collect the audit logs for analysis.
Use Case
Audit logs are special logs that record when and who called which application and accessed what data. And why the access was necessary.
The typical use case of Lumberjack is for an organization's insider risk (or insider threat) program. The organization will require its applications to write audit logs whenever there are employees calling the applications to access user data.
Architecture
Data access could happen on two routes:
- Via applications that connect to data storage. We can audit logs from such accesses application-level audit logs. Lumberjack provides Go and Java SDK to help write application-level audit logs via cloud logging. Find out more about how to use the SDK. It also provides a CLI tool for log validation, which can be used as a debugging tool or if you write logs through other means but want to use the same logging standards.
- Directly via data storage API. Google Cloud provides Cloud Audit Logs to capture such accesses. We also call these logs platform-level audit logs to differentiate from application-level audit logs.
For application-level audit logs, Lumberjack reuses the Google Cloud's AuditLog format in LogEntry's JSON payload.
Both application-level and platform-level audit logs should be routed via log sinks to a central audit log storage. We recommend using BigQuery as the central audit log storage. See the log sink filters.
Audit Logging in gRPC
Lumberjack SDK provides near-automated audit logging for gRPC services via gRPC interceptor. Find out more here.
Ingestion Service
By default, we recommend having Lumberjack SDK to write application-level audit logs to cloud logging. In case you prefer having a central log ingestion service, Lumberjack also provides a sample log ingestion service. Lumberjack SDK can be configured to write application-level audit logs to the ingestion service via a gRPC API instead. A central ingestion service could come in handy when you need to add common audit log processing logic.
Other Kinds of Audit Logging
The audit logging described above is the typical user data access audit logging. Lumberjack could also help with other kinds of audit logging:
- Consent Events: Audit logging user consent events (agreement and revocation). The request principal in such logs should be the user who's approving/revoking the consent agreement.
- System Events: Audit logging system produced events, e.g. GitHub workflow requesting GitHub token with elevated permissions. The request principal in such logs should be the system identity making the request.
Installation
Via abc CLI
You can leverage abc CLI to set up the basic infrastructure needed for this service. You will need to install abc CLI by following docs here.
You can customize to create the log sinks either in GCP projects, folders or organization. Below is the example of create the log sinks in GCP projects.
abc templates render \
-input='project_id=<YOUR_PROJECT_ID>' \
-input='lumberjack_release_version=<YOUR_LUMBERJACK_RELEASE_VERSION>' \
-input='log_sink_project_ids=[<YOUR_LOG_SINK_PROJECT_ID>,......]' \
-input='application_audit_logs_filter_file=<YOUR_APPLICATION_AUDIT_LOGS_FILTER_FILE>' \
-input='cloud_audit_logs_filter_file=<YOUR_CLOUD_AUDIT_LOGS_FILTER_FILE>' \
-input='terraform_state_bucket=<YOUR_TERRAFORM_STATE_BUCKET>' \
-input='terraform_state_prefix=<YOUR_TERRAFORM_STATE_PREFIX>' \
github.com/abcxyz/lumberjack.git//templates/lumberjack-e2e
Directories
¶
| Path | Synopsis |
|---|---|
|
clients
|
|
|
go/apis/v1alpha1
Package v1alpha1 contains versioned Lumberjack contracts, e.g.
|
Package v1alpha1 contains versioned Lumberjack contracts, e.g. |
|
go/pkg/audit
Package audit provides functionality to validate and emit application audit logs.
|
Package audit provides functionality to validate and emit application audit logs. |
|
go/pkg/auditerrors
Package auditerrors defines the sentinel errors for the project.
|
Package auditerrors defines the sentinel errors for the project. |
|
go/pkg/auditopt
Package auditopt configures a client by looking three locations to determine the config variables:
|
Package auditopt configures a client by looking three locations to determine the config variables: |
|
go/pkg/filtering
Package filtering provides a processor to filter audit log requests.
|
Package filtering provides a processor to filter audit log requests. |
|
go/pkg/justification
Package justification provides utils to validate justification tokens produced by JVS and populate audit logs with justification.
|
Package justification provides utils to validate justification tokens produced by JVS and populate audit logs with justification. |
|
go/pkg/remote
Package remote defines a remote audit log processor.
|
Package remote defines a remote audit log processor. |
|
go/pkg/security
Package security describes the authentication technology that the middleware investigates to autofill the principal in a log request.
|
Package security describes the authentication technology that the middleware investigates to autofill the principal in a log request. |
|
go/pkg/testutil
Package testutil provides utilities that are intended to enable easier and more concise writing of unit test code.
|
Package testutil provides utilities that are intended to enable easier and more concise writing of unit test code. |
|
go/pkg/trace
Package trace manages open telemetry trace exporter.
|
Package trace manages open telemetry trace exporter. |
|
go/test/shell
Package main contains a minimal Cloud Run HTTP server that emits an application audit log using the audit client.
|
Package main contains a minimal Cloud Run HTTP server that emits an application audit log using the audit client. |
|
go
Module
|
|
|
cmd
|
|
|
integration
|
|
|
internal
|
|
|
pkg
|
|
|
cli
Package cli implements the commands for the Lumberjack CLI.
|
Package cli implements the commands for the Lumberjack CLI. |
|
cloudlogging
Package cloudlogging pulls lumberjack/data access logs from GCP cloud logging.
|
Package cloudlogging pulls lumberjack/data access logs from GCP cloud logging. |
|
server
Package server implements the gRPC server of the audit log agent.
|
Package server implements the gRPC server of the audit log agent. |
|
validation
Package validation provides utils for lumberjack/data access logs validation.
|
Package validation provides utils for lumberjack/data access logs validation. |