lfgw

Label Filter Gateway (lfgw)

lfgw is a simple reverse proxy aimed at PromQL / MetricsQL metrics filtering based on OIDC roles.

It relies on VictoriaMetrics/metricsql for label filters manipulation in metric expressions (according to an ACL) before a request is proxied to Prometheus/VictoriaMetrics by httputil. The process is described in more details here.

Target setup: grafana (with OIDC integration) <-> lfgw <-> Prometheus/VictoriaMetrics.

Key features

• ACL-based request rewrites with implicit deny;
• a user can have multiple roles;
• support for autoconfiguration in environments, where OIDC-role names match names of namespaces ("assumed roles" mode; thanks to @aberestyak for the idea);
• automatic expression optimizations for non-full access requests;
• support for different headers with access tokens (Authorization, X-Forwarded-Access-Token, X-Auth-Request-Access-Token), which can be useful for tools like oauth2-proxy;
• requests to both /api/* and /federate endpoints are protected (=rewritten);
• requests to sensitive endpoints are blocked by default;
• compatible with both PromQL and MetricsQL.

Similar projects

Similar projects are described here.

Docker images

Docker images are published on ghcr.io/weisdd/lfgw.

Configuration

Example of keycloak + grafana + lfgw setup is described here.

Requirements for jwt-tokens

• OIDC-roles must be present in roles claim;
• Client ID specified via OIDC_CLIENT_ID must be present in aud claim (more details in environment variables section), otherwise token verification will fail.

Environment variables

Basic settings

(1*): since it's grafana who obtains jwt-tokens in the first place, the specified client id must also be present in the forwarded token (the aud claim).

Additional settings

ACL syntax

The file with ACL definitions (./acl.yaml by default) has a simple structure:

role: namespace, namespace2

For example:

team0: .*                # all metrics
team1: min.*, .*, stolon # all metrics, it's the same as .*
team2: minio             # only those with namespace="minio"
team3: min.*             # only those matching namespace=~"min.*"
team4: minio, stolon     # only those matching namespace=~"minio|stolon"
team5: min.*, stolon     # only those matching namespace=~"min.*|stolon"

To summarize, here are the key principles used for rewriting requests:

• .* - all requests are simply forwarded to an upstream;
• minio - all label filters with the namespace label are removed, then namespace="minio" is added;
• min.* - positive regex-match label filters (namespace=~"X") are removed, then namespace=~"mi.*" is added;
• minio, stolon - positive regex-match label filters (namespace=~"X") are removed, then namespace=~"minio|stolon" is added;
• min.*, stolon - positive regex-match label filters (namespace=~"X") are removed, then namespace=~"min.*|stolon" is added.

When deduplication is enabled, these queries will stay unmodified:

• min.*, stolon, query: request_duration{namespace="minio"} - a non-regexp label filter that matches policy;
• min.*, stolon, query: request_duration{namespace=~"minio"} - a "fake" regexp (no special symbols) label filter that matches policy;
• min.*, stolon, query: request_duration{namespace=~"min.*"} - a label filter is a subfilter of the policy.

Note: Regex matches are fully anchored. A match of env=~"foo" is treated as env=~"^foo$" (Source). Please, be careful, they are not expected to be used in ACLs.

Note: a user is free to have multiple roles matching the contents of acl.yaml. Basically, there are 3 cases:

• one role => a prepopulated LF is returned;
• multiple roles, one of which gives full access => a prepopulated LF, corresponding to the full access role, is returned;
• multiple "limited" roles => definitions of all those roles are merged together, and then lfgw generates a new LF. The process is the same as if this meta-definition was loaded through acl.yaml.

Licensing

lfgw code is licensed under MIT, though its dependencies might have other licenses. Please, inspect the modules listed in go.mod if needed.