monitor

monitor

Monitor the compliance of an asset to a configuration rule

1 Compile and test the microservice locally

1.1 Prepare local test

• Deploy the infrastructure prerequisites using the proposed Terraform module
• create-download a key JSON format for the created service account for this microservice
• set environment variables

export MONITOR_ENVIRONMENT="dev"
export MONITOR_PROJECT_ID="<your_project_id>"
export MONITOR_COMPLIANCE_STATUS_TOPIC_ID="ram-complianceStatus"
export MONITOR_VIOLATION_TOPIC_ID="ram-violation"
export MONITOR_START_PROFILER=true
export K_SERVICE="localgoapp"
export K_REVISION=${K_SERVICE}-$(git rev-parse HEAD)
export GOOGLE_APPLICATION_CREDENTIALS="<path_to_monitor_service_account_json_key_file>"

• Craft an HTTP POST request implementing CloudEvents format, using a tool like POSTMAN. Example data at the end of this page

Test example for POSTMAN

pm.test("Status code is 200", function () {
    pm.response.to.have.status(200);
});

1.2 Run the app locally and test it

Run monitor locally:

go run cmd/main.go

1.3 Check results

• When launching monitor locally it should log without errors the coldstart and init done log entries, like:

go run main.go 
{"microservice_name":"monitor","severity":"NOTICE","message":"coldstart","init_id":"44ce298c-fb19-4099-a834-c5ff0eb99fd8"}
{"microservice_name":"monitor","severity":"NOTICE","message":"init done","init_id":"44ce298c-fb19-4099-a834-c5ff0eb99fd8"}
Serving function...

• When posting the crafted HTTP request on localhost:8080/ it should
  • respond with 200 HTTP status code and empty body
  • log entries for one compliance status and zero to many violations depending on tested asset configuration and rule code.
  • publish the compliance status, and violation(s) to their respective Pubsub topics

To see published Pubsub messages subscribe to the two topics. Example:

gcloud pubsub subscriptions create test_compliance_status --topic=ram-complianceStatus --project=${MONITOR_PROJECT_ID}
gcloud pubsub subscriptions create test_violation --topic=ram-violation --project=${MONITOR_PROJECT_ID}
gcloud pubsub subscriptions pull test_compliance_status --auto-ack --project=${MONITOR_PROJECT_ID}
gcloud pubsub subscriptions pull test_violation --auto-ack --project=${MONITOR_PROJECT_ID}

2 build the microservice container image and test it locally

2.1 Build the container image locally and prepare to test

cd <repoRoot>
pack build --builder gcr.io/buildpacks/builder:v1 --env GOOGLE_FUNCTION_SIGNATURE_TYPE=cloudevent --env GOOGLE_FUNCTION_TARGET=EntryPoint monitor
docker images

Duplicate the previous test, and use a different port number in the URL, e.g. 8081

2.2 Run the container image locally and test it

Reference doc: Test a Cloud Run service locally using docker with GCP access

Make the key file readable by docker run process

chmod 644 <path to your service account key file>

PORT=8080 && docker run \
    -p 8081:${PORT} \
    -e PORT=${PORT} \
    -e K_SERVICE=dockerrun \
    -e K_CONFIGURATION=dockerrun \
    -e K_REVISION=dockerrun-$(git rev-parse HEAD) \
    -e MONITOR_ENVIRONMENT="dev" \
    -e MONITOR_PROJECT_ID=${MONITOR_PROJECT_ID} \
    -e MONITOR_COMPLIANCE_STATUS_TOPIC_ID=${MONITOR_COMPLIANCE_STATUS_TOPIC_ID} \
    -e MONITOR_VIOLATION_TOPIC_ID=${MONITOR_VIOLATION_TOPIC_ID} \
    -e MONITOR_START_PROFILER=true \
    -e GOOGLE_APPLICATION_CREDENTIALS=/tmp/keys/key.json \
    -v $GOOGLE_APPLICATION_CREDENTIALS:/tmp/keys/key.json:ro \
    monitor

Run test as previously for the local app, using the mapped port number.

2.3 Check results

Same as for the local app test.

3 Build the microservice container image remotely

3.1 Using Cloud Build CLI tool

cd ram/monitor
gcloud alpha builds submit . \
    --project=<your_build_project> \
    --pack=image=<your_container_iamge_repo_path>,builder=gcr.io/buildpacks/builder:v1,env=GOOGLE_FUNCTION_SIGNATURE_TYPE=cloudevent,env=GOOGLE_FUNCTION_TARGET=EntryPoint

3.2 Using GitLag CI

• Gitlab / ram group / settings / CICD / set up the following variables:
  • DEV_PROJECT_ID
  • DEV_SA_KEY_BUILD
  • QA_PROJECT_ID
  • SA_DEPLOY_QA_JSON_KEY

Sample data to craft a test request

Header

ce-source: //pubsub.googleapis.com/assetRule
ce-id: asset_rule_a
ce-specversion: 1.0
ce-type: google.cloud.pubsub.topic.v1.messagePublished
ce-time: 2021-07-27T10:47:00Z
Content-Type: application/json; charset=utf-8

Body

{
    "message": {
        "data": "<your_base64_encoded_data>",
        "messageId": "asset_rule_a",
        "publishTime": "2021-07-27T10:47:00.000000000Z"
    },
    "subscription": "projects/MY_PROJECT/subscriptions/MY_SUBSCRIPTION"
}

Data to be base64 encoded, example

{
    "rule": {
        "name": "monitor_bq_dataset_location",
        "deploymentTime": "2021-03-23T09:32:53.949581856Z",
        "regoModules": {
            "audit.rego": "\n#\n# Copyright 2024 Google LLC\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#      http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n\npackage validator.gcp.lib\n\n# RULE: the audit rule is result if its body {} evaluates to true\n# The rule body {} can be understood intuitively as: expression-1 AND expression-2 AND ... AND expression-N\naudit[result] {\n\t# iterate over each asset\n\tasset := input.assets[_]\n\ttrace(sprintf(\"asset name: %v\", [asset.name]))\n\n\t# assign the constrains in a variable\n\tconstraints := input.constraints\n\t# iterate over each constraint\n\tconstraint := constraints[_]\n\ttrace(sprintf(\"constraint kind: %v\", [constraint.kind]))    \n\n\t# use a custom function to retreive constraint.spec, if not defined returns a default value that is an empty object\n\tspec := _get_default(constraint, \"spec\", {})\n\t# use a custom function to retreive constraint.spec.match, if not defined returns a default value that is an empty objecy\n\tmatch := _get_default(spec, \"match\", {})\n\t# use a custom function to retreive constraint.spec.match.target, if not defined returns a default value that is an array with one target object targetting any organization, and childs\n\ttarget := _get_default(match, \"target\", [\"organization/*\"])\n\t# use a custom function to retreive constraint.spec.match.gcp, if not defined returns a default value that is an empty object\n\tgcp := _get_default(match, \"gcp\", {})\n\t# use a custom function to retreive constraint.spec.match.gcp.target, if not defined returns what we already got in target variable\n\tgcp_target := _get_default(gcp, \"target\", target)\n\t# iterate over each target and use builtin regex to check if the asset ancestry path matches one of them\n\t# TRUE if the asset ancestry path matches (regex) one of the target (iterate targets)\n\t# FALSE when the ancestry path do not matches at least one of the targer\n\ttrace(sprintf(\"asset.ancestry_path: %v\", [asset.ancestry_path]))\n\ttrace(sprintf(\"targets: %v\", [gcp_target]))\n\ttrace(sprintf(\"is in scope:\",[re_match(gcp_target[_], asset.ancestry_path)]))\n\tre_match(gcp_target[_], asset.ancestry_path)\n\n\t# use a custom function to retreive constraint.spec.match.exclude, if not defined returns a default value that is an empty array\n\texclude := _get_default(match, \"exclude\", [])\n\t# use a custom function to retreive constraint.spec.match.gcp.exclude, if not defined returns what we already got in exlucde variable\n\tgcp_exclude := _get_default(gcp, \"exclude\", exclude)\n\t# iterate over the exclusion list (the pattern) and use regex builtin function to check if the asset ancestry path (the value) matches one of then\n\t# assign to exclusion_match variable the virtual document generated by a rule which is a set built by using a comprehension\n\t# This set containts the asset ancestry path if it maches one of the exclusion\n\t# or is empty (set()) if the asset ancestry path does not matched any of the exclusion\n\texclusion_match := {asset.ancestry_path | re_match(gcp_exclude[_], asset.ancestry_path)}\n\ttrace(sprintf(\"exclusions: %v\", [gcp_exclude]))\n\ttrace(sprintf(\"Excluded if count exclusion_match \u003e 0: %v\", [count(exclusion_match)]))\n\t# this expression evaluate to true when count is zero, aka the ancestry path does not matches any of the exclusion, otherwise evaluates to false\n\tcount(exclusion_match) == 0\n\n\t# Use a with statement to programatically call the rego rule that is specified in the YAML constraint file\n\tviolations := data.templates.gcp[constraint.kind].deny with input.asset as asset\n\t\t with input.constraint as constraint\n\n\t# Iterates through each violation found\n\tviolation := violations[_]\n\t# if the asset is in target, and not excluded and at least one violation is founds, then returns for each violation a result object with the 4 following fields:\n\tresult := {\n\t\t\"asset\": asset.name,\n\t\t\"constraint\": constraint.metadata.name,\n\t\t\"constraint_config\": constraint,\n\t\t\"violation\": violation,\n\t}\n}\n\n# has_field returns whether an object has a field\n_has_field(object, field) {\n\tobject[field]\n}\n\n# False is a tricky special case, as false responses would create an undefined document unless\n# they are explicitly tested for\n_has_field(object, field) {\n\tobject[field] == false\n}\n\n_has_field(object, field) = false {\n\tnot object[field]\n\tnot object[field] == false\n}\n\n# get_default returns the value of an object's field or the provided default value.\n# It avoids creating an undefined state when trying to access an object attribute that does\n# not exist\n_get_default(object, field, _default) = output {\n\t_has_field(object, field)\n\toutput = object[field]\n}\n\n_get_default(object, field, _default) = output {\n\t_has_field(object, field) == false\n\toutput = _default\n}\n",
            "constraints.rego": "\n#\n# Copyright 2024 Google LLC\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#      http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n\npackage validator.gcp.lib\n\n# Function to fetch the constraint spec\n# Usage:\n# get_constraint_params(constraint, params)\n\nget_constraint_params(constraint) = params {\n\tparams := constraint.spec.parameters\n}\n\n# Function to fetch constraint info\n# Usage:\n# get_constraint_info(constraint, info)\n\nget_constraint_info(constraint) = info {\n\tinfo := {\n\t\t\"name\": constraint.metadata.name,\n\t\t\"kind\": constraint.kind,\n\t}\n}\n",
            "monitor_bq_dataset_location.rego": "# Copyright 2024 Google LLC\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#      http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\npackage templates.gcp.GCPBigQueryDatasetLocationConstraintV1\n\nimport data.validator.gcp.lib as lib\n\n############################################\n# Find BigQuery Dataset Location Violations\n############################################\ndeny[{\n\t\"msg\": message,\n\t\"details\": metadata,\n}] {\n\tconstraint := input.constraint\n\tlib.get_constraint_params(constraint, params)\n\n\t# Verify that resource is BigQuery dataset\n\tasset := input.asset\n\tasset.asset_type == \"bigquery.googleapis.com/Dataset\"\n\n\t# Check if resource is in exempt list\n\texempt_list := params.exemptions\n\tmatches := {asset.name} \u0026 cast_set(exempt_list)\n\tcount(matches) == 0\n\n\t# Check that location is in allowlist/denylist\n\ttarget_locations := params.locations\n\tasset_location := asset.resource.data.location\n\tlocation_matches := {asset_location} \u0026 cast_set(target_locations)\n\ttarget_location_match_count(params.mode, desired_count)\n\tcount(location_matches) == desired_count\n\n\tmessage := sprintf(\"%v is in a disallowed location.\", [asset.name])\n\tmetadata := {\"location\": asset_location}\n}\n\n#################\n# Rule Utilities\n#################\n\n# Determine the overlap between locations under test and constraint\n# By default (allowlist), we violate if there isn't overlap\ntarget_location_match_count(mode) = 0 {\n\tmode != \"denylist\"\n}\n\ntarget_location_match_count(mode) = 1 {\n\tmode == \"denylist\"\n}\n",
            "util.rego": "\n#\n# Copyright 2018 Google LLC\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#      http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n\npackage validator.gcp.lib\n\n# has_field returns whether an object has a field\nhas_field(object, field) {\n\tobject[field]\n}\n\n# False is a tricky special case, as false responses would create an undefined document unless\n# they are explicitly tested for\nhas_field(object, field) {\n\tobject[field] == false\n}\n\nhas_field(object, field) = false {\n\tnot object[field]\n\tnot object[field] == false\n}\n\n# get_default returns the value of an object's field or the provided default value.\n# It avoids creating an undefined state when trying to access an object attribute that does\n# not exist\nget_default(object, field, _default) = output {\n\thas_field(object, field)\n\toutput = object[field]\n}\n\nget_default(object, field, _default) = output {\n\thas_field(object, field) == false\n\toutput = _default\n}\n"
        },
        "constraints": [
            {
                "apiVersion": "constraints.gatekeeper.sh/v1alpha1",
                "kind": "GCPBigQueryDatasetLocationConstraintV1",
                "metadata": {
                    "name": "to_be_adapted",
                    "annotation": {
                        "category": "Personal Data Compliance",
                        "description": "BQ Dataset must be located in \u003cto_be_adapted\u003e for projects in \u003cto_be_adapted\u003e."
                    }
                },
                "spec": {
                    "severity": "critical",
                    "match": {
                        "exclude": null,
                        "target": [
                            "organization/"
                        ]
                    },
                    "parameters": {
                        "exemptions": [],
                        "locations": [
                            "EU",
                            "europe-north1",
                            "europe-west1",
                            "europe-west3",
                            "europe-west4"
                        ],
                        "mode": "allowlist"
                    }
                }
            }
        ]
    },
    "feedMessage": {
        "type": "cai",
        "asset": {
            "name": "//bigquery.googleapis.com/projects/<your_project_id>/datasets/brunore_ds_01",
            "owner": "",
            "violationResolver": "",
            "ancestryPathDisplayName": "ramtests.brunore.org/noncompliant/<your_project_id>",
            "ancestryPath": "organization/<org_id>/folder/<folder_number>/project/<your_project_numer>",
            "ancestry_path": "organization/<org_id>/folder/<folder_number>/project/<your_project_numer>",
            "ancestorsDisplayName": [
                "<your_project_id>",
                "noncompliant",
                "ramtests.brunore.org"
            ],
            "ancestors": [
                "projects/<your_project_numer>",
                "folders/<folder_number>",
                "organizations/<org_id>"
            ],
            "assetType": "bigquery.googleapis.com/Dataset",
            "asset_type": "bigquery.googleapis.com/Dataset",
            "iamPolicy": null,
            "iam_policy": null,
            "resource": {
                "version": "v2",
                "discovery_document_uri": "https://www.googleapis.com/discovery/v1/apis/bigquery/v2/rest",
                "discovery_name": "Dataset",
                "parent": "//cloudresourcemanager.googleapis.com/projects/<your_project_numer>",
                "data": {
                    "creationTime": "1604982398240",
                    "datasetReference": {
                        "datasetId": "brunore_ds_01",
                        "projectId": "<your_project_id>"
                    },
                    "id": "<your_project_id>:brunore_ds_01",
                    "kind": "bigquery#dataset",
                    "lastModifiedTime": "1605594631154",
                    "location": "US"
                },
                "location": "US"
            },
            "projectID": "<your_project_id>"
        },
        "window": {
            "startTime": "2021-07-21T10:09:17.141Z"
        },
        "deleted": false,
        "origin": "batch-export"
    },
    "step_stack": [
        {
            "step_id": "every-monday-at-01am10/2662791119629040",
            "step_timestamp": "2021-07-21T10:08:53.58Z"
        },
        {
            "step_id": "brunore-cai-exports-dev-003/dumpinventory_org<org_id>_bigquery_Dataset.dump/1626862157134339",
            "step_timestamp": "2021-07-21T10:09:17.141Z"
        },
        {
            "step_id": "dumpinventory_org<org_id>_bigquery_Dataset.dump/2662784568720615",
            "step_timestamp": "2021-07-21T10:09:17.302Z"
        }
    ]
}