secret-agent

secret-agent - Secret generator and manager for k8s

The secret-agent is a Kubernetes operator that generates the secrets required by the ForgeRock® Identity Platform. The secrets are stored in-cluster as Kubernetes secrets and can also be stored in a cloud secret manager.

Roadmap

Secret agent is now feature complete. Future updates will be limited to bug fixes.

Secret agent was originally designed to fulfill a short term obective to create and manage secrets for the ForgeRock platform running on Kubernetes. The longer term roadmap for platform secrets management is centered around:

• Direct integration with cloud secret management (GCP Secret Manager, Vault, etc.) using ForgeRock's commons secrets API.
• Management of certificates using https://certmanager.io. Cert-manager is the defacto Kubernetes standard for certificate management.
• Backup and restore of Kubernetes secrets using cloud native technologies such as https://velero.io, or vendor provided backup services (Backup for GKE).

Installation

To install the latest secret-agent release in a Kubernetes environment, run:

kubectl apply -f https://github.com/ForgeRock/secret-agent/releases/latest/download/secret-agent.yaml

Specific versions of the operator can be installed by running:

SA_VERSION=v0.1.0 kubectl apply -f https://github.com/ForgeRock/secret-agent/releases/download/${SA_VERSION}/secret-agent.yaml

Configuration

Once the operator is installed, new secrets can be generated by providing a secret agent configuration (SAC) object. The SAC is a custom kubernetes object monitored by the secret-agent operator. All the secrets’ specifications are defined through the SAC.

For more information on how to create a SAC, see the Secret Agent Configuration Schema and/or the Examples sections.

Once the SAC file has been created, it can be pushed to the cluster as with any other resource.

For example:

kubectl create -f config/samples/secret-agent_v1alpha1_secretagentconfiguration.yaml

It is important to note that the Kubernetes secrets produced by the secret-agent will be placed in the same namespace as the SAC. If similar secrets are desired in multiple namespaces, one SAC would be required per namespace.

Usage

Enabling Cloud Backup

The secret-agent can be configured to back up all the generated secrets in a cloud provider's secret manager solution. When this feature is enabled, secrets stored in the secret managers are considered the source of truth.

If a cloud provider has been configured, the operator will attempt to load the secret data from that cloud provider. If the secret is found in the cloud provider's secret manager, the operator will use the found data as the Kubernetes secret data. The operator will only generate new secrets if no secret data is found in the cloud provider.

The secret-agent supports the following cloud providers:

• Google Secret Manager
• AWS Secrets Manager
• Azure Key Vault

It is possible to run the secret-agent without setting up a cloud provider. This is useful when debugging or testing applications. To disable cloud provider support, set spec.appConfig.secretsManager to “none”. This is only possible if spec.appConfig.createKubernetesObjects is set to true.

In addition, it is possible to configure the secret-agent to store secrets in the secret manager without creating local Kubernetes secrets. This is useful if your applications can access the cloud secret manager directly and the secret-agent is only used to generate such secrets. To achieve this, set spec.appConfig.createKubernetesObjects to false. Do note that spec.appConfig.secretsManager cannot be set to "none".

In order to fetch and store secrets in the AWS Secrets Manager, the user must provide credentials with the necessary permissions.

Set up Cloud Backup With AWS Secret Manager

The secret-agent expects credentials to be discoverable via standard AWS mechanisms. These credentials can be provided in a number of ways as per the link.

However, when running inside AWS the preferred method should be to attach a serviceAccount to the deployment with properly scope policies on the role.

• Environment Variables:
  • AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN
  • AWS_WEB_IDENTITY_TOKEN_FILE -> this will be handled by the IAM controller for each deployment correctly set up annotations on the serviceAccount.
• Shared Credentials file: ~/.aws/credentials
• Shared Configuration file: ~/.aws/config
• EC2 Instance Metadata_v2: Obtains credentials from 169.254.169.254
  • This is not ideal as all the pods on the node will have access to the same policy, this may not be a desired outcome if topologySkew are not properly segragating workloads across different nodes/node pools.

Refer to AWS documentation for instructions on how to obtain credentials and grant necessary permissions to access the AWS Secrets Manager. The secret-agent needs to access read/write secrets. This can be achieved by allowing access to the arn:aws:iam::aws:policy/SecretsManagerReadWrite AWS managed policy.

When running outside of AWS environments it is possible to provide custom credentials via a Kubernetes secret. The secret reference is provided in the SAC in spec.appConfig.credentialsSecretName. In the default secret-agent deployment, the user is expected to publish the cloud credentials' secret in the same namespace as the operator. This target namespace can be changed by changing the runtime argument --cloud-secrets-namespace=[NS_NAME] located in the operator's manifest. If this argument is omitted completely, the namespace will default to the namespace of each SAC.

Once these credentials are posted to a Kubernetes secret, the next step is to configure the AWS Secret Manager using the SecretAgentConfiguration.

For example, the following configuration targets AWS Secret Manager in us-east-1:

apiVersion: secret-agent.secrets.forgerock.io/v1alpha1
kind: SecretAgentConfiguration
metadata:
  name: standard-forgerock-example
  namespace: test-sa
spec:
  appConfig:
    createKubernetesObjects: true
    credentialsSecretName: cloud-credentials [** optional**]
    secretsManager: AWS
    awsRegion: us-east-1

optional for AWS: The cloud-credentials secret referenced in spec.appConfig.credentialsSecretName would look like this:

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: cloud-credentials
  namespace: test-sa
data:
  AWS_ACCESS_KEY_ID: QU....[base64 encoded key].....GY=
  AWS_SECRET_ACCESS_KEY: cRB.....[base64 encoded access key].......BB==

Note: The maximum secret size supported by AWS is 65Kb For more information, see AWS documentation.

Set up Cloud Backup With GCP Secret Manager

The secret-agent expects credentials to be discoverable via standard GCP mechanisms. These credentials can be provided in a number of ways, including:

• Workload identity (recommended for GKE deployments)
• Credentials file with user accounts or service accounts.

Please refer to the GCP Documentation for instructions on how to create a service account with the necessary permissions to access the GCP Secrets Manager. The secret-agent needs access to read/write secrets. This can be achieved by assigning the Secret Manager Admin role to the service account provided.

Workload Identity is the recommended way to access Google Cloud services from applications running within GKE. For more information on how to enable workload identity see GCP Documentation.

In general, the user creates a Google Cloud service account with the proper role attached and enables workload identity in their GKE cluster. The Kubernetes services account is already created for you during the secret-agent deployment.

Run the following commands to enable workload identity for the secret-agent deployment:

PROJECTID=myproject #GCP project ID
GSA_NAME=mygcpserviceaccount #GCP service account name
# Create the GCP IAM policy binding
gcloud iam service-accounts add-iam-policy-binding --role roles/iam.workloadIdentityUser --member "serviceAccount:${PROJECTID}.svc.id.goog[secret-agent-system/secret-agent-controller-manager]" ${GSA_NAME}@${PROJECTID}.iam.gserviceaccount.com
# Annotate the Kubernetes service account
kubectl -n secret-agent-system annotate serviceaccounts secret-agent-controller-manager iam.gke.io/gcp-service-account=${GSA_NAME}@${PROJECTID}.iam.gserviceaccount.com

Note: in order to use workload identity, no spec.appConfig.credentialsSecretName should be provided. If credentials are provided, secret-agent will use the provided credentials instead.

The credentials are provided to the operator using a kubernetes secret under the GOOGLE_CREDENTIALS_JSON data key. The name of this secret is provided in spec.appConfig.credentialsSecretName. In the default secret-agent deployment, the operator will look for a secret with the provided name in the operator's own namespace. The user can specify a different namespace by setting the argument --cloud-secrets-namespace=[NS_NAME]. If this argument is omitted, the operator's default behavior is to fetch the credentials from the same namespace as the SAC.

The cloud-credentials secret referenced in spec.appConfig.credentialsSecretName would look like this:

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: cloud-credentials
  namespace: test-sa
data:
  GOOGLE_CREDENTIALS_JSON: .....[base64 encoded service account json].....

Once the necessary credentials are provided to secret-agent using workload identity or a credentials file, the next step is to configure the GCP Secret Manager using the SecretAgentConfiguration.

For example, the following configuration targets GCP Secret Manager for the example-project-id project:

apiVersion: secret-agent.secrets.forgerock.io/v1alpha1
kind: SecretAgentConfiguration
metadata:
  name: standard-forgerock-example
  namespace: test-sa
spec:
  appConfig:
    createKubernetesObjects: true
    credentialsSecretName: cloud-credentials [** skip if using workload identity **]
    secretsManager: GCP
    gcpProjectID: example-project-id

Set up Cloud Backup With Azure Key Vault

note: Azure's API response time on Key Vault is long and will delay the creation of secrets. It might be beneficial to deploy a SAC before long before deploying an application if use Azure Key Vault

The secret-agent uses credentials which are available using two different methods: Azure Managed Identities (recommended for Azure deployemnts) or explicit credentials. Explicit credentials are configured in a secret referenced in the SAC spec spec.appConfig.credentialsSecretName. Example Azure Configuration for a SAC:

apiVersion: secret-agent.secrets.forgerock.io/v1alpha1
kind: SecretAgentConfiguration
metadata:
  name: standard-forgerock-example
  namespace: test-sa
spec:
  appConfig:
    createKubernetesObjects: true
    credentialsSecretName: cloud-credentials [** optional**]
    secretsManager: Azure
    azureVaultName: secret-agent-vault

If no secret is provided in credentialsSecretName, the operator's Azure client will attempt to authenticate using managed identities. For more information, see Azure's documentation. This is the recommended configuration for deployments in Azure's AKS.

Otherwise, the credentials may be explicitly set in the credentialsSecretName secret. The service principle associated with the keys will need the role Key Vault Secrets Officer when using an RBAC policy based Key Vault.

Example credentials secret:

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: cloud-credentials
data:
  # AZURE_TENANT_ID: # OPTIONAL: Update if using Azure Key Vault
  # AZURE_CLIENT_ID: # OPTIONAL: Update if using Azure Key Vault
  # AZURE_CLIENT_SECRET: # OPTIONAL: Update if using Azure Key Vault

Note: The maximum secret size supported by Azure is 25Kb For more information, see Azure documentation.

Importing your own secrets

In addition to generating secrets, the secret-agent allow users to import their own secrets. This is especially useful for things like certificates and certificate authorities that can be referenced by other secrets in the SAC. For example, a user can import their organization's CA and use it to sign certificates generated by the secret-agent.

All that is required is to provide a Kubernetes secret with the same name and same key names as described in the SAC. It is important to note that if the cloud backup feature is enabled, the secret to be imported must be provided using the cloud manager's secret manager. The secret-agent will ignore local secrets if cloud backup is enabled.

Naming Convention For Cloud Backups

There is a naming convention used by secret-agent to store and read secrets from the cloud secret managers. In general, the names follow the format:

$prefix-$secretName-$keyName [If secretsManagerPrefix is provided]
$namespace-$secretName-$keyName [If no prefix is provided]

When using a SecretsManagerPrefix writes to a secret manager doesn't use namespace, make sure your prefixes are unique.

Due to cloud provider limitations, all /, . and _ characters in secret names are replaced by - when accessing the cloud secret managers.

For example, consider the following secret agent configuration:

---
apiVersion: secret-agent.secrets.forgerock.io/v1alpha1
kind: SecretAgentConfiguration
metadata:
  name: forgerock-sac
  namespace: dev
spec:
  appConfig:
    secretsManagerPrefix: "devCluster"
    awsRegion: us-east-1
    secretsManager: AWS
  secrets:
  - name: ds-passwords
    keys:
      - name: dirmanager.pw
        type: password

The secret generated by the SAC would be stored as devCluster-ds-passwords-dirmanager-pw in the AWS Secret Manager due to the secretsManagerPrefix. The same name would apply to other cloud providers.

Some key types require more than one secret to be backed up. Such key types require a public and private components stored separately. This is the case for key types: ca, keypair, ssh and keytool. These secrets use the same naming convention described above and append a suffix to the main name as constructed previously.

In the preceding table, $NAME follows the convention at the top of this section.

Examples

We provide a sample SAC that exercises all features of the secret-agent. See the samples folder.

Secret Agent Configuration Schema

The following tables list the configurable parameters of the secret agent configuration (SAC) and their default values.

App Config

Secret Config

Key Config

Keytool Aliases Config

Runtime Arguments

Running Tests

Tests can be run using make tests. Some of the tests exercise parts of the code that uses keytool, and kubebuildertools. Those must be installed locally in order to run the tests. Another option is to use docker:

• Ensure you're Kubernetes context is pointing to a test cluster, such as minikube, then
  • docker build -t gcr.io/forgerock-io/secret-agent-testing:latest -f --target=tester .
  • docker run -it --rm -v ${PWD}:/root/go/src/github.com/ForgeRock/secret-agent gcr.io/forgerock-io/secret-agent-testing:latest
  • go test ./...