argo-workflows-aws-plugin

argo-workflows-aws-plugin

Argo Workflows Executor Plugin for AWS Services, e.g. SageMaker Pipelines, Glue, etc.

Table of Contents

• Supported AWS Services
• Getting Started
  • Add IAM Role and Policy
  • Enable Executor Plugins
  • Installation
  • Add Workflow Template
  • Trigger Workflow
  • Uninstall Plugin
• References

Supported AWS Services

The following tables describe the implementation state for the protocol's RPC methods and database operations.

Getting Started

Add IAM Role and Policy

The plugin requires IAM role and policy to execute its operations.

The following CDK code add a role, which is later referenced in plugin.yaml manifest.

    const audClaim = `${cluster.clusterOpenIdConnectIssuer}:aud`;
    const subClaim = `${cluster.clusterOpenIdConnectIssuer}:sub`;

    const k8sConditions = new cdk.CfnJson(this, "KubeOIDCCondition", {
      value: {
        [audClaim]: "sts.amazonaws.com",
        // [subClaim]: "system:serviceaccount:kube-system:aws-node",
        [subClaim]: "system:serviceaccount:argo:awf-aws-executor-plugin",
      },
    });

    const awfPluginRole = new cdk.aws_iam.Role(this, "ArgoWorkflowsExecutorPluginRole", {
      roleName: `${stack.stackName}-awf-aws-executor-plugin`,
      assumedBy: new cdk.aws_iam.WebIdentityPrincipal(
        `arn:aws:iam::${cdk.Aws.ACCOUNT_ID}:oidc-provider/${cluster.clusterOpenIdConnectIssuer}`
      ).withConditions({
        StringEquals: k8sConditions,
      }),
    });

    awfPluginRole.addToPolicy(new cdk.aws_iam.PolicyStatement({
      effect: cdk.aws_iam.Effect.ALLOW,
      resources: ["arn:aws:sagemaker:*:*:pipeline/*"],
      actions: [
        "sagemaker:DescribePipeline",
        "sagemaker:StartPipelineExecution",
        "sagemaker:ListPipelineExecutionSteps",
        "sagemaker:DescribePipelineExecution",
        "sagemaker:ListPipelineExecutions",
        "sagemaker:ListPipelines"
      ]
    }));

Enable Executor Plugins

First, enable Executor Plugins:

kubectl patch deployment \
  workflow-controller \
  --namespace argo \
  --type='json' \
  -p='[{"op": "add", "path": "/spec/template/spec/containers/0/env/0", "value": {
    "name": "ARGO_EXECUTOR_PLUGINS",
    "value": "true",
}}]'

Next, restart:

kubectl -n argo set env deployment/workflow-controller ARGO_EXECUTOR_PLUGINS=true
kubectl rollout restart -n argo deployment workflow-controller

Installation

Download the plugin manifest:

wget https://raw.githubusercontent.com/greenpau/argo-workflows-aws-plugin/main/assets/plugin.yaml

Edit metadata.annotations.eks.amazonaws.com/role-arn in the ServiceAccount. (see DEVELOPMENT.md for more information about associated IAM role and policy)

Next, install the plugin:

kubectl apply -f plugin.yaml

The output follows:

serviceaccount/awf-aws-plugin-sa unchanged
clusterrole.rbac.authorization.k8s.io/argo-plugin-addition-role unchanged
clusterrolebinding.rbac.authorization.k8s.io/awf-aws-plugin-addition-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/awf-aws-plugin-binding unchanged
configmap/awf-aws-plugin created

List Argo Workflows Executor Plugins again:

$ kubectl get cm -l workflows.argoproj.io/configmap-type=ExecutorPlugin -n argo

NAME             DATA   AGE
awf-aws          2      34s

Get details about the plugins:

kubectl describe cm -l workflows.argoproj.io/configmap-type=ExecutorPlugin -n argo

Add Workflow Template

Create a workflow template:

kubectl apply -f https://raw.githubusercontent.com/greenpau/argo-workflows-aws-plugin/main/assets/amz-sagemaker-pipelines-workflow-template.yaml

Trigger Workflow

Start new workflow:

kubectl create -f https://raw.githubusercontent.com/greenpau/argo-workflows-aws-plugin/main/assets/amz-sagemaker-pipelines-workflow.yaml

The output follows:

workflow.argoproj.io/sm-pipelines-tswbm created

Review the status of the workflow by the its name, e.g. sm-pipelines-tswbm:

kubectl describe pod -n argo sm-pipelines-tswbm-1340600742-agent

Review logs of the containers (main, awf-aws) inside the pod:

kubectl logs -n argo -c main sm-pipelines-tswbm-1340600742-agent
kubectl logs -n argo -c awf-aws sm-pipelines-tswbm-1340600742-agent

Uninstall Plugin

If necessary, run the following commands to uninstall the plugin:

kubectl delete -f https://raw.githubusercontent.com/greenpau/argo-workflows-aws-plugin/main/assets/plugin.yaml

References

• Argo Workflows - Plugin Directory