rest_data_archiver

REST Data Archiver

This application is intended to provide a simple serverless function to dump the output of a REST API call into an S3 bucket on a regular basis. The result can be used for backup, analysis, or for access by another application. It is meant to be flexible in terms of the source and destination APIs. The primary source adapter implemented is a basic REST API, and the primary destination adapter is AWS S3. Since destinations have their own unique APIs and integration methods, each destination is developed individually to implement the Destination interface. The runtime for this application is configured using a config.json file. An example is provided named config.example.json.

Sources

REST API

Data sources coming from simple API calls can use the RestAPI source. Here are some examples of how to configure it:

Basic Authentication

{
  "Source": {
    "Type": "RestAPI",
    "AdapterConfig": {
      "Method": "GET",
      "BaseURL": "https://example.com",
      "AuthType": "basic",
      "Username": "username",
      "Password": "password",
      "UserAgent": "rest-data-archiver"
    }
  },
  "Sets": [
    {
      "Name": "Users",
      "Source": {
        "Path": "/users"
      },
      "Destination": {
      }
    }
  ]
}

Bearer Token Authentication

{
  "Source": {
    "Type": "RestAPI",
    "AdapterConfig": {
      "Method": "GET",
      "BaseURL": "https://example.com",
      "AuthType": "bearer",
      "Password": "token",
      "UserAgent": "rest-data-archiver"
    }
  }
}

Sets is configured the same as for basic authentication.

Salesforce OAuth Authentication

{
  "Source": {
    "Type": "RestAPI",
    "AdapterConfig": {
      "Method": "GET",
      "BaseURL": "https://login.salesforce.com/services/oauth2/token",
      "AuthType": "SalesforceOauth",
      "Username": "admin@example.com",
      "Password": "the-password",
      "ClientID": "put-your-client-id-here",
      "ClientSecret": "put-your-client-secret-here",
      "UserAgent": "rest-data-archiver"
    }
  },
  "Sets": [
    {
      "Name": "Contacts",
      "Source": {
        "Path": "/services/data/v20.0/query/?q=SELECT%20Email,Name%20FROM%20Contacts"
      },
      "Destination": {
      }
    }
  ]
}

Destinations

Amazon AWS S3

Using the S3 adapter, data from each Set will be saved as an object in the configured S3 bucket.

Here is an example for how to configure it:

{
  "Destination": {
    "Type": "S3",
    "AdapterConfig": {
      "BucketName": "my-archive-bucket",
      "AwsConfig": {
        "Region": "us-east-1",
        "AccessKeyId": "your-access-key-id",
        "SecretAccessKey": "secret-access-key"
      }
    }
  },
  "Sets": [
    {
      "Name": "Users",
      "Source": {
        "Path": "/users"
      },
      "Destination": {
        "ObjectNamePrefix": "users"
      }
    }
  ]
}

Email Alerts

Event Log events with a level of LOG_ALERT or LOG_EMERG will result in an email alert sent via AWS SES. Note that the LOG_EMERG level is 0, which is the Go zero-value. Any new event log created without a Level assigned will default to LOG_EMERG and could result in email alerts being sent.

The following is an example configuration:

{
  "Alert": {
    "AWSRegion": "us-east-1",
    "CharSet": "UTF-8",
    "ReturnToAddr": "no-reply@example.org",
    "SubjectText": "rest-data-archiver alert",
    "RecipientEmails": [
      "admin@example.org"
    ],
    "AWSAccessKeyID": "ABCD1234",
    "AWSSecretAccessKey": "abcd1234!@#$"
  }
}

Alternatively, AWS credentials can be supplied by the Serverless framework by adding the following configuration to serverless.yml:

provider:
  iamRoleStatements:
    - Effect: 'Allow'
      Action:
        - 'ses:SendEmail'
      Resource: "*"

Both authentication mechanisms are provided in the lambda-example directory, but only one is needed.

Exporting logs from CloudWatch

The log messages in CloudWatch can be viewed on the AWS Management Console. If an exported text or json file is needed, the AWS CLI tool can be used as follows:

aws configure
aws logs get-log-events \
   --log-group-name "/aws/lambda/lambda-name" \
   --log-stream-name '2019/11/14/[$LATEST]0123456789abcdef0123456789abcdef' \
   --output text \
   --query 'events[*].message'

Replace /aws/lambda/lambda-name with the actual log group name and 2019/11/14/[$LATEST]0123456789abcdef0123456789abcdef with the actual log stream. Note the single quotes around the log stream name to prevent the shell from interpreting the $ character. --output text can be changed to --output json if desired. Timestamps are available if needed, but omitted in this example by the --query string.