keystar

KeyStar, an HTTP and In-process Key Management Solution

KeyStar is a lightweight key management utility that provides a RESTful HTTP server. Additionally, KeyStar also provides an in-process key management library for Golang applications along with cryptographic wrapper utilities that make use of its internal key management. Presently, it only supports file system backed storage, but plans are in progress to support bbolt, SQLite, and eventually S3-compatible backends. Other options may be explored as time permits or requirements change (this includes etcd).

Available in this document are the following sections; missing sections will be added as time allows:

• Why KeyStar?
• Command Line
  • Configuration
  • Initialization
  • Other Commands
• API Concepts
  • Authentication and Tenancy
  • Key Storage Hierarchy
  • Key Spaces
  • Namespaces
  • Key Space and Accounting Limitations
  • Key Rings
  • Keys
    • Standard Keys
    • Composite Keys
    • Custom Keys
• Golang (in-process) API
  • Step-by-Step Examples
• HTTP API
  • Overview and Endpoints
    • Authentication and Authorization
    • Key Management: Global Namespace
    • Utility Requests
  • Authentication API
  • Key Management API
    • Retrieval
    • Creation or Retrieval (PUT)
    • Creation or Failure (POST)
    • Update
    • Rotation
    • Key or Key Ring Deletion
    • Deletion (Keys Only)
    • Templates
  • Utility API
    • Byte Generation
    • Key Generation
    • Signature Generation
• Cryptographic Utilities

Why KeyStar?

There are several key management libraries currently available across different languages and platforms. However, many of these libraries require complex configuration or have hard dependencies on other software that limit their utility for smaller projects. While KeyStar can be used for projects of any size, it was optimized for use as a key management service for small and mid-sized applications. KeyStar is perfect for projects that are too large to manage keys themselves, where key management might be more complex than necessary (large uWSGI applications with multiple processes), and where highly generalized key managers may impose a significant burden on the project's requirements, both in terms of complexity--as mentioned--and by consuming valuable developer time.

In particular, if you have a project that requires an external key management solution that can be queried during application start up--or periodically to refresh keys--but you have no need for extensive authentication, authorization, or multi-tenancy applications, KeyStar may be a better fit. This is especially true if your application is also written in Go: KeyStar can be included as part of your project and loaded directly. Optionally, in-process KeyStar instances can be configured to launch their own key management server(s) for remote clients or those written in other languages.

KeyStar also includes some helpful wrappers around Golang's cryptographic primitives for things like encrypted-and-signed data, highly scalable token generation (think password resets, etc), and other utilities to make the mundane use of crypto less boilerplate and more focused on the task at hand.

Command Line

Most access to KeyStar (including KeyStar clients and servers) can be managed from the keystar command and its sub-commands. Additionally, some initial configuration for setting up key spaces can only be performed with the keystar command.

As of the current version, the following sub-commands are defined:

• admin - KeyStar administrative tools.
• client - Client tools and utilities.
• keys - Key management tools, generators, and utilities.
• license - Prints any appropriate LICENSE texts for KeyStar and dependent libraries.
• server - KeyStar HTTP API server.

Note: Some commands are not yet finished or implemented. This is expected to change in the near future when KeyStar reaches its first official "release."

Configuration

Before KeyStar may be used, it is necessary to configure it. The current distribution of KeyStar should include a keystar.yaml.sample file that defines the host, port, and storage backend URI for the server. Copy the keystar.yaml.sample file to keystar.yaml and modify it accordingly.

KeyStar can be launched from any user account, or multiple accounts, on the system and provides mechanisms for discovering its configuration from any of a variety of locations. In particular, KeyStar searches:

• The current working directory.
• (Linux) The existence or contents of XDG_CONFIG_HOME/keystar/keystar.yaml, XDG_CONFIG_DIRS entries with the path /keystar/keystar.yaml, and /etc/keystar/keystar.yaml.
• (Windows) The existence or contents of %APPDATA%/Roaming/keystar/keystar.yaml

for its configuration file and will read it from the first location it finds, in that order.

Future versions of KeyStar may include a configuration that allows additional options, such as multiple servers, key spaces, or client profiles.

Initialization

Once KeyStar's configuration file, keystar.yaml, has been edited appropriately, the next step is to initialize the configured key space(s). To do so,run the keystar admin init command. This will return with either an error message if the initialization process failed, or it will indicate success and print the system account ID and secret tokens.

SAVE THE ACCOUNT ID AND TOKEN PAIR! Although you may use keystar admin init (and, eventually, other commands) to print out these values from the configured instance, you will need these to authenticate with the server when using the HTTP API.

Other Commands

Other commands KeyStar provides that may be of use are fairly limited, but we expect to add a variety of client and key commands for convenience. This section will be updated as these commands find their way into KeyStar's lexicon.

Of particular note as of this writing is the command keystar client authenticate. This command will read the system account ID token and its secret token, process the appropriate challenge-response cycle, and return a complete string for use as the Authorization: Bearer HTTP header. The output from this command can be pasted as an argument to cURL's -H flag and may be used to test or interact with the HTTP API.

API Concepts

Correct use of KeyStar requires understanding a few core concepts defining how it works and how it stores key information internally. Not all of these concepts are required for both the in-process and HTTP APIs but understanding most of the stack may nevertheless be useful.

Before we discuss the key data hierarchy, it is necessary to discuss authentication and KeyStar-level accounting.

Authentication and Tenancy

KeyStar defines two modes of operation for its authentication and accounting processes: Single-user occupancy or multi-tenancy. In single-user mode, user namespaces are disabled with only the global namespace and its cohorts accessible from clients. In multi-tenancy mode, accounts can be created that have access to their own "global" namespace and named namespaces, or they can be extended permissions to access the system-wide global namespace.

By using multi-tenancy mode, it is possible to support multiple applications within a single KeyStar instance by restricting some applications to their own namespace(s), granting access to only the global namespace, or granting an inheritance hierarchy to an application so keys not found in its local namespace will bubble up into the global namespace until they're located (or not).

However, single-user occupancy is likely the mode users will find most useful to their own application. Single-user occupancy is easier to configure, requires no additional maintenance (other than API key generation), yet still provides access to a variety of features for separating keys and key rings into their own distinct silos via namespacing. For in-process use of KeyStar, single-user occupancy is likely the best option.

Key Storage Hierarchy

KeyStar establishes the following hierarchy for key membership, in order:

• Key spaces
• Namespaces (global, named, and user)
• Key rings
• Keys

From top-to-bottom, this defines the a) file-system level storage or backend storage that is unique to each instance, b) namespaces that isolate individual keys and key rings from each other, and c) key rings containing related keys and the keys themselves.

In the following sections, we discuss each of these concepts.

Key Spaces

At the top level of KeyStar data structures is the key space. Key spaces contain all namespace identifiers: The global namespace, named global namespaces, and the associated user naemspaces. A single key space interacts with one (and only) one backend; this means that multiple key spaces could be defined within a single KeyStar instances that utilize multiple backends, directories, or other data stores. Key spaces also contain their own unique authentication and could be used to isolate separate instances, or enable multiple instances with single-user occupancy or multi-tenancy individually.

Namespaces

From the key space, we travel next to its subordinates: The namespace collection. Namespaces are unique key ring and key storage clusters segregated into the global namespace, which is the default for most key ring and key creation; named global namespaces, which are separate namespaces under the global namespace; and the user global and named user namespaces (available only if multi-tenancy is enabled).

Namespace access is defined by its type. The global namespace is used, by default, accessible strictly to the administrative account or its API keys. Under multi-tenancy modes, permissions may be given to individual accounts to access both their own namespaces or the system-wide global namespaces; or permissions may be restricted to prevent user accounts from accessing information outside their own namespace silos.

To provide a brief illustration of how namespaces can be used to separate key purposes within an application, consider the following:

• Global namespace -> Application keys
• Named global namespace -> "tokens" -> Application user tokens

In this instance, the application-level keys are stored directly within the global namespace, and therefore accessible from .KeySpace().Global(). We're also storing the application's user tokens in a separate global namespace (called "tokens") which would be accessible from .KeySpace().GlobalNamespace("tokens").

If this instance were configured for multi-tenancy, KeyStar accounts could be used for other services, such as separate applications; in this example, our main application would have access to the global namespace, and supporting applications would use KeyStar's "user" namespaces. (Do not confuse your application users with KeyStar's accounts!)

Key Space and Accounting Limitations

As of this writing, there are some limitations due to an incomplete implementation: First, a single KeyStar instance can only encapsulate a single KeySpace; second, KeyStar does not support multi-tenancy modes and can only be used as a single-occupancy application with the global or named global namespaces. "User" or "account" namespaces are not available. We plan on implementing both authentication, and therefore account namespaces, sometime after implementing the ability to configure multiple key spaces. This is because each key space will be able to contain its own accounting and will be isolated from other key spaces.

Key Rings

Subordinate to the namespace is the concept of key rings. Key rings are separate containers that hold keys intended for related purposes. This might include things like application crypto keys, signing keys, or secret keys. Keys within a given key ring all share the same expiration attributes (time-to-live) and automatic deletion or rotation settings. Note that one of the primary limitations of KeyStar is that you cannot control expiration of individual keys. While this may eventually be changed, we believe that controlling expiration at the key ring level delineates a clear boundary between related keys. Further, storing related keys within the same key ring provides a more transparent solution for quickly and cleanly rotating all application keys.

Another interesting side effect of storing related keys within a given key ring is thus: If you have an application that generates user tokens, such as for account password resets or the likes, creating a separate namespace for your users followed by a per-user key ring confers easier management of that user's key set.

The concept of key ring-level attributes rather than key-level attributes isn't set in stone and may eventually change.

Key rings also expose other utilities to client code, such as the ability to easily create a composite key or list the entire contents of the key ring. Key rings make it easy to delete all related keys with a single command and can be used to quickly disable a set of related keys.

Keys

Finally, we get to the topic of KeyStar's ultimate purpose: Key management and keys. Individually, keys are nothing more than a collection of metadata and a base64-encoded series of bytes. However, these keys can have important purposes, such as use for an application's secret key, encrypting or signing payloads, or token generation. Keys can be described as existing in three flavors: Standard keys, composite keys, and custom keys.

Standard Keys

Most applications will make use of KeyStar's "standard" keys at some point in time. In particular, for signing binary chunks or similar, standard keys may be used for virtually any purpose. These are the simplest key and comprise only a single binary chunk

Composite Keys

Composite keys are a sort of "meta" key that are implemented identically to "standard" keys (above) but have special meaning when combined. In most cryptographic solutions, two keys are required to successfully encrypt and validate a payload: The cipher key, which is used to encrypt or decrypt the data; and the MAC key, which is used to sign or verify the encrypted payload. KeyStar takes this a step further by exposing utility functions for creating composite keys that contain both and can have their lengths defined separately.

Internally, composite keys are merely standard keys with a unique suffix defining their purpose. Some API calls make no distinction between composite keys and their standard counterparts, and may expose this feature when listing a key ring's content.

Custom Keys

Less often used but still useful is the concept of a custom key. A custom key is a key whose encoded data chunk may contain base64-encoded binary data, string data, or anything in between. This data is user-submitted and stored identically to other key types with the exception that key decoding may fail. Calling .Bytes() on a custom key whose content is unknown may cause the base64 decoding process to fail and return an error. Therefore, custom keys are slightly more inconvenient than other types, primarily because they are user-defined. Custom keys are also ignored during key rotation.

Golang (in-process) API

KeyStar provides a direct API for use with in-process key management. This is the same interface used by the HTTP API server; therefore, the HTTP API service is itself a KeyStar client.

Please be warned: The API is currently in a state of flux and isn't expected to stabilize until an official version tag is released--probably around v0.1 or greater.

API documentation will be posted eventually. For now, enjoy some examples.

Step-by-Step Examples

KeyStar clients start with the initialization of KeyStar:

import (
  "git.pluggableideas.com/destrealm/go/keystar"
  kt "git.pluggableideas.com/destrealm/go/keystar/types"
)

func main() {
  ks := keystar.NewKeyStar(
    &kt.Options{
      BackendURI: "file://.keys",
    }
  )

}

This configures KeyStar to use the file backend with a key storage directory of .keys.

Next, namespaces are managed via the KeySpace struct, obtained via a call to .KeySpace(). Future versions may allow for the configuration of multiple KeySpaces. To obtain the global namespace or a named global namespace:

ks := keystar.NewKeyStar(
  &kt.Options{
    BackendURI: "file://.keys",
  }
)
global := ks.KeySpace().Global()
named, err := ks.KeySpace().GlobalNamespace("testing")
if err != nil {
  // Handle errors during the creation of named global namespaces.
}

Once a namespace is obtained, key rings and keys can be managed accordingly:

import (
  "git.pluggableideas.com/destrealm/go/keystar/types/api"
)

// Get or create a key ring from the global namespace (a TTL of "0" means no
// expiration):
kr, err := global.GetOrCreateKeyRing("test-keyring", 0)
if err != nil {
  // Handle key ring creation or retrieval errors.
}

// Get or create a key 32 bytes in length:
key, err := kr.GetOrCreate("demo", 32)
if err != nil {
  // Handle key creation or retrieval errors.
}

// Get or create a composite key with a cipher key length of 32 and an HMAC key
// length of 128:
ckey, err := kr.GetOrCreateComposite("demo-composite", 32, 128)
if err != nil {
  // Handler key creation or retrieval errors.
}

// List all keys.
if keys, err := kr.List(); err == nil {
  for _, k := range keys {
    fmt.Println("Key: %s (%d bytes): %s", k.Name, k.Length, k.Encoded)
  }
}

// To retrieve the contents of a key, decoded into a byte slice:
b := key.MustGetBytes()

// If the idea of a panic while decoding a key worries you, errors can be
// handled manually:
if b, err := key.Bytes(); err != nil {
  // ...
}

// Key decoding should never panic under ordinary conditions, but custom keys
// (created as follows) will fail if the `Encoded` value is not base64-encoded.
custom := &api.Key{
  Name: "sample-custom",
  Custom: true,
  Encoded: "This is a custom key.",
}
custom.Length = len(custom.Encoded)

// ... do other things.

// ...then add the pre-generated key to the key ring.
if err = kr.Add(custom); err != nil {
  // Handle errors...
}

HTTP API

KeyStar's HTTP API makes use of the GET, POST, PUT, DELETE, and PATCH verbs. Some creative license is taken in our interpretation of what these verbs mean, and their behaviors deviate slightly due to the limits imposed by storing or retrieving random data like keys to ensure idempotency[^1]. In particular, such deviations are present in POST and PUT. POST instructs KeyStar to create a key or generate an error if the key already exists. PUT is used instead to create or retrieve a key and will always return the same result for each query or generate an error if the request body contains key metadata that doesn't match the key's on-disk data structures (this behavior may change to a replacement rather than error in the future).

Overview and Endpoints

This section illustrates actions, endpoints, and verbs. The following are routes defined by KeyStar:

Authentication and authorization

Action	Verb	Endpoint
Request authentication token	GET	/authorize/{id:string}[?duration={int}]
Submit challenge response	POST	/authorize/{id:string}

Key Management: Global Namespace

Action	Verb	Endpoint(s)
Retrieve key	GET	/keyring/{keyring:string}
		/keyring/{keyring:string}/{key:string}
		/{namespace:string}/keyring/{keyring:string}
		/{namespace:string}/keyring/{keyring:string}/{key:string}
Update key	PATCH	/keyring/{keyring:string}/{key:string}
		/{namespace:string}/keyring/{keyring:string}/{key:string}
Create or retrieve key	PUT	/keyring/{keyring:string}/{key:string}
		/{namespace:string}/keyring/{keyring:string}/{key:string}
Create or fail if exists	POST	/keyring
		/{namespace:string}/keyring
Rotate key ring	POST	/rotate/{keyring:string}
		/{namespace:string}/rotate/{keyring:string}
Delete key or key ring	DELETE	/keyring
		/{namespace:string}/keyring
Delete key	DELETE	/keyring/{keyring:string}/{key:string}
		/{namespace:string}/keyring/{keyring:string}/{key:string}
Create using template	POST	/template/
		/tmeplate/{namespace:string}

Note: All key management endpoints can be prefixed with the path /global/ to avoid confusion that may be caused when user namespaces are introduced. For example, retrieving a key can be performed via either the /keyring/{keyring:string} or /global/keyring/{keyring:string} endpoints.

Utility Requests

Action	Verb	Endpoint(s)
Generate random bytes	GET	/generate/bytes
Generate a key entitied	POST	/generate/key
Generate a composite key	POST	/generate/composite-key
Sign submitted data	POST	/generate/signature

Note that signing submitted data via the /generate/sign endpoint does not allow the submission of arbitrary data by anyone. All utility endpoints are authenticated and require the appropriate Authorization: Bearer header to be present for the given user. Moreover, uploaded data is limited to the default POST body size limit (usually 10 megabytes). If more data is required, it's a better idea to submit a signature of the data and sign that.

Authentication API

KeyStar's authentication API is based around a challenge-response mechanism whereby the requesting client submits its ID token, the server responds with a random base64-encoded string of bytes as its challenge, and the client then signs this challenge with its own key using HMAC SHA512_256. Since both the client and server share the same secret tokens for a given client ID, the server is then able to validate the submitted signature using its copy. If the authentication process is successful, a token will be returned for use as an Authorization: Bearer token that will validate all future requests. The bearer token must be present in all requests to endpoints other than /authorize.

The initial authentication request must be submitted to the /authorize/{id:string} endpoint using a GET request where {id:string} is the client's identification token. This will return a response similar to the following:

{
    "challenge": "<base64 encoded-challenge token>"
}

Challenge tokens will only persist for 5 minutes (300 seconds). If a more limited-duration token is required, the optional query string variable duration may be specified. For example, requesting a challenge token that is valid for no more than 10 seconds may be obtained by querying /authorize/{id:string}?duration=10.

Clients must then submit a POST request containing the following response to the /authorize/{id:string} endpoint:

{
    "challenge": "<original base64-encoded challenge token>",
    "response": "<base64-encoded signature>",
    "algorithm": "(sha512_256|sha512_385|sha512|sha256):default:sha512_256"
}

If the algorithm field is not specified, challenge authorization will default to sha512_256. As of this writing, only sha512_256 is implemented with others soon to be included.

After the challenge-response has been submitted via POST, the server will return one of several possible values:

• 400, bad request; indicates the request was missing an appropriate field or could not be parsed, such as invalid JSON or base64-encoding.
• 500, internal server error; indicates that an error occurred while validating the challenge that the server could not account for. This may be due to improper configuration or failure to run keystar admin init.
• 200, plus a JSON-encoded response (see below).

Once successful, the server will return a JSON response containing a base64-encoded token for use in the Authorization: Bearer client header:

{
    "authorization": "<base64-encoded authorization token>"
}

This token may be used more or less indefinitely. Timeouts or requests to limit the token's duration will eventually be supported.

Identification tokens for single-occupancy configurations are analogous to the tokens created for the administrative account and are generated with the keystar admin init command. This command can be run multiple times if the system keys have already been generated to retrieve the administrative ID and secret tokens. Note that for multi-tenancy configurations, tokens may have to be generated by the administrative account. Details on how user authentication will work in the future are currently being fleshed out. Specialized API tokens are planned in the future that will be analogous to ID/secret pairs and will be usable in either single occupancy or multi-tenant configurations, with the administrative or user accounts able to create more specialized tokens with their own permissions.

Key Management API

Key management is broken down into eight separate categories:

• Retrieval
• Update
• Creation or Retrieval (PUT)
• Creation or Failure (POST)
• Rotation
• Key or Key Ring Deletion
• Deletion (Keys Only)
• Templates

Each request URI supports optional namespace members, as a string, which define unique namespaces for a given set of key rings. Presently, two types of namespace containers are planned: Global, which is currently available via the /global/ root path (or, optionally, leaving the root path absent); and /user/, which will define a user-level namespace or global namespace. Only the global namespace is currently implemented.

Retrieval

Key retrieval can be used to retrieval all keys that are members of a specified key ring, individual keys, or composite keys. As mention earlier, composite keys are a special type of key of which two independent components comprise the parent "key." These component keys are intended for use as cipher and HMAC keys for cryptographic functions (encrypting and decrypting) and signing (signature creation and validation). At present, KeyStar makes no distinction between standard and composite keys, storing both as independent members of a given key ring.

API retrieval is performed via GET requests made to the endpoints:

• /keyring/{keyring:string}[?key=<name:string>[&type=composite]]
• /keyring/{keyring:string}/{key:string}[?type=composite]
• /{namespace:string}/keyring/{keyring:string}[?key=<name:string>[&type=composite]]
• /{namespace:string}/keyring/{keyring:string}/{key:string}[?type=composite]

As illustrated above, endpoints that do not specify the key name as part of the request URI can have the name provided via a query variable called key. Optionally, composite keys may be requested by appending the query string type=composite. Presently, composite and key are the only valid values for type; all other values will result in a 400 status code returned by the server.

If no name is provided, KeyStar will return all keys associated with the requested key ring.

The {namespace:string} URI prefix instructs KeyStar to return keys assigned to the given namespace.

Key retrieval returns two schemas depending on the value of key, whether provided via the request URI or query string. Individual fields may be absent if their value is considered empty (zero, empty string, or false).

For specific keys (this is the Key schema):

Field name	Type	Always Present	Description
name	string	yes	Key name
length	integer	yes	Length in bytes
created	string	yes	ISO 8601 timestamp in UTC (date + time)
encoded	string	yes	Base64-encoded key data
ttl	integer	no	Time to live before expiration
delete_after	integer	no	Delete key after expiration (seconds)
rotate_after	integer	no	Rotate key after expiration (seconds)

For composite keys the individual key schema is as above but is contained within an encapsulating parent object as follows and individual key members do not contain a name field:

Field name	Type	Always Present	Description
name	string	yes	Composite key name
cipher	Key	yes	Cipher key contents; uses above schema
hmac	Key	yes	HMAC key contents; uses above schema

In the above table, the type Key is a placeholder for the stand alone key schema but is absent the name field.

For listing the contents of a key ring, the returned JSON schema is an array containing Key types. Composite keys are currently not handled separately as there is currently no way to differentiate them from ordinary keys without persisting additional data with the key. Further, composite keys may be stored with names identical to stand alone keys:

Key retrieval presently returns the following HTTP status codes:

Code	Condition or Description
400	If an incorrect key type was specified
401	If the Authorization: Bearer token was absent
404	If the specified namespace, key ring, or key do not exist
500	If an error was encountered or the response could not be encoded
404	If the namespace or key ring could not be found or keys could not be listed
500	If the key ring could not be rotated or the response could not be encoded

Creation or Retrieval (PUT)

KeyStar provides two options for creating keys--this is one such option. By submitting key options via a PUT request, KeyStar will attempt to create the key with the specified configuration, or if the key already exists, it will return the key matching the request. At present, if the specified key already exists but its creation options differed from those requested by the client, an error will be returned. This may be changed in the future to follow PUT semantics more precisely.

This method of key creation is known as "create-or-retrieve."

API create-or-retrieval for keys may be done by submitting PUT requests to the following endpoints:

• /keyring/{keyring:string}/{key:string}[?type=composite]
• /{namespace:string}/keyring/{keyring:string}/{key:string}[?type=composite]

where both endpoints may be prefixed with /global to specify the global namespace and to avoid confusion when user namespaces are introduced. Omitting the /global prefix also selects the global namespace.

For key creation, both the key ring and key names must be specified. An optional namespace identifier may be supplied as well to create a key ring within the requested namespace. All of the options, namespace, keyring, and key are string values. These values will be URL-decoded before submission to the backend and may contain unicode characters.

Composite keys may be generated with this endpoint by supplying the query string type=composite. Presently, composite and key are the only valid values for type; all other values will result in a 400 status code.

PUT requests must contain the headers Content-Type: application/json or Content-Type: text/json to be correctly parsed. application/json is preferred. If the Content-Type header is not present, the server will return a 400 BAD REQUEST.

Create-or-retrieve endpoints must be supplied with a JSON-encoded request body. For individual keys, the schema may contain the following fields:

Field name	Type	Required	Description
length	integer	yes	Requested key length in bytes
ttl	integer	no	Time to live in seconds; expires when exhausted
delete_after	integer	no	Delete key after expiration (seconds)
rotate_after	integer	no	Rotate key after expiration (seconds)

For composite keys, the schema may instead contain:

Field name	Type	Required	Description
cipher_length	integer	yes	Cipher key length in bytes
hmac_length	integer	yes	HMAC key length in bytes
ttl	integer	no	Time to live in seconds; expires when exhausted
delete_after	integer	no	Delete key after expiration (seconds)
rotate_after	integer	no	Rotate key after expiration (seconds)

Note: Only the length field or cipher_length and hmac_length fields are required for ordinary keys and composite keys, respectively. All other fields are optional and may be omitted.

Create-or-retrieve returns a schema similar to GET requests.

For stand alone keys:

Field name	Type	Always Present	Description
name	string	yes	Key name
length	integer	yes	Key length
created	string	yes	ISO 8601 timestamp in UTC (date + time)
encoded	string	yes	Base64-encoded key data
ttl	integer	no	Time to live before expiration (seconds)
delete_after	integer	no	Delete key after expiration (seconds)
rotate_after	integer	no	Rotate key after expiration (seconds)

For composite keys the individual key schema is as above but is contained within an encapsulating parent object as follows and individual key members do not contain a name field:

Field name	Type	Always Present	Description
name	string	yes	Composite key name
cipher	Key	yes	Cipher key contents; uses above schema
hmac	Key	yes	HMAC key contents; uses above schema

In the above table, the type Key is a placeholder for the stand alone key schema but is absent the name field.

In the following examples, we use cURL to create an assortment of keys. The server is configured to use localhost and the default port (9911). For brevity we omit the Authorization: Bearer and Content-Type: application/json headers. If you're following along with these examples, you will need to run the command keystar client authenticate, copy the generated Authorization: Bearer header and supply the appropriate content type, e.g.:

$ curl -H "Authorization: Bearer <token string>" -H 'Content-Type: application/json' ... <url>

Example 1: Create a key with a length of 32 and no TTL within the key ring "testing" using the key name "demo":

$ curl -d '{"length":32}' -X PUT http://localhost:9911/keyring/testing/demo

Example 2: Create a composite key named "demo-composite" with a cipher key of length 32 and an HMAC key of length 128 within a key ring named "test-composite:"

$ curl -d '{"cipher_length":32,"hmac_length":128}' -X PUT http://localhost:9911/keyring/test-composite/demo-composite?type=composite

Example 3: Create a key named "ttl-demo" under the key ring "expires" with a length of 16 and a TTL of 5 minutes (300 seconds):

$ curl -d '{"length":16,"ttl":300}' -X PUT http://localhost:9911/keyring/expires/ttl-demo

Example 4: Create a new key ring "expires" under the namespace "demo" with a key named identically above to example 3 (ttl-demo) with similar attributes except for the length:

$ curl -d '{"length":32,"ttl":300}' -X PUT http://localhost:9911/demo/keyring/expires/ttl-demo

Create-or-retrieve returns the following HTTP status codes:

Code	Condition or description
400	If a required field is missing, the request body could not be parsed, or an attempt is made to overwrite a custom key
401	If the Authorization: Bearer header is absent
404	If a failure occurs while loading the key ring, namespace, or key[^2]
500	If a failure occurs while updating the key or encoding the response

Creation or Failure (POST)

Creation-or-failure is the second method KeyStar exposes for creating keys. This method, using POST, will attempt to create a key with the specified name and properties, and will fail if the key already exists on disk. This method is useful if you wish to avoid overwriting existing keys.

Although similar to the create-or-retrieval API, the create-or-failure API does not allow specifying the key ring or key name in the request URI. Instead, these values must be supplied via the POST request body.

• /keyring[?type=composite]
• /{namespace:string}/keyring[?type=composite]

As with create-or-retrieve, both endpoints may be prefixed with /global for specifying the global namespace. Omitting this prefix is allowable and synonymous.

Composite keys may be generated with this endpoint by supplying the query string type=composite with the request. Presently, composite and key are the only valid values for type; all other values will result in a 400 BAD REQUEST status code.

POST requests, as with PUT, must contain the headers Content-Type: application/json or Content-Type: text/json to be correctly parsed. application/json is preferred. if the Content-Type header is omitted, the server will return a 400 BAD REQUEST.

Create-or-failure must be supplied with a JSON-encoded request body. For individual keys, the schema is similar to PUT but must include the name and keyring fields:

Field name	Type	Required	Description
name	string	yes	Key name
keyring	string	yes	Key ring
length	integer	yes	Requested key length in bytes
ttl	integer	no	Time to live in seconds; expires when exhausted
delete_after	integer	no	Delete key after expiration (seconds)
rotate_after	integer	no	Rotate key after expiration (seconds)

For composite keys, the schema may instead contain:

Field name	Type	Required	Description
name	string	yes	Key name
keyring	string	yes	Key ring
cipher_length	integer	yes	Cipher key length in bytes
hmac_length	integer	yes	HMAC key length in bytes
ttl	integer	no	Time to live in seconds; expires when exhausted
delete_after	integer	no	Delete key after expiration (seconds)
rotate_after	integer	no	Rotate key after expiration (seconds)

Create-or-failure returns a schema similar to GET and PUT requests.

For stand alone keys:

Field name	Type	Always Present	Description
name	string	yes	Key name
length	integer	yes	Key length
created	string	yes	ISO 8601 timestamp in UTC (date + time)
encoded	string	yes	Base64-encoded key data
ttl	integer	no	Time to live before expiration (seconds)
delete_after	integer	no	Delete key after expiration (seconds)
rotate_after	integer	no	Rotate key after expiration (seconds)

For composite keys the individual key schema is as above but is contained within an encapsulating parent object as follows and individual key members do not contain a name field:

Field name	Type	Always Present	Description
name	string	yes	Composite key name
cipher	Key	yes	Cipher key contents; uses above schema
hmac	Key	yes	HMAC key contents; uses above schema

In the above table, the type Key is a placeholder for the stand alone key schema but is absent the name field.

In the following examples, we use cURL to create an assortment of keys. The server is configured to use localhost and the default port (9911). For brevity we omit the Authorization: Bearer and Content-Type: application/json headers. If you're following along with these examples, you will need to run the command keystar client authenticate, copy the generated Authorization: Bearer header and supply the appropriate content type, e.g.:

$ curl -H "Authorization: Bearer <token string>" -H 'Content-Type: application/json' ... <url>

Reminder: If the request method (-X) is not specified, cURL will default to POST.

Example 1: Create a key with a length of 32 and no TTL within the key ring "testing" using the key name "demo":

$ curl -d '{"keyring":"testing","name":"demo","length":32}' http://localhost:9911/keyring

Example 2: Create a composite key named "demo-composite" with a cipher key of length 32 and an HMAC key of length 128 within a key ring named "test-composite:"

$ curl -d '{"keyring":"test-composite","name":"demo-composite","cipher_length":32,"hmac_length":128}' http://localhost:9911/keyring?type=composite

Example 3: Create a key named "ttl-demo" under the key ring "expires" with a length of 16 and a TTL of 5 minutes (300 seconds):

$ curl -d '{"keyring":"expires","name":"ttl-demo","length":16,"ttl":300}' http://localhost:9911/keyring

Example 4: Create a new key ring "expires" under the namespace "demo" with a key named identically above to example 3 (ttl-demo) with similar attributes except for the length:

$ curl -d '{"keyring":"expires","name":"ttl-demo","length":32,"ttl":300}' http://localhost:9911/demo/keyring

Create-or-failure returns the following HTTP status codes:

Code	Condition or description
400	If a required field is missing or the request body could not be parsed
401	If the required Authorization: Bearer token is absent
404	If the specified namespace could not be found or retrieving the key ring fails[^3]
500	If the key could not be created or encoded to JSON during the response cycle

Update

KeyStar's update semantics use a slightly modified JSON PATCH (RFC 6902) format. In particular, not all operations are applicable to KeyStar's storage mechanics, nor is the "path" field fully analogous to those attributes exposed on our data structures and is not entirley useful for modifying keys. (For example, keys cannot be modified directly, only the key ring.)

Our modifications therefore limit "path" to only those values that can be modified following key creation, and we limit the operations to test, add, replace (synonym for add), and remove.

The update API for keys may be performed by submitting PATCH requests containing RFC 6902-formatted JSON patch sets to the following endpoints:

• /keyring/{keyring:string}/{key:string}
• /{namespace:string}/keyring/{keyring:string}/{key:string}

where both endpoints may be prefixed with /global to specify the global namespace and to avoid confusion when user namespaces are introduced. Omitting the /global prefix also selects the global namespace.

Acceptable operations are:

Operation	Description
test	Tests the specified value against the specified key ring or key
add	Updates the value of the specified field provided by path
replace	Synonym for add
remove	Resets the specified values to their defaults

The path field for PATCH updates may be one of ttl, delete_after, rotate_after, length, or encoded. However, the precise inclusion of these fields depends on the operation. The following is a support matrix for operations and what fields they allow:

Operation	Fields	Remarks
test	ttl, delete_after, rotate_after, length	Tests the values of these fields specifically, and returns an error if no match is found
add	encoded, ttl, delete_after, rotate_afte	Allows changing of the specified fields; encoded is supported only for custom keys
replace	same as add	same as add
rmeove	ttl, delete_after, rotate_after	Only those fields that can be sensibly "reset" are included; even for custom keys, encoded cannot be reset, and the key must instead be deleted

Other operations such as move or copy will return errors.

The update API returns a schema similar to GET, PUT, and POST requests.

For stand alone keys:

Field name	Type	Always Present	Description
name	string	yes	Key name
length	integer	yes	Key length
created	string	yes	ISO 8601 timestamp in UTC (date + time)
encoded	string	yes	Base64-encoded key data
ttl	integer	no	Time to live before expiration (seconds)
delete_after	integer	no	Delete key after expiration (seconds)
rotate_after	integer	no	Rotate key after expiration (seconds)

Currently, composite keys are not supported and therefore cannot be changed via this interface. You may have noticed an absence of the expected query stringof key names. variables.

The update API returns the following HTTP status codes:

Code	Condition or Description
400	If a malformed patch set was submitted or the request could not be parsed
401	If the Authorization: Bearer token was absent
404	If the namespace, key ring, or key could not be found
500	If the update fails for any reason or the response could not be encoded

Rotation

Key rotation is a mechanism for cycling out old secrets and replacing them with new data. This is generally used if a key is to be retired from use or if it is suspected that the key data has been leaked. KeyStar provides a mechanism for rotating keys via the HTTP API so clients can perform this task manually if required.

At present, automatic rotation is not supported. Likewise, multiple key versions are not currently stored nor is any method exposed for retrieving prior copies of a rotated key. Both of these features are planned in the near future.

Key rotation may be performed by submitting a POST request to the following endpoints:

• /rotate/{keyring:string}
• /{namespace:string}/rotate/{keyring:string}

Note that only the keyring and namespace values may be present. As an artifact of how KeyStar stores keys within key rings, individual keys cannot be rotated; only the entire key ring can be rotated manually. Automatic key rotation will eventually provide a method for rotation only those keys that have expired, but this is not currently exposed via the HTTP API for manual intervention.

Currently, to trigger a key rotation, an empty POST request body must be submitted to one of the above endpoints, depending on the key ring's location in the namespace hierarchy. Eventually, additional options may be provided for changing rotation behaviors or for rotating only those keys that have expired.

Key rotation returns a schema similar to GET requests listing all keys. Individual keys are defined by the following schema in a JSON-encoded format:

Field name	Type	Always Present	Description
name	string	yes	Key name
length	integer	yes	Key length
created	string	yes	ISO 8601 timestamp in UTC (date + time)
encoded	string	yes	Base64-encoded key data
ttl	integer	no	Time to live before expiration (seconds)
delete_after	integer	no	Delete key after expiration (seconds)
rotate_after	integer	no	Rotate key after expiration (seconds)

As with other request methods, empty fields (those containing a zero, an empty string, or other "falsey" values) are omitted.

For listing the contents of a key ring, the returned JSON schema is an array containing Key types. Composite keys are currently not handled separately as there is currently no way to differentiate them from ordinary keys without persisting additional data with the key. Further, composite keys may be stored with names identical to stand alone keys:

Key rotation currently defines the following HTTP status codes in its returned responses:

Code	Condition or Description
401	If the Authorization: Bearer token was absent
404	If the namespace or key ring could not be found or keys could not be listed
500	If the key ring could not be rotated or the response could not be encoded

Key or Key Ring Deletion

Key or key ring deletion can be accomplished through distinct endpoints, each with greater granularity. This provides clients with a way of either a) submitting the entire POST data with no indication via the request URI which key or key ring is going to be deleted or b) increasing specificity for visually recognizing which key or key ring is being deleted. All of the endpoints specified below achieve the same outcome and require the same request body contents.

• /keyring/
• /{namespace:string}/keyring/
• /keyring/{keyring:string}/
• /{namespace:string}/keyring/{keyring:string}/
• /keyring/{keyring:string}/{key:string}
• /{namespace:string}/keyring/{keyring:string}/{key:string}

As with all other endpoints, each URI path may be prefixed with /global for specifying the global namespace. Omitting this prefix is allowable and synonymous.

Bear in mind that unlike other actions in the KeyStar HTTP API, each of the endpoints above both require the same request body and are handled by the same code. The only difference is immediate visual recognition of intent; therefore, it is advisable to either use the endpoint that reflects the closest "intent to delete" or use the bare endpoint and carefully commented JSON data to communicate intent.

POST requests, as with all other actions, must contain the headers Content-Type: application/json or Content-Type: text/json to be correctly parsed. application/json is preferred. if the Content-Type header is omitted, the server will return a 400 BAD REQUEST.

However, unlike other actions, DELETE does not accept query string values for specifying the type of key to delete. Instead, composite keys must be specified via the post body.

DELETE accepts the following JSON schema. Either the key ring name or the key ring name and the key name must be specified, depending on which is requested for deletion. If these values are also specified in the request URI, both must match.

Field name	Type	Required	Description
keyring	string	yes	Key ring to delete.
key	string	yes*	Key to delete.
type	string	yes*	"Composite" if deleting composite keys or empty.

Required values marked with an asterisk (*) are only required if the intent is to delete a key or a specific type of key. In particular, the key value is also required if the requested endpoint also contains the key.

At present the only valid values accepted by type arer composite and key. If type is not specified or is empty, the value is assumed to be key. Any other value will result in a 400 BAD REQUEST status code.

On success, DELETE returns the simplest type of all the endpoints: {"status":"ok"}. No other information is provided.

This may change in future iterations to include the name(s) of the key ring or keys that were deleted.

DELETE defines the following HTTP status codes on failure.

Code	Condition or description
400	If the request could not be parsed or a required value was absent.
404	If an invalid namespace, key ring, or key were specified.
500	If the server is unable to remove the key or key ring.

Templates

Templates provide a means of generating a default set of keys and key rings for KeyStar. This is useful for applications that need a list of specific predetermined keys after, for example, installation or their initial setup. Templates can be used to quickly established a known set of key rings and their keys for other uses as well and can be used to generate composite keys.

The template system exposes the following endpoints:

• /template/
• /template/{namespace:string}

As with most other endpoints, namespace may be specified to restrict the key ring and key creation to a specific namespace. Otherwise, the global namespace will be used instead. Likewise, both endpoints may be prefixed with /global to avoid confusion with other namespace categories, but its inclusion is not required.

Be aware that POSTing to the template endpoints is not idempotent and returns a different response depending on its internal state. Keys that do not exist, as described by the template, will be created; all others will be ignored. Responses from the endpoint will include a list of keys and their state.

There are two schemas that comprise a template request body: The outer definition and the per-key descriptor. The outer definition is defined as follows:

Field	Type	Required	Description
version	integer	yes	Template version identifier. Currently 1.
template	array	yes	Key descriptors. See below.

The key descriptor is as follows, per key:

Field	Type	Required	Description
keyring	string	yes	Key ring name.
ttl	integer	no	Time to live before expiration.
keys	array	no	Key definition. Not required when creating key rings. See below.

And finally, key definitions:

Field	Type	Required	Description
name	string	yes	Key name.
length	integer	yes*	Length in bytes. Required for non-composite keys.
cipher	integer	yes*	Cipher key length. Required for composite keys.
hmac	integer	yes*	HMAC key length. Required for composite keys.
composite	boolean	no	true for composite keys; false or absent otherwise.

The above schemas are information dense and contain cross-references. Therefore an example may be more helpful.

In the following illustration, we create an empty key ring tokens; a key ring application containing two keys, one which is a composite key, and another that is a stand alone key; and finally, we create another key ring named users that contains three separate keys.

{"version": 1, "template": [
  {"keyring": "tokens", "ttl": 86400},
  {"keyring": "application", "keys": [
    {"name": "auth", "cipher": 32, "hmac": 128, "composite": true},
    {"name": "download", "length": 128},
  ]},
  {"keyring": "users", "ttl": 86400, "keys": [
    {"name": "user1", "length": 32},
    {"name": "user2", "length": 32},
    {"name": "user3", "length": 32}
  ]}
]}

Template requests return a schema that communicates whether or not the request failed for a given key ring or key. Responses contain an array where each entry is defined as:

Field	Type	Required	Description
keyring	string	yes	Key ring name.
key	string	no	Key name. Only provided if defined in the request.
state	string	yes	Key state; one of ok, exists, or failed.

Template requests may return one of the following HTTP status codes:

Code	Condition or description
400	If the request body could not be parsed or an invalid version was specified
401	If the Authorization: Bearer header was absent.
404	If the requested namespace (when submitted to the namespaced endpoint) does not exist
500	If the server cannot create the key ring, key, or the response could not be encoded

Utility API

The utility API provides convenience interfaces for generating keys, randomness, and a handful of other helpful tools. To provide additional protections against unwanted access, all of these endpoints require authentication via the Authorization: Bearer header. None of these endpoints persist submitted data or generated responses. In the case of the utility key generation functions, the submission semantics are similar to those used by PUT and POST.

Endpoints

The following endpoints are defined by the utility API:

• /generate/bytes
• /generate/key
• /generate/composite-key
• /generate/signature

We'll discuss each of these endpoints in sequence.

Byte Generation

The byte generation endpoint /generate/bytes accepts only GET requests and requires the query string variable count to be present. If count is absent or contains invalid values, this endpoint will return a 400 BAD REQUEST.

Byte generation returns a simple JSON structure containing the following schema:

Field	Type	Description
bytes	string	Base64-encoded random bytes

Future implementations may provide options for changing the encoding type.

No more than 65536 bytes may be requested at a time.

This endpoint may be configured by administrators to rate limit requests.

Key Generation

Key generation offers semantics similar to PUT and POST with the exception that none of the generated keys are stored. These endpoints are mostly useful for testing clients or parsers and provide additional features for manipulating fields that aren't present with the key management API.

These endpoints accept JSON schemas identical to those of PUT and POST in the key management API, but the returned values may be manipulated with the addition of query string variables as outlined below.

For standard key creation, the submitted JSON object must adhere to this structure:

Field name	Type	Required	Description
length	integer	yes	Requested key length in bytes
name	string	no	Optional key name to return in data
ttl	integer	no	Time to live in seconds; expires when exhausted
delete_after	integer	no	Delete key after expiration (seconds)
rotate_after	integer	no	Rotate key after expiration (seconds)

For composite key creation, the submitted JSON must be as follows:

Field name	Type	Required	Description
cipher_length	integer	yes	Cipher key length in bytes
hmac_length	integer	yes	HMAC key length in bytes
name	string	no	Optional key name to return in data
ttl	integer	no	Time to live in seconds; expires when exhausted
delete_after	integer	no	Delete key after expiration (seconds)
rotate_after	integer	no	Rotate key after expiration (seconds)

Note the absence of key and key ring names.

To control key behavior, the following query string variables may be set:

Field name	Type	Description
created	string	Sets the creation date to the specified value.
count	integer	Generates count keys using the submitted data as a template.
randomize	boolean	Randomize fields in returned data. Requires count > 0.

Both key generation endpoints accept either POST or PUT and behave the same accordingly. As with the key management API, fields that are set to a zero (or empty) value are omitted from the returned data.

Pay particular attention to the count and randomize query values. If count is set to a value greater than 0 and randomize is true, all key data returned will contain random values regardless of the request body content. Otherwise, count will merely return similar copies of the requested key, randomizing only the encoded data.

count cannot be larger than 10 and key lengths cannot exceed 65536.

This endpoint may be configured by administrators to rate limit requests.

Signature Generation

Signature generation provides a means of signing input data with the authenticated user's secret key. Other keys may be specified as part of the request, but the primary use case for this endpoint is to provide a means of testing client signatures.

This endpoint only accepts POST requests and may be limited to a request body of no more than 10 megabytes. For larger files, consider signing only the hash.

The POST request body must contain the JSON-formatted schema:

Field name	Type	Required	Description
data	string	yes	Source data to sign. Must be a string or base64-encoded.
type	string	no	Data type; one of "string" or "base64." Default: "base64."
keyring	string	no	Key ring name if using a key other than the authenticated user's secret key
key	string	no	Key name if using a key other than the authenticated user's secret key
algorithm	string	no	Signature algorithm. One of "sha512", "sha512_224", "sha512_256", "sha224", or "sha256".

Responses from this endpoint will contain the following JSON-formatted schema:

Field name	Type	Description
signature	string	Signature derived from submitted data
algorithm	string	Algorithm used to generate signature

Cryptographic Utilities

KeyStar provides an assortment of cryptographic utilities. These interface directly with KeyStar's key types (primarily composite keys for convenience) and provide features such as encrypt-then-MAC and time-based token generation. Wrappers such as these may be used for encrypting cookies or user session data; in the case of time-based token generation, this may be useful for generating password reset tokens and similar without littering the host application database with ephemeral data. As a bonus example, KeyStar's key namespacing features allow applications to create isolated key stores specifically for generating user tokens that automatically invalidate via key rotation, rendering them completely unusable in the event an adversary attempts to attack the token signature to control its contents.

[^1]: This needs to be a word.

[^2]: A 500-level error may be more appropriate for this circumstance, but we elected to use 404 instead since it's otherwise impossible to differentiate to the client the nature of the error. 404 responses to PUT requests may require administrators to enable the debug log to determine the nature of the error. This response type may be changed in a future iteration to something more appropriate.

[^3]: As with the previous footnote, a 500-level error may be more appropriate for this circumstance. Although it may make little sense from the perspective of a pure-RESTful API, at least it's consistent!