kog

package module
v1.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 6, 2022 License: Apache-2.0 Imports: 45 Imported by: 0

README

Kog - process workflow engine (PWE) executor

A cross-platform CLI to execute a PWE job.

Execute Jobs

Kog supports executing a job either via docker directly or as a kubernetes job

  • --executor docker default.
    • Set in DOCKER_HOST environment variable to a valid docker host, for example tcp://localhost:2375
  • --executor k8s
    • This uses the KUBECONFIG env var and the current context to determine which cluster and namespace to use
Workflows

There are few workflows checked in under data folder. The simplest workflow to execute is data/test/success.yaml, which executes echo command in alpine container.

kog run -f https://gitlab.com/f5-pwe/kog/raw/master/data/test/success.yaml
kog run -f /local/path/to/workflow.json

The --file|-f flag works for both remote and local files encoded in either json or yaml

Workflow Action Container Specification
  1. Containers MUST have an entrypoint that reads the KOG_CONTEXT env var as a base64 encoded json serialized Context
  2. Containers MUST print to stdout a prefixed line that contains the json serialized ActionResult
    • eg KOG:{"result":"success","rc":0,"message":"auto success","context":{"action_name":"auto-success","correlation_id":"btzb6nwmekwj","run_id":"btzb6nwmengl","step_name":"start","task_name":"success test"}
    • the context from the ActionResult is passed into the next step in the workflow
  3. IF you support the docker executor type your Docker file MUST have an ENV KOG_ENV_CONTEXT=true
    • Example Dockerfile
    • This is just to tell the docker executor to not use stdin to pass the context, this is how it used to be done and I did not want to break backwards compatibility
Context

Every kog job and step has a Context that it is executed within. This context is passed to each workflow step, which can then use this context to configure details about how this step is configured. Each step returns a context that is used in the next step, so it is common to put results from this step into the context before returning it so the next step can use those details

  • --context /path/to/context.json

The --context flag works for both remote and local files encoded in either json or yaml

Environment Variables

There are three was to pass env vars to a workflow step

  1. --env VARIABLE=something
  2. --env-file /path/to/data.env this works for both remote and local files in the dotenv file format
  3. --import-current-env you SHOULD always use this with --ignore-env to exclude env vars you do NOT want passed to your workflow step

On docker the env variables are all passed in via --env and is thus inherently insecure, do NOT pass passwords, tokens, etc this way unless you are 100% running on a local docker daemon and are 100% sure the box is secure

On kubernetes all the env variables are saves as a kubernetes Secret then loaded into the job via the normal kubernetes methods. This is only remotely more secure then the docker method, so still do not pass passwords or tokens this way.

Volume Mounts

The docker executor supports mounting volumes into each step action, the symantics are identical to the docker --volume flag

Kubernetes does not yet support volume mounting. If you are interested in this feature feel free to do a MR

Logging

By default, Kog logs to console in logfmt format. It is possible to switch format to JSON by settings --log-formatter flag to json. The log settings will also be propagated to all spawned containers using passed context.

Remote Logging

To enable log aggregation, it is useful to set --remote-logger flag to syslog. You can also specify the remote logger address --remote-logger-addr 10.0.0.15:514 which is useful if you are passing your logs to logstash or some other log aggregation tool Enabling remote logger does not disable local logs, but adds a handler to ship logs to a remote location.

Status Notification

To enable step result notifications, set --notifier flag to either console, or https://url.example.com/kog/notifications This will enable sending a notification on every step start to keep track of a job progress. You can specify multiple notifiers for each run

HTTP Notifiers

passing a url to the notifier flag will enable kog to make an http POST request to that URL with the json serialized pwe.Job object as the body

Return Codes

By default the kog run command will use the job status to map to return codes, use --use-job-rc=false to always return 0 (except for issues with the kog command itself)

Upgrading and Switching to specific versions

As of v1.4.0 kog supports the upgrade and switch-to commands

  • kog upgrade to upgrade to the latest version
  • kog switch-to --version #.#.# to switch to a specific version
  • kog switch-to --list to list the latest 10 versions and show which version you are on.
$ kog switch-to --list
> 1.5.0
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0

Dev Environment Setup

  1. Install Go language on your system. You can use built-in package manager system and look for go or golang. For more advanced instruction, see official documentation.
  2. Create the folder for the project, this is important since Go expects packages to be under
mkdir -p ~/projects/f5-pwe
cd ~/projects//f5-pwe
  1. Clone this project
git clone https://gitlab.com/f5-pwe/kog
  1. Init the dependencies
cd kog
make init
  1. Compile and install kog into current directory
make build
  1. Verify you can run kog
./bin/kog-$GOOS-$GOARCH run -f https://gitlab.com/f5-pwe/kog/raw/master/data/test/success.yaml

Documentation

Overview

Copyright 2018 F5 Networks

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func DecodeResults 

func DecodeResults(result []byte, logger log.Logger) (actions.ActionResult, error)

func DecodeStepResults 

func DecodeStepResults(ctx variables.Context, logger log.Logger) (results []actions.ActionResult, err error)

func ExecuteStep 

func ExecuteStep(executor Executor, step workflow.WorkflowStep, job *pwe.Job, ctx variables.Context,
	logger log.Logger, notifierHooks *NotifyHooks, extraEnv map[string]string, volumes []string) actions.ActionResult

func FileOrURLReader added in v0.0.2 

func FileOrURLReader(taskFilename string, logger log.Logger) (reader io.Reader, err error)

func NewStepResult 

func NewStepResult(result actions.ActionStatus, message string) actions.ActionResult

func NextStep 

func NextStep(step workflow.WorkflowStep, code actions.ActionStatus) string

func ParseContext added in v0.0.2 

func ParseContext(data []byte) (ctx variables.Context, err error)

func ParseJob 

func ParseJob(data []byte) (job *pwe.Job, err error)

Types

type Executor added in v1.2.2 

type Executor interface {
	Prepare(job *pwe.Job) error
	CreateJob(inputData, correlationId, jobName, stepName, runId string, action actions.Action, labels map[string]string, volumes []string) error
	GetJob() Job
	AttachIO() error
	SendInput(inputData string) error
	ProcessOutput(resultText *string)
	Wait(timeout time.Duration) (int, error)
	Cleanup() error
}

func NewDockerExecutor added in v1.2.2 

func NewDockerExecutor(logger log.Logger) (e Executor, err error)

func Newk8sExecutor added in v1.2.2 

func Newk8sExecutor(logger log.Logger) (e Executor, err error)

type Job added in v1.2.2 

type Job interface {
	ID() string
	Input(input io.WriteCloser) io.WriteCloser
	Output(output io.ReadCloser) io.ReadCloser
}

type JobResult 

type JobResult struct {
	ID            string        `json:"correlation_id" yaml:"correlation_id"`
	Name          string        `json:"task_name" yaml:"task_name"`
	Status        pwe.JobStatus `json:"status" yaml:"status"`
	Message       string        `json:"message" yaml:"message"`
	NotifyStatus  string        `json:"notify_status" yaml:"notify_status"`
	NotifyMessage string        `json:"notify_message" yaml:"notify_message"`
}

func ExecuteJob 

func ExecuteJob(executor Executor, job *pwe.Job, logger log.Logger, notifierHooks *NotifyHooks, extraEnv map[string]string, volumes []string) (retval JobResult)

type NotifyHook added in v1.2.2 

type NotifyHook interface {
	FireNotification(job *pwe.Job) error
	Logger() log.Logger
}

func NewConsoleHook added in v1.2.2 

func NewConsoleHook(logger log.Logger) NotifyHook

func NewHTTPNotifyHook added in v1.2.2 

func NewHTTPNotifyHook(baseURL string, logger log.Logger) NotifyHook

type NotifyHooks added in v1.2.2 

type NotifyHooks []NotifyHook

func (*NotifyHooks) AddNotification added in v1.2.2 

func (h *NotifyHooks) AddNotification(hook NotifyHook)

func (NotifyHooks) FireNotification added in v1.2.2 

func (h NotifyHooks) FireNotification(job *pwe.Job) error

Source Files

View all Source files

Directories

Path Synopsis
cmd
kog

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.