If you ever worried "What would happen if my monitoring/alerting systems fail?" then search no further, you're at the right place

Key Features • How To Use • Credits • License

Quick description

DeepSentinel is a low-level, server-agent software that monitors crucial machine and service states in a scalable and concurrent way that can process incoming data in real-time, track the health status of various services on multiple machines, and trigger alerts based on the absence of expected signals within given time intervals.

DeepSentinel is thought and coded to fit very niche use-cases where a reliability risk affects your monitoring and alerting systems ; don't expect it to replace your Datadog or Grafana cloud instances!

Key Features

• server runs fully in-ram with a low ressource footprint
  • Can be hosted on a high SLA serverless provider
  • Doesn't need disk access (but appreciate it to persist the auth-token)
• agent pushes simple JSON payloads via HTTP/S as an alive signal
• Both server and agent are monitoring themselves for any fatal error
• agent daemonize itself and runs no matter what
  • You only need to configure server address, machine name and auth token
  • No actions required on the server-side
• agent daemon is live configurable and live unregisterable

How To Use

Install Server

Via Docker

On any instance that has Docker installed you can run the following commands :

docker run \
    -v /your/host/path:/etc/deepsentinel \
    -p <host_port>:5000 \
    -e DEEPSENTINEL_ADDRESS=0.0.0.0 \
    -e DEEPSENTINEL_PORT=5000 \
    -e DEEPSENTINEL_PROBE_INACTIVITY_DELAY=10s \
    -e DEEPSENTINEL_DEGRADED_TO_FAILED=20 \
    -e DEEPSENTINEL_FAILED_TO_ALERT_LOW=30 \
    -e DEEPSENTINEL_ALERT_LOW_TO_ALERT_HIGH=50 \
    -e DEEPSENTINEL_LOGGING_LEVEL=info \
    -e DEEPSENTINEL_LOW_ALERT_PROVIDER=pagerduty \
    -e DEEPSENTINEL_HIGH_ALERT_PROVIDER=pagerduty \
    -e DEEPSENTINEL_PAGERDUTY_API_KEY=<pd_api_key> \
    -e DEEPSENTINEL_PAGERDUTY_INTEGRATION_KEY=<pd_integration_key> \
    -e DEEPSENTINEL_PAGERDUTY_INTEGRATION_URL=<pd_integration_url> \
    ghcr.io/equals215/deepsentinel-server:latest

Replace latest by any tag you want to run. Docker Image tag == Repo tag.
Adapt environment variables based on the config you want to apply.

You will have to grab the auth token from either the logs or the config file bind mounted to your host

Via binaries

1. Get the server binary that matches your system under the Release section.

• With wget :

wget https://github.com/equals215/deepsentinel/releases/download/v0.0.3-untested/deepsentinel-server-linux-amd64 -O deepsentinel-server && \
chmod +x deepsentinel-server

• With curl :

curl -o deepsentinel-server https://github.com/equals215/deepsentinel/releases/download/v0.0.3-untested/deepsentinel-server-linux-amd64 && \
chmod +x deepsentinel-server

1. Now you have to configure the server :

Create the config file :

mkdir -p /etc/deepsentinel/
nano /etc/deepsentinel/server-config.json

{
  "address": "0.0.0.0",
  "alertlow-to-alerthigh": 30,
  "degraded-to-failed": 10,
  "failed-to-alertlow": 20,
  "high-alert-provider": "pagerduty",
  "logging-level": "info",
  "low-alert-provider": "",
  "no-alert": false,
  "pagerduty": {
    "api-key": "...",
    "integration-key": "...",
    "integration-url": "..."
  },
  "port": "5000",
  "probe-inactivity-delay": "5s"
}

1. Now that you generated the configuration you can daemonize it if your system supports systemd or launchd :

./deepsentinel-server daemon install

1. The server should now be able to accept incoming connections. Remember to grab the auth token from either the logs or the config file at /etc/deepsentinel/server-config.json — you will need it to configure agents.

Install Agent

As the agent is supposed to be run as close to the system as possible, it's not a good practice to run it inside a Docker container, hence why there is not Docker container for it 🤠

1. Get the agent binary that matches your system under the Release section.

• With wget :

wget https://github.com/equals215/deepsentinel/releases/download/v0.0.3-untested/deepsentinel-agent-linux-amd64 -O deepsentinel-agent && \
chmod +x deepsentinel-agent

• With curl :

curl -o deepsentinel-agent https://github.com/equals215/deepsentinel/releases/download/v0.0.3-untested/deepsentinel-agent-linux-amd64 && \
chmod +x deepsentinel-agent

1. Now you have to configure the agent using the install command :

The following command will ask you for auth-token, server-address and machine-name in order to set up properly. It will then generate it's config file located in /etc/deepsentinel/agent-config.json and daemonize itself if your system supports systemd or launchd

sudo ./deepsentinel-agent install

1. Your agent should now be sending alive signals to the server. Check the server's logs to ensure that everything is setup properly.

Credits

TBD

License

GNU-GPLv3

GitHub @equals215  ·  LinkedIn Thomas Foubert