calert

Send Alertmanager notifications to Google Chat (and more!)

calert uses Alertmanager webhook receiver to receive alerts payload, and pushes this data to Google Chat webhook endpoint.

Grab the latest release from Releases.

To run:

./calert.bin --config config.toml

You can find the list of docker images here

docker pull ghcr.io/mr-karan/calert:latest

latest is the default Ubuntu 24.04 image. It includes a shell, CA certificates, and timezone data for production debugging. latest-scratch contains the same static binary, certificates, and timezone data in a smaller scratch image, but has no shell or package tools.

Here's an example docker-compose config with a custom message.tmpl mounted inside the container:

  calert:
    image: ghcr.io/mr-karan/calert:latest
    ports:
      - "6000:6000"
    volumes:
      - ./message.tmpl:/etc/calert/message.tmpl

Here's an example docker-compose config with a custom config.toml mounted inside the container:

  calert:
    image: ghcr.io/mr-karan/calert:latest
    ports:
      - "6000:6000"
    volumes:
      - ./config.toml:/app/config.sample.toml

Refer to config.sample.toml for instructions on how to configure calert.

All the config variables can also be supplied as Environment Variables by prefixing CALERT_ and replacing . (period) with __ (double underscores).

Example:

• app.address would become CALERT_APP__ADDRESS

calert can load a map of different providers. The unique identifier for the provider is the room name. Each provider has it's own configuration, based on it's provider_type. Currently calert supports Google Chat but can support arbitary providers as well.

calert supports Go templates for formatting alert messages. Templates have access to all alert fields and several helper functions.

For rich formatting with colors and structured layouts, use Google Chat's CardsV2 format by defining a cardsV2 template block:

{{- define "cardsV2" -}}
{
  "cardId": {{ json (printf "alert-%s" .Fingerprint) }},
  "card": {
    "header": {
      "title": {{ json .Labels.alertname }},
      "subtitle": {{ json (.Status | Title) }}
    },
    "sections": [{
      "widgets": [{
        "decoratedText": {
          "text": {{ json .Annotations.description }}
        }
      }]
    }]
  }
}
{{- end -}}

Google Chat's simple text webhook supports limited formatting:

• Bold: *text*
• Italic: _text_
• Strikethrough: ~text~
• Monospace: `text`

Note: HTML tags (like <font color="...">) and standard emoji shortcodes (:warning:) are not supported in simple text messages. For colors and rich formatting, use CardsV2 templates instead.

• Alertmanager has the ability of group similar alerts together and fire only one event, clubbing all the alerts data into one event. calert leverages this and sends all alerts in one message by looping over the alerts and passing data in the template. You can configure the rules for grouping the alerts in alertmanager.yml config. You can read more about it here.

• Configure Alertmanager config file (alertmanager.yml) and give the address of calert web-server. You can refer to the official documentation for more details.

You can refer to the following config block to route webhook alerts to calert:

route:
    receiver: 'calert'
    group_wait: 30s
    group_interval: 60s
    repeat_interval: 15m
    group_by: ['room', 'alertName']

receivers:
    - name: 'calert'
      webhook_configs:
      - url: 'http://calert:6000/dispatch'

Alertmanager's repeat_interval controls how often alerts are re-sent while still firing. Deterministic thread keys keep the same alert lifecycle in the same thread and do not rotate when thread_ttl elapses. In group mode, thread_ttl controls only retention of deduplication and resolved-status state.

If you want fewer repeated messages:

• Increase repeat_interval in Alertmanager config
• Increase thread_ttl in group mode only when deduplication and resolved-status state should be retained longer

When using Kubernetes AlertmanagerConfig CRD, the receiver name is automatically prefixed with namespace/config-name/receiver. Use the room_name query parameter to override:

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: my-config
  namespace: monitoring
spec:
  receivers:
    - name: prod_alerts
      webhookConfigs:
        - url: http://calert:6000/dispatch?room_name=prod_alerts

calert ships with a basic support for sending multiple related alerts under a same thread, working around the limitations by Alertmanager.

Alertmanager currently doesn't send any Unique Identifier for each Alert. The use-case of sending related alerts under the same thread is helpful to triage similar alerts and see all their different states (Firing, Resolved) for people consuming these alerts. calert tries to solve this by:

• Use the alert fingerprint and startsAt fields to derive a deterministic thread key.
• Firing and resolved updates for the same alert lifecycle share a thread across restarts and replicas.
• A refire with a new startsAt receives a new thread key.

The above is per-alert threading (threading_mode = "alert", the default). calert also supports threading one whole Alertmanager group into a single thread with an aggregated message, and sharing dedup state across active-active instances via Redis. See docs/group-threading.md for setup.

calert exposes various metrics in the Prometheus exposition format.

Here's a list of internal app metrics available at /metrics:

It also exposes Go process metrics in addition to app metrics, which you can use to monitor the performance of calert.

calert exposes an health endpoint at /ping.

A few notes on v2 migration:

v2 is a complete rewrite from scratch and is a breaking release. The configuration has changed extensively. Please refer to latest config.sample.toml for a complete working example of the config.

In case you're simply experimenting with calert config changes and you don't wish to send actual notifications, you can set dry_run=true in each provider.

Apart from the config, calert now determines the room based on the receiver specified in Alertmanager config. Previously, the room was identified with ?room query parameter in each HTTP request. However, since the Alert payload contains the receiver name, it's better to extract this information from the labels instead.

Here's an example of how Alertmanager config looks like. Notice the value of receiver (prod_alerts) should match one of provider.<room_name> (eg provider.prod_alerts) in your config.toml):

receivers:
    - name: 'prod_alerts'
      webhook_configs:
      - url: 'http://calert:6000/dispatch'

PRs on Feature Requests, Bug fixes are welcome. Feel free to open an issue and have a discussion first.

For deployment manifests like Helm, Kustomize, Nomad etc - they're placed under contrib folder and generally manintained by the community.

LICENSE