Notifications

A rule decides when to alert. Notifications decide where the alert goes — Slack, email, PagerDuty, or a custom webhook. Both alerting engines in the stack can notify, but they use different delivery mechanisms, so you configure each one separately:

Alerts from	Delivered by	Configured in
OpenSearch rules (log rules / classic monitors)	Notification channels attached as trigger actions	OpenSearch Dashboards → Notifications, then referenced in the rule
Prometheus rules (metric rules, SLO burn-rate)	Alertmanager receivers + routing tree	docker-compose/alertmanager/alertmanager.template.yml

Alerts are stored even without a channel

Out of the box, every firing Prometheus alert is indexed into the alertmanager-alerts OpenSearch index by the default receiver, so you have a searchable history before you wire up any external channel. External delivery (Slack/email/PagerDuty) is what you add on top.

OpenSearch rules — notification channels

OpenSearch log rules (and classic monitors) notify through actions attached to their triggers. An action points at a notification channel managed by the OpenSearch Notifications plugin, which supports Slack, Microsoft Teams, Amazon Chime, email (SMTP/SES), Amazon SNS, and custom webhooks.

One rule, many channels

Actions fire per trigger, so a single rule can route different severities to different channels — for example, page on-call for critical and post to a Slack channel for warning.

Set up a channel and attach it:

1. In OpenSearch Dashboards, open Notifications and create a channel — pick the type (Slack, email, webhook, …) and provide its endpoint (for Slack, an incoming webhook URL).
2. Send a test message from the channel screen to confirm connectivity.
3. When you create or edit a log rule, open the trigger’s Notification actions section and choose Add another action.
4. Select the channel, and optionally customize the message with the trigger’s context (rule name, severity, results).

For channel management details, see the OpenSearch Notifications documentation.

Prometheus rules — Alertmanager receivers

Prometheus metric rules (and SLO burn-rate rules) are evaluated by the Cortex ruler and sent to Alertmanager, which groups, deduplicates, and routes them to receivers. The stack ships a working config at:

docker-compose/alertmanager/alertmanager.template.yml

It defines a routing tree, a default opensearch-webhook receiver that indexes alerts into OpenSearch, and placeholder receivers for Slack, email, and PagerDuty that you fill in with real credentials. Credentials for the OpenSearch receiver are injected from .env (OPENSEARCH_USER / OPENSEARCH_PASSWORD) at container startup.

View the live routing in Dashboards

You don’t have to read the YAML to see how alerts are currently routed. The Routing tab of the Unified Alerts View renders the active Alertmanager definition for the Prometheus data source — the route tree, grouping and timing, and every configured receiver — as a read-only view. Edit the template file to change it; use the Routing tab to confirm what’s in effect.

Add a Slack receiver

The template already includes a dummy-slack receiver — replace the placeholder api_url with a real Slack incoming webhook URL:

receivers:
  - name: 'slack-notifications'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/<workspace>/<channel>/<token>'
        channel: '#alerts'
        send_resolved: true
        title: '[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}'
        text: |
          {{ range .Alerts }}
          *Severity:* {{ .Labels.severity }}
          *Service:* {{ .Labels.service_name }}
          *Summary:* {{ .Annotations.summary }}
          {{ end }}

Then add a route so matching alerts reach it. Routes are evaluated top-down, first match wins:

route:
  receiver: 'opensearch-webhook'   # default catch-all
  routes:
    - match:
        severity: critical
      receiver: 'slack-notifications'

Add an email receiver

receivers:
  - name: 'email-notifications'
    email_configs:
      - to: 'oncall@example.com'
        from: 'alertmanager@observability-stack.local'
        smarthost: 'smtp.example.com:587'
        auth_username: 'alertmanager@example.com'
        auth_password: 'CHANGE_ME'
        require_tls: true
        send_resolved: true

Add a PagerDuty receiver

receivers:
  - name: 'pagerduty-notifications'
    pagerduty_configs:
      - routing_key: '<your-integration-routing-key>'
        send_resolved: true
        severity: '{{ .CommonLabels.severity }}'
        description: '{{ .CommonAnnotations.summary }}'

Apply and validate

1. Edit docker-compose/alertmanager/alertmanager.template.yml.

2. Validate the config before restarting:

   docker exec alertmanager amtool check-config /tmp/alertmanager.yml

3. Restart Alertmanager to pick up the change:

   docker compose restart alertmanager

4. Alertmanager’s own UI is available at http://localhost:9093 — use it to see active alerts, grouping, and any silences.

Suppressing noise

To stop a noisy Prometheus alert without deleting its rule, use an Alertmanager silence (in the :9093 UI or via amtool silence add), or route it to the built-in null receiver. The shipped config also uses inhibition rules to suppress warnings when a critical alert fires for the same service.

For the full receiver reference (webhook, OpsGenie, VictorOps, and more), see the Alertmanager configuration docs.

Verify delivery

After wiring a channel, confirm it works end to end:

• OpenSearch rules — trigger the condition (or use Run preview on the rule) and confirm the notification arrives; the alert also appears in Explore Alerts.

• Prometheus rules — check the target channel received the alert, and cross-check http://localhost:9093 for it in Alertmanager. Every alert is also queryable in OpenSearch:

  curl -sk -u admin:$OPENSEARCH_PASSWORD \
    "https://localhost:9200/alertmanager-alerts/_search?pretty"

If nothing arrives, see the alert troubleshooting checklist.

Related

• Unified Alerts View — create rules and triage the alerts these channels deliver.
• Routing — the read-only view of the Alertmanager route tree.
• SLOs — SLO burn-rate alerts route through Alertmanager like any other Prometheus rule.