prom alertmanager setup

alerts-alertmanager | alerts-vmalert | alerts-acceptance

requirements

prepare some webhook e.g. for slack

docker setup

let’s first get alert mgmt up and running

cd ~/dev/
vi alertmanager.yml

global:
  slack_api_url: https://hooks.slack.com/services/...
  resolve_timeout: 1s

route:
  # match all
  receiver: 'slack'

  # no alert grouping for now
  #group_by:
  #  - alertname
  #group_interval: 5m
  #group_wait: 30s
  #repeat_interval: 12h

receivers:
- name: 'slack'
  slack_configs:
  - send_resolved: true
    channel: '#alarms'
    # keep default title
    # https://github.com/prometheus/alertmanager/blob/main/template/default.tmpl#L4
    text: >-
     {{ range .Alerts -}}
     {{ .Annotations.dashboard }}
     {{ end }}

ready to go

docker rm -f alertmanager
docker run --name alertmanager -d \
    -p 9093:9093 \
    -v $HOME/dev/alertmanager.yml:/etc/alertmanager/alertmanager.yml \
    quay.io/prometheus/alertmanager
docker logs alertmanager
netstat -lntup | grep :9093

setup a tunnel and reach the Web UI

ssh THAT-DOCKER-HOST -l USER -L 9093:localhost:9093

http://localhost:9093/#/alerts

usage

watch -n1 docker logs alertmanager --tail=10

reload

docker restart alertmanager

resources

https://github.com/prometheus/alertmanager

https://prometheus.io/docs/alerting/latest/alertmanager/

https://prometheus.io/docs/alerting/latest/configuration/

https://prometheus.io/docs/alerting/latest/notification_examples/