alerts-alertmanager | alerts-vmalert | alerts-acceptance
prepare some webhook e.g. for slack
let’s first get alert mgmt up and running
cd ~/dev/ vi alertmanager.yml
global: slack_api_url: https://hooks.slack.com/services/... resolve_timeout: 1s route: # match all receiver: 'slack' # no alert grouping for now #group_by: # - alertname #group_interval: 5m #group_wait: 30s #repeat_interval: 12h receivers: - name: 'slack' slack_configs: - send_resolved: true channel: '#alarms' # keep default title # https://github.com/prometheus/alertmanager/blob/main/template/default.tmpl#L4 text: >- {{ range .Alerts -}} {{ .Annotations.dashboard }} {{ end }}
docker rm -f alertmanager docker run --name alertmanager -d \ -p 9093:9093 \ -v $HOME/dev/alertmanager.yml:/etc/alertmanager/alertmanager.yml \ quay.io/prometheus/alertmanager docker logs alertmanager netstat -lntup | grep :9093
setup a tunnel and reach the Web UI
ssh THAT-DOCKER-HOST -l USER -L 9093:localhost:9093 http://localhost:9093/#/alerts
watch -n1 docker logs alertmanager --tail=10
reload
docker restart alertmanager
https://github.com/prometheus/alertmanager
https://prometheus.io/docs/alerting/latest/alertmanager/
https://prometheus.io/docs/alerting/latest/configuration/
https://prometheus.io/docs/alerting/latest/notification_examples/