vmetrics // alerts setup

install | vmetrics | metrics injection | dashboard and acceptance

DEPRECATED - we do NOT use vmetrics alert mgmt and prefer to do it ourselves

requirements

alert mgmt

let’s first get alert mgmt up and running

vi alertmanager.yml

global:
  slack_api_url: https://hooks.slack.com/services/...

route:
  receiver: 'slack-notifications'
  #group_by: [alertname]

receivers:
- name: 'slack-notifications'
  slack_configs:
  - send_resolved: true
    channel: '#team-devops-alerts'
    text: 'test - {{ .GroupLabels.alertname }}

chmod 600 alertmanager.yml

#mkdir -p /data/alertmanager/

docker rm -f alertmanager
docker run --name alertmanager -d \
    -p 9093:9093 \
    -v $HOME/alertmanager.yml:/etc/alertmanager/alertmanager.yml \
    quay.io/prometheus/alertmanager
    #-v /data/alertmanager:/alertmanager \
docker logs alertmanager
netstat -lntup | grep :9093

setup a tunnel and reach the Web UI

ssh minikube -L 9093:localhost:9093

http://localhost:9093/#/alerts

vmalert

vi alerts.yaml

groups:
- name: alerts
  rules:
  - alert: cpu usage hits the roof
    expr: log_metric_gauge_cpu_p > 95
    # :mean5m
    for: 5m

ready

vi /etc/rc.local

echo starting vmalert
nohup vmalert-prod -rule=/root/alerts.yaml -datasource.url=http://localhost:8428 \
    -notifier.showURL \
    -notifier.suppressDuplicateTargetErrors \
    -notifier.url http://10.1.0.20:9093 \
    > /var/log/vmalert.log &

#-notifier.blackhole

acceptance

read the logs on vmetrics / vmalert

tail -F /var/log/vmalert.log

read the logs on alert mgmt

watch docker logs alertmanager --tail=5

watch the metrics on vmetrics Web UI

ssh vmetrics -L 8428:localhost:8428

http://localhost:8428/vmui/

MetricsQL> log_metric_gauge_cpu_p

stress the shit out of a flb client host

stress --cpu 3

ops

vmalert

reload alerts.yaml

pgrep -a vmalert
kill -HUP `pgrep vmalert`

resources

alert mgmt

https://github.com/prometheus/alertmanager

https://prometheus.io/docs/alerting/latest/alertmanager/

https://prometheus.io/docs/alerting/latest/configuration/

https://prometheus.io/docs/alerting/latest/notification_examples/

vmalert

https://docs.victoriametrics.com/vmalert.html

https://docs.victoriametrics.com/guides/guide-vmanomaly-vmalert.html