minikube // ingress-nginx logs towards opensearch

warning

nginx sends syslog logs as RFC 3164

poc setup

infra

#minikube delete --all
minikube start --driver=docker
minikube addons enable metrics-server

app

kubectl create deployment web --image=gcr.io/google-samples/hello-app:1.0
kubectl expose deployment web --type=ClusterIP --port=8080
kubectl get deploy -o wide
kubectl get svc -o wide

ingress

minikube addons enable ingress
kubectl get pods -n ingress-nginx
vi ingress-logs.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-logs
spec:
  rules:
    - host: hello-world.info
      http:
    paths:
      - path: /
        pathType: Prefix
        backend:
          service:
        name: web
        port:
          number: 8080
  #ingressClassName: nginx

kubectl apply -f ingress-logs.yaml
kubectl get ingress
kubectl get svc -o wide
kubectl get endpoints

check namespace and configmap name

kubectl -n ingress-nginx get pods
kubectl -n ingress-nginx describe pods | grep configmap

==> namespace is ingress-nginx

==> configmap is ingress-nginx-controller

setup the custom configs

vi ingress-logs-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  enable-syslog: "true"
  syslog-host: "10.1.0.35"
  syslog-port: "5140"

kubectl apply -f ingress-logs-configmap.yaml

poc acceptance

are configs in place?

pod=`kubectl -n ingress-nginx get pods | grep ingress-nginx-controller | awk '{print $1}'`
kubectl -n ingress-nginx exec -ti $pod -- bash

grep syslog /etc/nginx/nginx.conf

make sure the log server is reachable (minikube routes to the outside)

ping -c1 10.1.0.35
^D

log server receives as RFC 3164

logger --udp --server 10.1.0.35 --port 5140 --rfc3164 -- test from minikube

finally, do the nginx logs arrive fine?

curl --resolve hello-world.info:80:192.168.49.2 -i http://hello-world.info/THIS-IS-A-TEST

==> notice facility and level/severity local7.info

system.local7.info: {"host":"ingress-nginx-controller-7799c6795f-d5gb2","ident":"nginx","message":"192.168.49.1 - - [30/Oct/2023:11:24:04 +0000] \"GET /THIS-IS-A-TEST HTTP/1.1\" 200 60 \"-\" \"curl/7.88.1\" 94 0.001 [default-web-8080] [] 10.244.0.3:8080 60 0.000 200 1cbe2cc4e464c25e41d89e293f9a8a9c"}

shooting troubles

[warn]: #0 failed to parse message data="<190>Oct 30 11:22:40 ingress-nginx-controller-7799c6795f-d5gb2 nginx: 192.168.49.1 - - [30/Oct/2023:11:22:40 +0000] \"GET /THIS-IS-A-TEST HTTP/1.1\" 200 60 \"-\" \"curl/7.88.1\" 94 0.001

[default-web-8080] [] 10.244.0.3:8080 60 0.001 200 d1629b463a8388fedbdbd26668b7373a"

==> switch to RFC 3164 on the syslog server

resources

https://kubernetes.github.io/ingress-nginx/examples/customization/custom-configuration/ ==> sample yaml

https://kubernetes.github.io/ingress-nginx/user-guide/basic-usage/

https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/log-format/

https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/

https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#syslog-host

https://kubernetes.github.io/ingress-nginx/user-guide/cli-arguments/ ==> --internal-logger-address

nginx

https://nginx.org/en/docs/syslog.html

https://getdocs.org/Nginx/docs/latest/syslog