setup fluent-bit

parse ingress logs towards opensearch

intro

this manifest deploys as daemonset so every node gets one

nmap -p 9200 10.1.0.30 # opensearch
nmap -p 9200 10.1.0.36 # elastic v7

preparation

get into the node that holds the ingress and check what files you wanna grab

docker exec -ti minikube bash

    cd /var/log/containers/
    ls -lF *ingress*
tail ingress-nginx-controller*log
    ^D

setup

git clone https://github.com/fluent/helm-charts.git
cd helm-charts/charts/fluent-bit/
vi values.yaml

tune a few things – note we switched to debug image – note we grab only ingress logs – note the parser name has inverted nginx - ingress naming – note the index target matches template hence will be created on-the-fly – note we have a containerd runtime hence using cri input parser

  repository: cr.fluentbit.io/fluent/fluent-bit
  tag: latest-debug

    [INPUT]
        name tail
        path /var/log/containers/ingress-nginx-controller*log
        #parser docker
        parser cri
        tag kube
        mem_buf_limit 5MB
        skip_long_lines on

    [FILTER]
        name parser
        match kube
        #key_name log
        key_name message
        parser k8s-nginx-ingress
        reserve_data true

    [OUTPUT]
        name stdout
        match kube

see flb-k8s-ingress for splitting stdout and stderr

for opensearch we would have

    [OUTPUT]
        Name opensearch
        Match kube.*
        Host 10.1.0.30
        port 9200
        tls on
        tls.verify off
    tls.debug 3
        http_user admin
        http_passwd PASSWORD
        index logs-dev-ingress
        suppress_type_name on
        #replace_dots on

as for elasticsearch we would have

    [FILTER]
        name modify
        match kube.*
        rename host vhost

    [OUTPUT]
        Name es
        Match kube.*
        Host 10.1.0.36
    #tls on
    #tls.verify off
        logstash_format off
        index logs-dev-ingress
        suppress_type_name on
        trace_error on

deploy

#helm uninstall fluent-bit
helm template fluent-bit . | grep image:
helm template fluent-bit . | less # check indentation
helm template fluent-bit . | grep namespace
helm template fluent-bit . --debug | less
helm install --dry-run fluent-bit .
helm install fluent-bit .
# --namespace ingress...
kubectl get ds
#kubectl describe ds fluent-bit | less

custom log format (optional)

see k8s-ingress-custom

acceptance

see k8s-ingress-logs-acceptance

shooting troubles

object mapping for [host] tried to parse field [host] as object, but found a concrete value

==> rename field host to something else – apparently it’s already in use somehow

resources

https://docs.fluentbit.io/manual/installation/kubernetes

https://fluentbit.io/blog/2020/12/29/5-minute-guide-to-deploying-fluent-bit-on-kubernetes/

setup fluent-bit

intro

preparation

setup

custom log format (optional)

acceptance

shooting troubles

resources

inputs

parsers

filters

outputs

troubles

troubles - host object

diy

efk

moar

type

journald