setting up fluent-bit on k8s

parse ingress logs towards opensearch

warning

this manifest deploys as daemonset so every node gets one

get into the nodes which hold the ingress and check what files you wanna grab

docker exec -ti minikube bash

    #cd /var/log/containers/
    cd /var/log/pods/
ls -lF *ingress-nginx*/controller/
tail *ingress-nginx*/controller/0.og
    ^D

custom log format (optional)

see k8s-ingress-custom

setup

git clone https://github.com/fluent/helm-charts.git
cd helm-charts/charts/fluent-bit/
ls -lF values.yaml

grab and tune our jinja2 sample

cd ../
wget https://pub.nethence.com/logging/fluentbit/templates/flb-values.yaml.j2
vi flb-values.yaml.j2
cd fluent-bit/

proceed with the chart values generation

export log_host=...
export log_port=9200
export log_http_user=...
export log_http_passwd=...
export priclass=...

j2 -e log_host -e log_port -e log_http_user -e log_http_passwd -e priclass ../flb-values.yaml.j2 > flb-values.yaml

deploy the pods

helm template test-flb ./ --values=flb-values.yaml | grep image:
helm template test-flb ./ --values=flb-values.yaml | less # check indentation
# --debug

kubectx
kubens -c

helm template test-flb ./ --values=flb-values.yaml | grep namespace
# --namespace ...

#helm uninstall test-flb
helm install test-flb ./ --values=flb-values.yaml --dry-run
helm install test-flb ./ --values=flb-values.yaml
#helm upgrade test-flb ./ --values=flb-values.yaml

kubectl get ds
kubectl describe ds test-flb | less

watch -n1 "kubectl get pods | grep ^test-flb"
for pod in `kubectl get pods | grep ^test-flb | awk '{print $1}'`; do
    kubectl logs $pod --tail=1
done; unset pod

acceptance

see k8s-ingress-logs-acceptance

additional notes

helpful for debugging

get into the first listed test-flb pod

kubectl exec -ti `kubectl get pods | grep ^test-flb | head -1 | awk '{print $1}'` -- bash

troubleshooting

happens with elastic

object mapping for [host] tried to parse field [host] as object, but found a concrete value

==> rename field host to something else – apparently it’s already in use somehow

resources

https://docs.fluentbit.io/manual/installation/kubernetes

https://fluentbit.io/blog/2020/12/29/5-minute-guide-to-deploying-fluent-bit-on-kubernetes/

setting up fluent-bit on k8s

warning

custom log format (optional)

setup

acceptance

additional notes

helpful for debugging

troubleshooting

resources

inputs

parsers

filters

outputs

troubles

troubles - host object

diy

efk

moar

type

journald