setup fluentd log server

intro

fluentd can handle both, rfc3164 (default) and rfc5424.

install

apt install curl sudo
curl -fsSL https://toolbelt.treasuredata.com/sh/install-debian-bookworm-fluent-package5-lts.sh | sh

setup

cd /etc/fluent/
mv -i fluentd.conf fluentd.conf.dist
grep -vE '^$|^#' fluentd.conf.dist > fluentd.conf.clean
grep -vE '^$|^#' fluentd.conf.dist > fluentd.conf
vi fluentd.conf

enable syslog and don’t forget to setup an event for those

<source>
  @type syslog
  port 5140
  bind 0.0.0.0
  tag system
  <parse>
    message_format rfc5424
  </parse>
</source>

<match system.**>
  @type stdout
</match>

ready to go

systemctl restart fluentd.service
systemctl status fluentd.service
netstat -lntup | grep 514

acceptance

on the fluentd server

tail -F /var/log/fluent/fluentd.log

from another host on that subnet

nmap -sU -p 5140 10.1.0.35

==> open|filtered however you will see trace on the server side (fluentd.log)

logger --udp --server 10.1.0.35 --port 5140 -- test from minikube
# --rfc3164

==> no need to specify a tag from the client side, ident field is there anyways

resources

https://docs.fluentd.org/input/syslog

https://github.com/fluent/fluent-plugin-opensearch

install

https://docs.fluentd.org/installation

https://docs.fluentd.org/installation/install-by-deb

concepts

https://docs.fluentd.org/deployment/logging

setup

https://docs.fluentd.org/configuration/routing-examples

https://docs.fluentd.org/input/syslog

https://docs.fluentd.org/quickstart/life-of-a-fluentd-event

https://docs.fluentd.org/how-to-guides/parse-syslog

https://www.mezmo.com/learn-kubernetes/syslog-logging-with-fluentd-secure-logging-done-right

k8s

https://docs.fluentd.org/configuration/config-file

https://github.com/fluent-plugin-systemd/fluent-plugin-systemd/issues/1

https://stackoverflow.com/questions/43558042/permission-issue-when-reading-container-logs-with-fluentd-in-kubernetes

troubles

https://docs.fluentd.org/quickstart/faq ==> need to define match

alternatives

https://kube-logging.dev/docs/configuration/


FW https://medium.com/swlh/fluentd-simplified-eb5f19416e37

https://stackoverflow.com/questions/75039303/fluentd-problem-with-regex-while-parsing-log


HOME | GUIDES | LECTURES | LAB | SMTP HEALTH | HTML5 | CONTACT
Copyright © 2024 Pierre-Philipp Braun