setting up file-d

tested on debian12

warning // lessons learned

file-d wants valid ssl certs from the data nodes – you might use your own CA, tho

install

grab latest binary

    ver=0.29.0
wget https://github.com/ozontech/file.d/releases/download/v$ver/file.d_${ver}_linux_amd64.deb
    dpkg -i file.d_${ver}_linux_amd64.deb
apt-get -f install

requirements

you will need to root CA certificate from one of the data nodes

from your workstation

scp opensearch0:/etc/opensearch/root-ca.pem ./
chmod 444 root-ca.pem
scp root-ca.pem file-d:/etc/file.d/

back to the file-d node

make sure the datanode resolves according to the ssl SAN

vi /etc/hosts

10.1.0.30       node-0.example.com

ping -c1 node-0.example.com
nmap -p 9200 node-0.example.com

mwe pipeline setup

welcome pipeline outputs to elastic/osearch

# how many workers?
grep ^proc /proc/cpuinfo | tail -1

cd /etc/file.d/
cp -pi config.yaml config.yaml.dist
vi config.yaml

  welcome:
    input:
      type: file
      persistence_mode: async
      watching_dir: /tmp/
      filename_pattern: welcome.json
      offsets_file: /tmp/offsets-welcome.yaml
      offsets_op: continue
    actions:
      - type: set_time
        field: "@timestamp"
      - type: remove_fields
        fields: [time_iso8601]
    output:
      type: elasticsearch
      endpoints: ["https://node-0.example.com:9200"]
      ca_cert: /etc/file.d/root-ca.pem
      index_format: "nginx-test-access"
      username: admin
      password: PASSWORD
      connection_timeout: 1s
      workers_count: 3
      batch_op_type: create
      fatal_on_failed_insert: true

chmod 600 config*

acceptance

file.d --config=/etc/file.d/config.yaml

echo '{ "testfield": "test ok" }' >> /tmp/welcome.json

nginx pipeline setup

install nginx and setup json escapes

apt install nginx curl
cd /etc/nginx/
cp -pi nginx.conf nginx.conf.dist
vi nginx.conf

http {
    # https://pub.nethence.com/www/nginx
    log_format json escape=json '{'
        ...
    '}';

cd sites-available/
cp default default.dist
vi default

server {
    access_log /var/log/nginx/access.json json;

nginx pipeline (same output as above)

  nginx_access:
    input:
      type: file
      persistence_mode: async
      watching_dir: /var/log/nginx/
      filename_pattern: "*.json"
      offsets_file: /tmp/offsets-nginx.yaml
      offsets_op: continue
    actions:
      - type: set_time
        field: "@timestamp"
      - type: remove_fields
        fields: [time_iso8601]
      - type: rename
        override: false
        host: vhost
      - type: modify
        sensor: file.d@file-d

acceptance

    file.d --config=/etc/file.d/config.yaml

curl -sI localhost | head -1

ready to go

systemctl restart file.d
systemctl status file.d
systemctl enable file.d

additional notes

if you want to grab all the logs, tail from head

      offsets_op: reset

resources

https://github.com/ozontech/file.d/tags

https://github.com/ozontech/file.d/blob/master/docs/configuring.md https://ozontech.github.io/file.d/#/docs/configuring

https://github.com/ozontech/file.d/blob/master/docs/examples.md

https://ozontech.github.io/file.d/#/plugin/output/elasticsearch/README

actions

https://ozontech.github.io/file.d/#/plugin/action/set_time/README

https://ozontech.github.io/file.d/#/plugin/action/remove_fields/README