index/stream template mappings & field-types

logging | elk | fluentbit

advanced setup

see mappings

audithack-tpl   sshguard journal and custom audit-peers scripts

cloudflare-tpl  cloudflare instant logs incl. ASN
        4-byte ASNs --> ClientASN as integer

ingress-tpl ingress-nginx stdout and stderr

nginx-tpl   standalone nginx

suricata-tpl    eve json output
        enforcing text for the HEX fields (otherwise might be mistaken as numeric)
        date detection for flow.start and flow.end

notes

we use the detect numeric feature to avoid filling-in every text and numeric fields. all we have to do is to fix HEX fields and define

• IP
• GEO_POINT
• and eventually layer 4 port fields (short)

our samples are here: https://pub.nethence.com/bin/logging/

resources

elastic

https://www.elastic.co/guide/en/elasticsearch/reference/7.17/indices-get-template.html

https://www.elastic.co/guide/en/elasticsearch/reference/7.17/index-templates.html

https://www.elastic.co/guide/en/elasticsearch/reference/7.17/indices-get-settings.html

osearch

https://opensearch.org/docs/latest/field-types/

https://opensearch.org/docs/latest/field-types/supported-field-types/index/

https://opensearch.org/docs/latest/field-types/supported-field-types/numeric/

https://opensearch.org/docs/latest/field-types/supported-field-types/flat-object/

total fields limit

https://www.elastic.co/guide/en/elasticsearch/reference/7.17/mapping-settings-limit.html

https://stackoverflow.com/questions/55372330/what-does-limit-of-total-fields-1000-in-index-has-been-exceeded-means-in

cloudflare instant logs ASN

https://www.networkworld.com/article/760079/understanding-4-byte-autonomous-system-numbers.html