setup audit-trails at yandex cloud

architecture

here’s the expected data flow

trail ==> yandex data-stream ==> YDB ==> data transfer ==> opensearch

note there’s retention enabled by default with YDB

deployment steps

create data stream to receive the trails
create the trail
prepare opensearch mapping
setup data transfer

data stream

YDB

name trail-ydb
and defaults

stream

name audit-trails
and defaults

trail

name trail
data stream     audit-trails
service account     ACCOUNT-HERE (with role yds.editor)
and defaults

opensearch mapping

just a casual index template (NOT DATA-STREAM)

audit-trails*

with settings e.g.

{
  "index.mapping.total_fields.limit": "2000",
  "index.number_of_shards": "3",
  "index.number_of_replicas": "0",
  "index.plugins.index_state_management.rollover_alias": "audit-trails-rolloverfix"
}

with mapping – (not sure about dynamic and numeric detection, it may cause troubles if some fields have HEX codes that would show up as numeric first)

{
  "numeric_detection": true,
  "dynamic": "true",
  "properties": {
    "details": {
      "type": "flat_object"
    }
  }
}

data transfer

endpoint src

Source
YDB - trails-ydb

endpoint dst

Target
opensearch
no cleanup
sanitize

transfer

name        trail
type        replication
(defaults)
src     endpoint src
dst     endpoint dst

acceptance

check that data is arriving in the index (not data-stream).

once everything is fine, eventually include audit-trails in a mgmt policy.

troubleshooting

Missing alias or not the write index when rollover

==> remove alias AND/OR add index.plugins.index_state_management.rollover_alias (not sure if both are necessary)

resources

https://cloud.yandex.ru/ru/docs/data-transfer/tutorials/trails-to-os ==> YDB to opensearch / RU

https://cloud.yandex.ru/ru/docs/managed-opensearch/tutorials/trails-to-opensearch ==> data streams to opensearch / RU

https://cloud.yandex.com/en/docs/managed-opensearch/tutorials/trails-to-opensearch ==> idem / EN

https://cloud.yandex.com/en/docs/data-transfer/troubleshooting/#opensearch ==> data transfer troubles

rollover fixup

https://repost.aws/knowledge-center/opensearch-failed-rollover-index

https://opensearch.org/docs/latest/dashboards/im-dashboards/rollover/

https://forum.opensearch.org/t/rollover-alias-question/2959/6

https://forum.opensearch.org/t/ism-policies-not-getting-applied/9258

https://opensearch.org/docs/latest/im-plugin/ism/policies/#actions

https://forum.opensearch.org/t/missing-alias-or-not-the-write-index-when-rollover/8092