setup audit-trails at yandex cloud

architecture

here’s the expected data flow

YDB ==> yandex stream ==> data transfer ==> opensearch

note there’s retention enabled by default with YDB

deployment steps

create data stream to receive the trails
create the trail
prepare opensearch mapping
setup data transfer

YDB

name audit-trails

Tail & Stream

name audit-trails

(create data stream)
name audit-trails

data stream     audit-trails
service account     ACCOUNT-HERE with role yds.editor
mgmt events     V
    folder      test

opensearch mapping

create an index template, NOT A DATA STREAM

audit-trails-indextpl

pattern: audit-trails*

with settings e.g.

{
  "index.mapping.total_fields.limit": "2000",
  "index.number_of_shards": "3",
  "index.number_of_replicas": "0",
  "index.plugins.index_state_management.rollover_alias": "audit-trails-rolloverfix"
}

with mapping – (not sure about dynamic and numeric detection, it may cause troubles if some fields have HEX codes that would show up as numeric first)

{
  "numeric_detection": true,
  "dynamic": "true",
  "properties": {
    "details": {
      "type": "flat_object"
    }
  }
}

opensearch user role

role

name            audit-trails
cluster perm        unlimited
index           audit-trails*
index perm      unlimited

data transfer

endpoint src

name            audit-trails-src
database        yandex stream - audit-trails
service account     ACCOUNT-HERE with role yds.editor
conversion rules    audittrails.v1 parser

endpoint dst

name            audit-trails-dst
database        opensearch - opensearch
cleanup         none required
sanitize        true (not sure that's needed)

transfer

name        audit-trails-transfer
type        replication
src     endpoint src
dst     endpoint dst

acceptance

check that data is arriving in the index (not data-stream).

once everything is fine, eventually include audit-trails in a mgmt policy.

troubleshooting

Missing alias or not the write index when rollover

==> remove alias AND/OR add index.plugins.index_state_management.rollover_alias (not sure if both are necessary)

resources

https://cloud.yandex.ru/ru/docs/data-transfer/tutorials/trails-to-os ==> YDB to opensearch / RU

https://cloud.yandex.ru/ru/docs/managed-opensearch/tutorials/trails-to-opensearch ==> data streams to opensearch / RU

https://cloud.yandex.com/en/docs/managed-opensearch/tutorials/trails-to-opensearch ==> idem / EN

https://cloud.yandex.com/en/docs/data-transfer/troubleshooting/#opensearch ==> data transfer troubles

rollover fixup

https://repost.aws/knowledge-center/opensearch-failed-rollover-index

https://opensearch.org/docs/latest/dashboards/im-dashboards/rollover/

https://forum.opensearch.org/t/rollover-alias-question/2959/6

https://forum.opensearch.org/t/ism-policies-not-getting-applied/9258

https://opensearch.org/docs/latest/im-plugin/ism/policies/#actions

https://forum.opensearch.org/t/missing-alias-or-not-the-write-index-when-rollover/8092