ism index patterns based on indexAlias/index #5117

[kranthikirang]

Description

We will be able to update the index_patterns of the ISM policy based on the indexAlias created (more useful when its dynamic and/or with rollover feature) when user supplies ism_policy_file. This will not interfere when user wants to manage the policy with superset of wildcard (which should be a manual step). This commit will only ensure policy's index_patterns are updated when user supplies policy_json_file.

Example:
route:
- cluster-utils: '/route_name == "cluster"'
sink:

• opensearch:
  hosts: [ "https://opensearch-node1:9200","https://opensearch-node2:9200" ]

  Change to your credentials

  username: "admin"
  password: "PGjGcHKfQW1U"
  insecure: true
  dlq_file: /usr/share/data-prepper/log/data-prepper/dlq.log

  Add a certificate file if you are accessing an OpenSearch cluster with a self-signed certificate

  #cert: /path/to/cert
  normalize_index: true
  index: cluster-utils_eastus2_${cluster_name}
  index_type: custom
  template_type: v1
  template_file: /usr/share/data-prepper/templates/template-file.json
  ism_policy_file: /usr/share/data-prepper/policies/ism-policy.json
  routes: [cluster-utils]

Where using data prepper we will create ism policy (if user supplies wildcard cluster-utils*) each time we receive a different cluster_name and will fail because opensearch will not allow having same patterns in multiple policies. Also keeping a single policy wouldn't make sense if we want to manage via AP. with this change data-prepper will create new policy with index pattern as cluster-utils_eastus2_${cluster_name}* which will in-align to index template index_patterns. Also, previous functionality will also work as index_patterns are specific to the indexAlias. However users will have to re-run the policies on the older indexes created.

Extremely useful with rollover through ISM policy.

Issues Resolved

Resolves #5117

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[kranthikirang]

Can anyone look into this PR please? @dlvenable

[dlvenable]

Thank you @kranthikirang for this contribution. I have a few questions and then some specific comments.

What was the ISM policy this created for your index?

Did you try with any variables that are not at the end?