[BUG] Index template_file and template_content truncates index pattern incorrectly

[graytaylor0]

Describe the bug

Index patterns are being created with a truncated index pattern from the index template index patterns passed in the config

This bug occurs when there is no ism_policy_file passed to the source. This is here in the code (

data-prepper/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/IndexManagerFactory.java

Line 103 in 463fa45

this.ismPolicyManagementStrategy = new NoIsmPolicyManagement(openSearchClient, restHighLevelClient);

) where the decision to use NoIsmPolicyManagement class is made. This class has the following method which does not add a pattern back (

data-prepper/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/NoIsmPolicyManagement.java

Line 43 in 463fa45

return Collections.singletonList(indexAlias);

), which is called here (after removing the pattern suffix) (

data-prepper/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/AbstractIndexManager.java

Line 230 in 463fa45

final List<String> indexPatterns = ismPolicyManagementStrategy.getIndexPatterns(indexPrefixWithoutTrailingDash);

). With an ism policy file, this method will be called instead, which adds back the pattern (

data-prepper/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/IsmPolicyManagement.java

Line 145 in 463fa45

return Collections.singletonList(indexAlias + "-*");

)

To Reproduce
Steps to reproduce the behavior:

1. Create a pipeline with the following opensearch sink configuration

sink:
  - opensearch:
       index: "test-index-1"
       template_type: "index-template"
       template_content: |
          {
            "index_patterns": [
              "test-*"
            ],
            "template": {
              "aliases": {
                  "my_test_logs": {}
              },
              "settings": {
                  "number_of_shards": 5,
                  "number_of_replicas": 2,
                  "refresh_interval": -1
              },
              "mappings": {
                "properties": {
                  "timestamp": {
                    "type": "date",
                    "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
                  },
                  "value": {
                    "type": "double"
                  }
                }
              }
            }
          }

2. Run data prepper and then go to OpenSearch Dashboards to look at the created index template

GET _index_template/test-index-template

You will see that the index_pattern in the template is only test and not test-*

Expected behavior
To not truncate the index pattern

Additional context

[LHozzan]

Maybe same here.