[BUG] Indices with large mappings can grind Opensearch Dashboards to a halt

[tronboto]

Describe the bug

A cluster that contains indices with large mappings, such as Filebeat or Metricbeat indices, can cause significant slowdown and unresponsiveness in Opensearch Dashboards.

To Reproduce

1. Set up a 2-node development cluster as per the docs here:

https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/#sample-docker-compose-file-for-development

Note that I updated the heap size to 5g for each node and I'm running this on a c8.m16 VM.

2. Export the index template for filebeat 7.12.1:

docker run docker.elastic.co/beats/filebeat-oss:7.12.1 filebeat export template --noilm > template.json

3. Run the following python script that creates the index template and 500 indices based on this template (requires elasticsearch==7.12.1 to be installed):

import json
import logging

from elasticsearch import Elasticsearch

logging.basicConfig(level=logging.INFO)

es = Elasticsearch()

# load template from disk
with open("template.json") as f:
    template = json.load(f)

# create template
es.indices.put_template("filebeat-7.12.1-template", template)

# create indices
for i in range(1, 601):
    es.indices.create(f"filebeat-7.12.1-{i:06d}")

4. Open Dev Tools and type:

GET _cat/health

Press Ctrl-<enter> around 50 times consecutively to run the requests in quick succession.

5. Try opening Discover in a new tab

If things still seem ok, try doing step 4 again.

Expected behavior

• I expect those _cat/health requests to return very quickly and not take tens of seconds and sometimes even time out. Note that if you comment out the template loading step in the script so that the indices are created with no mappings, then the requests do return instantly. Also, even if the template is loaded, executing a python script that runs _cat/health many times in succession doesn't experience the same slowdown.
• I expect the Discover app to load and not just hang indefinitely.
• Note that I can't reproduce this behaviour in 2.10.0. It's slower than I would like, but the requests don't take as long and it doesn't become unresponsive.

OpenSearch Version
2.16.0

Dashboards Version
2.16.0

Plugins

Using the stock opensearchproject/opensearch-dashboards:2.16.0 image.

Screenshots

N/A

Host/Environment (please complete the following information):

• Windows 10
• Chrome 128.0.6613.120

Additional context

N/A

[tronboto]

Hey @LDrago27 . I noticed you added the needs more info label. Is there anything in particular you want to know?

[tronboto]

Closing in favour of #8417