[BUG] Exception "e_o_f_exception read past EOF" returned for sub-query with multi_match nested into dis_max

[martin-gaievski]

What is the bug?

Error below returned in response

"type": "e_o_f_exception",
"reason": "read past EOF (pos=4250): MemorySegmentIndexInput(path=\"/home/ec2-user/opensearch/data/

error appears when query is quite complex and for certain query phases. Example of the query structure:

"hybrid": {
          "queries": [
                {
                    "bool": {
                          "boost": 1,
                          "should": {
                                "dis_max": {
                                      "queries": [
                                          "bool": {
                                                    "boost": 1.1,
                                                    "filter": [],
                                                    "minimum_should_match": "1",
                                                    "should": {
                                                        "dis_max": {
                                                            "queries": [
                                                                {
                                                                    "multi_match": { }
                                                                },
                                                                {
                                                                    "multi_match": { }
                                                                },
                                                                {
                                                                    "match": { }
                                                                }
                                                            ],
                                                            "tie_breaker": 0
                                                        }
                                                    }
                                                }

Below is example of zoomed in section of multi_match sub-query:

                                             "should": {
                                                        "dis_max": {
                                                            "queries": [
                                                                {
                                                                    "multi_match": {
                                                                        "_name": "new/banner/match_in_phrases",
                                                                        "boost": 5,
                                                                        "fields": [
                                                                            "sentences.in_english"
                                                                        ],
                                                                        "fuzziness": "AUTO:3,11",
                                                                        "fuzzy_transpositions": true,
                                                                        "max_expansions": 50,
                                                                        "minimum_should_match": "60%",
                                                                        "operator": "AND",
                                                                        "prefix_length": 0,
                                                                        "query": "{{query_text}}",
                                                                        "type": "most_fields"
                                                                    }
                                                                },

What is the expected behavior?

No errors, query should return meaningful results with documetns and scores

What is your host/environment?

Issue present in 2.12 release

Do you have any additional context?

1. In a local test cluster error can manifestate itself differently. There is an exception below in a log and cluster crushes.

» ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [integTest-0] fatal error in thread [opensearch[integTest-0][search][T#14]], exiting
»  java.lang.AssertionError: null
»       at org.apache.lucene.search.TermScorer.score(TermScorer.java:85) ~[lucene-core-9.9.2.jar:9.9.2 a2939784c4ca60bc28bf488b5479c02fc2e5e22c - 2024-01-25 09:51:09]
»       at org.apache.lucene.search.DisjunctionSumScorer.score(DisjunctionSumScorer.java:41) ~[lucene-core-9.9.2.jar:9.9.2 a2939784c4ca60bc28bf488b5479c02fc2e5e22c - 2024-01-25 09:51:09]
»       at org.apache.lucene.search.DisjunctionScorer.score(DisjunctionScorer.java:178) ~[lucene-core-9.9.2.jar:9.9.2 a2939784c4ca60bc28bf488b5479c02fc2e5e22c - 2024-01-25 09:51:09]
»       at org.apache.lucene.search.DisjunctionMaxScorer.score(DisjunctionMaxScorer.java:65) ~[lucene-core-9.9.2.jar:9.9.2 a2939784c4ca60bc28bf488b5479c02fc2e5e22c - 2024-01-25 09:51:09]
»       at org.apache.lucene.search.DisjunctionScorer.score(DisjunctionScorer.java:178) ~[lucene-core-9.9.2.jar:9.9.2 a2939784c4ca60bc28bf488b5479c02fc2e5e22c - 2024-01-25 09:51:09]
»       at org.apache.lucene.search.ConjunctionScorer.score(ConjunctionScorer.java:61) ~[lucene-core-9.9.2.jar:9.9.2 a2939784c4ca60bc28bf488b5479c02fc2e5e22c - 2024-01-25 09:51:09]
»       at org.apache.lucene.search.DisjunctionMaxScorer.score(DisjunctionMaxScorer.java:65) ~[lucene-core-9.9.2.jar:9.9.2 a2939784c4ca60bc28bf488b5479c02fc2e5e22c - 2024-01-25 09:51:09]
»       at org.apache.lucene.search.DisjunctionScorer.score(DisjunctionScorer.java:178) ~[lucene-core-9.9.2.jar:9.9.2 a2939784c4ca60bc28bf488b5479c02fc2e5e22c - 2024-01-25 09:51:09]
»       at org.apache.lucene.search.ReqOptSumScorer.score(ReqOptSumScorer.java:266) ~[lucene-core-9.9.2.jar:9.9.2 a2939784c4ca60bc28bf488b5479c02fc2e5e22c - 2024-01-25 09:51:09]
»       at org.opensearch.neuralsearch.query.HybridQueryScorer.score(HybridQueryScorer.java:67) ~[?:?]

2. If same sub-queries put into dis_max clause then it works without exceptions