[Reporting/CSV Export] exported search data is randomly missing rows

[tsullivan]

EDIT: removed the outdated description.

Steps to reproduce:

1. Use the latest snapshot of Elasticsearch 8.0.0
2. Load the test data and test saved objects:

   export TEST_KIBANA_URL=http://elastic:changeme@localhost:5601
   export TEST_ES_URL=http://elastic:changeme@localhost:9200
   node scripts/kbn_archiver.js --config x-pack/test/functional/config.js load x-pack/test/functional/fixtures/kbn_archiver/reporting/ecommerce
   node scripts/es_archiver.js --config x-pack/test/functional/config.js load x-pack/test/functional/es_archives/reporting/ecommerce
   

3. Open Discover and view the ecommerce index
4. Set the timepicker range to Jun 2, 2019 @ 18:54:23.161 / Jul 22, 2019 @ 07:03:07.850
   • search covers 4,675 hits
5. From the menu at the top, click Share > CSV Reports > Generate CSV
6. The exported CSV file contains less than 4,675 rows

Update: view #113675 (comment) for steps on running the skipped test.

[elasticmachine]

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[tsullivan]

Thanks to research done by @jloleysens, we're realizing that these test failures are due to ES returning shard failures. That explanation lines up to the evidence that we see random rows of CSV missing, only some of the time. I have also seen CSV rows that weren't missing, but had null values in random cells where there should have been 0 - and that kind of thing.

We have a few fixes to aim for:

• Stabilize the tests: adjust the tests to perform fewer scrolls (use a smaller date range and return back a smaller number of total records)
• Handle the shard failures when they happen, by capturing them in the generate_csv module, and add them as warning text to the completed reports.
  • Elasticsearch provides the shard failures in a search response, separately from the hits information.
• Use the point-in-time API to page through sets of data for CSV export.
  • Scan-and-Scroll may still be necessary as a backup strategy when exporting non-timebased data.

[tsullivan]

In stabilizing the CSV tests, we are only going to keep a single test that is responsible for testing a "large" export. That test is going to be skipped until this issue is resolved: 428c0d9