[microsoft_exchange_online_message_trace] Remove event.start/end

[chrisberkhout]

Proposed commit message

[microsoft_exchange_online_message_trace] Remove event.start/end

Don't set `event.start` and `event.end` to `StartDate` and `EndDate`,
because those are query parameters, not properties of the event itself.

Also, extend the description of the `additional_look_back` setting to
note that up to 24h may be necessary (according to the API
documentation[1]).

[1]: https://learn.microsoft.com/en-us/previous-versions/office/developer/o365-enterprise-developers/jj984335%28v%3doffice.15%29

Discussion

Elsewhere I suggested standard solutions for pagination when there is:

1. a delay between data creation and data availability, and/or
2. an unstable a period after initial creation of the data.

I suggested switching from the look-back strategy (introduced in #8717) to the standard solutions, in order to reduce fetching of the same data multiple times (and reduce indexing failures due to duplicate _id values).

However, the standard solution for delayed data availability isn't possible, because there is no creation or update time associated with the log entries we receive. StartDate and EndDate are query parameters. In the API documentation these fields are marked "[In]" rather than "[In/Out]" (input parameters and report output columns). We just see them echoed back in the response.

This PR cleans up incorrect population of ECS fields with StartDate and EndDate.

Shall we drop microsoft.online_message_trace.StartDate and microsoft.online_message_trace.EndDate as well?

The standard solution for an unstable period is complicated by the fact that this period can be quite long. In #8717 it was reported to usually be less than an hour, but some users on the Internet report delays of multiple hours and the API documentation says up to 24 hours.

So, regarding pagination and duplicate _id values, the choice is between:

• a long (up to 24h) delay for all data (by only fetching data once it's known to be there), or
• indexing rejections due to duplicate _id values (using the look-back strategy).

In this case I think the look-back strategy is not a bad one.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues

• Relates microsoft_exchange_online_message_trace: indexing errors due to duplicate documents #11911

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 23b8b66

cc @chrisberkhout

[ShourieG]

LGTM from my end but I think we should wait for @efd6 to review since he has more context with regards to this PR that introduced these fields.

[efd6]

Thanks

[elastic-vault-github-plugin-prod]

Package microsoft_exchange_online_message_trace - 1.25.3 containing this change is available at https://epr.elastic.co/package/microsoft_exchange_online_message_trace/1.25.3/