[Amazon Security Lake] - OCSF v1.1 update with major refactor & adding support for dynamic template and mappings & system tests

[ShourieG]

Type of change

• Enhancement

Proposed commit message

With the upgrade of OCSF schemas, we are enhancing our support to meet compatibility requirements for OCSF v1.1. We are also reworking the ingest pipeline to incorporate dynamic templates and mappings to enable faster OCSF upgrades in future.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• Added support for User Inventory Info class events.
• Added Terraform based system tests (requires elastic package change to work)
• Rework ingestion pipelines to incorporate dynamic templates and dynamic mappings.
• Add support for more event classes introduced in OCSF v1.1.
• Add new profiles and objects as required based on OCSF v1.1 updates.
  - [ ] Update dashboards wherever required. (dashboards are at the category level and atm after inspection no changes are required since they operate on shared values).
• Updated documentation
• Removed system test configs

NOTE

• Due to the nature and structure of the OCSF schema, this integration has limitations on how deep the mappings run. Some important objects like 'Actor', 'User' and 'Product' have more fleshed-out mappings compared to others which get flattened after the initial 2-3 levels of nesting to keep them maintainable in a YAML format. This will evolve on a need-by-need basis going forward.

• The CI tests will pass once the respective elastic-package changes are implemented as defined here.

Approach

• Segregated objects like user,actor,process,device,file,network and some more into their separate files across all data streams.
• Implemented dynamic templates on few objects which aligned to current pipeline capabilities.
• Implemented terraform based multi bucket system tests.
• Added all new ocsf v1.1 classes and objects across data streams.
• Fixed existing errors and issues in mappings and timestamp parsing across all data streams.
• Cleaned up existing codebase and made it more maintainable.

How to review this PR

Due to the scale of the changes, intermittent merges with main to resolve conflicts and reworks all across the board, re-writing the git history and consolidating the commits with git rebase is proving to be really challenging, hence I suggest the following approach to review this PR:-

1. Complete review of the terraform based deployer, which will later be used in system tests after elastic-package changes are available.
2. Prioritize Commits with the "dynamic" keyword which is specific to some dynamic template implementations.
3. Prioritize Commits withe the keywords "updated", "converted", "segregated", "fixed".
4. Prioritise reviewing the pipeline changes as they contain core logic.
5. The commits with "added" keywords signify addition of new mappings for ocsf v1.1. These are quite large and often times redundant to review completely due to the nature of OCSF. Having said that if you personally feel any of the mappings are worth reviewing please go ahead.
6. Ignore commits with keywords "resolved", "merged", "test", "initial", "trying" .

Some commits have certain elements that could stand out and might have been reworked/removed later down stream. In those scenarios, feel free to review in the complete context or reach out to me in case of any confusion.

How to test this PR locally

Related issues

• Closes [Amazon Security Lake] Add support for new objects and event classes, profiles and update schemas accordingly #10740, [Amazon Security Lake]Re-work ingestion pipelines to support dynamic mappings #10219
• Relates [Meta][Amazon Security Lake] Supporting OCSF v1.1 #9607, [Amazon Security Lake]Perform a pass over respective dashboards and update/add any dashboard element or tables if required #10266

System Tests

--- Test results for package: amazon_security_lake - START ---
╭──────────────────────┬─────────────┬───────────┬──────────────────────┬────────┬─────────────────╮
│ PACKAGE              │ DATA STREAM │ TEST TYPE │ TEST NAME            │ RESULT │    TIME ELAPSED │
├──────────────────────┼─────────────┼───────────┼──────────────────────┼────────┼─────────────────┤
│ amazon_security_lake │ event       │ system    │ application-activity │ PASS   │ 3m12.321247667s │
│ amazon_security_lake │ event       │ system    │ discovery            │ PASS   │ 3m10.478033084s │
│ amazon_security_lake │ event       │ system    │ findings             │ PASS   │ 3m13.906812333s │
│ amazon_security_lake │ event       │ system    │ iam                  │ PASS   │ 3m13.214461166s │
│ amazon_security_lake │ event       │ system    │ network-activity     │ PASS   │ 3m10.608428458s │
│ amazon_security_lake │ event       │ system    │ system-activity      │ PASS   │ 3m12.344728625s │
╰──────────────────────┴─────────────┴───────────┴──────────────────────┴────────┴─────────────────╯

Screenshots

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elastic-sonarqube]

Quality Gate failed

Failed conditions
1.0% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

[chrisberkhout]

Couple of general comments.

I looked at the README and dashboard diffs.

[efd6]

There is a lot of colour and movement in this change which makes it difficult to be confident of the review. I have looked at:

• updated root org templates
• reworked 'org' object mapping as tynamic template for all data streams
• segregated process fields in 'findings', added 'actor' fields for new class support, ignore _dev folder
• added fulnerability findings support and segregated 'resource' group into it's own file
• added ntp activity event class, deprecated proxy event class, aded proxy_endpoint field, uupdated network activity class and segregated endpoint event mappings into separate files across all data streams. updated ocsf object as necessary across respective data streams
• added os patch state event class, segregated device fields across all data streams, added new fields to support newly added event class
• added datastore activity event class, segregated actor, user & metadata fields across all data streams, flattened ldap fields in event data stream to make room for more fields
• added support for detection finding event class, segregated and mapped finding_info in findings data stream
• added support of compliance finding event class, segregated and updated resources object group, added new objects as required
• segregated and expanded api object across all data streams, added support for incitent findings event class
• added support for Device Config State Change event class, updated schema version in comment and dashboard links to 1.1.0
• added support for scan activity event class
• segregated file fields across required data streams, added support for file hosting activity class
• added cwe & epss objects as flattened to cve object
• converted feature object to follow dynamic mapping rules across all data streams
• converted feature object to follow dynamic mapping rules across all data streams
• added some missing fields after locally running system tests for discovery datastream
• reworked terrform deployer to support multi-bucket based system tests
• updated docs and changelog
• fixed timestamp issues across all data streams, added all system tests and updated missing mappings accorgingly

There are a variety of comments and suggestions. I'll take another look again tomorrow.

[ShourieG]

@efd6, I've updated the PR with the suggested changes

[efd6]

LGTM but please wait for @chrisberkhout

[chrisberkhout]

I checked over the things I raised when I started a review earlier, and Dan's comments, and had a quick look at everything else, focusing on the field definition files, the testing setup and just checking the general structure of things.

No big issues. Please feel free to disagree with me on points I've raised. This seems pretty done.

[chrisberkhout]

I forgot to mention, regarding type: keyword for arrays of strings...

Elasticsearch will always let you provide an array of values rather than a single value.

ECS can say that a field should be array rather than a single value (example) and elastic-package will enforce that (doc), but Elasticsearch doesn't require anything different in field definitions.

That may gradually change in the future: [Meta] Better handling of single-valued fields elasticsearch#80825.

[ShourieG]

@chrisberkhout, I've addressed all the PR suggestions except the system tests. Tests don't seem to be terminating with assert.hit_count: 0, so I've not updated that as of yet.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 69b2f19

History

• 💚 Build #17391 succeeded 06209ba
• 💚 Build #15019 succeeded 5794401
• 💚 Build #14728 succeeded 2261431
• 💔 Build #14641 failed 360c3d8
• 💔 Build #14630 failed dd90df2
• 💔 Build #14525 failed 19ffbf7

cc @ShourieG

[elastic-sonarqube]

Quality Gate failed

Failed conditions
1.2% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

[chrisberkhout]

🚢

[elastic-vault-github-plugin-prod]

Package amazon_security_lake - 2.0.0 containing this change is available at https://epr.elastic.co/search?package=amazon_security_lake