From 5aabacba7ff7f5719531846547526271fce6a937 Mon Sep 17 00:00:00 2001 From: Sourin Paul <82123779+SourinPaul@users.noreply.github.com> Date: Wed, 12 Jul 2023 15:16:55 -0700 Subject: [PATCH 01/32] Create asset.yml --- rfcs/text/0041/asset.yml | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 rfcs/text/0041/asset.yml diff --git a/rfcs/text/0041/asset.yml b/rfcs/text/0041/asset.yml new file mode 100644 index 000000000..498ed4c38 --- /dev/null +++ b/rfcs/text/0041/asset.yml @@ -0,0 +1,26 @@ +# Licensed to Elasticsearch B.V. under one or more contributor +# license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright +# ownership. Elasticsearch B.V. licenses this file to you under +# the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +--- +- name: asset + fields: + - name: asset.name + level: extended + type: keyword + short: + example: 88a1f0ed-5ae5-41ee-af6b-41921c311872 + description: > + Asset name of .... From 5274c02bd01449ecef34c5ab7448bd0bcff5c68a Mon Sep 17 00:00:00 2001 From: Sourin Paul <82123779+SourinPaul@users.noreply.github.com> Date: Wed, 12 Jul 2023 15:18:10 -0700 Subject: [PATCH 02/32] Create host.yml --- rfcs/text/0041/host.yml | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 rfcs/text/0041/host.yml diff --git a/rfcs/text/0041/host.yml b/rfcs/text/0041/host.yml new file mode 100644 index 000000000..050aaeafb --- /dev/null +++ b/rfcs/text/0041/host.yml @@ -0,0 +1,26 @@ +# Licensed to Elasticsearch B.V. under one or more contributor +# license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright +# ownership. Elasticsearch B.V. licenses this file to you under +# the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +--- +- name: host + fields: + - name: host + level: extended + type: keyword + short: + example: .. + description: > + Asset name of .... From fca40be534a98246483b7589bb271ba3c4063cf0 Mon Sep 17 00:00:00 2001 From: Sourin Paul <82123779+SourinPaul@users.noreply.github.com> Date: Wed, 12 Jul 2023 16:39:07 -0700 Subject: [PATCH 03/32] Update 0041-asset-integration.md Initial touch --- rfcs/text/0041-asset-integration.md | 31 +++++++++++++++++++---------- 1 file changed, 21 insertions(+), 10 deletions(-) diff --git a/rfcs/text/0041-asset-integration.md b/rfcs/text/0041-asset-integration.md index 7fe8ee9b3..7227540fb 100644 --- a/rfcs/text/0041-asset-integration.md +++ b/rfcs/text/0041-asset-integration.md @@ -1,7 +1,7 @@ # 0041: Asset Integration -- Stage: **0 (strawperson)** +- Stage: **1 (Draft)** - Date: **2023-07-07** + + This proposal extends the existing ECS field set to store inventory metadata for hosts and users from external application repositories. Using ECS to store such fields will improve metadata querying and retrieval across various use cases. Terminologies: @@ -25,11 +29,6 @@ This proposal includes the following: This proposal will also facilitate storing host and user inventory within the security solution (the entity store). - - - @@ -135,13 +134,17 @@ Stage 2: Add or update all remaining field definitions. The list should now be e Stage 1: Describe at a high-level how these field changes will be used in practice. Real world examples are encouraged. The goal here is to understand how people would leverage these fields to gain insights or solve problems. ~1-3 paragraphs. --> -* As part of Entity Analytics, we are ingesting metadata about Users and from various external vendor applications. We are storing all ingested metadata in Elasticsearch. After we map these fields to ECS, we will enrich these ingested events for risk-scoring scenarios (e.g., context enrichments) and detecting advanced analytics (UBA) use cases. +* As part of Entity Analytics, we are ingesting metadata about Users and from various external vendor applications. We are storing all ingested metadata in Elasticsearch. After we map these fields to ECS, we will enrich these ingested events for risk-scoring scenarios (e.g., context enrichments) and detecting advanced analytics (UEBA) use cases. + +### Example of Hosts and Users stored in ES * This schema will persist `Observed` (queried) entities from the ingested security log dataset in an Entity store. This entity store can be further extended to meet broader Asset Management needs. * Additional enrichment use cases for existing prebuilt detection rules will leverage these ECS fields. + + ## Source data + +### Real-world mapping of an Identity provider + +#### Okta Users +#### AzureAD Hosts + + ## Scope of impact -* We have a couple of fleet integrations under development. We want them to use these proposed ECS before being released. * Schema/ field sets defined here focus on asset inventory data sources. Additional fields may need to be appended (ideally within this RFC lifecycle) to support the entity store needs. * Due diligence is needed to avoid the proliferation of field sets and validate business requirements. * In stage1, @jasonrhodes identified fields from o11y use cases and a potential conflict: https://github.com/elastic/ecs/pull/2215#pullrequestreview-1498781860 @@ -216,9 +227,8 @@ The following are the people that consulted on the contents of this RFC. * @andrewkroh | subject matter expert * @jamiehynds | subject matter expert * @lauravoicu | subject matter expert -* @MikePaquette | subject matter expert +* @MikePaquette | sponsor * @sourinpaul | sponsor -* ? * Stage 0: https://github.com/elastic/ecs/pull/2215 +* Stage 1: https://github.com/elastic/ecs/pull/NNN -This proposal will also facilitate storing host and user inventory within the security solution (the entity store). +This proposal will also facilitate storing host and user inventory within the security solution (the entity store). Schema/ field sets defined here focus on asset inventory data sources. Additional fields may need to be appended (ideally within this RFC lifecycle) to support the entity store needs. -### Real-world mapping of an Identity provider - -#### Okta Users +### Examples of Real-world mapping: + +#### Mapping User object from Okta (partial): +```yml +- set: + field: asset.status + copy_from: entityanalytics_okta.user.status + tag: set_asset_status + ignore_empty_value: true + - set: + field: user.profile.status + copy_from: entityanalytics_okta.user.status + tag: set_user_profile_status + ignore_empty_value: true + - date: + field: okta.created + target_field: entityanalytics_okta.user.created + tag: date_user_created + formats: + - ISO8601 + if: ctx.okta?.created != null && ctx.okta.created != '' + on_failure: + - remove: + field: okta.created + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - set: + field: user.account.create_date + copy_from: entityanalytics_okta.user.created + tag: set_user_account_create_date + ignore_empty_value: true + - set: + field: asset.create_date + copy_from: entityanalytics_okta.user.created + tag: set_asset_create_date + ignore_empty_value: true + - date: + field: okta.activated + target_field: entityanalytics_okta.user.activated + tag: date_user_activated + formats: + - ISO8601 + if: ctx.okta?.activated != null && ctx.okta.activated != '' + on_failure: + - remove: + field: okta.activated + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - set: + field: user.account.activated_date + copy_from: entityanalytics_okta.user.activated + tag: set_user_account_activated_date + ignore_empty_value: true + - date: + field: okta.statusChanged + target_field: entityanalytics_okta.user.status_changed + tag: date_user_status_changed + formats: + - ISO8601 + if: ctx.okta?.statusChanged != null && ctx.okta.statusChanged != '' + on_failure: + - remove: + field: okta.statusChanged + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' +``` + + #### AzureAD Hosts @@ -207,8 +323,6 @@ The goal here is to research and understand the impact of these changes on users Stage 1: Identify potential concerns, implementation challenges, or complexity. Spend some time on this. Play devil's advocate. Try to identify the sort of non-obvious challenges that tend to surface later. The goal here is to surface risks early, allow everyone the time to work through them, and ultimately document resolution for posterity's sake. --> -* Schema/ field sets defined here focus on asset inventory data sources. Additional fields may need to be appended (ideally within this RFC lifecycle) to support the entity store needs. -* Due diligence is needed to avoid the proliferation of field sets and validate business requirements. * In stage1, @jasonrhodes identified fields from o11y use cases and a potential conflict: https://github.com/elastic/ecs/pull/2215#pullrequestreview-1498781860 * Stage 0: https://github.com/elastic/ecs/pull/2215 -* Stage 1: https://github.com/elastic/ecs/pull/NNN +* Stage 1: https://github.com/elastic/ecs/pull/2233 +* Fields required for storing host and user metadata as the Elastic Security entity store/ index. -This proposal will also facilitate storing host and user inventory within the security solution (the entity store). Schema/ field sets defined here focus on asset inventory data sources. Additional fields may need to be appended (ideally within this RFC lifecycle) to support the entity store needs. +We will create new enhancement RFCs to extend these schemas as needed. -* In stage1, @jasonrhodes identified fields from o11y use cases and a potential conflict: https://github.com/elastic/ecs/pull/2215#pullrequestreview-1498781860 +~~* In stage1, @jasonrhodes identified fields from o11y use cases and a potential conflict: https://github.com/elastic/ecs/pull/2215#pullrequestreview-1498781860~~ +--> Resolution: Exclude `asset.ean`, `asset.parents`, and `asset.children` from this RFC proposal and reintroduce these fields at a later time. Refer to: [[PR comment]](https://github.com/elastic/ecs/pull/2233#issuecomment-1917633738). -- Stage: **1 (Draft)** -- Date: **2023-07-07** +- Stage: **2 (Candidate)** +- Date: **2024-02-22** -* Fields required for storing host and user metadata as the Elastic Security entity store/ index. +* Fields required for storing host and user metadata as the Elastic Security entity store/ index. We will create new enhancement RFCs to extend these schemas as needed. @@ -76,7 +76,7 @@ user.account.status.deprovisioned | boolean | true/ false | A flag indicating if user.account.password_change_date | date | June 5, 2023 @ 18:25:57.000 | Last date/time when account password was updated **Update:** -Updated proposal to redact the below field. ECS guidance is to reuse existing organization.* fields instead +Updated proposal to redact the below field. ECS guidance is to reuse existing organization.* fields instead > user.profile.organization | keyword | Elasticsearch Inc. | Organization name associated with the account. @@ -234,7 +234,7 @@ Stage 2: Included a real world example source document. Ideally this example com Stage 3: Add more real world example source documents so we have at least 2 total, but ideally 3. Format as described in stage 2. --> -### Examples of Real-world mapping: +### Examples of Real-world mapping: #### Mapping User object from Okta into ECS (partial): ```yml @@ -705,7 +705,7 @@ processors: ``` - + #### AzureAD Hosts @@ -775,7 +775,7 @@ e.g.: * Stage 0: https://github.com/elastic/ecs/pull/2215 -* Stage 1: https://github.com/elastic/ecs/pull/2233 +* Stage 2: https://github.com/elastic/ecs/pull/2233