github: Add latest transforms for issues, dependabot, code_scanning, and secret_scanning alerts

packages/github/_dev/build/docs/README.md

@@ -10,7 +10,7 @@ The GitHub audit log records all events related to the GitHub organization. See
 
 To use this integration, the following prerequisites must be met:
  - You must be an organization owner.
- - You must be using Github Enterprise Cloud.
+ - You must be using GitHub Enterprise Cloud.
  - You must use a Personal Access Token with `read:audit_log` scope.
 
 *This integration is not compatible with GitHub Enterprise server.*
@@ -22,7 +22,7 @@ To use this integration, the following prerequisites must be met:
 
 ### Code Scanning
 
-The Code Scanning lets you retrieve all security vulnerabilities and coding errors from a repository setup using Github Advanced Security Code Scanning feature. See [About code scanning](https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/about-code-scanning) for more details.
+The Code Scanning lets you retrieve all security vulnerabilities and coding errors from a repository setup using GitHub Advanced Security Code Scanning feature. See [About code scanning](https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/about-code-scanning) for more details.
 
 To use this integration, GitHub Apps must have the `security_events` read permission. 
 Or use a personal access token with the `security_events` scope for private repos or `public_repo` scope for public repos. See [List code scanning alerts](https://docs.github.com/en/enterprise-cloud@latest/rest/code-scanning#list-code-scanning-alerts-for-a-repository)
@@ -34,7 +34,7 @@ Or use a personal access token with the `security_events` scope for private repo
 
 ### Secret Scanning
 
-The Github Secret Scanning lets you retrieve secret scanning for advanced security alerts from a repository setup using Github Advanced Security Secret Scanning feature. See [About Secret scanning](https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/about-secret-scanning) for more details.
+The GitHub Secret Scanning lets you retrieve secret scanning for advanced security alerts from a repository setup using GitHub Advanced Security Secret Scanning feature. See [About Secret scanning](https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/about-secret-scanning) for more details.
 
 To use this integration, GitHub Apps must have the `secret_scanning_alerts` read permission. 
 Or you must be an administrator for the repository or for the organization that owns the repository, and you must use a personal access token with the `repo` scope or `security_events` scope. For public repositories, you may instead use the `public_repo` scope. See [List secret scanning alerts](https://docs.github.com/en/enterprise-cloud@latest/rest/secret-scanning#list-secret-scanning-alerts-for-a-repository)
@@ -45,7 +45,7 @@ Or you must be an administrator for the repository or for the organization that
 
 ### Dependabot
 
-The Github Dependabot lets you retrieve known vulnerabilites in dependencies from a repository setup using Github Advanced Security Dependabot feature. See [About Dependabot](https://docs.github.com/en/code-security/dependabot/dependabot-alerts) for more details.
+The GitHub Dependabot lets you retrieve known vulnerabilites in dependencies from a repository setup using GitHub Advanced Security Dependabot feature. See [About Dependabot](https://docs.github.com/en/code-security/dependabot/dependabot-alerts) for more details.
 
 To use this integration, you must be an administrator for the repository or for the organization that owns the repository, and you must use a personal access token with the `repo` scope or `security_events` scope. For public repositories, you may instead use the `public_repo` scope. See [Authenticating with GraphQL](https://docs.github.com/en/graphql/guides/forming-calls-with-graphql#authenticating-with-graphql) and [Token Issue](https://github.com/dependabot/feedback/issues/169)
 
@@ -55,11 +55,11 @@ To use this integration, you must be an administrator for the repository or for
 
 ### Issues
 
-The Github Issues datastream lets you retrieve github issues, including pull requests, issue assignees, comments, labels, and milestones. See [About Issues](https://docs.github.com/en/rest/issues/issues?apiVersion=latest) for more details. You can retrieve issues for specific repository or for entire organization. Since Github API considers pull requests as issues, users can use `github.issues.is_pr` field to filter for only pull requests. 
+The GitHub Issues datastream lets you retrieve github issues, including pull requests, issue assignees, comments, labels, and milestones. See [About Issues](https://docs.github.com/en/rest/issues/issues?apiVersion=latest) for more details. You can retrieve issues for specific repository or for entire organization. Since GitHub API considers pull requests as issues, users can use `github.issues.is_pr` field to filter for only pull requests. 
 
 All issues including `closed` are retrieved by default. If users want to retrieve only `open` requests, you need to change `State` parameter to `open`.
 
-To use this integration, users must use Github Apps or Personal Access Token with `read` permission to repositories or organization. Please refer to [Github Apps Permissions Required](https://docs.github.com/en/rest/overview/permissions-required-for-github-apps?apiVersion=latest) and [Personal Access Token Permissions Required](https://docs.github.com/en/rest/overview/permissions-required-for-fine-grained-personal-access-tokens?apiVersion=latest) for more details.
+To use this integration, users must use GitHub Apps or Personal Access Token with `read` permission to repositories or organization. Please refer to [GitHub Apps Permissions Required](https://docs.github.com/en/rest/overview/permissions-required-for-github-apps?apiVersion=latest) and [Personal Access Token Permissions Required](https://docs.github.com/en/rest/overview/permissions-required-for-fine-grained-personal-access-tokens?apiVersion=latest) for more details.
 
 {{fields "issues"}}
 

packages/github/changelog.yml

@@ -1,4 +1,30 @@
 # newer versions go on top
+- version: "2.0.0"
+  changes:
+    - description: Update fields inside fingerprint processor in code_scanning, secret_scanning, and dependabot to ingest all event updates.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/11518
+    - description: Reformat fields to add package-fields.yml across all datastreams.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/11518
+    - description: Remove github.state, github.severity due to inconsistency across datastreams. Update dashboards to use corresponding datastream-level fields instead.
+      type: breaking-change
+      link: https://github.com/elastic/integrations/pull/11518
+    - description: Remove event.action field from code_scanning, secret_scanning, dependabot, and issues as it is redundant. Update dashboards to use data_stream.dataset instead.
+      type: breaking-change
+      link: https://github.com/elastic/integrations/pull/11518
+    - description: Add latest transforms for github issues, dependabot, code_scanning, and secret_scanning alerts.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/11518
+    - description: Add navigation to all dashboards.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/11518
+    - description: Upgrade legacy visualization to latest for code_scanning and secret_scanning.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/11518
+    - description: Change dashboards to point to destination index for issues, dependabot, code_scanning, and secret_scanning alerts.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/11518
 - version: "1.29.3"
   changes:
     - description: Use triple-brace Mustache templating when referencing variables in ingest pipelines.

packages/github/data_stream/code_scanning/elasticsearch/ingest_pipeline/default.yml

@@ -4,12 +4,6 @@ processors:
   - set:
       field: ecs.version
       value: '8.11.0'
-  - set:
-      field: event.action
-      value: "code_scanning"
-  - set:
-      field: event.kind
-      value: "alert"
   - rename:
       field: message
       target_field: event.original
@@ -21,12 +15,19 @@ processors:
   - fail:
       if: "!(ctx.github.code_scanning instanceof Map)"
       message: Missing JSON object
-  - fingerprint:
-      fields:
-        - github.code_scanning.number
-        - github.code_scanning.updated_at
-      target_field: "_id"
+  - remove:
+      field: 
+        - event.kind
       ignore_missing: true
+      description: Fields defined as constant_keyword are removed from _source for storage efficiency.
+  - append:
+      field: event.type
+      value: creation
+      if: ctx.github?.code_scanning?.fixed_at == null && ctx.github?.code_scanning?.dismissed_at == null
+  - append:
+      field: event.type
+      value: deletion
+      if: ctx.github?.code_scanning?.fixed_at != null || ctx.github?.code_scanning?.dismissed_at != null
   - date:
       field: github.code_scanning.created_at
       formats:
@@ -48,6 +49,13 @@ processors:
       timezone: UTC
       target_field: "@timestamp"
       if: ctx.github.code_scanning.updated_at != null
+  - date:
+      field: github.code_scanning.dismissed_at
+      formats:
+        - ISO8601
+      timezone: UTC
+      target_field: github.code_scanning.dismissed_at
+      if: ctx.github?.code_scanning?.dismissed_at != null
   - rename:
       target_field: _temp
       field: github.code_scanning.repository
@@ -135,24 +143,22 @@ processors:
       target_field: github.code_scanning.number
       if: ctx.github.code_scanning.number == null
       ignore_missing: true
+  - fingerprint:
+      fields:
+        - github.repository.owner.login
+        - github.repository.name
+        - github.code_scanning.number
+        - github.code_scanning.created_at
+        - github.code_scanning.updated_at
+        - github.code_scanning.dismissed_at
+      target_field: "_id"
+      ignore_missing: true
   - lowercase:
       field: github.code_scanning.state
       ignore_missing: true
   - lowercase:
       field: github.code_scanning.rule.security_severity_level
       ignore_missing: true
-  - set:
-      field: github.severity
-      value: "{{{github.code_scanning.rule.security_severity_level}}}"
-      if: ctx.github.code_scanning.rule?.security_severity_level != null
-  - set:
-      field: github.severity
-      value: "undefined"
-      if: ctx.github.severity == null
-  - set:
-      field: github.state
-      value: "{{{github.code_scanning.state}}}"
-      if: ctx.github.code_scanning.state != null
   - rename:
       target_field: _temp.dismissed_by
       field: github.code_scanning.dismissed_by

packages/github/data_stream/code_scanning/fields/agent.yml

@@ -6,18 +6,18 @@
   fields:
     - name: containerized
       type: boolean
-      description: >
-        If the host is a container.
-
+      description: If the host is a container.
     - name: os.build
       type: keyword
       example: "18D109"
-      description: >
-        OS build information.
-
+      description: OS build information.
     - name: os.codename
       type: keyword
       example: "stretch"
-      description: >
-        OS codename, if any.
-
+      description: OS codename, if any.
+- name: input.type
+  type: keyword
+  description: Input Type.
+- name: log.offset
+  type: long
+  description: Log Offset.

packages/github/data_stream/code_scanning/fields/base-fields.yml

@@ -1,23 +1,16 @@
 - name: data_stream.type
-  type: constant_keyword
-  description: Data stream type.
+  external: ecs
 - name: data_stream.dataset
-  type: constant_keyword
-  description: Data stream dataset name.
+  external: ecs
 - name: data_stream.namespace
-  type: constant_keyword
-  description: Data stream namespace.
+  external: ecs
 - name: event.module
   type: constant_keyword
-  description: Event module
   value: github
+  external: ecs
 - name: event.dataset
   type: constant_keyword
-  description: Event dataset
   value: github.code_scanning
-- name: "@timestamp"
-  type: date
-  description: Event timestamp.
-- name: input.type
-  type: keyword
-  description: Type of Filebeat input.
+  external: ecs
+- name: '@timestamp'
+  external: ecs

packages/github/data_stream/code_scanning/fields/ecs.yml

@@ -0,0 +1,4 @@
+# Define ECS constant fields as constant_keyword
+- name: event.kind
+  type: constant_keyword
+  value: alert