[O11y] Migrate infraobs packages to ecs@mappings #10171

harnish-elastic · 2024-06-18T05:32:39Z

Enhancement

Proposed commit message

Migration performed using ecs-update. Minor manual changes are performed.

event.category or event.type is expecting value as an array instead of string. Hence provided the value as an array.
generated pipeline and system tests.
If the package is containing kibana version configuration as below,
```
conditions:
  kibana.version: "^8.10.0"
```
Updated these configuration to,
```
conditions:
  kibana:
    version: "^8.10.0"
```

Command

  go run github.com/andrewkroh/go-examples/ecs-update@014b35dfe4c9832b51e7c909a39a48257d6a005d \
    -ecs-version=8.11.0 \
    -ecs-git-ref=v8.11.0 \
    -fields-yml-drop-ecs \
    -kibana-version=^8.13.0 \
    -drop-import-mappings \
    -pr=10171 \
    -owner=elastic/obs-infraobs-integrations \
    packages/*

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

TSDB Testing

Performed TSDB testing for some packages. I have put the index templates in this doc.
Performed live data collection to make sure there are no event drops.

efd6 · 2024-06-18T05:36:03Z

s/10135/10171/ in PR description. Also suggest using sha 014b35dfe4c9832b51e7c909a39a48257d6a005d for the tool; f7c19f8f3fa105fe1d869ed34f4d6c76c75bce2c is out of date with some fixes.

To fix up issues the best thing to do is reset --hard origin/main and re-run the command with the correct values, rather than polishing the PR.

harnish-elastic · 2024-06-18T05:45:23Z

s/10135/10171/ in PR description. Also suggest using sha 014b35dfe4c9832b51e7c909a39a48257d6a005d for the tool; f7c19f8f3fa105fe1d869ed34f4d6c76c75bce2c is out of date with some fixes.

To fix up issues the best thing to do is reset --hard origin/main and re-run the command with the correct values, rather than polishing the PR.

Cool, let me do that. Thanks!

elasticmachine · 2024-06-18T06:02:56Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

efd6 · 2024-06-18T06:13:32Z

The issues with the sample events will need addressing. The approach that I have taken is to do manual fix-ups guided by the failures that I see after running the tool (for example here) and then run the tool after those fixes are merged.

…anual_ecs_mapping_migration Conflicts: packages/cassandra/changelog.yml

…anual_ecs_mapping_migration Conflicts: packages/rabbitmq/changelog.yml

niraj-elastic · 2024-07-02T10:03:02Z

packages/oracle_weblogic/data_stream/access/elasticsearch/ingest_pipeline/default.yml

    ignore_empty_value: true
    ignore_failure: true
 - append:
    field: event.category
-    value:
-    - log
+    value: ["network"]


@harnish-elastic Can you please explain this change?

The change was related to event.category will not be longer support the value log. So based on data stream's logs, the network value suits. Hence updated the value!

Can you share what the exact error here and since when has this category been removed ?

oracle_weblogic/access test-access.log:
[0] parsing field value failed: field "event.category"'s value "log" is not one of the allowed values (api, authentication, configuration, database, driver, email, file, host, iam, intrusion_detection, library, malware, network, package, process, registry, session, threat, vulnerability, web)

ishleenk17 · 2024-07-05T04:08:22Z

packages/oracle_weblogic/data_stream/access/elasticsearch/ingest_pipeline/default.yml

    ignore_empty_value: true
    ignore_failure: true
 - append:
    field: event.category
-    value:
-    - log
+    value: ["network"]


Can you share what the exact error here and since when has this category been removed ?

packages/rabbitmq/data_stream/exchange/fields/ecs.yml

…anual_ecs_mapping_migration Conflicts: packages/prometheus/changelog.yml packages/prometheus/docs/README.md packages/prometheus/manifest.yml

elasticmachine · 2024-07-05T05:57:55Z

💚 Build Succeeded

Buildkite Build
Commit: 2236426

History

💔 Build #13274 failed 21ba3c9
💚 Build #12919 succeeded 982c0ce
💚 Build #12822 succeeded 7eac098
💔 Build #12805 failed 7eac098
💚 Build #12768 succeeded 3701399
💔 Build #12764 failed 8570ead

cc @harnish-elastic

elastic-sonarqube · 2024-07-05T05:58:07Z

Quality Gate failed

Failed conditions
38.9% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

agithomas · 2024-07-05T12:52:41Z

packages/oracle_weblogic/data_stream/admin_server/elasticsearch/ingest_pipeline/default.yml

    ignore_empty_value: true
    ignore_failure: true
 - append:
    field: event.category
-    value:
-    - log
+    value: ["iam"]


When i look at the attached screenshot of this PR, i do not think that we must limit the event.category to iam only.

To unblock this PR, lets keep in "iam" for now.
If we come to a consensus that we need to add more categories, we can do that later as we are appending here.

tetianakravchenko

prometheus.remote_write - LGTM!

ishleenk17

Looks good!

niraj-elastic

LGTM

elasticmachine · 2024-07-08T10:18:17Z

Package cassandra - 1.14.0 containing this change is available at https://epr.elastic.co/search?package=cassandra

elasticmachine · 2024-07-08T10:18:18Z

Package nats - 1.7.0 containing this change is available at https://epr.elastic.co/search?package=nats

elasticmachine · 2024-07-08T10:18:19Z

Package oracle_weblogic - 1.7.0 containing this change is available at https://epr.elastic.co/search?package=oracle_weblogic

elasticmachine · 2024-07-08T10:18:19Z

Package prometheus - 1.18.0 containing this change is available at https://epr.elastic.co/search?package=prometheus

elasticmachine · 2024-07-08T10:18:20Z

Package rabbitmq - 1.15.0 containing this change is available at https://epr.elastic.co/search?package=rabbitmq

elasticmachine · 2024-07-08T10:18:21Z

Package spring_boot - 1.6.0 containing this change is available at https://epr.elastic.co/search?package=spring_boot

elasticmachine · 2024-07-08T10:18:22Z

Package sql - 0.5.0 containing this change is available at https://epr.elastic.co/search?package=sql

elasticmachine · 2024-07-08T10:18:22Z

Package stan - 1.7.0 containing this change is available at https://epr.elastic.co/search?package=stan

elasticmachine · 2024-07-08T10:18:23Z

Package statsd_input - 0.4.0 containing this change is available at https://epr.elastic.co/search?package=statsd_input

harnish-elastic added 8 commits June 17, 2024 18:54

[stan] - change to ECS version [email protected]

77bede2

[spring_boot] - change to ECS version [email protected]

2179327

[prometheus] - change to ECS version [email protected]

aced679

[rabbitmq] - change to ECS version [email protected]

654bf69

[nats] - change to ECS version [email protected]

5d6e52c

[cassandra] - change to ECS version [email protected]

4da71b5

[oracle_weblogic] - change to ECS version [email protected]

8e4b711

add changelog.yml

ef31550

harnish-elastic self-assigned this Jun 18, 2024

update PR link

3a341f2

harnish-elastic added 2 commits June 18, 2024 12:08

cassandra generate system tests

20820b4

nats - generate system tests

53365dc

harnish-elastic force-pushed the manual_ecs_mapping_migration branch from c98e090 to 53365dc Compare June 18, 2024 06:54

harnish-elastic added 7 commits June 18, 2024 12:44

[oracle_weblogic] - generate system tests

0cfd7ac

[spring_boot] - generate system tests

dc6585f

[stan] - elastic-package check

fc903e7

[stan] - generate system tests

2a437da

[sql_input] - change to ECS version [email protected]

4e9533d

[statsd_input] - change to ECS version [email protected]

d62e037

Merge branch 'main' of github.com:harnish-elastic/integrations into m…

917295b

…anual_ecs_mapping_migration Conflicts: packages/cassandra/changelog.yml

harnish-elastic requested review from milan-elastic and niraj-elastic June 21, 2024 09:51

niraj-elastic and others added 2 commits June 24, 2024 17:01

update readme

8570ead

Merge branch 'main' of github.com:harnish-elastic/integrations into m…

3701399

…anual_ecs_mapping_migration Conflicts: packages/rabbitmq/changelog.yml

harnish-elastic marked this pull request as ready for review June 25, 2024 06:40

harnish-elastic requested a review from a team as a code owner June 25, 2024 06:40

niraj-elastic requested changes Jul 2, 2024

View reviewed changes

harnish-elastic requested a review from niraj-elastic July 4, 2024 05:09

ishleenk17 reviewed Jul 5, 2024

View reviewed changes

harnish-elastic added 3 commits July 5, 2024 10:52

remove invalid field from ecs.yml in rabbitmq

21ba3c9

Merge branch 'main' of github.com:harnish-elastic/integrations into m…

6b4091e

…anual_ecs_mapping_migration Conflicts: packages/prometheus/changelog.yml packages/prometheus/docs/README.md packages/prometheus/manifest.yml

elastic-package check

2236426

harnish-elastic requested a review from ishleenk17 July 5, 2024 11:08

agithomas reviewed Jul 5, 2024

View reviewed changes

zmoog approved these changes Jul 5, 2024

View reviewed changes

tetianakravchenko approved these changes Jul 5, 2024

View reviewed changes

ishleenk17 approved these changes Jul 8, 2024

View reviewed changes

harnish-elastic requested a review from agithomas July 8, 2024 06:13

niraj-elastic approved these changes Jul 8, 2024

View reviewed changes

ishleenk17 merged commit ca32ce2 into elastic:main Jul 8, 2024
4 of 5 checks passed

andrewkroh added Integration:cassandra Cassandra Integration:nats NATS Integration:oracle_weblogic Oracle WebLogic Integration:prometheus Prometheus Integration:rabbitmq RabbitMQ Logs and Metrics labels Jul 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[O11y] Migrate infraobs packages to ecs@mappings #10171

[O11y] Migrate infraobs packages to ecs@mappings #10171

harnish-elastic commented Jun 18, 2024 •

edited

Loading

efd6 commented Jun 18, 2024 •

edited

Loading

harnish-elastic commented Jun 18, 2024

elasticmachine commented Jun 18, 2024 •

edited

Loading

efd6 commented Jun 18, 2024

niraj-elastic Jul 2, 2024

harnish-elastic Jul 2, 2024

ishleenk17 Jul 5, 2024

harnish-elastic Jul 5, 2024

ishleenk17 Jul 5, 2024

elasticmachine commented Jul 5, 2024

elastic-sonarqube bot commented Jul 5, 2024

agithomas Jul 5, 2024

ishleenk17 Jul 8, 2024

tetianakravchenko left a comment

ishleenk17 left a comment

niraj-elastic left a comment

elasticmachine commented Jul 8, 2024

elasticmachine commented Jul 8, 2024

elasticmachine commented Jul 8, 2024

elasticmachine commented Jul 8, 2024

elasticmachine commented Jul 8, 2024

elasticmachine commented Jul 8, 2024

elasticmachine commented Jul 8, 2024

elasticmachine commented Jul 8, 2024

elasticmachine commented Jul 8, 2024

[O11y] Migrate infraobs packages to ecs@mappings #10171

[O11y] Migrate infraobs packages to ecs@mappings #10171

Conversation

harnish-elastic commented Jun 18, 2024 • edited Loading

Proposed commit message

Checklist

TSDB Testing

efd6 commented Jun 18, 2024 • edited Loading

harnish-elastic commented Jun 18, 2024

elasticmachine commented Jun 18, 2024 • edited Loading

🚀 Benchmarks report

efd6 commented Jun 18, 2024

niraj-elastic Jul 2, 2024

Choose a reason for hiding this comment

harnish-elastic Jul 2, 2024

Choose a reason for hiding this comment

ishleenk17 Jul 5, 2024

Choose a reason for hiding this comment

harnish-elastic Jul 5, 2024

Choose a reason for hiding this comment

ishleenk17 Jul 5, 2024

Choose a reason for hiding this comment

elasticmachine commented Jul 5, 2024

💚 Build Succeeded

History

elastic-sonarqube bot commented Jul 5, 2024

Quality Gate failed

agithomas Jul 5, 2024

Choose a reason for hiding this comment

ishleenk17 Jul 8, 2024

Choose a reason for hiding this comment

tetianakravchenko left a comment

Choose a reason for hiding this comment

ishleenk17 left a comment

Choose a reason for hiding this comment

niraj-elastic left a comment

Choose a reason for hiding this comment

elasticmachine commented Jul 8, 2024

elasticmachine commented Jul 8, 2024

elasticmachine commented Jul 8, 2024

elasticmachine commented Jul 8, 2024

elasticmachine commented Jul 8, 2024

elasticmachine commented Jul 8, 2024

elasticmachine commented Jul 8, 2024

elasticmachine commented Jul 8, 2024

elasticmachine commented Jul 8, 2024

harnish-elastic commented Jun 18, 2024 •

edited

Loading

efd6 commented Jun 18, 2024 •

edited

Loading

elasticmachine commented Jun 18, 2024 •

edited

Loading