Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[MS Exchange Server] Added fields to ignore if empty. #12417

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[MS Exchange Server] Added fields to ignore if empty. #12417

wants to merge 1 commit into from
+2 −0

Conversation

matthewscherer
Copy link

@matthewscherer matthewscherer commented Jan 21, 2025 •
edited by andrewkroh
Loading

Proposed commit message

A customer came across a couple log lines with missing networkmessageid and senderaddress fields. This in turn rejected the logs messages. Allowing these two fields to be empty or otherwise missing will allow them to pass the ingest function.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

@matthewscherer matthewscherer added the bug Something isn't working, use only for issues label Jan 21, 2025
@matthewscherer matthewscherer requested a review from a team as a code owner January 21, 2025 20:58
@andrewkroh andrewkroh added bugfix Pull request that fixes a bug issue Integration:microsoft_exchange_server Microsoft Exchange Server Team:Security-Windows Platform Security Windows Platform Team [elastic/sec-windows-platform] and removed bug Something isn't working, use only for issues labels Jan 21, 2025
@elasticmachine
Copy link

Pinging @elastic/sec-windows-platform (Team:Security-Windows Platform)

@elastic-vault-github-plugin-prod
Copy link

🚀 Benchmarks report

Package microsoft_exchange_server 👍(3) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
messagetracking 12195.12 10204.08 -1991.04 (-16.33%) 💔

To see the full report comment with /test benchmark fullreport

@elastic-sonarqube Elastic SonarQube
Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

@marc-gr
Copy link
Contributor

marc-gr commented Jan 28, 2025

changes lgtm but it requires manifest.yml version bump and the changelog.yml entry (eg https://github.com/elastic/integrations/pull/12381/files#diff-22d0d39b8e0908029d837bafe11c9ca489aee4b752de6546b3fc678ce7b370f1)

@andrewkroh
Copy link
Member

changes lgtm but it requires manifest.yml version bump and the changelog.yml

My recommended way to do this is to use

elastic-package changelog add \
--link https://github.com/elastic/integrations/pull/12417 \
--type bugfix \
--next patch \
--description 'Handle events where `networkmessageid` or `senderaddress` are not present.'

andrewkroh
andrewkroh reviewed Jan 31, 2025
View reviewed changes
Copy link
Member

@andrewkroh andrewkroh left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change needs an accompanying test case where a new log sample{s} with empty senderaddress/networkmessageid is added to a packages/microsoft_exchange_server/data_stream/messagetracking/_dev/test/pipeline/*.log file.

Here's a patch with those changes. The "expected" files were re-generated using elastic-package test pipeline -v -d messagetracking -g.

diff --git a/packages/microsoft_exchange_server/data_stream/messagetracking/_dev/test/pipeline/test-messagetracking.log b/packages/microsoft_exchange_server/data_stream/messagetracking/_dev/test/pipeline/test-messagetracking.log
index 6a83935be5..8b2045d40c 100644
--- a/packages/microsoft_exchange_server/data_stream/messagetracking/_dev/test/pipeline/test-messagetracking.log
+++ b/packages/microsoft_exchange_server/data_stream/messagetracking/_dev/test/pipeline/test-messagetracking.log
@@ -2,3 +2,5 @@
 2024-01-25T15:16:09.949Z,10.11.12.14,exchange-mail.my.domain.com,10.11.12.14,exchange-mail,08DC1DB12C345BE5;2024-01-25T15:16:09.544Z;0,exchange-mail\Default exchange-mail,SMTP,RECEIVE,70912345566403,<[email protected]>,1e6eb197-c6b4-1234-1b69-56dc1db88f50,[email protected],,7229,1,,,vzdump backup status (host01.my.domain.com): backup successful,[email protected],[email protected],0cA: ,Incoming,,10.11.12.13,10.11.12.14,S:ProxyHop1=exchange-mail.my.domain.com(10.11.12.14);S:MessageValue=MediumHigh;S:Replication=Failed;S:FirstForestHop=exchange-mail.my.domain.com;S:FromEntity=Internet;S:ProxiedClientIPAddress=10.11.12.13;S:ProxiedClientHostname=host01.my.domain.com;S:DeliveryPriority=Normal;S:AccountForest=my.domain.com,Email,05503123-c5b9-46fe-1234-56dc1db88f8f,15.02.0330.005
 2024-01-25T15:16:14.415Z,10.11.12.14,exchange-mail.my.domain.com,10.11.12.14,exchange-mail,08DC1DB12C345BE9;2024-01-25T15:16:12.885Z;0,exchange-mail\Default exchange-mail,SMTP,RECEIVE,70912345566407,<[email protected]>,c95b5dd1-f520-1234-e6dc-56dc1db8914d,[email protected],,8251,1,,,vzdump backup status (pve-vhost01.my.domain.com): backup successful,[email protected],[email protected],0cA: ,Incoming,,10.11.12.15,10.11.12.14,S:ProxyHop1=exchange-mail.my.domain.com(10.11.12.14);S:MessageValue=MediumHigh;S:Replication=Failed;S:FirstForestHop=exchange-mail.my.domain.com;S:FromEntity=Internet;S:ProxiedClientIPAddress=10.11.12.15;S:ProxiedClientHostname=pve-vhost01.my.domain.com;S:DeliveryPriority=Normal;S:AccountForest=my.domain.com,Email,d6aef52d-0e05-1234-e29b-56dc1db89238,15.02.0330.005
 2024-01-07T00:00:07.463Z,192.168.0.1,exchange,192.168.0.2,exchange.example.com,;250 [email protected][Hostname=exchange.example.com];ClientSubmitTime:,Intra-Organization SMTP Send Connector,SMTP,SEND,29519319995411,[email protected],0b7099ea-cb95-1234-328e-08dc5f139ac8,[email protected],250 2.1.5Recipient OK,38663,1,,,ein Titel,[email protected],[email protected],2024-01-07T00:00:05.535Z;LSRV=exchange.example.com:TOTAL-HUB=1.921|SMR=0.127(SMRDE=0.002|SMRC=0.125(SMRCL=0.105|X-SMRCR=0.125))|CAT=1.698(CATOS=0.018(CATSM=0.017(CATSM-Malware Agent=0.017))|CATRESL=0.004|CATORES=1.567(CATRS=1.566(CATRS-ScanMail Routing Agent=0.117|CATRS-Transport Rule Agent=0.002(X-ETREX=0.002)|CATRS-Index Routing Agent=1.444))|CATORT=0.108(CATRT=0.107(CATRT-Journal Agent=0.107)))|QDM=0.010|SMSC=0.006(X-SMSDR=0.011)|SMS=0.076(SMSMBXD=0.071),Originating,,,,S:E2ELatency=1.928;S:MsgRecipCount=1;S:IncludeInSla=True;S:Microsoft.Exchange.Transport.MailRecipient.RequiredTlsAuthLevel=Opportunistic;S:IsSmtpResponseFromExternalServer=False;S:DeliveryPriority=Normal;S:AccountForest=example.com,Email,a7ae9ef9-e10c-4111-19bf-08dc0f111bee,15.01.2507.035
+2025-01-14T09:29:05.327Z,216.160.83.56,HELLOWORLD,175.16.199.1,global-mail-onmicrosoft-com.mail.protection.outlook.com,";250 2.6.0 <[email protected]> [InternalId=5450313518243, Hostname=AAAAAAAAAAAAA.bbbbbbbb.prod.outlook.com] 141301 bytes in 0.101, 1361.827 KB/sec Queued mail for delivery;ClientSubmitTime:",Outbound to Office 365 - cb1ce4e7-cbb6-46e8-8a4c-e1f2efbdd202,SMTP,SENDEXTERNAL,8774618205228,<[email protected]>,cb1ce4e7-cbb6-46e8-8a4c-e1f2efbdd202,[email protected],250 2.1.5 Recipient OK,136349,1,,,John Doe,,<>,2025-01-14T09:28:50.814Z;SRV=HELLOWORLD.foo.example.com:TOTAL-FE=0.074|SMR=0.072(SMRPI=0.002(SMRPI-FrontendProxyAgent=0.002))|SMS=0.002;SRV=HELLOWORLD.foo.example.com:TOTAL-HUB=14.439|SMR=0.279(SMRDE=0.174|SMRC=0.105(SMRCL=0.105|X-SMRCR=0.016))|CAT=0.046(CATOS=0.018(CATSM=0.018(CATSM-Malware Agent=0.017))|CATRESL=0.003|CATORES=0.021(CATRS=0.021(CATRS-Prioritization Agent=0.002|CATRS-Index Routing Agent=0.017))|CATORT=0.002(CATRT=0.002(CATRT-CodeTwo Exchange Rules Transport Agent=0.001)))|QDE=13.818|SMS=0.291,Incoming,,,,S:E2ELatency=14.513;S:ExternalSendLatency=0.402;S:ToEntity=Internet;S:FromEntity=Internet;S:MsgRecipCount=1;S:IncludeInSla=True;S:Microsoft.Exchange.Transport.MailRecipient.RequiredTlsAuthLevel=DomainValidation;S:Microsoft.Exchange.Transport.MailRecipient.EffectiveTlsAuthLevel=DomainValidation;S:IsSmtpResponseFromExternalServer=True;S:DeliveryPriority=Normal;S:OriginalFromAddress=<>;S:AccountForest=wgs.wuerth.com,Email,82f8feae-ef9b-41bf-8500-1e10946655ae,15.01.2507.039
+2025-01-14T09:28:39.334Z,2a02:cf40::0000:1234:5678:9abc,HELLOWORLD,,,"MDB:f4adaa08-ff49-4bca-ba70-68e9b80597b0, Mailbox:84dd42d9-8814-4758-a8bb-e8765d62ec90, Event:190620893, MessageClass:IPM.Note, CreationTime:2025-01-14T09:28:39.321Z, ClientType:MOMT, SubmissionAssistant:MailboxTransportSubmissionEmailAssistant",,STOREDRIVER,NOTIFYMAPI,,,,,,,,,,,[email protected],,2025-01-14T09:28:39.321Z;LSRV=HELLOWORLD.foo.example.com:TOTAL-SUB=0.013|SA=0.013|MTSS-PEN=0.000,,,,,S:ItemEntryId=00-00-00-00-39-E7-E4-05-8C-C2-08-4C-AC-96-9A-D0-DE-C4-85-EE-07-00-D8-00-A6-A0-EC-B2-39-4D-BD-2F-31-4C-F7-4F-3F-4C-00-00-00-14-1A-99-00-00-46-7D-7B-CD-1B-EA-FC-40-9C-37-A9-7E-B0-DD-4C-16-00-08-65-53-76-46-00-00,,74364e3b-32b7-4108-9a2c-6dda84007459,15.01.2507.039
diff --git a/packages/microsoft_exchange_server/data_stream/messagetracking/_dev/test/pipeline/test-messagetracking.log-expected.json b/packages/microsoft_exchange_server/data_stream/messagetracking/_dev/test/pipeline/test-messagetracking.log-expected.json
index 1ee68f0e1d..83c61ba3a5 100644
--- a/packages/microsoft_exchange_server/data_stream/messagetracking/_dev/test/pipeline/test-messagetracking.log-expected.json
+++ b/packages/microsoft_exchange_server/data_stream/messagetracking/_dev/test/pipeline/test-messagetracking.log-expected.json
@@ -6,15 +6,14 @@
                 "direction": "Incoming",
                 "from": {
                     "address": [
-                        "[email protected]"
+                        "[email protected]",
+                        "[email protected]"
                     ]
                 },
                 "local_id": "2fd37dca-1234-5bfb-175d-08dc1db88f52",
                 "message_id": "<[email protected]>",
                 "sender": {
-                    "address": [
-                        "[email protected]"
-                    ]
+                    "address": "[email protected]"
                 },
                 "subject": "Undelivered Mail Returned to Sender",
                 "to": {
@@ -24,7 +23,7 @@
                 }
             },
             "event": {
-                "ingested": "2024-09-04T11:53:25.104693004Z",
+                "ingested": "2025-01-31T22:06:04.860204507Z",
                 "original": "2024-01-25T15:16:09.843Z,,,,exchange-mail,No suitable shadow servers,,SMTP,HAREDIRECTFAIL,70971234566456,<[email protected]>,2fd37dca-1234-5bfb-175d-08dc1db88f52,[email protected],,15054,1,,,Undelivered Mail Returned to Sender,[email protected],[email protected],,Incoming,,,,S:DeliveryPriority=Normal;S:[email protected];S:AccountForest=my.domain.com,Email,dc69df25-1234-564c-41c4-08dc1db88f7f,15.02.0330.005"
             },
             "message": "2024-01-25T15:16:09.843Z,,,,exchange-mail,No suitable shadow servers,,SMTP,HAREDIRECTFAIL,70971234566456,<[email protected]>,2fd37dca-1234-5bfb-175d-08dc1db88f52,[email protected],,15054,1,,,Undelivered Mail Returned to Sender,[email protected],[email protected],,Incoming,,,,S:DeliveryPriority=Normal;S:[email protected];S:AccountForest=my.domain.com,Email,dc69df25-1234-564c-41c4-08dc1db88f7f,15.02.0330.005",
@@ -69,9 +68,7 @@
                 "local_id": "1e6eb197-c6b4-1234-1b69-56dc1db88f50",
                 "message_id": "<[email protected]>",
                 "sender": {
-                    "address": [
-                        "[email protected]"
-                    ]
+                    "address": "[email protected]"
                 },
                 "subject": "vzdump backup status (host01.my.domain.com): backup successful",
                 "to": {
@@ -81,7 +78,7 @@
                 }
             },
             "event": {
-                "ingested": "2024-09-04T11:53:25.104705446Z",
+                "ingested": "2025-01-31T22:06:04.86022234Z",
                 "original": "2024-01-25T15:16:09.949Z,10.11.12.14,exchange-mail.my.domain.com,10.11.12.14,exchange-mail,08DC1DB12C345BE5;2024-01-25T15:16:09.544Z;0,exchange-mail\\Default exchange-mail,SMTP,RECEIVE,70912345566403,<[email protected]>,1e6eb197-c6b4-1234-1b69-56dc1db88f50,[email protected],,7229,1,,,vzdump backup status (host01.my.domain.com): backup successful,[email protected],[email protected],0cA: ,Incoming,,10.11.12.13,10.11.12.14,S:ProxyHop1=exchange-mail.my.domain.com(10.11.12.14);S:MessageValue=MediumHigh;S:Replication=Failed;S:FirstForestHop=exchange-mail.my.domain.com;S:FromEntity=Internet;S:ProxiedClientIPAddress=10.11.12.13;S:ProxiedClientHostname=host01.my.domain.com;S:DeliveryPriority=Normal;S:AccountForest=my.domain.com,Email,05503123-c5b9-46fe-1234-56dc1db88f8f,15.02.0330.005"
             },
             "message": "2024-01-25T15:16:09.949Z,10.11.12.14,exchange-mail.my.domain.com,10.11.12.14,exchange-mail,08DC1DB12C345BE5;2024-01-25T15:16:09.544Z;0,exchange-mail\\Default exchange-mail,SMTP,RECEIVE,70912345566403,<[email protected]>,1e6eb197-c6b4-1234-1b69-56dc1db88f50,[email protected],,7229,1,,,vzdump backup status (host01.my.domain.com): backup successful,[email protected],[email protected],0cA: ,Incoming,,10.11.12.13,10.11.12.14,S:ProxyHop1=exchange-mail.my.domain.com(10.11.12.14);S:MessageValue=MediumHigh;S:Replication=Failed;S:FirstForestHop=exchange-mail.my.domain.com;S:FromEntity=Internet;S:ProxiedClientIPAddress=10.11.12.13;S:ProxiedClientHostname=host01.my.domain.com;S:DeliveryPriority=Normal;S:AccountForest=my.domain.com,Email,05503123-c5b9-46fe-1234-56dc1db88f8f,15.02.0330.005",
@@ -131,9 +128,7 @@
                 "local_id": "c95b5dd1-f520-1234-e6dc-56dc1db8914d",
                 "message_id": "<[email protected]>",
                 "sender": {
-                    "address": [
-                        "[email protected]"
-                    ]
+                    "address": "[email protected]"
                 },
                 "subject": "vzdump backup status (pve-vhost01.my.domain.com): backup successful",
                 "to": {
@@ -143,7 +138,7 @@
                 }
             },
             "event": {
-                "ingested": "2024-09-04T11:53:25.1047078Z",
+                "ingested": "2025-01-31T22:06:04.86022384Z",
                 "original": "2024-01-25T15:16:14.415Z,10.11.12.14,exchange-mail.my.domain.com,10.11.12.14,exchange-mail,08DC1DB12C345BE9;2024-01-25T15:16:12.885Z;0,exchange-mail\\Default exchange-mail,SMTP,RECEIVE,70912345566407,<[email protected]>,c95b5dd1-f520-1234-e6dc-56dc1db8914d,[email protected],,8251,1,,,vzdump backup status (pve-vhost01.my.domain.com): backup successful,[email protected],[email protected],0cA: ,Incoming,,10.11.12.15,10.11.12.14,S:ProxyHop1=exchange-mail.my.domain.com(10.11.12.14);S:MessageValue=MediumHigh;S:Replication=Failed;S:FirstForestHop=exchange-mail.my.domain.com;S:FromEntity=Internet;S:ProxiedClientIPAddress=10.11.12.15;S:ProxiedClientHostname=pve-vhost01.my.domain.com;S:DeliveryPriority=Normal;S:AccountForest=my.domain.com,Email,d6aef52d-0e05-1234-e29b-56dc1db89238,15.02.0330.005"
             },
             "message": "2024-01-25T15:16:14.415Z,10.11.12.14,exchange-mail.my.domain.com,10.11.12.14,exchange-mail,08DC1DB12C345BE9;2024-01-25T15:16:12.885Z;0,exchange-mail\\Default exchange-mail,SMTP,RECEIVE,70912345566407,<[email protected]>,c95b5dd1-f520-1234-e6dc-56dc1db8914d,[email protected],,8251,1,,,vzdump backup status (pve-vhost01.my.domain.com): backup successful,[email protected],[email protected],0cA: ,Incoming,,10.11.12.15,10.11.12.14,S:ProxyHop1=exchange-mail.my.domain.com(10.11.12.14);S:MessageValue=MediumHigh;S:Replication=Failed;S:FirstForestHop=exchange-mail.my.domain.com;S:FromEntity=Internet;S:ProxiedClientIPAddress=10.11.12.15;S:ProxiedClientHostname=pve-vhost01.my.domain.com;S:DeliveryPriority=Normal;S:AccountForest=my.domain.com,Email,d6aef52d-0e05-1234-e29b-56dc1db89238,15.02.0330.005",
@@ -193,9 +188,7 @@
                 "local_id": "0b7099ea-cb95-1234-328e-08dc5f139ac8",
                 "message_id": "[email protected]",
                 "sender": {
-                    "address": [
-                        "[email protected]"
-                    ]
+                    "address": "[email protected]"
                 },
                 "subject": "ein Titel",
                 "to": {
@@ -205,7 +198,7 @@
                 }
             },
             "event": {
-                "ingested": "2024-09-04T11:53:25.104709787Z",
+                "ingested": "2025-01-31T22:06:04.860224882Z",
                 "original": "2024-01-07T00:00:07.463Z,192.168.0.1,exchange,192.168.0.2,exchange.example.com,;250 [email protected][Hostname=exchange.example.com];ClientSubmitTime:,Intra-Organization SMTP Send Connector,SMTP,SEND,29519319995411,[email protected],0b7099ea-cb95-1234-328e-08dc5f139ac8,[email protected],250 2.1.5Recipient OK,38663,1,,,ein Titel,[email protected],[email protected],2024-01-07T00:00:05.535Z;LSRV=exchange.example.com:TOTAL-HUB=1.921|SMR=0.127(SMRDE=0.002|SMRC=0.125(SMRCL=0.105|X-SMRCR=0.125))|CAT=1.698(CATOS=0.018(CATSM=0.017(CATSM-Malware Agent=0.017))|CATRESL=0.004|CATORES=1.567(CATRS=1.566(CATRS-ScanMail Routing Agent=0.117|CATRS-Transport Rule Agent=0.002(X-ETREX=0.002)|CATRS-Index Routing Agent=1.444))|CATORT=0.108(CATRT=0.107(CATRT-Journal Agent=0.107)))|QDM=0.010|SMSC=0.006(X-SMSDR=0.011)|SMS=0.076(SMSMBXD=0.071),Originating,,,,S:E2ELatency=1.928;S:MsgRecipCount=1;S:IncludeInSla=True;S:Microsoft.Exchange.Transport.MailRecipient.RequiredTlsAuthLevel=Opportunistic;S:IsSmtpResponseFromExternalServer=False;S:DeliveryPriority=Normal;S:AccountForest=example.com,Email,a7ae9ef9-e10c-4111-19bf-08dc0f111bee,15.01.2507.035"
             },
             "message": "2024-01-07T00:00:07.463Z,192.168.0.1,exchange,192.168.0.2,exchange.example.com,;250 [email protected][Hostname=exchange.example.com];ClientSubmitTime:,Intra-Organization SMTP Send Connector,SMTP,SEND,29519319995411,[email protected],0b7099ea-cb95-1234-328e-08dc5f139ac8,[email protected],250 2.1.5Recipient OK,38663,1,,,ein Titel,[email protected],[email protected],2024-01-07T00:00:05.535Z;LSRV=exchange.example.com:TOTAL-HUB=1.921|SMR=0.127(SMRDE=0.002|SMRC=0.125(SMRCL=0.105|X-SMRCR=0.125))|CAT=1.698(CATOS=0.018(CATSM=0.017(CATSM-Malware Agent=0.017))|CATRESL=0.004|CATORES=1.567(CATRS=1.566(CATRS-ScanMail Routing Agent=0.117|CATRS-Transport Rule Agent=0.002(X-ETREX=0.002)|CATRS-Index Routing Agent=1.444))|CATORT=0.108(CATRT=0.107(CATRT-Journal Agent=0.107)))|QDM=0.010|SMSC=0.006(X-SMSDR=0.011)|SMS=0.076(SMSMBXD=0.071),Originating,,,,S:E2ELatency=1.928;S:MsgRecipCount=1;S:IncludeInSla=True;S:Microsoft.Exchange.Transport.MailRecipient.RequiredTlsAuthLevel=Opportunistic;S:IsSmtpResponseFromExternalServer=False;S:DeliveryPriority=Normal;S:AccountForest=example.com,Email,a7ae9ef9-e10c-4111-19bf-08dc0f111bee,15.01.2507.035",
@@ -237,6 +230,93 @@
             "tags": [
                 "preserve_original_event"
             ]
+        },
+        {
+            "@timestamp": "2025-01-14T09:29:05.327Z",
+            "client": {
+                "domain": "HELLOWORLD",
+                "ip": "216.160.83.56"
+            },
+            "email": {
+                "direction": "Incoming",
+                "local_id": "cb1ce4e7-cbb6-46e8-8a4c-e1f2efbdd202",
+                "message_id": "<[email protected]>",
+                "subject": "John Doe",
+                "to": {
+                    "address": [
+                        "[email protected]"
+                    ]
+                }
+            },
+            "event": {
+                "ingested": "2025-01-31T22:06:04.860225882Z",
+                "original": "2025-01-14T09:29:05.327Z,216.160.83.56,HELLOWORLD,175.16.199.1,global-mail-onmicrosoft-com.mail.protection.outlook.com,\";250 2.6.0 <[email protected]> [InternalId=5450313518243, Hostname=AAAAAAAAAAAAA.bbbbbbbb.prod.outlook.com] 141301 bytes in 0.101, 1361.827 KB/sec Queued mail for delivery;ClientSubmitTime:\",Outbound to Office 365 - cb1ce4e7-cbb6-46e8-8a4c-e1f2efbdd202,SMTP,SENDEXTERNAL,8774618205228,<[email protected]>,cb1ce4e7-cbb6-46e8-8a4c-e1f2efbdd202,[email protected],250 2.1.5 Recipient OK,136349,1,,,John Doe,,<>,2025-01-14T09:28:50.814Z;SRV=HELLOWORLD.foo.example.com:TOTAL-FE=0.074|SMR=0.072(SMRPI=0.002(SMRPI-FrontendProxyAgent=0.002))|SMS=0.002;SRV=HELLOWORLD.foo.example.com:TOTAL-HUB=14.439|SMR=0.279(SMRDE=0.174|SMRC=0.105(SMRCL=0.105|X-SMRCR=0.016))|CAT=0.046(CATOS=0.018(CATSM=0.018(CATSM-Malware Agent=0.017))|CATRESL=0.003|CATORES=0.021(CATRS=0.021(CATRS-Prioritization Agent=0.002|CATRS-Index Routing Agent=0.017))|CATORT=0.002(CATRT=0.002(CATRT-CodeTwo Exchange Rules Transport Agent=0.001)))|QDE=13.818|SMS=0.291,Incoming,,,,S:E2ELatency=14.513;S:ExternalSendLatency=0.402;S:ToEntity=Internet;S:FromEntity=Internet;S:MsgRecipCount=1;S:IncludeInSla=True;S:Microsoft.Exchange.Transport.MailRecipient.RequiredTlsAuthLevel=DomainValidation;S:Microsoft.Exchange.Transport.MailRecipient.EffectiveTlsAuthLevel=DomainValidation;S:IsSmtpResponseFromExternalServer=True;S:DeliveryPriority=Normal;S:OriginalFromAddress=<>;S:AccountForest=wgs.wuerth.com,Email,82f8feae-ef9b-41bf-8500-1e10946655ae,15.01.2507.039"
+            },
+            "message": "2025-01-14T09:29:05.327Z,216.160.83.56,HELLOWORLD,175.16.199.1,global-mail-onmicrosoft-com.mail.protection.outlook.com,\";250 2.6.0 <[email protected]> [InternalId=5450313518243, Hostname=AAAAAAAAAAAAA.bbbbbbbb.prod.outlook.com] 141301 bytes in 0.101, 1361.827 KB/sec Queued mail for delivery;ClientSubmitTime:\",Outbound to Office 365 - cb1ce4e7-cbb6-46e8-8a4c-e1f2efbdd202,SMTP,SENDEXTERNAL,8774618205228,<[email protected]>,cb1ce4e7-cbb6-46e8-8a4c-e1f2efbdd202,[email protected],250 2.1.5 Recipient OK,136349,1,,,John Doe,,<>,2025-01-14T09:28:50.814Z;SRV=HELLOWORLD.foo.example.com:TOTAL-FE=0.074|SMR=0.072(SMRPI=0.002(SMRPI-FrontendProxyAgent=0.002))|SMS=0.002;SRV=HELLOWORLD.foo.example.com:TOTAL-HUB=14.439|SMR=0.279(SMRDE=0.174|SMRC=0.105(SMRCL=0.105|X-SMRCR=0.016))|CAT=0.046(CATOS=0.018(CATSM=0.018(CATSM-Malware Agent=0.017))|CATRESL=0.003|CATORES=0.021(CATRS=0.021(CATRS-Prioritization Agent=0.002|CATRS-Index Routing Agent=0.017))|CATORT=0.002(CATRT=0.002(CATRT-CodeTwo Exchange Rules Transport Agent=0.001)))|QDE=13.818|SMS=0.291,Incoming,,,,S:E2ELatency=14.513;S:ExternalSendLatency=0.402;S:ToEntity=Internet;S:FromEntity=Internet;S:MsgRecipCount=1;S:IncludeInSla=True;S:Microsoft.Exchange.Transport.MailRecipient.RequiredTlsAuthLevel=DomainValidation;S:Microsoft.Exchange.Transport.MailRecipient.EffectiveTlsAuthLevel=DomainValidation;S:IsSmtpResponseFromExternalServer=True;S:DeliveryPriority=Normal;S:OriginalFromAddress=<>;S:AccountForest=wgs.wuerth.com,Email,82f8feae-ef9b-41bf-8500-1e10946655ae,15.01.2507.039",
+            "microsoft": {
+                "exchange": {
+                    "connectorid": "Outbound to Office 365 - cb1ce4e7-cbb6-46e8-8a4c-e1f2efbdd202",
+                    "customdata": "S:E2ELatency=14.513;S:ExternalSendLatency=0.402;S:ToEntity=Internet;S:FromEntity=Internet;S:MsgRecipCount=1;S:IncludeInSla=True;S:Microsoft.Exchange.Transport.MailRecipient.RequiredTlsAuthLevel=DomainValidation;S:Microsoft.Exchange.Transport.MailRecipient.EffectiveTlsAuthLevel=DomainValidation;S:IsSmtpResponseFromExternalServer=True;S:DeliveryPriority=Normal;S:OriginalFromAddress=<>;S:AccountForest=wgs.wuerth.com",
+                    "eventid": "SENDEXTERNAL",
+                    "internalmessageid": "8774618205228",
+                    "logid": "82f8feae-ef9b-41bf-8500-1e10946655ae",
+                    "messageinfo": "2025-01-14T09:28:50.814Z;SRV=HELLOWORLD.foo.example.com:TOTAL-FE=0.074|SMR=0.072(SMRPI=0.002(SMRPI-FrontendProxyAgent=0.002))|SMS=0.002;SRV=HELLOWORLD.foo.example.com:TOTAL-HUB=14.439|SMR=0.279(SMRDE=0.174|SMRC=0.105(SMRCL=0.105|X-SMRCR=0.016))|CAT=0.046(CATOS=0.018(CATSM=0.018(CATSM-Malware Agent=0.017))|CATRESL=0.003|CATORES=0.021(CATRS=0.021(CATRS-Prioritization Agent=0.002|CATRS-Index Routing Agent=0.017))|CATORT=0.002(CATRT=0.002(CATRT-CodeTwo Exchange Rules Transport Agent=0.001)))|QDE=13.818|SMS=0.291",
+                    "networkmessageid": "cb1ce4e7-cbb6-46e8-8a4c-e1f2efbdd202",
+                    "recipientcount": 1,
+                    "recipientstatus": "250 2.1.5 Recipient OK",
+                    "returnpath": "<>",
+                    "schemaversion": "15.01.2507.039",
+                    "source": "SMTP",
+                    "sourcecontext": ";250 2.6.0 <[email protected]> [InternalId=5450313518243, Hostname=AAAAAAAAAAAAA.bbbbbbbb.prod.outlook.com] 141301 bytes in 0.101, 1361.827 KB/sec Queued mail for delivery;ClientSubmitTime:",
+                    "transporttraffictype": "Email"
+                }
+            },
+            "network": {
+                "bytes": 136349
+            },
+            "server": {
+                "domain": "global-mail-onmicrosoft-com.mail.protection.outlook.com",
+                "ip": "175.16.199.1"
+            },
+            "tags": [
+                "preserve_original_event"
+            ]
+        },
+        {
+            "@timestamp": "2025-01-14T09:28:39.334Z",
+            "client": {
+                "domain": "HELLOWORLD",
+                "ip": "2a02:cf40::0000:1234:5678:9abc"
+            },
+            "email": {
+                "from": {
+                    "address": [
+                        "[email protected]"
+                    ]
+                },
+                "sender": {
+                    "address": "[email protected]"
+                }
+            },
+            "event": {
+                "ingested": "2025-01-31T22:06:04.86022684Z",
+                "original": "2025-01-14T09:28:39.334Z,2a02:cf40::0000:1234:5678:9abc,HELLOWORLD,,,\"MDB:f4adaa08-ff49-4bca-ba70-68e9b80597b0, Mailbox:84dd42d9-8814-4758-a8bb-e8765d62ec90, Event:190620893, MessageClass:IPM.Note, CreationTime:2025-01-14T09:28:39.321Z, ClientType:MOMT, SubmissionAssistant:MailboxTransportSubmissionEmailAssistant\",,STOREDRIVER,NOTIFYMAPI,,,,,,,,,,,[email protected],,2025-01-14T09:28:39.321Z;LSRV=HELLOWORLD.foo.example.com:TOTAL-SUB=0.013|SA=0.013|MTSS-PEN=0.000,,,,,S:ItemEntryId=00-00-00-00-39-E7-E4-05-8C-C2-08-4C-AC-96-9A-D0-DE-C4-85-EE-07-00-D8-00-A6-A0-EC-B2-39-4D-BD-2F-31-4C-F7-4F-3F-4C-00-00-00-14-1A-99-00-00-46-7D-7B-CD-1B-EA-FC-40-9C-37-A9-7E-B0-DD-4C-16-00-08-65-53-76-46-00-00,,74364e3b-32b7-4108-9a2c-6dda84007459,15.01.2507.039"
+            },
+            "message": "2025-01-14T09:28:39.334Z,2a02:cf40::0000:1234:5678:9abc,HELLOWORLD,,,\"MDB:f4adaa08-ff49-4bca-ba70-68e9b80597b0, Mailbox:84dd42d9-8814-4758-a8bb-e8765d62ec90, Event:190620893, MessageClass:IPM.Note, CreationTime:2025-01-14T09:28:39.321Z, ClientType:MOMT, SubmissionAssistant:MailboxTransportSubmissionEmailAssistant\",,STOREDRIVER,NOTIFYMAPI,,,,,,,,,,,[email protected],,2025-01-14T09:28:39.321Z;LSRV=HELLOWORLD.foo.example.com:TOTAL-SUB=0.013|SA=0.013|MTSS-PEN=0.000,,,,,S:ItemEntryId=00-00-00-00-39-E7-E4-05-8C-C2-08-4C-AC-96-9A-D0-DE-C4-85-EE-07-00-D8-00-A6-A0-EC-B2-39-4D-BD-2F-31-4C-F7-4F-3F-4C-00-00-00-14-1A-99-00-00-46-7D-7B-CD-1B-EA-FC-40-9C-37-A9-7E-B0-DD-4C-16-00-08-65-53-76-46-00-00,,74364e3b-32b7-4108-9a2c-6dda84007459,15.01.2507.039",
+            "microsoft": {
+                "exchange": {
+                    "customdata": "S:ItemEntryId=00-00-00-00-39-E7-E4-05-8C-C2-08-4C-AC-96-9A-D0-DE-C4-85-EE-07-00-D8-00-A6-A0-EC-B2-39-4D-BD-2F-31-4C-F7-4F-3F-4C-00-00-00-14-1A-99-00-00-46-7D-7B-CD-1B-EA-FC-40-9C-37-A9-7E-B0-DD-4C-16-00-08-65-53-76-46-00-00",
+                    "eventid": "NOTIFYMAPI",
+                    "logid": "74364e3b-32b7-4108-9a2c-6dda84007459",
+                    "messageinfo": "2025-01-14T09:28:39.321Z;LSRV=HELLOWORLD.foo.example.com:TOTAL-SUB=0.013|SA=0.013|MTSS-PEN=0.000",
+                    "schemaversion": "15.01.2507.039",
+                    "source": "STOREDRIVER",
+                    "sourcecontext": "MDB:f4adaa08-ff49-4bca-ba70-68e9b80597b0, Mailbox:84dd42d9-8814-4758-a8bb-e8765d62ec90, Event:190620893, MessageClass:IPM.Note, CreationTime:2025-01-14T09:28:39.321Z, ClientType:MOMT, SubmissionAssistant:MailboxTransportSubmissionEmailAssistant"
+                }
+            },
+            "tags": [
+                "preserve_original_event"
+            ]
         }
     ]
-}
\ No newline at end of file
+}
diff --git a/packages/microsoft_exchange_server/data_stream/messagetracking/elasticsearch/ingest_pipeline/default.yml b/packages/microsoft_exchange_server/data_stream/messagetracking/elasticsearch/ingest_pipeline/default.yml
index 1b9e5c7d38..efcc35b3bb 100644
--- a/packages/microsoft_exchange_server/data_stream/messagetracking/elasticsearch/ingest_pipeline/default.yml
+++ b/packages/microsoft_exchange_server/data_stream/messagetracking/elasticsearch/ingest_pipeline/default.yml
@@ -52,10 +52,10 @@ processors:
     field: email.local_id
     copy_from: microsoft.exchange.networkmessageid
     ignore_empty_value: true
-- append:
+- set:
     field: email.sender.address
-    value: "{{{microsoft.exchange.senderaddress}}}"
-    ignore_failure: true
+    copy_from: microsoft.exchange.senderaddress
+    ignore_empty_value: true
 - dissect:
     field: microsoft.exchange.customdata
     pattern: "%{}S:OriginalFromAddress=%{_tmp.email.from.address};"
@@ -63,13 +63,12 @@ processors:
 - append:
     field: email.from.address
     value: "{{{_tmp.email.from.address}}}"
-    if: ctx._tmp?.email?.from?.address != null && ctx?._tmp?.email?.from?.address != "<>"
+    if: ctx._tmp?.email?.from?.address != null && ctx._tmp.email.from.address != "<>"
     ignore_failure: true
 - append:
     field: email.from.address
     value: "{{{microsoft.exchange.senderaddress}}}"
-    if: ctx.email?.from?.address == null
-    ignore_failure: true
+    if: ctx.microsoft?.exchange?.senderaddress != null && ctx.microsoft.exchange.senderaddress != ""
 - remove:
     field: microsoft.exchange.senderaddress
     ignore_missing: true
@@ -84,7 +83,7 @@ processors:
     ignore_failure: true
     ignore_missing: true
 - set:
-    field: event.ingested 
+    field: event.ingested
     copy_from: _ingest.timestamp
     ignore_failure: true
 - remove:

...rosoft_exchange_server/data_stream/messagetracking/elasticsearch/ingest_pipeline/default.yml
Comment on lines 55 to 57
- append:
field: email.sender.address
value: "{{{microsoft.exchange.senderaddress}}}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to ECS, email.sender.address should be a scalar. So I think we should have

- set:
    field: email.sender.address
    copy_from: microsoft.exchange.senderaddress
    ignore_empty_value: true

This also handles the case where senderaddress is empty.

...rosoft_exchange_server/data_stream/messagetracking/elasticsearch/ingest_pipeline/default.yml
@@ -71,6 +72,7 @@ processors:
ignore_failure: true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The above append processor needs to handle the case where senderaddress is emtpy. The current if condition looks wrong.

- append:
    field: email.from.address
    value: "{{{microsoft.exchange.senderaddress}}}"
    if: ctx.microsoft?.exchange?.senderaddress != null && ctx.microsoft.exchange.senderaddress != ""

@andrewkroh andrewkroh changed the title Added fields to ignore if empty. [MS Exchange Server] Added fields to ignore if empty. Jan 31, 2025
@qcorporation qcorporation force-pushed the allow_empty_fields_in_the_exchange_integration branch from cbae7cb to c30036b Compare February 3, 2025 22:34
@qcorporation qcorporation requested review from a team as code owners February 3, 2025 22:34
@elasticmachine
Copy link

💔 Build Failed

Failed CI Steps

History

@andrewkroh andrewkroh added Integration:1password 1Password Integration:abnormal_security Abnormal Security New Integration Issue or pull request for creating a new integration package. Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations] labels Feb 3, 2025
@elasticmachine
Copy link

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewkroh andrewkroh andrewkroh left review comments

@faec faec Awaiting requested review from faec faec was automatically assigned from elastic/elastic-agent-data-plane

@khushijain21 khushijain21 Awaiting requested review from khushijain21 khushijain21 was automatically assigned from elastic/elastic-agent-data-plane

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
bugfix Pull request that fixes a bug issue Integration:microsoft_exchange_server Microsoft Exchange Server New Integration Issue or pull request for creating a new integration package. Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations] Team:Security-Windows Platform Security Windows Platform Team [elastic/sec-windows-platform]
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@matthewscherer @elasticmachine @marc-gr @andrewkroh