ENH: Add ability to copy more fields from the report

Sometimes collectors sets more context information, e.g.
the source file name. Currently, parsers usually remove
those non-standard information. With this change, we let
the user decide to copy more information to the event.

In addition, the currently copied fields were documented.

CHANGELOG.md

@@ -32,6 +32,8 @@
 - `intelmq.bots.parsers.shadowserver._config`:
   - Fetch schema before first run (PR#2482 by elsif2, fixes #2480).
 - `intelmq.bots.parsers.dataplane.parser`: Use `  |  ` as field delimiter, fix parsing of AS names including `|` (PR#2488 by DigitalTrustCenter).
+- all parsers: add `copy_custom_fields` parameter allowing copying additional fields from the report, e.g. `extra.file_name`.
+  (PR# by Kamil Mankowski).
 
 #### Experts
 - `intelmq.bots.experts.sieve.expert`:

docs/user/bots.md

@@ -1331,6 +1331,17 @@ tweet text is sent separately and if allowed, links to pastebin are followed and
 
 ## Parser Bots
 
+If not set differently during parsing, all parser bots copy the following fields from the report to an event:
+
+  - `feed.accuracy`
+  - `feed.code`
+  - `feed.documentation`
+  - `feed.name`
+  - `feed.provider`
+  - `feed.url`
+  - `rtir_id`
+  - `time.observation`
+
 ### Common parameters
 
 #### `default_fields`
@@ -1346,6 +1357,18 @@ defaults_fields:
   protocol.transport: tcp
 ```
 
+#### `copy_custom_fields`
+
+(optional, list) List of additional fields to be copy from the report (only applied if parsing the
+event doesn't set the value).
+
+Example usage:
+
+```yaml
+copy_custom_fields:
+  - extra.file_name
+```
+
 ---
 
 ### Abuse.ch Feodo Tracker <div id="intelmq.bots.parsers.abusech.parser_feodotracker" />

intelmq/lib/bot.py

@@ -1082,6 +1082,7 @@ class ParserBot(Bot):
     _default_message_type = 'Report'
 
     default_fields: Optional[dict] = {}
+    copy_custom_fields: Optional[list] = []
 
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
@@ -1245,6 +1246,13 @@ def process(self):
                         for key, value in self.default_fields.items():
                             event.add(key, value, overwrite=False)
 
+                if self.copy_custom_fields:
+                    for key in self.copy_custom_fields:
+                        if key not in report:
+                            continue
+                        for event in events:
+                            event.add(key, report.get(key), overwrite=False)
+
             except Exception:
                 self.logger.exception('Failed to parse line.')
                 self.__failed.append((traceback.format_exc(), self._current_line))

intelmq/tests/lib/test_parser_bot.py

@@ -167,6 +167,19 @@ def test_bad_default_fields_parameter_2(self):
         self.assertAnyLoglineEqual(message="Invalid value of key 'source.port' in default_fields parameter.",
                                    levelname="ERROR")
 
+    def test_copy_custom_fields_from_report(self):
+        """Allow copying custom fields from the report message to support more context from reports"""
+        report = {**EXAMPLE_SHORT, "extra.file_name": "file.txt", "extra.field2": "value2"}
+        self.input_message = report
+
+        self.run_bot(parameters={"copy_custom_fields":
+                                 ["extra.file_name", "extra.not_exists"]})
+
+        output_message = EXAMPLE_EVENT.copy()
+        output_message["extra.file_name"] = "file.txt"
+        self.assertMessageEqual(0, output_message)
+
+
     def test_missing_raw(self):
         """ Test DummyParserBot with missing raw. """
         self.input_message = EXAMPLE_EMPTY_REPORT