Create log data set monitoring docs (#3960)

docs/en/serverless/logging/log-monitoring.mdx

@@ -79,6 +79,13 @@ The following resources provide information on viewing and monitoring your logs:
 - <DocLink slug="/serverless/observability/discover-and-explore-logs">Discover and explore</DocLink>: Discover and explore all of the log events flowing in from your servers, virtual machines, and containers in a centralized view.
 - <DocLink slug="/serverless/observability/aiops-detect-anomalies">Detect log anomalies</DocLink>: Use ((ml)) to detect log anomalies automatically.
 
+## Monitor data sets
+
+The **Data Set Quality** page provides an overview of your data sets and their quality.
+Use this information to get an idea of your overall data set quality, and find data sets that contain incorrectly parsed documents.
+
+<DocLink id="serverlessObservabilityMonitorDatasets">Monitor data sets</DocLink>
+
 ## Application logs
 
 Application logs provide valuable insight into events that have occurred within your services and applications.

docs/en/serverless/logging/monitor-datasets.mdx

@@ -0,0 +1,64 @@
+---
+id: serverlessObservabilityMonitorDatasets
+slug: /serverless/observability/monitor-datasets
+title: Monitor log data set quality
+description: Monitor log data sets to find degraded documents.
+tags: [ 'serverless', 'observability', 'how-to' ]
+---
+
+<p><DocBadge template="technical preview" /></p>
+<p><DocBadge template="beta" /></p>
+
+The **Data Set Quality** page provides an overview of your log data sets.
+Use this information to get an idea of your overall log data set quality and find data sets that contain incorrectly parsed documents.
+Access the Data Set Quality page from the main ((kib)) menu and go to **Stack Management** → **Data Set Quality**.
+
+<DocImage size="2" url="../images/logs-dataset-overview.png" alt="Screen capture of the data set overview" />
+
+<DocCallOut title="Requirements">
+  Users with the `viewer` role can view the Data Sets Quality summary. To view the Active Data Sets and Estimated Data summaries, users need the `monitor` [index privilege](((ref))/security-privileges.html#privileges-list-indices) for the `logs-*-*` index.
+</DocCallOut>
+
+The quality of your data sets is based on the percentage of degraded documents in each data set.
+A degraded document in a data set contains the [`_ignored`](((ref))/mapping-ignored-field.html) property because one or more of its fields were ignored during indexing.
+Fields are ignored for a variety of reasons.
+For example, when the [`ignore_malformed`](((ref))/mapping-ignored-field.html.html) parameter is set to true, if a document field contains the wrong data type, the malformed field is ignored and the rest of the document is indexed.
+
+From the data set table, you'll find information for each data set such as its namespace, when the data set was last active, and the percentage of degraded docs.
+The percentage of degraded documents determines the data set's quality according to the following scale:
+
+* Good (<DocImage flatImage alt="Good icon" url="../images/green-dot-icon.png" />): 0% of the documents in the data set are degraded.
+* Degraded (<DocImage flatImage alt="Degraded icon" url="../images/yellow-dot-icon.png" />): Greater than 0% and up to 3% of the documents in the data set are degraded.
+* Poor (<DocImage flatImage alt="Poor icon" url="../images/red-dot-icon.png" />): Greater than 3% of the documents in the data set are degraded.
+
+Opening the details of a specific data set shows the degraded documents history, a summary for the data set, and other details that can help you determine if you need to investigate any issues.
+
+## Investigate issues
+The Data Set Quality page has a couple of different ways to help you find ignored fields and investigate issues.
+From the data set table, you can expand a data set's details, and view commonly ignored fields and information about those fields.
+You can also open a data set in Logs Explorer to find ignored fields in individual logs.
+
+### Find ignored fields in data sets
+To expand the details of a dataset with poor or degraded quality and view ignored fields:
+
+1. From the data set table, click <DocIcon type="expand" title="expand icon" /> next to a data set with poor or degraded quality.
+1. From the details, scroll down to **Degraded fields**.
+
+The **Degraded fields** section shows fields that have been ignored, the number of documents that contain ignored fields, and the timestamp of last occurrence of the field being ignored.
+
+### Find ignored fields in individual logs
+To use Logs Explorer to find ignored fields in individual logs:
+
+1. Find data sets with degraded documents using the **Degraded Docs** column of the data sets table.
+1. Click the percentage in the **Degraded Docs** column to open the data set in Logs Explorer.
+
+The **Documents** table in Logs Explorer is automatically filtered to show documents that were not parsed correctly.
+Under the *actions* column, you'll find the degraded document icon (<DocIcon type="indexClose" title="degraded document icon" />).
+
+Now that you know which documents contain ignored fields, examine them more closely to find the origin of the issue:
+
+1. Under the **actions** column, click <DocIcon type="expand" title="expand icon" /> to open the log details.
+1. Select the **JSON** tab.
+1. Scroll towards the end of the JSON to find the `ignored_field_values`.
+
+Here, you'll find all of the `_ignored` fields in the document and their values, which should provide some clues as to why the fields were ignored.

docs/en/serverless/logging/view-and-monitor-logs.mdx

@@ -77,4 +77,10 @@ The following actions help you filter and focus on specific fields in the log de
 * **Filter for value (<DocIcon type="plusInCircle" title="filter for value icon" />):** Show logs that contain the specific field value.
 * **Filter out value (<DocIcon type="minusInCircle" title="filter out value icon" />):** Show logs that do _not_ contain the specific field value.
 * **Filter for field present (<DocIcon type="filter" title="filter for present icon" />):** Show logs that contain the specific field.
-* **Toggle column in table (<DocIcon type="listAdd" title="toggle column in table icon" />):** Add or remove a column for the field to the main Logs Explorer table.
+* **Toggle column in table (<DocIcon type="listAdd" title="toggle column in table icon" />):** Add or remove a column for the field to the main Logs Explorer table.
+
+## View log data set details
+
+From the main ((kib)) menu, go to **Stack Management** → **Data Set Quality* to view more details about your data sets and monitor their overall quality.
+
+Refer to <DocLink id="serverlessObservabilityMonitorDatasets">Monitor data sets</DocLink> for more information.

docs/en/serverless/serverless-observability.docnav.json

@@ -65,6 +65,11 @@
           "classic-sources": ["enObservabilityMonitorLogs"],
           "classic-skip": true
         },
+        {
+          "slug": "/serverless/observability/monitor-datasets",
+          "classic-sources": ["enObservabilityMonitorDatasets"],
+          "classic-skip": true
+        },
         {
           "slug": "/serverless/observability/run-log-pattern-analysis",
           "classic-sources": ["enKibanaRunPatternAnalysisDiscover"]