Add steps and details for running 'unprivileged' Elastic Agent (#1087)

* Add docs and reqs for running unprivileged agent

* Fill in a few details

* Touchup

* touchup

* Add note about adding agent user to higher privileges group

* Update for Blake's comments

* Add warning & troubleshooting for sudo issue

* update command reference

* Add 'Agent and dashboard behaviors in unprivileged mode' section

* Address Blake's comments

* fixup

* Remove resolution for disk volume issue

* Update three dashboard issues

* fixup

* Fix Linux 'System integration' issue resolutions

docs/en/ingest-management/commands.asciidoc

@@ -569,6 +569,7 @@ elastic-agent install --url <string>
                       [--proxy-url <string>]
                       [--staging <string>]
                       [--tag <string>]
+                      [--unprivileged]
                       [global-flags]
 ----
 
@@ -611,6 +612,7 @@ elastic-agent install --fleet-server-es <string>
                       [--proxy-url <string>]
                       [--staging <string>]
                       [--tag <string>]
+                      [--unprivileged]
                       [--url <string>] <3>
                       [global-flags]
 ----
@@ -778,6 +780,11 @@ use these tags to filter the list of agents in {fleet}.
 NOTE: Currently, there is no way to remove or edit existing tags. To change the
 tags, you must unenroll the {agent}, then re-enroll it using new tags.
 
+`--unprivileged`::
+Run {agent} without full superuser privileges. 
+This option is useful in organizations that limit `root` access on Linux or macOS systems, or `admin` access on Windows systems.
+For details and limitations for running {agent} in this mode, refer to <<elastic-agent-unprivileged>>.
+
 `--url <string>`::
 {fleet-server} URL to use to enroll the {agent} into {fleet}.
 

docs/en/ingest-management/elastic-agent/elastic-agent-unprivileged-mode.asciidoc

@@ -0,0 +1,184 @@
+[[elastic-agent-unprivileged]]
+= Run {agent} without administrative privileges
+
+Beginning with {stack} version 8.15, {agent} is no longer required to be run by a user with superuser privileges. You can now run agents in an `unprivileged` mode that does not require `root` access on Linux or macOS, or `admin` access on Windows. Being able to run agents without full administrative privileges is often a requirement in organizations where this kind of access is often very limited.
+
+In general, agents running without full administrative privileges will perform and behave exactly as those run by a superuser. There are certain integrations and datastreams that are not available, however. If an integration requires root access, this is <<unprivileged-integrations,indicated on the integration main page>>.
+
+// Add mention of the System integration data streams.
+
+You can also <<unprivileged-change-mode,change the privilege mode>> of an {agent} after it has been installed.
+
+Refer to <<unprivileged-command-behaviors>> and <<unprivileged-running>> for the requirements and steps associated with running an agent without full `root` or `admin` superuser privileges.
+
+* <<unprivileged-running>>
+* <<unprivileged-command-behaviors>>
+* <<unprivileged-integrations>>
+* <<unprivileged-view-mode>>
+* <<unprivileged-change-mode>>
+
+[discrete]
+[[unprivileged-running]]
+== Run {agent} in `unprivileged` mode
+
+To run {agent} without administrative privileges you use exactly the same commands that you use for {agent} otherwise, with one exception. When you run the <<elastic-agent-install-command,`elastic-agent install`>> command, add the `--unprivileged` flag. For example:
+
+[source,shell]
+----
+elastic-agent install \
+  --url=https://cedd4e0e21e240b4s2bbbebdf1d6d52f.fleet.eu-west-1.aws.cld.elstc.co:443 \
+  --enrollment-token=NEFmVllaa0JLRXhKebVKVTR5TTI6N2JaVlJpSGpScmV0ZUVnZVlRUExFQQ== \
+  --unprivileged
+----
+
+IMPORTANT: On Linux systems, when you install {agent} using the `--unprivileged` flag, {agent} commands should not be run with `sudo`.
+Doing so may result in <<agent-sudo-error,an error>> due to the agent not having the required privileges.
+
+[discrete]
+[[unprivileged-command-behaviors]]
+== Agent and dashboard behaviors in unprivileged mode
+
+In addition to the <<unprivileged-integrations,integrations that are not available>> when {agent} is run in unpriviledged mode, certain data streams are also not available. The following tables show, for different operating systems, the impact when the agent does not have full administrative privileges. In most cases the limitations can be mediated by granting permissions for a user or group to the files indicated.
+
+.macOS
+[options,header]
+|===
+|Action |Behavior in unprivileged mode |Resolution
+
+|Run {agent} with the System integration
+|Log file error: `Unexpected file opening error: Failed opening /var/log/system.log: open /var/log/system.log: permission denied`.
+|Give read permission to the `elastic-agent` group for the `/var/log/system.log` file to fix this error.
+
+|Run {agent} with the System integration
+|On the `[Logs System] Syslog` dashboard, the `Syslog events by hostname`, `Syslog hostnames and processes` and `Syslog logs` visualizations are are missing data.
+|Give read permission to the `elastic-agent` group for the `/var/log/system.log` file to fix the missing visualizations.
+
+|Run {agent} with the System integration
+|On the `[Metrics System] Host overview` dashboard, only the processes run by the `elastic-agent-user` user are shown in the CPU and memory usage lists. 
+|To fix the missing processes in the visualization lists you can add add the `elastic-agent-user` user to the system `admin` group. Note that while this mitigates the issue, it also grants `elastic-agent user` with more permissions than may be desired.
+
+|Run {agent} and access the {agent} dashboards
+|On the `[Elastic Agent] Agents info` dashboard, visualizations including `Most Active Agents` and `Integrations per Agent` are missing data.
+|To fix the missing data in the visualizations you can add add the `elastic-agent-user` user to the system `admin` group. Note that while this mitigates the issue it also grants `elastic-agent user` with more permissions than may be desired.
+
+|Run {agent} and access the {agent} dashboards
+|On the `[Elastic Agent] Integrations` dashboard, visualizations including `Integration Errors Table`, `Events per integration` and `Integration Errors` are missing data.
+|To fix the missing data in the visualizations you can add add the `elastic-agent-user` user to the system `admin` group. Note that while this mitigates the issue it also grants `elastic-agent user` with more permissions than may be desired.
+
+|===
+
+.Linux
+[options,header]
+|===
+|Action |Behavior in unprivileged mode |Resolution
+
+|Run {agent} with the System integration
+|Log file error: `[elastic_agent.filebeat][error] Harvester could not be started on new file: /var/log/auth.log.1, Err: error setting up harvester: Harvester setup failed. Unexpected file opening error: Failed opening /var/log/auth.log.1: open /var/log/auth.log.1: permission denied`
+|To avoid the error you can add add the `elastic-agent-user` user to the `adm` group. Note that while this mitigates the issue it also grants `elastic-agent user` with more permissions than may be desired.
+
+|Run {agent} with the System integration
+|Log file error: `[elastic_agent.metricbeat][error] error getting filesystem usage for /run/user/1000/gvfs: error in Statfs syscall: permission denied`
+|To avoid the error you can add add the `elastic-agent-user` user to the `adm` group. Note that while this mitigates the issue it also grants `elastic-agent user` with more permissions than may be desired.
+
+|Run {agent} with the System integration
+|On the `[Logs System] Syslog` dashboard, the `Syslog events by hostname`, `Syslog hostnames and processes` and `Syslog logs` visualizations are are missing data.
+|To fix the missing data in the visualizations you can add add the `elastic-agent-user` user to the `adm` group. Note that while this mitigates the issue it also grants `elastic-agent user` with more permissions than may be desired.
+
+|Run {agent} and access the {agent} dashboards
+|On the `[Elastic Agent] Agents info` dashboard, visualizations including `Most Active Agents` and `Integrations per Agent` are missing data.
+|Giving read permission to the `elastic-agent` group for the `/var/log/system.log` file will partially fix the visualizations, but errors may still occur because the `elastic-agent-user` does not have read access to files in the `/run/user/1000/` directory.
+// It'd be nice if we can expand on this, even if just to say why that read access can't be given.
+
+|Run {agent} and access the {agent} dashboards
+|On the `[Elastic Agent] Integrations` dashboard, visualizations including `Integration Errors Table`, `Events per integration` and `Integration Errors` are missing data.
+|Give read permission to the `elastic-agent` group for the `/var/log/system.log` file to fix the missing visualizations.
+
+|===
+
+.Windows
+[options,header]
+|===
+|Action |Behavior in unprivileged mode |Resolution
+
+|Run {agent} with the System integration
+|Log file error: `failed to open Windows Event Log channel "Security": Access is denied`
+|Add the `elastic-agent-user` user to the `Event Log Users` group to fix this error.
+
+|Run {agent} with the System integration
+|Log file error: `cannot open new key in the registry in order to enable the performance counters: Access is denied`
+|Update the permissions for the `HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PartMgr` registry to fix this error.
+
+|Run {agent} with the System integration
+|Most of the System and {agent} dashboard visualizations are missing all data.
+|Add the `elastic-agent-user` user to the `Event Log Users` group and update the permissions for the `HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PartMgr` registry to fix the missing visualizations.
+
+Note that the `elastic-agent-user` user may still not have access to all processes, so the lists in the `Top processes by CPU usage` and `Top processes by memory usage` visualizations may be incomplete. 
+
+|Run {agent} with the System integration
+|On the `[Metrics System] Host overview` dashboard, the `Disk usage` visualizations are missing data. 
+|This occurs because direct access to the disk or a volume is restricted and not available to users without administrative privileges. Refer to link:https://learn.microsoft.com/en-us/windows/win32/secbp/running-with-special-privileges[Running with Special Privileges] in the Microsoft documentation for details.
+
+|===
+
+[discrete]
+[[unprivileged-integrations]]
+== Using Elastic integrations
+
+// Add mention of the System integration data streams.
+
+Most Elastic integrations support running {agent} in unprivileged mode. For the exceptions, any integration that requires {agent} to have root privileges has the requirement indicated at the top of the integration page in {kib}:
+
+[role="screenshot"]
+image::images/integration-root-requirement.png[Elastic Defend integration page showing root requirement]
+
+As well, a warning is displayed in {kib} if you try to add an integration that requires root privileges to an {agent} policy that has agents enrolled in unprivileged mode.
+
+[role="screenshot"]
+image::images/unprivileged-agent-warning.png[Warning indicating that root privileged agent is required for an integration]
+
+Examples of integrations that require {agent} to have administrative privileges are:
+
+* link:https://docs.elastic.co/en/integrations/endpoint[{elastic-defend}]
+* link:https://docs.elastic.co/integrations/auditd_manager[Auditd Manager]
+* link:https://docs.elastic.co/integrations/fim[File Integrity Monitoring]
+* link:https://docs.elastic.co/integrations/network_traffic[Network Packet Capture]
+* link:https://docs.elastic.co/integrations/system_audit[System Audit]
+* link:https://docs.elastic.co/integrations/profiler_agent[Universal Profiling Agent]
+
+[discrete]
+[[unprivileged-view-mode]]
+== Viewing an {agent} privilege mode
+
+For any {agent} policy you can view the number of agents that are currently running in privileged or unprivileged mode:
+
+. In {fleet}, open the **Agent policies** tab.
+
+. Click the agent policy to view the policy details.
+
+The number of agents enrolled with the policy is shown. Hover over the link to view the number of privileged and unpriviled agents.
+
+[role="screenshot"]
+image::images/privileged-and-unprivileged-agents.png[Agent policy tab showing 1 unprivileged agent and 0 privileged enrolled agents]
+
+In the event that the {agent} policy has integrations installed that require root privileges, but there are agents running without root privileges, this is shown in the tooltip.
+
+[role="screenshot"]
+image::images/root-integration-and-unprivileged-agents.png[Agent policy tab showing 1 unprivileged agent and 0 privileged enrolled agents]
+
+[discrete]
+[[unprivileged-change-mode]]
+== Changing an {agent}'s privilege mode
+
+If an agent doesn't have the right level of privilege to read a data source, you can adjust the agent's privileges by adding `elastic-agent-user` to the user group that has privileges to read the data source.
+
+As background, when you run {agent} in `unprivileged` mode, one user and one group are created on the host. The same names are used for all operating systems:
+
+* `elastic-agent-user`: The user that is created and that the {agent} service runs as.
+* `elastic-agent`: The group that is created. Any user in this group has access to control and communicate over the control protocol to the {agent} daemon.
+
+For example:
+
+. When you install {agent} with the `--unprivileged` setting, the `elastic-agent-user` user and the `elastic-agent` group are created automatically.
+. If you then want your user `myuser` to be able to run an {agent} command such as `elastic-agent status`, add the `myuser` user to the `elastic-agent` group.
+. Then, once added to the group, the `elastic-agent status` command will work. Prior to that, the user `myuser` running the command will result in a permission error that indicates a problem communicating with the control socket.
+

docs/en/ingest-management/index.asciidoc

@@ -63,6 +63,8 @@ include::elastic-agent/ingest-pipeline-kubernetes.asciidoc[leveloffset=+3]
 
 include::elastic-agent/configuration/env/container-envs.asciidoc[leveloffset=+3]
 
+include::elastic-agent/elastic-agent-unprivileged-mode.asciidoc[leveloffset=+2]
+
 include::elastic-agent/install-agent-msi.asciidoc[leveloffset=+2]
 
 include::elastic-agent/installation-layout.asciidoc[leveloffset=+2]

docs/en/ingest-management/tab-widgets/download.asciidoc

@@ -7,8 +7,12 @@ endif::[]
 
 ifeval::["{release-state}"!="unreleased"]
 
-IMPORTANT: To simplify upgrading to future versions of {agent}, we recommended
-that you use the tarball distribution instead of the DEB distribution.
+[IMPORTANT]
+====
+* To simplify upgrading to future versions of {agent}, we recommended
+that you use the tarball distribution instead of the RPM distribution.
+* You can install {agent} in an `unprivileged` mode that does not require `root` privileges. Refer to <<elastic-agent-unprivileged>> for details.
+====
 
 ["source","sh",subs="attributes"]
 ----
@@ -28,8 +32,12 @@ endif::[]
 
 ifeval::["{release-state}"!="unreleased"]
 
-IMPORTANT: To simplify upgrading to future versions of {agent}, we recommended
+[IMPORTANT]
+====
+* To simplify upgrading to future versions of {agent}, we recommended
 that you use the tarball distribution instead of the RPM distribution.
+* You can install {agent} in an `unprivileged` mode that does not require `root` privileges. Refer to <<elastic-agent-unprivileged>> for details.
+====
 
 ["source","sh",subs="attributes"]
 ----

docs/en/ingest-management/troubleshooting/troubleshooting.asciidoc

@@ -59,8 +59,7 @@ Find troubleshooting information for {fleet}, {fleet-server}, and {agent} in the
 * <<pgp-key-download-fail>>
 * <<fleet-server-integration-removed>>
 * <<agent-oom-k8s>>
-
-
+* <<agent-sudo-error>>
 
 
 [discrete]
@@ -807,3 +806,20 @@ To detect the problem, run the `kubectl describe pod` command and check the resu
 ----
 
 To resolve the problem, allocate additional memory to the agent and then restart it.
+
+[discrete]
+[[agent-sudo-error]]
+== Error when running {agent} commands with `sudo`
+
+On Linux systems, when you install {agent} <<elastic-agent-unprivileged,without administrative privileges>>, that is, using the `--unprivileged` flag,
+{agent} commands should not be run with `sudo`. Doing so may result in an error due to the agent not having the required privileges.
+
+For example, when you run {agent} with the `--unprivileged` flag, running the `elastic-agent inspect` command will result in an error like the following:
+
+[source,sh]
+----
+Error: error loading agent config: error loading raw config: fail to read configuration /Library/Elastic/Agent/fleet.enc for the elastic-agent: fail to decode bytes: cipher: message authentication failed
+----
+
+To resolve this, either run {agent} without the `--unprivileged` flag so that it has administrative access, or run the {agent} commands without the `sudo` prefix.
+