Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve logging section #679

Merged
merged 2 commits into from
Dec 21, 2024
Merged

Improve logging section #679

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
080e67c
logging section revisions
NoahMaizels Dec 3, 2024
da9b661
Improve logging section
NoahMaizels Dec 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
268 changes: 140 additions & 128 deletions docs/bee/working-with-bee/logs-and-files.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,172 +1,184 @@
---
title: Logs and Files
id: logs-and-files
---

### Linux
# Logging in Bee

If you have installed Bee on Linux using a package manager you will now be able to manage your Bee service using `systemctl`.
This section introduces logging in Bee, detailing log locations, exporting logs, setting the general verbosity level, and setting fine-grained verbosity for individual loggers.

## Log Locations

:::warning
Bee logs by default can be quite verbose, and over time may occupy disk space in the gigabytes range. You may wish to practice log rotation to prevent excessive disk utilization.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add an external link to log rotation?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed both in #681

Thanks for the feedback!

:::

### **Linux (Package Manager Installation)**
When installed via a package manager (e.g., `APT`, `RPM`), Bee runs as a **systemd service**, and logs are managed by the system journal, **journalctl**.

View logs using:
```bash
systemctl status bee
journalctl --lines=100 --follow --unit bee
```

Export all logs as JSON:
```bash
journalctl --unit bee --output=json > bee-logs.json
```
● bee.service - Bee - Ethereum Swarm node
Loaded: loaded (/lib/systemd/system/bee.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2020-11-20 23:50:15 GMT; 6s ago

Export logs for a specific time range:
```bash
journalctl --since "1 hour ago" --output=json --unit bee > bee-logs.json
```

Logs are available using the `journalctl` command:
Learn more about `journalctl` usage and filtering logs in this [tutorial](https://www.digitalocean.com/community/tutorials/how-to-use-journalctl-to-view-and-manipulate-systemd-logs) from DigitalOcean.


### **macOS (Homebrew Installation)**

For a Homebrew installation on macOS, logs are saved to:
```bash
journalctl --lines=100 --follow --unit bee
/usr/local/var/log/swarm-bee/bee.log
```

View logs in real time:
```bash
tail -f /usr/local/var/log/swarm-bee/bee.log
```

```text
INFO[2021-02-09T18:55:11Z] swarm public key 03379f7aa673b7f03737064fd23ba1453619924a4602e70bbccc133ba67d0968bd
DEBU[2021-02-09T18:55:11Z] using existing libp2p key
DEBU[2021-02-09T18:55:11Z] using existing pss key
INFO[2021-02-09T18:55:11Z] pss public key 03bae655ce94431e1f2c2de8d017f88c8c5c293ef0057379223084aba9e318596e
INFO[2021-02-09T18:55:11Z] using ethereum address 99c9e7868d22244106a5ffbc2f5d6b7c88e2c85a
INFO[2021-02-09T18:55:14Z] using default factory address for chain id 5: f0277caffea72734853b834afc9892461ea18474
INFO[2021-02-09T18:55:14Z] no chequebook found, deploying new one.
WARN[2021-02-09T18:55:15Z] cannot continue until there is sufficient ETH (for Gas) and at least 10 BZZ available on 99c9e7868d22244106a5ffbc2f5d6b7c88e2c85a

### **Docker**

Docker saves **stdout** and **stderr** output as JSON files by default. The logs are stored under:

```
/var/lib/docker/containers/<container-id>/<container-id>-json.log
```

### MacOS
View logs in real time:
```bash
docker logs -f <container-name>
```

Services are managed using Homebrew services.
Export logs to a file:
```bash
docker logs <container-name> > bee-logs.json
```

Export logs for a specific time range:
```bash
brew services restart swarm-bee
docker logs --since "30m" <container-name> > bee-logs.json
```

Logs are available at `/usr/local/var/log/swarm-bee/bee.log`
See [Docker documentation](https://docs.docker.com/reference/cli/docker/container/logs/) for additional options.


### **Shell Script**

For a shell script-installed Bee started using `bee start`, logs are sent to **stdout** and **stderr** by default, which means they will appear in the terminal. They are **not saved to disk by default**.

To save logs to a file, redirect **stdout** and **stderr**:

```bash
tail -f /usr/local/var/log/swarm-bee/bee.log
bee start --password <password> > bee.log 2>&1 &
```

View recent logs and follow for updates:
```bash
tail -f bee.log
```

## Data Locations
## Logging Levels

Bee supports the following log levels:

| Level | Description |
|-------------|------------------------------------|
| `0=silent` | No logs. |
| `1=error` | Critical errors only. |
| `2=warn` | Warnings and errors. |
| `3=info` | General operational logs (default).|
| `4=debug` | Detailed diagnostic logs. |
| `5=trace` | Highly granular logs for debugging.|

### Behavior of Log Levels

Log levels are cumulative: setting a higher verbosity includes all lower levels.
For example, `debug` will output logs at `debug`, `info`, `warn`, and `error` levels.


### Bee
## Setting Verbosity

Configuration files are stored in `/etc/bee/`. State, chunks and other data are stored in `/var/lib/bee/`
The general verbosity level can be set using the `verbosity` configuration option in order to display all log messages up to the selected level of verbosity.

### **YAML Config File**
Set the `verbosity` parameter in `config.yaml`:

```yaml
# Log verbosity: 0=silent, 1=error, 2=warn, 3=info, 4=debug, 5=trace
verbosity: debug
```

### **Command Line Flag**
Specify verbosity when starting Bee:

```bash
bee start --verbosity debug
```

## Logging Guidelines
### **Environment Variable**
Set `BEE_VERBOSITY` before starting Bee:

The Bee logs provide a robust information on the workings of a Bee node which are useful both to node operators and to Bee developers. Log messages have four levels described below, and logs can be adjusted for verbosity and granularity to suit the needs of the user.
```bash
export BEE_VERBOSITY=debug
bee start
```

### Log Levels

The log messages are divided into four basic levels:
## Fine-Grained Logging Control

- `Error` - Errors in the node. Although the node operation may continue, the error indicates that it should be addressed.
- `Warning` - Warnings should be checked in case the problem recurs to avoid potential damage.
- `Info` - Informational messages useful for node operators that do not indicate any fault or error.
- `Debug` - Information concerning program logic decisions, diagnostic information, internal state, etc. which is primarily useful for developers.
Bee allows fine-grained control of logging levels for specific subsystems using the **`/loggers` API endpoint**. This enables adjustments without restarting the node.

There is a notion of `V-level` attached to the `Debug` level. `V-levels` provide a simple way of changing the verbosity of debug messages. `V-levels` provide a way for a given package to distinguish the relative importance or verbosity of a given log message. Then, if a particular logger or package logs too many messages, the package can simply change the `V` level for that logger.
### **1. Retrieving Loggers List**

### Logging API usage
Retrieve the list of active loggers and their verbosity levels:
```bash
curl http://localhost:1633/loggers | jq
```

In the current Bee code base, it is possible to change the granularity of logging for some services on the fly without the need to restart the node. These services and their corresponding loggers can be found using the `/loggers` endpoint. Example of the output:
The list of loggers includes detailed entries for each subsystem. Below is an example for the `node/api` logger:

```json
{
"tree": {
"node": {
"/": {
"api": {
"+": [
"info|node/api[0][]>>824634933256"
]
},
"batchstore": {
"+": [
"info|node/batchstore[0][]>>824634933256"
]
},
"leveldb": {
"+": [
"info|node/leveldb[0][]>>824634933256"
]
},
"pseudosettle": {
"+": [
"info|node/pseudosettle[0][]>>824634933256"
]
},
"pss": {
"+": [
"info|node/pss[0][]>>824634933256"
]
},
"storer": {
"+": [
"info|node/storer[0][]>>824634933256"
]
}
},
"+": [
"info|node[0][]>>824634933256"
]
}
},
"loggers": [
{
"logger": "node/api",
"verbosity": "info",
"subsystem": "node/api[0][]>>824634933256",
"id": "bm9kZS9hcGlbMF1bXT4-ODI0NjM0OTMzMjU2"
},
{
"logger": "node/storer",
"verbosity": "info",
"subsystem": "node/storer[0][]>>824634933256",
"id": "bm9kZS9zdG9yZXJbMF1bXT4-ODI0NjM0OTMzMjU2"
},
{
"logger": "node/pss",
"verbosity": "info",
"subsystem": "node/pss[0][]>>824634933256",
"id": "bm9kZS9wc3NbMF1bXT4-ODI0NjM0OTMzMjU2"
},
{
"logger": "node/pseudosettle",
"verbosity": "info",
"subsystem": "node/pseudosettle[0][]>>824634933256",
"id": "bm9kZS9wc2V1ZG9zZXR0bGVbMF1bXT4-ODI0NjM0OTMzMjU2"
},
{
"logger": "node",
"verbosity": "info",
"subsystem": "node[0][]>>824634933256",
"id": "bm9kZVswXVtdPj44MjQ2MzQ5MzMyNTY="
},
{
"logger": "node/leveldb",
"verbosity": "info",
"subsystem": "node/leveldb[0][]>>824634933256",
"id": "bm9kZS9sZXZlbGRiWzBdW10-PjgyNDYzNDkzMzI1Ng=="
},
{
"logger": "node/batchstore",
"verbosity": "info",
"subsystem": "node/batchstore[0][]>>824634933256",
"id": "bm9kZS9iYXRjaHN0b3JlWzBdW10-PjgyNDYzNDkzMzI1Ng=="
}
]
"logger": "node/api",
"verbosity": "info",
"subsystem": "node/api[0][]>>824634474528",
"id": "bm9kZS9hcGlbMF1bXT4-ODI0NjM0NDc0NTI4"
}
```

The recorders come in two versions. The first is the tree version and the second is the flattened version. The `subsystem` field is the unique identifier of the logger. The `id` field is a version of the `subsystem` field encoded in base 64 for easier reference to a particular logger. The node name of the version tree is composed of the subsystem with the log level prefix and delimited by the `|` character. The number in the first square bracket indicates the logger's V-level.
- **`id`**: The Base64-encoded identifier used to adjust the logger’s verbosity.
- **`verbosity`**: The current log level.


### **2. Adjusting Logger Verbosity**

You can dynamically adjust the log level for any logger without restarting Bee.

**Syntax**:
```bash
curl -X PUT http://localhost:1633/loggers/<id>/<verbosity>
```

- **`<id>`**: The Base64-encoded logger name retrieved from `/loggers`.
- **`<verbosity>`**: Desired log level (`none`, `error`, `warn`, `info`, `debug`, `trace`).

**Example**: Set `node/api` to `debug`:
```bash
curl -X PUT http://localhost:1633/loggers/bm9kZS9hcGlbMF1bXT4-ODI0NjM0NDc0NTI4/debug
```

The logger endpoint uses HTTP PUT requests to modify the verbosity of the logger(s). The request must have the following parameters `/loggers/{subsystem}/{verbosity}`. The `{subsytem}` parameter is the base64 version of the subsytem field or regular expression corresponding to multiple subsystems. Since the loggers are arranged in tree structure, it is possible to turn on/off or change the logging level of the entire tree or just its branches with a single command. The verbosity can be one of `none`, `error`, `warning`, `info`, `debug` or a number in the range `1` to `1<<<31 - 1` to enable the verbosity of a particular V-level, if available for a given logger. A value of `all` will enable the highest verbosity of V-level.
### Log Level Behavior Note

Examples:
Log levels are cumulative. When a logger is set to a specific level, it will include all log messages at that level and below.

`curl -XPUT http://localhost:1633/loggers/bm9kZS8q/none` - will disable all loggers; `bm9kZS8q` is base64 encoded `node/*` regular expression.
For example:
- Setting a logger to `info` will show logs at `info`, `warn`, and `error`.
- Logs at higher levels (`debug` and `trace`) will **not** be displayed.

`curl -XPUT http://localhost:1633/loggers/bm9kZS9hcGlbMV1bXT4-ODI0NjM0OTMzMjU2/error` - will set the verbosity of the logger with the subsystem `node/api[1][]>>824634933256` to `error`.