Skip to content

Commit

Permalink
Update tedge-log-management plugin spec
Browse files Browse the repository at this point in the history 
Signed-off-by: Rina Fujino <[email protected]>
  • Loading branch information
@rina23q
rina23q committed Jul 18, 2023
1 parent 50fd1f5 commit 47f7738
Showing 1 changed file with 32 additions and 47 deletions.
79 changes: 32 additions & 47 deletions docs/src/references/tedge-log-management.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,27 +7,27 @@ draft: true

# Log file management plugin

Thin-edge provides a command plugin to fetch log files from the device.
Thin-edge provides an operation plugin to fetch log files from the device.

* Log file management feature is provided with a `tedge-log-plugin` which runs as a daemon on thin-edge.
* The log file management feature is provided with a `tedge-log-plugin` which runs as a daemon on thin-edge.
* The device owner can define the list of log files in the plugin's configuration file named `tedge-log-plugin.toml`.
* Each entry in the the `tedge-log-plugin.toml` file contains a log `type` and a `path` pattern,
* Each entry in the `tedge-log-plugin.toml` file contains a log `type` and a `path` pattern,
where the `type` is used to represent the logical group of log files matching the `path` pattern.
* On receipt of a log file upload command for a given `type`,
* Upon receiving a log file upload command for a given `type`,
the log files for that type are retrieved using the `path` pattern defined in this `tedge-log-plugin.toml`,
matched against the requested time range, search text and maximum line count.
* The plugin uploads the requested log file to the tedge file transfer repository,
which url is given by the received log upload command.
* The list of managed log files in `tedge-log-plugin.toml` can be updated both locally as well as from cloud, for instance, by using the configuration management feature.
* However, the plugin provides no direct connection to clouds, which is responsibility of another component, i.e. cloud mapper.
* The plugin has dependency on the `tedge.toml` device configuration file to get the MQTT hostname, port and the device identifier.
* The plugin uploads the requested log file to the tedge file transfer repository.
Its url is given by the received log upload command.
* The list of managed log files in `tedge-log-plugin.toml` can be updated both locally as well as from clouds, for instance, by using the configuration management feature.
* However, the plugin provides no direct connection to clouds, which is the responsibility of another component, i.e. the cloud mapper.
* The plugin has a dependency on the `tedge.toml` configuration file to get the MQTT hostname, port, and device identifier.

## Installation

As part of this plugin installation:
* On systemd enabled devices, the service definition file for this `tedge-log-plugin` daemon is also installed as part of this plugin installation.
* On systemd-enabled devices, the service definition file for this `tedge-log-plugin` daemon is also installed.

Once installed, the `tedge-log-plugin` is run as a daemon on the device listening to log update commands on `<root>/<identifier>/cmd/log_upload/+` MQTT topic.
Once installed, the `tedge-log-plugin` runs as a daemon on the device, listening to log update commands on the [`<root>/<identifier>/cmd/log_upload/+` MQTT topic](mqtt-topic-design.md).

## Configuration

Expand All @@ -46,10 +46,13 @@ files = [
```

The `tedge-log-plugin` parses this configuration file on startup for all the `type` values specified,
and sends the supported log types message to MQTT local broker on `<root>/<identifier>/cmd/log_upload` topic with a retained flag as follows:
and sends the supported log types message to the MQTT local broker on the `<root>/<identifier>/cmd/log_upload` topic with a retained flag.
The `<root>/<identifier>` is defined in `tedge.toml` file as `device.topic`.

Given that `device.topic` is set to `te/device/main//`, the message to declare the supported log types is as follows.

```sh te2mqtt
tedge mqtt pub -r '<root>/<identifier>/cmd/log_upload' '{
tedge mqtt pub -r 'te/device/main///cmd/log_upload' '{
"types" : [ "mosquitto", "software-management", "c8y_CustomOperation" ]
}'
```
Expand All @@ -64,12 +67,11 @@ then a JSON message with an empty array for the `types` field is sent, indicatin

## Handling log upload commands

The plugin establishes the MQTT connection to a broker for `mqtt.bind.address` and `mqtt.bind.port` from the `tedge.toml` device configuration.
Then, it subscribes to the `<root>/<identifier>/cmd/log_upload/+` topic, listening for
log file upload commands.
The `<root>/<identifier>` is defined in `tedge.toml` file as the `xxx` key.
The plugin establishes an MQTT connection to the broker using the `mqtt.bind.address` and `mqtt.bind.port` values from the `tedge.toml` configuration.
Then, it subscribes to the `<root>/<identifier>/cmd/log_upload/+` topic, listening for log file upload commands.
The `<root>/<identifier>` is defined in `tedge.toml` file as the `device.topic` key.

For instance, if the `<root>/<identifier>` is defined as `te/device/main//`, the plugin subscribes to the following topic.
For instance, if `device.topic` is set to `te/device/main//`, the plugin subscribes to the following topic.

```sh te2mqtt
tedge mqtt sub 'te/device/main///cmd/log_upload/+'
Expand All @@ -94,23 +96,23 @@ retrieves the log files using the `target` glob pattern provided in the plugin c
including only the ones modified within the date range(`2013-06-22T17:03:14.000+02:00` to `2013-06-23T18:03:14.000+02:00`),
with the content filtered by the search text(`ERROR`) and the maximum line count(`1000`).

This filtered content is then uploaded to the given url as `uploadUrl` via HTTP PUT request.
This filtered content is then uploaded to the given URL as `uploadUrl` via an HTTP PUT request.

During the process, the plugin updates the command status via MQTT
by publishing a retained message to the topic `<root>/<identifier>/cmd/log_upload/<id>`,
which is the same topic as the one where the command is received.
which is the same topic where the command is received.

On a reception of a new log file upload command, the plugin updates the status to `executing `.
On a reception of the HTTP success status code after uploading a file to the file transfer repository, the plugin updates the status to `successful`.
If any unexpected error occurs, the plugin updates the status to `failed` with `reason`.
On the reception of a new log file upload command, the plugin updates the status to `executing `.
On the reception of the HTTP success status code after uploading a file to the file transfer repository, the plugin updates the status to `successful`.
If any unexpected error occurs, the plugin updates the status to `failed` with a `reason`.

So, the operation status update message for the above example looks like below.
Thus, the operation status update message for the above example looks like below.

```sh te2mqtt
tedge mqtt pub -r 'te/device/main///cmd/log_upload/1234' '{
"status": "failed",
"reason": "The target log file for 'mosquitto' does not exist.",
"uploadUrl": "http://127.0.0.1:8000/tedge/file-transfer/main/log_upload/mosquitto",
"uploadUrl": "http://127.0.0.1:8000/tedge/file-transfer/main/log_upload/mosquitto-1234",
"logType": "mosquitto",
"dateFrom": "2013-06-22T17:03:14.000+02:00",
"dateTo": "2013-06-22T18:03:14.000+02:00",
Expand Down Expand Up @@ -152,7 +154,7 @@ USAGE:
OPTIONS:
--config-dir <CONFIG_DIR>
[default: /etc/tedge/plugins]
[default: /etc/tedge]
--debug
Turn-on the debug log level.
Expand All @@ -165,31 +167,14 @@ OPTIONS:
-V, --version
Print version information
The thin-edge `CONFIG_DIR` is used to store:
* tedge-log-plugin.toml - the configuration file that specifies which logs to be retrieved
The thin-edge `CONFIG_DIR` is used:
* to find where to connect the MQTT bus: `tedge config get mqtt.bind.address` and `tedge config get mqtt.bind.port`
* to find where to publish/subscribe: `tedge config get device.topic`
* to store tedge-log-plugin.toml: the configuration file that specifies which logs to be retrieved
```

## Logging

The `tedge-log-plugin` reports progress and errors to the OS journal which can be retrieved using `journalctl`.

## Check points for developer (this section will be deleted later)

* Stop creating any supported operation file from the plugin. Mapper will address it on the reception of the log type declaration message.
* Renaming the toml file from `c8y-log-plugin.toml` to `tedge-log-plugin.toml`.
* The location of the toml file. Change from `/etc/tedge/c8y/c8y-log-plugin.toml` to `/etc/tedge/plugins/tedge-log-plugin.toml`.
* Create a `tedge-log-plugin.toml` file with example contents if it does not exist on startup.
* All MQTT messages must be published with a retain flag.
* All JSON deserializer must accept unknown fields.

### Drop 1 only

* The `<root>/<identifier>` is hard-coded to `te/device/main//`, thus, supporting only main device. So,
* The topic for log type declaration: hard-code to `te/device/main///cmd/log_upload`.
* The topic for receiving/updating operation: hard-code to `te/device/main///cmd/log_upload/+`. (`+` is for a command id)
* Use `mqtt.bind.address` and `mqtt.bind.port` from tedge config API for MQTT connection.

### Drop 2

* Get `<root/identifier>` from `xxx` of tedge config API. (The key `xxx` must be specified later.)

0 comments on commit 47f7738

Please sign in to comment.