beatshipper

Since there’s no way to send GNU zip files through filebeat, this service will be responsible for checking if there are new .gz files based on a path that we’ll explode using globbing, decompress them, send them using the filebeat service with the provided configuration and store them in a local file registry.

It can also be used for non-compressed files.

Requirements

🔔 Go Version 1.18 or higher

Installation

• Move to the cloned repository and install with:

$ go install

$ beatshipper -h
Sends data based on paths that will be exploded using GLOB
with the possibility of passing GNU Zip files also.

Usage:
  beatshipper [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  help        Help about any command
  send        Send the message as a beat

Flags:
  -h, --help      help for beatshipper
  -v, --version   version for beatshipper

Use "beatshipper [command] --help" for more information about a command.

$ beatshipper --version
beatshipper version 0.0.1

Configuration

We should create the configuration in any of the following both locations:

• /etc/beatshipper/beatshipper-conf.yml
• $HOME/.config/beatshipper-conf.yml

An example is located in the root of the project. We should add the following fields:

• host
• port
• path array of paths that will be exploded using GLOB
• registry: the name of the file where we'll store the processed files.
• recheck time of recheck (re-process) if there are new pending files to send
• logsource very useful in logstash to handle conditions based on the name of the source

Configuration through SystemD

We have created two SystemD files:

• service file: it specifies the binary that will execute the service
• timer file: we are gonna execute the service as timer.

By default the timer will launch the service in one shoot type every 30 minutes (*:0/30). If we wanna change the time we should change the OnCalendar directime in the timer file.

These files are being coppied to the following directory /lib/systemd/system/ if we install the generated RPM package.

We can enable and start the timer with the following commands:

$ systemctl enable beatshipper.timer
$ systemctl start beatshipper.timer

We can check if the timer has been activated:

$ systemctl list-timers | grep beatshipper -B1
NEXT                        LEFT          LAST                        PASSED       UNIT                           ACTIVATES
Mon 2022-12-26 16:00:00 CET 27min left    Mon 2022-12-26 15:30:32 CET 2min 12s ago beatshipper.timer              beatshipper.service

$ journalctl -u beatshipper.timer
dic 26 15:30:32 user systemd[1]: Started Beatshipper execution schedule.

Usage:

If we have the right configuration, we can start the program using the following command - It will read the configuration, start checking if there are pending packages that aren't in the registry and send them to the server that is listening using beats- e.g. a logstash instance with the beats module -:

$ beatshipper send
2022/11/18 21:04:53 shipper.go:77: Conexion successful with: localhost:5044
2022/11/18 21:04:53 shipper.go:82: Processing: /jenkinst-test
jenkinst-test/jenkins-logs/rcj/test/sub-test/sub-sub-test/var/log/extra/rpm-list.txt.gz
2022/11/18 21:04:53 shipper.go:82: Processing: /jenkinst-test
jenkinst-test/jenkins-logs/rcj/test/sub-test/sub-sub-test10/var/log/extra/rpm-list.txt.gz
2022/11/18 21:04:53 shipper.go:118: Chunk slice into: 5 slices
2022/11/18 21:04:53 shipper.go:96: Sending batch of data...
2022/11/18 21:04:53 shipper.go:96: Sending batch of data...
2022/11/18 21:04:53 shipper.go:96: Sending batch of data...
2022/11/18 21:04:53 shipper.go:101: Conexion closed
2022/11/18 21:04:53 registry.go:75: Added to registry: /jenkinst-test
jenkinst-test/jenkins-logs/rcj/test/sub-test/sub-sub-test/var/log/extra/rpm-list.txt.gz
2022/11/18 21:04:53 registry.go:75: Added to registry: /jenkinst-test
jenkinst-test/jenkins-logs/rcj/test/sub-test/sub-sub-test10/var/log/extra/rpm-list.txt.gz

Registry

Files will be stored in the following way:

$ jq < shipper_registry.json
{
  "ParsedFiles": [
    {
      "name": "/jenkinst-test/jenkins-logs/rcj/test/sub-test/sub-sub-test/var/log/extra/rpm-list.txt.gz",
      "date": "2022-11-18T21:04:53.773777134+01:00"
    },
    {
      "name": "/jenkinst-test/jenkins-logs/rcj/test/sub-test/sub-sub-test10/var/log/extra/rpm-list.txt.gz",
      "date": "2022-11-18T21:04:53.773781056+01:00"
    },
  ]
}