Skip to content

Commit

Permalink
Implement workflow for automatic ECS templates generation (#586)
Browse files Browse the repository at this point in the history 
* Add initial ecs Dockerfile

* Add basic entrypoint script

* Add dependencies installation to Dockerfile

* Remove unused variables

Rename local variables to lower case

* Remove upload related code on generate.sh tool

* Fix compatibility issues with latest python version

Update generate.sh script to use env variables

* Remove unnecesary script

* Rename ECS source directory

* Implement docker compose to ecs generator tool

* Working docker compose file for ECS generator

* Add doscstrings to Dockerfile

* Add script to execute the ECS generator tool

* Update ecs generator launcher script

* Mitigate warnings on container down and stop actions

Stop container after each execution

* Rename ecs util to mapping-generator

* Implement generate-ecs-mapping GHA workflow to auto-generate PRs with the updated template

* Update GHA plugins version

* Validate ECS generation workflow

* Fix generator.sh path on Dockerfile and update GHA to work with multiple modules at once

Filter out dirs that dont correspond to any template

* Define specific version for the ubuntu runner

Update GHA to only run when a json or yml file is edited

* Update ecs generator with latest changes

* Reove duplicated generate.sh tool

Move mapping generator to main ecs directory

* Update tool location paths

* Update GHA to use the new tool

Update ecs README with new tool usage

Update ECS generator script name

Remove unused index upload option

Fix issue with exit codes on branch checking

* Upload resulting templates as artifacts

Improve PR body styling

* Fix typo in ecs/README.md

* Remove hardcoded ECS version

Update GHA workflow to only work on official wazuh-indexer repo

Add PR generation tool description at main ECS README

* Add docstrings to functions

* Move repository check conditional to be job-level

* Test workflow

* Fix short description

* Fix attempt of relative path

* Update vulnerability.yml to validate ecs workflow

Signed-off-by: Kevin Ledesma <[email protected]>

* Test changes on inventory-hardware subset.yml

Signed-off-by: Kevin Ledesma <[email protected]>

* Update for Wazuh Indexer bot

* Undo changes to to trigger the build

* Update configure_git() condition

* Trigger automation

* Create ssh folder

* Fix template artifact upload

Add notify block in GHA with the PR link

* Fix ECS template backup directory

* Resolve the full path of the output directory

* Add ECS generator badge to README

* Fix body formatting

* Revert changes to ECS defintions

---------

Signed-off-by: Kevin Ledesma <[email protected]>
Co-authored-by: Álex Ruiz <[email protected]>
  • Loading branch information
@QU3B1M @AlexRuiz7
QU3B1M and AlexRuiz7 authored Jan 23, 2025
1 parent 12195a7 commit bdc2b0b
Show file tree
Hide file tree
Showing 10 changed files with 635 additions and 233 deletions.
48 changes: 48 additions & 0 deletions .github/workflows/generate-ecs-mappings.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
name: ECS Generator

on:
push:
paths:
- "ecs/**/*.json"
- "ecs/**/*.yml"

jobs:
run-ecs-generator:
if: github.repository == 'wazuh/wazuh-indexer'
runs-on: ubuntu-24.04

steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 2

- name: Extract branch name
shell: bash
run: echo "branch=${GITHUB_HEAD_REF:-${GITHUB_REF#refs/heads/}}" >> $GITHUB_OUTPUT
id: branch-name

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Set up Docker Compose
run: sudo apt-get install docker-compose

- name: Generate PR to wazuh-indexer-plugins
id: generate-pr
env:
GITHUB_TOKEN: ${{ secrets.ACTION_TOKEN }}
COMMITER_EMAIL: ${{ secrets.INDEXER_BOT_EMAIL }}
COMMITTER_USERNAME: "Wazuh Indexer Bot"
SSH_PRIVATE_KEY: ${{ secrets.INDEXER_BOT_PRIVATE_SSH_KEY }}
SSH_PUBLIC_KEY: ${{ secrets.INDEXER_BOT_PUBLIC_SSH_KEY }}
run: |
bash ecs/scripts/generate-pr-to-plugins.sh \
-b ${{ steps.branch-name.outputs.branch }} \
-o ecs-templates
- name: Upload artifact
uses: actions/upload-artifact@v4
with:
name: ecs-templates
path: "ecs-templates"
1 change: 1 addition & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
[![GHA precommit](https://github.com/opensearch-project/OpenSearch/actions/workflows/precommit.yml/badge.svg)](https://github.com/opensearch-project/OpenSearch/actions/workflows/precommit.yml)
[![Jenkins gradle check job](https://img.shields.io/jenkins/build?jobUrl=https%3A%2F%2Fbuild.ci.opensearch.org%2Fjob%2Fgradle-check%2F&label=Jenkins%20Gradle%20Check)](https://build.ci.opensearch.org/job/gradle-check/)
[![Build packages](https://github.com/wazuh/wazuh-indexer/actions/workflows/build.yml/badge.svg)](https://github.com/wazuh/wazuh-indexer/actions/workflows/build.yml)
[![ECS Generator](https://github.com/wazuh/wazuh-indexer/actions/workflows/generate-ecs-mappings.yml/badge.svg)](https://github.com/wazuh/wazuh-indexer/actions/workflows/generate-ecs-mappings.yml)


- [Welcome!](#welcome)
Expand Down
161 changes: 61 additions & 100 deletions ecs/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,71 +4,22 @@ This script generates the ECS mappings for the Wazuh indices.

### Requirements

- ECS repository clone. The script is meant to be launched from the root level of that repository.
- `Python` 3.6 or higher + `venv` module
- `jq`
- [Docker Compose](https://docs.docker.com/compose/install/)

### Folder structure

There is a folder for each module. Inside each folder, there is a `fields` folder with the required
files to generate the mappings. These are the inputs for the ECS generator.
There is a folder for each module. Inside each folder, there is a `fields` folder with the required files to generate the mappings. These are the inputs for the ECS generator.

### Usage

1. Get a copy of the ECS repository at the same level as the `wazuh-indexer` repo:

```console
git clone [email protected]:elastic/ecs.git
```

2. Install the dependencies:

```console
cd ecs
python3 -m venv env
source env/bin/activate
pip install -r scripts/requirements.txt
```

2. Copy the `generate.sh` script to the root level of the ECS repository.

```console
cp generate.sh ../../ecs
cd ../../ecs
bash generate.sh
```

Expected output:
```
Usage: generate.sh <ECS_VERSION> <INDEXER_SRC> <MODULE> [--upload <URL>]
* ECS_VERSION: ECS version to generate mappings for
* INDEXER_SRC: Path to the wazuh-indexer repository
* MODULE: Module to generate mappings for
* --upload <URL>: Upload generated index template to the OpenSearch cluster. Defaults to https://localhost:9200
Example: generate.sh v8.11.0 ~/wazuh-indexer states-vulnerabilities --upload https://indexer:9200
```

3. Use the `generate.sh` script to generate the mappings for a module. The script takes 3 arguments,
plus 2 optional arguments to upload the mappings to the `wazuh-indexer`. Both, composable and legacy mappings
are generated. For example, to generate the mappings for the `states-vulnerabilities` module using the
ECS version `v8.11.0` and assuming that path of this repository is `~/wazuh/wazuh-indexer`:

1. Execute the mapping-generator tool
```bash
./generate.sh v8.11.0 ~/wazuh/wazuh-indexer states-vulnerabilities
```

The tool will output the folder where they have been generated.

```console
Loading schemas from git ref v8.11.0
Running generator. ECS version 8.11.0
Mappings saved to ~/wazuh/wazuh-indexer/ecs/states-vulnerabilities/mappings/v8.11.0
bash ecs/generator/mapping-generator.sh run <MODULE_NAME>
```

4. When you are done. Exit the virtual environment.

```console
deactivate
2. (Optional) Run the tool's cleanup
> The tool stops the container automatically, but it is recommended to run the down command if the tool is not going to be used anymore.
```bash
bash ecs/generator/mapping-generator.sh down
```
### Output
Expand All @@ -80,8 +31,8 @@ For our use case, the most important files are under `mappings/<ECS_VERSION>/gen
- `template.json`: Elasticsearch compatible index template for the module
- `opensearch-template.json`: OpenSearch compatible index template for the module
The original output is `template.json`, which is not compatible with OpenSearch by default. In order
to make this template compatible with OpenSearch, the following changes are made:
The original output is `template.json`, which is not compatible with OpenSearch by default.
In order to make this template compatible with OpenSearch, the following changes are made:
- The `order` property is renamed to `priority`.
- The `mappings` and `settings` properties are nested under the `template` property.
Expand All @@ -104,64 +55,74 @@ Notes:
### Adding new mappings
The easiest way to create mappings for a new module is to take a previous one as a base.
Copy a folder and rename it to the new module name. Then, edit the `fields` files to
match the new module fields.
Copy a folder and rename it to the new module name. Then, edit the `fields` files to match the new module fields.
The name of the folder will be the name of the module to be passed to the script. All 3 files
are required.
The name of the folder will be the name of the module to be passed to the script. All 3 files are required.
- `fields/subset.yml`: This file contains the subset of ECS fields to be used for the module.
- `fields/template-settings-legacy.json`: This file contains the legacy template settings for the module.
- `fields/template-settings.json`: This file contains the composable template settings for the module.
### Event generator
For testing purposes, the script `generate_events.py` can be used to generate events for a given module.
Currently, it is only able to generate events for the `states-vulnerabilities` module. To support other
modules, please extend of refactor the script.

The script prompts for the required parameters, so it can be launched without arguments:
Each module contains a Python script to generate events for its module. The script prompts for the required parameters, so it can be launched without arguments:
```bash
./event_generator.py
```
The script will generate a JSON file with the events, and will also ask whether to upload them to the
indexer. If the upload option is selected, the script will ask for the indexer URL and port, credentials,
and index name.

The script will generate a JSON file with the events, and will also ask whether to upload them to the indexer. If the upload option is selected, the script will ask for the indexer URL and port, credentials, and index name.
The script uses log file. Check it out for debugging or additional information.
---
### Automatic PR creation tool
The `generate-pr-to-plugins.sh` script found in the `ecs/scripts` folder is a tool that detects modified ECS modules, generates new templates, commits the changes to a target repository, and creates or updates a pull request.
#### Requirements
- Docker Compose
- GitHub CLI (`gh`)
#### Usage
To use the script, run the following command:
```sh
./update-ecs-templates.sh -t <GITHUB_TOKEN>
```
**Options**
- `-b <BRANCH_NAME>`: (Optional) Branch name to create or update the pull request. Default is current branch.
- `-t <GITHUB_TOKEN>`: (Optional) GitHub token to authenticate with the GitHub API. If not provided, the script will use the `GITHUB_TOKEN` environment variable.
#### Script Workflow
1. **Validate Dependencies**
- Checks if the required commands (`docker`, `docker-compose`, and `gh`) are installed.
2. **Detect Modified Modules**
- Fetches and extracts modified ECS modules by comparing the current branch with the base branch.
- Identifies relevant ECS modules that have been modified.
3. **Run ECS Generator**
- Runs the ECS generator script for each relevant module to generate new ECS templates.
4. **Clone Target Repository**
- Clones the target repository (`wazuh/wazuh-indexer-plugins`) if it does not already exist.
- Configures Git and GitHub CLI with the provided GitHub token.
5. **Commit and Push Changes**
- Copies the generated ECS templates to the appropriate directory in the target repository.
- Commits and pushes the changes to the specified branch.
6. **Create or Update Pull Request**
- Creates a new pull request or updates an existing pull request with the modified ECS templates.
#### References
- [ECS repository](https://github.com/elastic/ecs)
- [ECS usage](https://github.com/elastic/ecs/blob/main/USAGE.md)
- [ECS field reference](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html)

### All-in-one script

```bash
#!/bin/bash

indices=(
agent
alerts
command
states-fim
states-inventory-hardware
states-inventory-hotfixes
states-inventory-networks
states-inventory-packages
states-inventory-ports
states-inventory-processes
states-inventory-system
states-vulnerabilities
)

ECS="v8.11.0"
WI_REPO_PATH=~/wazuh/wazuh-indexer

for index in "${indices[@]}"; do
bash generate.sh $ECS $WI_REPO_PATH "$index"
done
```
130 changes: 0 additions & 130 deletions ecs/generate.sh
View file

This file was deleted.

Loading

0 comments on commit bdc2b0b

Please sign in to comment.