Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

7 doc/nld proofread #19

Open
wants to merge 72 commits into
base: master
Choose a base branch
from
Open
Changes from 3 commits
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
099bbaa
docs(hwpc-sensor): refactor documentation, remove Destination/Source …
Oct 4, 2024
bb5ec29
docs(smartwatts): refactor documentation, remove Destination/Source r…
Oct 4, 2024
41ef0b3
docs(getting-started): Add necessary example to run a complete PowerA…
Oct 7, 2024
fb73d58
docs(getting-started): Documents the Getting-Started instructions thr…
Oct 8, 2024
bfd3e51
Add hwpc-sensor.drawio
Inkedstinct Oct 8, 2024
522fec0
Add hwpc-sensor.drawio
Inkedstinct Oct 8, 2024
81d6146
docs(hwpc-sensor): Switch diagrams to Noun Project icons
Oct 8, 2024
2790e42
docs(overview): Clarifies PowerAPI framework components and status
Oct 8, 2024
de31f7f
docs(source/destination abstractions): Remove Source / Destination ab…
Oct 8, 2024
dff504b
docs(reports): Add JSON Schema for Reports basis
Oct 9, 2024
777f5ed
docs(grafana): Redirect to Grafana documentation as GUI changes frequ…
Oct 9, 2024
6497233
docs(hwpc-sensor): Include Daniel suggestions of providing examples w…
Oct 9, 2024
3396b98
docs(getting-started): Refer only 'process' as atomic monitored element
Oct 9, 2024
88cd731
docs(global): Improve examples snippet
Oct 9, 2024
b667972
docs(getting-started): Add start of getting started script
Zetos11 Oct 9, 2024
dc7de38
docs(mkdocs): Add local development tips
Oct 10, 2024
e692d32
docs(getting-started): Document a 'one-liner' way for quick test of P…
Oct 10, 2024
9d861df
Merge remote-tracking branch 'origin/7_doc/nld_proofread' into 7_doc/…
Zetos11 Oct 11, 2024
e7fadff
docs(getting-started): Add docker compose gestion
Zetos11 Oct 14, 2024
ffd5f26
docs(getting-started): Fix archive compatibility with version<3.10
Zetos11 Oct 16, 2024
9819bbf
docs(getting-started): Early design for the pretty print idea
Zetos11 Oct 16, 2024
a0d8de3
docs(getting-started): Pretty print improvement and start.py update
Zetos11 Oct 16, 2024
8455db5
docs(getting-started): Getting started archive is ready for use
Zetos11 Oct 17, 2024
d243f8b
docs(getting-started): Refactor Preety_print + change to start
Zetos11 Oct 17, 2024
9571781
docs(getting-started): Adapt getting started doc structure
Zetos11 Oct 21, 2024
fec3645
docs(getting-started): Change to the preparation part
Zetos11 Oct 21, 2024
8a1f7f0
docs(getting-started): Beginning update
Zetos11 Oct 21, 2024
1e8fa2f
fix(overview): Fix markdown list
Oct 21, 2024
671beb4
fix(overview): Add highlights for 'quick overview' with pretty_print.py
Oct 21, 2024
b6fe076
fix(getting-started): Add highlights for 'quick overview' with pretty…
Oct 21, 2024
72b3734
feat(getting-started/script): Add signal handler for SIGINT in order …
Oct 21, 2024
ebae5b0
fix(getting-started): Add git clone instruction instead of archive do…
Oct 21, 2024
72706cd
Merge branch '7_doc/nld_proofread' of github.com:Inkedstinct/powerapi…
Oct 22, 2024
c108291
docs(getting_started): Make cgroup part optionnal with more explicit …
Oct 22, 2024
bf4cf46
docs(reference): Update overview.md
Oct 29, 2024
99ecc97
docs(getting-started): Added cpu mapping
Zetos11 Oct 29, 2024
707e727
docs(getting-started): Removed start.sh a,d stop.sh
Zetos11 Oct 29, 2024
451f504
docs(reference): Update smartwatts.md
Oct 29, 2024
b4079f6
docs(reference): Update hwpc-sensor.md
Oct 29, 2024
30d99ad
docs(hwpc-sensor): Fix sensor overview background for dark mode
Oct 30, 2024
9ee9476
docs(getting-started): Add archive option for getting-started
Oct 30, 2024
577f40e
docs(getting-started): Add CLI commands for archive option in getting…
Oct 30, 2024
10f4227
docs(getting-started): Fix dead links
Oct 30, 2024
622ae68
docs(getting-started): Add Docker install documentation
Oct 30, 2024
9e8b2db
docs(references): Remove configuration-file part for Formulas, curren…
Oct 30, 2024
a2121d0
docs(overview): Fix SmartWatts reports unit
Oct 30, 2024
7909f08
docs(getting-started): Add UID / GUID information
Oct 30, 2024
fcfb376
docs(hwpc-sensor): Add Socket output as example + minor fixes
Oct 30, 2024
b05e74e
docs(getting-started): Add info on how to stop script
Oct 30, 2024
dedfdf2
docs(all): Fix HWPCReports -> HWPC Reports, same for PowerReports
Oct 30, 2024
64e6b4a
docs(smartwatts): Add CSV/CSV examplesw
Oct 30, 2024
4c548b6
docs(getting-started): Synchro archive with cloned
Oct 30, 2024
a5212f1
Merge remote-tracking branch 'origin/7_doc/nld_proofread' into 7_doc/…
Zetos11 Oct 30, 2024
7f63b2e
docs(getting-started): Merged pretty print with start.py
Zetos11 Oct 30, 2024
872d82f
docs(getting-started): Improve clarity and formatting
Oct 30, 2024
c4b7ed8
docs(hwpc-sensor): Typo
Oct 30, 2024
a70ce29
docs(smartwatts): Typo
Oct 30, 2024
b91e44d
docs(getting-started): Typo
Oct 30, 2024
203af14
docs(getting-started): Fix option for UID/GUID
Zetos11 Nov 5, 2024
e7dffa0
docs(getting-started): Clean up + Archive update
Zetos11 Nov 5, 2024
8b82ae3
fix: Correct issue related to csv directory permission by deleting/cr…
roda82 Jan 8, 2025
061b87b
fix: Remove user from formula
roda82 Jan 8, 2025
2133fef
refactor: Update some variables and functions names
roda82 Jan 9, 2025
6863ff8
style: Add white line to pretty print
roda82 Jan 9, 2025
ca3f3b5
refactor: Force usage of bash shell for system commands
roda82 Jan 9, 2025
a68b808
refactor: Force usage of bash shell for system commands
roda82 Jan 9, 2025
d1cf296
refactor: Define global variable for minimum execution time
roda82 Jan 9, 2025
dde7768
fix: Change umask value for current user
roda82 Jan 9, 2025
9075a30
fix: Change umask value for current user
roda82 Jan 9, 2025
1f2de03
docs(getting-started): Add exception gestion for CPU choice
Zetos11 Jan 9, 2025
eba10ed
docs(smartwatts): Fix HWPC Report -> HWPCReport, same for Power report
KellianL Jan 30, 2025
4242ffc
docs(smartwatts): Fix examples CSV/CSV
KellianL Feb 12, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -2,3 +2,23 @@
PowerAPI website use [MKDocs](https://www.mkdocs.org/).

You can push your changes on `master` branch to deploy them on the website.

## Local development

When redacting changes for this documentation, you may want to visualize the
output the way it will be presented in the final website once compiled.

One way to do so is to:

1. Install the necessary packages : mkdocs, mkdocs-material, mkdocs-material-extensions
This can be done with: `pip install mkdocs mkdocs-material mkdocs-material-extensions`

2. Have a local copy of this repository (taking care of *checkout-ing* the right ref)

3. From the CLI, have this particular repository as PWD

4. Use mkdocs in order to serve locally (default on http://localhost:8000)
This can be done with: `mkdocs serve -o`

5. Check the URL it serves to, targets will be rebuild on modifications in the
current directory or in any nested elements (use -W to add paths to be considered for hot reloading)
286 changes: 62 additions & 224 deletions docs/getting_started.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,28 @@
# Getting started

!!! info "Pre-Requisites"

**In order to follow this tutorial, you will need several elements ready on
the target server:
- A compatible processor
- A python installation ready
- Docker & Docker-Compose ready
- Root access**

!!! warning "Testing purpose tutorial"

This quick Getting-Started will guide you to get a quick view of PowerAPI
capabilities.
To do so in a light way, **the final output displayed is not the
intended use of the tools for an everyday deployment**, you'll only get quick & concise
statistics on the tested period.

## Define elements to monitor

PowerAPI being a monitoring tool for energy consumption, we will need to define
the necessary elements to monitor. An example is also given if you do not already
have a process you wish to monitor.

### Create a cGroup

We need a subset of running processes to be monitored. For this, we use the
@@ -28,20 +49,16 @@ cgclassify -g perf_event:new_cgroup_name PID
with `PID`, the pid of the process you want to monitor. If you want to monitor a
process composed of many processes, replace PID with `$(pidof process_name)`.

### Example process to monitor
### Installing a process to monitor

[stress-ng](https://wiki.ubuntu.com/Kernel/Reference/stress-ng) can be used to
generate load on one's system.
An example usage, once installed :

```sh
```sh
cgcreate -g perf_event:stress-ng-cgroup
stress-ng --cpu 1 --timeout 5m
```

The PID can be found using :

```sh
STRESS_PID="$(pgrep stress-ng)"
cgclassify -g perf_event:stress-ng-cgroup $!
```

## Which components to get a complete stack
@@ -51,21 +68,15 @@ If you wish to get started as soon as possible, the following archive will allow
1. A MongoDB instance to store the [Sensor](./reference/sensors/hwpc-sensor.md)
Reports

2. An InfluxDB2 instance to store the [Formulas](./reference/formulas/smartwatts.md)

3. An [HWPC-Sensor](./reference/sensors/hwpc-sensor.md) that outputs its
[HWPCReports](./reference/reports/report.md#HWPCReport) in a MongoDB Database,
within the HWPCReport Collection.

4. A [SmartWatts](./reference/formulas/smartwatts.md) that streams the
[HWPCReports](./reference/reports/report.md#HWPCReport) from the MongoDB
Database Collection, processes it and outputs its
[PowerReports](./reference/reports/report.md#PowerReports) in an InfluxDB2
Database within the PowerReport Collection

5. A Grafana instance to visualize the
[PowerReports](./reference/reports/report.md#PowerReports) from the InfluxDB2
instance
[PowerReports](./reference/reports/report.md#PowerReports) as CSV files for a
quick glimpse

## Preparation

@@ -78,99 +89,27 @@ unzip powerapi-stack.zip && cd powerapi-stack

From this archive, you will have all the necessary files to get started, let us break down each elements.

### Docker-Compose

```yaml
# ./docker-compose.yaml

version: '3.8'
services:
mongodb:
image: mongo:latest
container_name: mongodb
ports:
- "27017:27017"
volumes:
- mongo_data:/data/db
environment:
MONGO_INITDB_DATABASE: powerapi
command: ["mongod"]
networks:
- powerapi_network

influxdb:
image: influxdb:2.0
container_name: influxdb
ports:
- "8086:8086"
volumes:
- influxdb_data:/var/lib/influxdb2
environment:
INFLUXDB_DB: powerapi
INFLUXDB_ADMIN_USER: admin
INFLUXDB_ADMIN_PASSWORD: admin123
INFLUXDB_USER: powerapi_user
INFLUXDB_PASSWORD: powerapi_password
networks:
- powerapi_network

hwpc_sensor:
image: powerapi/hwpc-sensor
container_name: hwpc_sensor
volumes:
- ./config/hwpc-sensor-config.json:/hwpc-sensor-config.json
command: --config-file /hwpc-sensor-config.json
networks:
- powerapi_network
depends_on:
- mongodb

smartwatts:
image: powerapi/smartwatts
container_name: smartwatts
volumes:
- ./config/smartwatts-config.json:/smartwatts-config.json
command: --config /smartwatts-config.json
networks:
- powerapi_network
depends_on:
- mongodb
- influxdb

grafana:
image: grafana/grafana:latest
container_name: grafana
ports:
- "3000:3000"
environment:
- GF_SECURITY_ADMIN_USER=admin
- GF_SECURITY_ADMIN_PASSWORD=admin
volumes:
- grafana_data:/var/lib/grafana
- ./config/provisioning:/etc/grafana/provisioning
networks:
- powerapi_network
depends_on:
- influxdb

volumes:
mongo_data:
influxdb_data:

networks:
powerapi_network:
driver: bridge
### Archive content

```sh
|powerapi-stack/
|--docker-compose.yaml
|--configs.d/
|----hwpc-sensor-config.json
|----smartwatts-config.json
|--start.py
|--reports.d/
|----powerreports.csv
```

### HWPC-Sensor Configuration
#### HWPC-Sensor Configuration

As described in the [HWPC-Sensor Documentation](./reference/sensors/hwpc-sensor.md#global-parameters)
several parameters can be set, both globally and for specific Groups monitored.
The provided docker-compose.yaml file use configuration files to set those parameters.
An example configuration file for HWPC-Sensor is given below and available in the archive presented [above](./getting_started.md#preparation) :

```json
# ./hwpc-sensor-config.json
```json title="powerapi-stack/hwpc-sensor-config.json"

{
"verbose": false,
@@ -201,111 +140,30 @@ several parameters can be set for the Formulas.
The provided docker-compose.yaml file use configuration files to set those parameters.
An example configuration file for SmartWatts is given below and available in the archive presented [above](./getting_started.md#preparation) :

```json
# ./smartwatts-config.json

```json title="powerapi-stack/smartwatts-config.json"
{
"cpu_tdp": 125,
"cpu_base_clock": 100,
"cpu_base_freq": 2100, # Use lscpu and look for CPU MHz field to know the correct value
"cpu_error_threshold": 2.0,
"dram_error_threshold": 2.0,
"learn_min_samples_required": 10,
"learn_history_window_size": 60,
"sensor_reports_frequency": 1000,
"input_mongo": {
"uri": "mongodb://mongodb:27017/powerapi",
"collection": "HWPCReports"
"verbose": true,
"stream": true,
"input": {
"puller": {
"model": "HWPCReport",
"type": "mongodb",
"uri": "mongodb://127.0.0.1",
"db": "powerapi",
"collection": "HWPCReports"
}
},
"output_influx": {
"uri": "http://influxdb:8086",
"org": "powerapi_org",
"bucket": "PowerReports",
"token": "INFLUXDB_API_TOKEN_GENERATED"
}
}
```

### Grafana Configuration

Grafana has to be configured to use our InfluxDB2 instance as a datasource,
it should also know our dashboard file in order to visualize it.

#### Filesystem structure

The following filesystem structure is presented in the archive presented [above](./getting_started.md#preparation) : :


```sh
|powerapi-stack/
|--grafana_config/
|----provisioning/
|------dashboards/
|--------dashboard.yaml
|--------reports.json
|------datasources/
|--------datasource.yaml
|...
```

#### Datasource file

The following file gives information to Grafana to use InfluxDB2 as Datasource:

```yaml
# ./grafana_config/provisioning/datasources/datasource.yaml

apiVersion: 1
datasources:
- name: InfluxDB
type: influxdb
access: proxy
url: http://influxdb:8086
isDefault: true
database: PowerReports
user: powerapi_user
password: powerapi_password
jsonData:
version: Flux
organization: powerapi_org
defaultBucket: PowerReports
token: INFLUXDB_API_TOKEN_GENERATED
editable: true
```

#### Dashboard files

The following files give information to Grafana:

1. Where to search pre-configured Dashboards
2. How to configure our Dashboards

```yaml
# ./grafana_config/provisioning/dashboards/dashboard.yaml

apiVersion: 1
providers:
- name: 'default'
folder: ''
type: file
options:
path: /etc/grafana/provisioning/dashboards
```

```json
# ./grafana_config/provisioning/dashboards/reports.json

{
"id": null,
"title": "PowerAPI Dashboard",
"tags": [],
"timezone": "browser",
"schemaVersion": 16,
"version": 0,
"panels": [
{
"output": {
"pusher_power": {
"type": "csv",
"directory": "reports.d",
"files": "powerreports.csv"
}
]
},
"cpu-base-freq": 1900,
"cpu-error-threshold": 2.0,
"disable-dram-formula": true,
"sensor-reports-frequency": 1000
}
```

@@ -314,26 +172,6 @@ providers:
Once all set, you shall be able to initiate the stack with :

```sh
docker-compose -d up
python3 start.py
```

This first spin-up will create the DBs and the Sensor.
Both SmartWatts Formulas and Grafana will struggle as they need an InfluxDB API key.
To resolve this, you can run :
```sh
influx auth create --org powerapi_org --read-buckets --write-buckets
```

You will obtain a token to replace the *INFLUXDB_API_TOKEN_GENERATED* placeholder in the `smartwatts-config.json` file and in `grafana_config/provisioning/datasources/datasource.yaml`

Once both changed, another `docker-compose -d up` should do the trick to start the
remaining components.

Thus, once deployed, you can access [Grafana](https://localhost:3000) by default
on http://localhost:8080 with its basic credentials : `admin/admin`.

In the dashboards sections, basic dashboards will be provisionned with PowerReports
from SmartWatts providing real time insights about you consumption.

Feel free to try yo make your own visualization and take a look at
[this documentation](./reference/grafana/grafana.md) for further details.