Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: Update the docs #31

Merged
merged 1 commit into from
Jul 23, 2024
Merged

docs: Update the docs #31

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
121 changes: 121 additions & 0 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
# Contributing to ProxLB (PLB)

Thank you for considering contributing to ProxLB! We appreciate your help in improving the efficiency and performance of Proxmox clusters. Below are guidelines for contributing to the project.

## Table of Contents

- [Contributing to ProxLB (PLB)](#contributing-to-proxlb-plb)
- [Table of Contents](#table-of-contents)
- [Creating an Issue](#creating-an-issue)
- [Running Linting](#running-linting)
- [Running Tests](#running-tests)
- [Add Changelogs](#add-changelogs)
- [Submitting a Pull Request](#submitting-a-pull-request)
- [Code of Conduct](#code-of-conduct)
- [Getting Help](#getting-help)

## Creating an Issue

If you encounter a bug, have a feature request, or have any suggestions, please create an issue in our GitHub repository. To create an issue:

1. **Go to the [Issues](https://github.com/gyptazy/proxlb/issues) section of the repository.**
2. **Click on the "New issue" button.**
3. **Select the appropriate issue template (Bug Report, Feature Request, or Custom Issue).**
4. **Provide a clear and descriptive title.**
5. **Fill out the necessary details in the issue template.** Provide as much detail as possible to help us understand and reproduce the issue or evaluate the feature request.

## Running Linting
Before submitting a pull request, ensure that your changes sucessfully perform the lintin. ProxLB uses [flake8] for running tests. Follow these steps to run tests locally:

1. **Install pytest if you haven't already:**
```sh
pip install fake8
```

2. **Run the lintin:**
```sh
python3 -m flake8 proxlb
```

Linting will also be performed for each PR. Therefore, it might make sense to test this before pushing locally.

## Running Tests

Before submitting a pull request, ensure that your changes do not break existing functionality. ProxLB uses [pytest](https://docs.pytest.org/en/stable/) for running tests. Follow these steps to run tests locally:

1. **Install pytest if you haven't already:**
```sh
pip install pytest
```

2. **Run the tests:**
```sh
pytest
```

Ensure all tests pass before submitting your changes.

## Add Changelogs
ProxLB uses the [Changelog Fragments Creator](https://github.com/gyptazy/changelog-fragments-creator) for creating the overall `CHANGELOG.md` file. This changelog file is being generated from the files placed in the https://github.com/gyptazy/ProxLB/tree/main/.changelogs/ directory. Each release is represented by its version number where additional yaml files are being placed and parsed by the CFC tool. Such files look like:

```
added:
- Add option to rebalance by assigned VM resources to avoid overprovisioning. [#16]
```

Every PR should contain such a file describing the change to ensure this is also stated in the changelog file.

## Submitting a Pull Request

We welcome your contributions! Follow these steps to submit a pull request:

1. **Fork the repository to your GitHub account.**
2. **Clone your forked repository to your local machine:**
```sh
git clone https://github.com/gyptazy/proxlb.git
cd proxlb
```

Please prefix your PR regarding its type. It might be:
* doc
* feature
* fix

It should also provide the issue id to which it is related.

1. **Create a new branch for your changes:**
```sh
git checkout -b feature/10-add-new-cool-stuff
```

2. **Make your changes and commit them with a descriptive commit message:**
```sh
git add .
git commit -m "feature: Adding new cool stuff"
```

3. **Push your changes to your forked repository:**
```sh
git push origin feature/10-add-new-cool-stuff
```

4. **Create a pull request from your forked repository:**
- Go to the original repository on GitHub.
- Click on the "New pull request" button.
- Select the branch you pushed your changes to and create the pull request.

Please ensure that your pull request:

- Follows the project's coding style and guidelines.
- Includes tests for any new functionality.
- Updates the documentation as necessary.

## Code of Conduct

By participating in this project, you agree to abide by our [Code of Conduct](CODE_OF_CONDUCT.md). Please read it to understand the expected behavior and responsibilities when interacting with the community.

## Getting Help

If you need help or have any questions, feel free to reach out by creating an issue or by joining our [discussion forum](https://github.com/gyptazy/proxlb/discussions). You can also refer to our [documentation](https://github.com/gyptazy/ProxLB/tree/main/docs) for more information about the project or join our [chat room](https://matrix.to/#/#proxlb:gyptazy.ch) in Matrix.

Thank you for contributing to ProxLB! Together, we can enhance the efficiency and performance of Proxmox clusters.
28 changes: 26 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
- [Introduction](#introduction)
- [Video of Migration](#video-of-migration)
- [Features](#features)
- [How does it work?](#how-does-it-work)
- [Usage](#usage)
- [Dependencies](#dependencies)
- [Options](#options)
Expand All @@ -35,6 +36,7 @@
- [Misc](#misc)
- [Bugs](#bugs)
- [Contributing](#contributing)
- [Support](#support)
- [Author(s)](#authors)

## Introduction
Expand Down Expand Up @@ -78,6 +80,11 @@ Automated rebalancing reduces the need for manual actions, allowing operators to
* Periodically (daemon)
* Proxmox Web GUI Integration (optional)

## How does it work?
ProxLB is a load-balancing system designed to optimize the distribution of virtual machines (VMs) and containers (CTs) across a cluster. It works by first gathering resource usage metrics from all nodes in the cluster through the Proxmox API. This includes detailed resource metrics for each VM and CT on every node. ProxLB then evaluates the difference between the maximum and minimum resource usage of the nodes, referred to as "Balanciness." If this difference exceeds a predefined threshold (which is configurable), the system initiates the rebalancing process.

Before starting any migrations, ProxLB validates that rebalancing actions are necessary and beneficial. Depending on the selected balancing mode — such as CPU, memory, or disk — it creates a balancing matrix. This matrix sorts the VMs by their maximum used or assigned resources, identifying the VM with the highest usage. ProxLB then places this VM on the node with the most free resources in the selected balancing type. This process runs recursively until the operator-defined Balanciness is achieved. Balancing can be defined for the used or max. assigned resources of VMs/CTs.

## Usage
Running PLB is easy and it runs almost everywhere since it just depends on `Python3` and the `proxmoxer` library. Therefore, it can directly run on a Proxmox node, dedicated systems like Debian, RedHat, or even FreeBSD, as long as the API is reachable by the client running PLB.

Expand Down Expand Up @@ -218,7 +225,15 @@ docker run -it --rm -v $(pwd)/proxlb.conf:/etc/proxlb/proxlb.conf proxlb
```

### Logging
ProxLB uses the `SystemdHandler` for logging. You can find all your logs in your systemd unit log or in the journalctl.
ProxLB uses the `SystemdHandler` for logging. You can find all your logs in your systemd unit log or in the `journalctl`. In default, ProxLB only logs critical events. However, for further understanding of the balancing it might be useful to change this to `INFO` or `DEBUG` which can simply be done in the [proxlb.conf](https://github.com/gyptazy/ProxLB/blob/main/proxlb.conf#L14) file by changing the `log_verbosity` parameter.

Available logging values:
| Verbosity | Description |
|------|:------:|
| DEBUG | This option logs everything and is needed for debugging the code. |
| INFO | This option provides insides behind the scenes. What/why has been something done and with which values. |
| WARNING | This option provides only warning messages, which might be a problem in general but not for the application itself. |
| CRITICAL | This option logs all critical events that will avoid running ProxLB. |

## Motivation
As a developer managing a cluster of virtual machines for my projects, I often encountered the challenge of resource imbalance. Nodes within the cluster would become unevenly loaded, with some nodes being overburdened while others remained underutilized. This imbalance led to inefficiencies, performance bottlenecks, and increased operational costs. Frustrated by the lack of an adequate solution to address this issue, I decided to develop the ProxLB (PLB) to ensure better resource distribution across my clusters.
Expand Down Expand Up @@ -257,7 +272,16 @@ Container Images for Podman, Docker etc., can be found at:
Bugs can be reported via the GitHub issue tracker [here](https://github.com/gyptazy/ProxLB/issues). You may also report bugs via email or deliver PRs to fix them on your own. Therefore, you might also see the contributing chapter.

### Contributing
Feel free to add further documentation, to adjust already existing one or to contribute with code. Please take care about the style guide and naming conventions.
Feel free to add further documentation, to adjust already existing one or to contribute with code. Please take care about the style guide and naming conventions. You can find more in our [CONTRIBUTING.md](https://github.com/gyptazy/ProxLB/blob/main/CONTRIBUTING.md) file.

### Support
If you need assistance or have any questions, we offer support through our dedicated [chat room](https://matrix.to/#/#proxlb:gyptazy.ch) in Matrix and on Reddit. Join our community for real-time help, advice, and discussions. Connect with us in our dedicated chat room for immediate support and live interaction with other users and developers. You can also visit our [Reddit community](https://www.reddit.com/r/Proxmox/comments/1e78ap3/introducing_proxlb_rebalance_your_vm_workloads/) to post your queries, share your experiences, and get support from fellow community members and moderators. You may also just open directly an issue [here](https://github.com/gyptazy/ProxLB/issues) on GitHub. We are here to help and ensure you have the best experience possible.

| Support Channel | Link |
|------|:------:|
| Matrix | [#proxlb:gyptazy.ch](https://matrix.to/#/#proxlb:gyptazy.ch) |
| Reddit | [Reddit community](https://www.reddit.com/r/Proxmox/comments/1e78ap3/introducing_proxlb_rebalance_your_vm_workloads/) |
| GitHub | [ProxLB GitHub](https://github.com/gyptazy/ProxLB/issues) |

### Author(s)
* Florian Paul Azim Hoberg @gyptazy (https://gyptazy.ch)
54 changes: 54 additions & 0 deletions docs/03_FAQ.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,57 @@ Jul 06 10:25:16 build01 proxlb[7285]: proxlb: Error: [python-imports]: Could no

Debian/Ubuntu: apt-get install python3-proxmoxer
If the package is not provided by your systems repository, you can also install it by running `pip3 install proxmoxer`.

### How does it work?
ProxLB is a load-balancing system designed to optimize the distribution of virtual machines (VMs) and containers (CTs) across a cluster. It works by first gathering resource usage metrics from all nodes in the cluster through the Proxmox API. This includes detailed resource metrics for each VM and CT on every node. ProxLB then evaluates the difference between the maximum and minimum resource usage of the nodes, referred to as "Balanciness." If this difference exceeds a predefined threshold (which is configurable), the system initiates the rebalancing process.

Before starting any migrations, ProxLB validates that rebalancing actions are necessary and beneficial. Depending on the selected balancing mode — such as CPU, memory, or disk — it creates a balancing matrix. This matrix sorts the VMs by their maximum used or assigned resources, identifying the VM with the highest usage. ProxLB then places this VM on the node with the most free resources in the selected balancing type. This process runs recursively until the operator-defined Balanciness is achieved. Balancing can be defined for the used or max. assigned resources of VMs/CTs.

### Logging
ProxLB uses the `SystemdHandler` for logging. You can find all your logs in your systemd unit log or in the `journalctl`. In default, ProxLB only logs critical events. However, for further understanding of the balancing it might be useful to change this to `INFO` or `DEBUG` which can simply be done in the [proxlb.conf](https://github.com/gyptazy/ProxLB/blob/main/proxlb.conf#L14) file by changing the `log_verbosity` parameter.

Available logging values:
| Verbosity | Description |
|------|:------:|
| DEBUG | This option logs everything and is needed for debugging the code. |
| INFO | This option provides insides behind the scenes. What/why has been something done and with which values. |
| WARNING | This option provides only warning messages, which might be a problem in general but not for the application itself. |
| CRITICAL | This option logs all critical events that will avoid running ProxLB. |

### Motivation
As a developer managing a cluster of virtual machines for my projects, I often encountered the challenge of resource imbalance. Nodes within the cluster would become unevenly loaded, with some nodes being overburdened while others remained underutilized. This imbalance led to inefficiencies, performance bottlenecks, and increased operational costs. Frustrated by the lack of an adequate solution to address this issue, I decided to develop the ProxLB (PLB) to ensure better resource distribution across my clusters.

My primary motivation for creating PLB stemmed from my work on my BoxyBSD project, where I consistently faced the difficulty of maintaining balanced nodes while running various VM workloads but also on my personal clusters. The absence of an efficient rebalancing mechanism made it challenging to achieve optimal performance and stability. Recognizing the necessity for a tool that could gather and analyze resource metrics from both the cluster nodes and the running VMs, I embarked on developing ProxLB.

PLB meticulously collects detailed resource usage data from each node in a Proxmox cluster, including CPU load, memory usage, and local disk space utilization. It also gathers comprehensive statistics from all running VMs, providing a granular understanding of the workload distribution. With this data, PLB intelligently redistributes VMs based on memory usage, local disk usage, and CPU usage. This ensures that no single node is overburdened, storage resources are evenly distributed, and the computational load is balanced, enhancing overall cluster performance.

As an advocate of the open-source philosophy, I believe in the power of community and collaboration. By sharing solutions like PLB, I aim to contribute to the collective knowledge and tools available to developers facing similar challenges. Open source fosters innovation, transparency, and mutual support, enabling developers to build on each other's work and create better solutions together.

Developing PLB was driven by a desire to solve a real problem I faced in my projects. However, the spirit behind this effort was to provide a valuable resource to the community. By open-sourcing PLB, I hope to help other developers manage their clusters more efficiently, optimize their resource usage, and reduce operational costs. Sharing this solution aligns with the core principles of open source, where the goal is not only to solve individual problems but also to contribute to the broader ecosystem.

### Packages / Container Images
Ready to use packages can be found at:
* https://cdn.gyptazy.ch/files/amd64/debian/proxlb/
* https://cdn.gyptazy.ch/files/amd64/ubuntu/proxlb/
* https://cdn.gyptazy.ch/files/amd64/redhat/proxlb/
* https://cdn.gyptazy.ch/files/amd64/freebsd/proxlb/

Container Images for Podman, Docker etc., can be found at:
| Version | Image |
|------|:------:|
| latest | cr.gyptazy.ch/proxlb/proxlb:latest |

### Bugs
Bugs can be reported via the GitHub issue tracker [here](https://github.com/gyptazy/ProxLB/issues). You may also report bugs via email or deliver PRs to fix them on your own. Therefore, you might also see the contributing chapter.

### Contributing
Feel free to add further documentation, to adjust already existing one or to contribute with code. Please take care about the style guide and naming conventions. You can find more in our [CONTRIBUTING.md](https://github.com/gyptazy/ProxLB/blob/main/CONTRIBUTING.md) file.

### Support
If you need assistance or have any questions, we offer support through our dedicated [chat room](https://matrix.to/#/#proxlb:gyptazy.ch) in Matrix and on Reddit. Join our community for real-time help, advice, and discussions. Connect with us in our dedicated chat room for immediate support and live interaction with other users and developers. You can also visit our [Reddit community](https://www.reddit.com/r/Proxmox/comments/1e78ap3/introducing_proxlb_rebalance_your_vm_workloads/) to post your queries, share your experiences, and get support from fellow community members and moderators. You may also just open directly an issue [here](https://github.com/gyptazy/ProxLB/issues) on GitHub. We are here to help and ensure you have the best experience possible.

| Support Channel | Link |
|------|:------:|
| Matrix | [#proxlb:gyptazy.ch](https://matrix.to/#/#proxlb:gyptazy.ch) |
| Reddit | [Reddit community](https://www.reddit.com/r/Proxmox/comments/1e78ap3/introducing_proxlb_rebalance_your_vm_workloads/) |
| GitHub | [ProxLB GitHub](https://github.com/gyptazy/ProxLB/issues) |
Loading