fix: Fix anti-affinity rules not evaluating a new and different node correctly

[gyptazy]

fix: Fix anti-affinity rules not evaluating a new and different node correctly.

General

This PR fixes an issue where a new and different node was not correctly evaluated when working with anti-affinity rules. It also applies and adds some more logging and fixes wrong logging options that were previously defined.

Test

<6> ProxLB: Info: [rebalancing-tags-group-exclude]: Rebalancing of VM test03-small is needed due to anti-affinity group policy.
<6> ProxLB: Info: [random-node-getter]: New random node pmx01 evaluated in run 1.
<6> ProxLB: Info: [random-node-getter]: New random node pmx03 evaluated in run 2.
<6> ProxLB: Info: [random-node-getter]: New random node pmx01 evaluated in run 3.
<6> ProxLB: Info: [random-node-getter]: New random node pmx01 evaluated in run 4.
<6> ProxLB: Info: [random-node-getter]: New random node pmx03 evaluated in run 5.
<6> ProxLB: Info: [random-node-getter]: New random node pmx02 evaluated in run 6.
<6> ProxLB: Info: [rebalancing-tags-group-exclude]: Rebalancing VM test03-small to node pmx02 due to anti-affinity group policy.
<6> ProxLB: Info: [rebalancing-vm-calculator]: Balancing calculations done.
<6> ProxLB: Info: [vm-rebalancing-executor]: No rebalancing needed.
<6> ProxLB: Info: [cli-output-generator-dry-run]: Starting dry-run to rebalance vms to their new nodes.
<6> ProxLB: Info: [cli-output-generator-dry-run]: Printing cli output of VM rebalancing.
             VM   Current Node   Rebalanced Node   Current Storage   Rebalanced Storage   VM Type
<6> ProxLB: Info: [cli-output-generator-table-dryn-run]:              VM   Current Node   Rebalanced Node   Current Storage   Rebalanced Storage   VM Type
   test03-small          pmx01             pmx02         N/A (N/A)            N/A (N/A)        vm
<6> ProxLB: Info: [cli-output-generator-table-dryn-run]:    test03-small          pmx01             pmx02         N/A (N/A)            N/A (N/A)        vm
<6> ProxLB: Info: [post-validations]: All post-validations succeeded.
<6> ProxLB: Info: [daemon]: Not running in daemon mode. Quitting.

Fixes

Fixes: #67

[adminsyspro]

Hi,

I just tried PR #68 but I have this error :

<6> ProxLB: Info: [api-connection]: API connection succeeded to host: 10.99.99.10.
<6> ProxLB: Info: [only-on-master-executor]: No master only rebalancing is defined. Skipping validation.
<6> ProxLB: Info: [node-statistics]: Added node Proxmox-3AZ-1-B.
<6> ProxLB: Info: [node-statistics]: Added node Proxmox-3AZ-1-A.
<6> ProxLB: Info: [node-statistics]: Added node Proxmox-3AZ-1-C.
<6> ProxLB: Info: [node-statistics]: Created node statistics.
<6> ProxLB: Info: [api-get-vm-tags]: Got VM/CT tag plb_antiaffinity_fw for VM FW2 from API.
<6> ProxLB: Info: [api-get-vm-include-exclude-tags]: Got PLB exclude group.
<6> ProxLB: Info: [vm-statistics]: Getting disk information for vm FW2.
<6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for FW2 found.
<6> ProxLB: Info: [vm-statistics]: Added vm FW2.
<6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM ClusterControl from API.
<6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM VM-Test-3 from API.
<6> ProxLB: Info: [vm-statistics]: Getting disk information for vm VM-Test-3.
<6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for VM-Test-3 found.
<6> ProxLB: Info: [vm-statistics]: Added vm VM-Test-3.
<6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM VM-Test-1 from API.
<6> ProxLB: Info: [vm-statistics]: Getting disk information for vm VM-Test-1.
<6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for VM-Test-1 found.
<6> ProxLB: Info: [vm-statistics]: Added vm VM-Test-1.
<6> ProxLB: Info: [api-get-vm-tags]: Got VM/CT tag for VM Test-VM-2 from API.
<6> ProxLB: Info: [vm-statistics]: Getting disk information for vm Test-VM-2.
<6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for Test-VM-2 found.
<6> ProxLB: Info: [vm-statistics]: Added vm Test-VM-2.
<6> ProxLB: Info: [api-get-vm-tags]: Got VM/CT tag plb_antiaffinity_fw for VM FW1 from API.
<6> ProxLB: Info: [api-get-vm-include-exclude-tags]: Got PLB exclude group.
<6> ProxLB: Info: [vm-statistics]: Getting disk information for vm FW1.
<6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for FW1 found.
<6> ProxLB: Info: [vm-statistics]: Added vm FW1.
<6> ProxLB: Info: [vm-statistics]: Created VM statistics.
<4> ProxLB: Warning: [node-update-statistics]: Node Proxmox-3AZ-1-B is overprovisioned for disk by 164%.
<4> ProxLB: Warning: [node-update-statistics]: Node Proxmox-3AZ-1-B is overprovisioned for disk by 328%.
<4> ProxLB: Warning: [node-update-statistics]: Node Proxmox-3AZ-1-B is overprovisioned for disk by 492%.
<4> ProxLB: Warning: [node-update-statistics]: Node Proxmox-3AZ-1-B is overprovisioned for disk by 656%.
<4> ProxLB: Warning: [node-update-statistics]: Node Proxmox-3AZ-1-B is overprovisioned for disk by 820%.
<6> ProxLB: Info: [node-update-statistics]: Updated node resource assignments by all VMs.
<6> ProxLB: Info: [storage-statistics]: Added storage PBS-RBX-MASTER.
<6> ProxLB: Info: [storage-statistics]: Storage CephStorageVM support CTs.
<6> ProxLB: Info: [storage-statistics]: Storage CephStorageVM support VMs.
<6> ProxLB: Info: [storage-statistics]: Updateing storage CephStorageVM support to CTs and VMs.
<6> ProxLB: Info: [storage-statistics]: Added storage CephStorageVM.
<6> ProxLB: Info: [storage-statistics]: Storage CephStorageVM support CTs.
<6> ProxLB: Info: [storage-statistics]: Storage CephStorageVM support VMs.
<6> ProxLB: Info: [storage-statistics]: Updateing storage CephStorageVM support to CTs and VMs.
<6> ProxLB: Info: [storage-statistics]: Added storage CephStorageVM.
<6> ProxLB: Info: [storage-statistics]: Added storage PBS-RBX-MASTER.
<6> ProxLB: Info: [storage-statistics]: Storage CephStorageVM support CTs.
<6> ProxLB: Info: [storage-statistics]: Storage CephStorageVM support VMs.
<6> ProxLB: Info: [storage-statistics]: Updateing storage CephStorageVM support to CTs and VMs.
<6> ProxLB: Info: [storage-statistics]: Added storage CephStorageVM.
<6> ProxLB: Info: [storage-statistics]: Added storage PBS-RBX-MASTER.
<6> ProxLB: Info: [storage-statistics]: Created storage statistics.
<6> ProxLB: Info: [balancing-method-validation]: Valid balancing method: memory
<6> ProxLB: Info: [balancing-mode-validation]: Valid balancing method: used
<6> ProxLB: Info: [balanciness-validation]: Rebalancing for memory is not needed. Highest usage: 93% | Lowest usage: 86%.
<6> ProxLB: Info: [rebalancing-tags-group-exclude]: Rebalancing of VM FW2 is needed due to anti-affinity group policy.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 1.
<6> ProxLB: Info: [rebalancing-tags-group-exclude]: Rebalancing VM FW2 to node Proxmox-3AZ-1-A due to anti-affinity group policy.
<6> ProxLB: Info: [rebalancing-tags-group-exclude]: Rebalancing of VM VM-Test-3 is needed due to anti-affinity group policy.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 1.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-B evaluated in run 2.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-B evaluated in run 3.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 4.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 5.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-B evaluated in run 6.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 7.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-C evaluated in run 8.
<6> ProxLB: Info: [rebalancing-tags-group-exclude]: Rebalancing VM VM-Test-3 to node Proxmox-3AZ-1-C due to anti-affinity group policy.
<6> ProxLB: Info: [rebalancing-tags-group-exclude]: Rebalancing of VM VM-Test-1 is needed due to anti-affinity group policy.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 1.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 2.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 3.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-B evaluated in run 4.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-B evaluated in run 5.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-B evaluated in run 6.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 7.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-C evaluated in run 8.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 9.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-B evaluated in run 10.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-B evaluated in run 11.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-C evaluated in run 12.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-B evaluated in run 13.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-C evaluated in run 14.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 15.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-C evaluated in run 16.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 17.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 18.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 19.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 20.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 21.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 22.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-B evaluated in run 23.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 24.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-C evaluated in run 25.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 26.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-B evaluated in run 27.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-B evaluated in run 28.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-C evaluated in run 29.
Traceback (most recent call last):
File "/usr/bin/proxlb", line 1483, in
main()
File "/usr/bin/proxlb", line 1463, in main
node_statistics, vm_statistics = balancing_vm_calculations(proxlb_config['vm_balancing_method'], proxlb_config['vm_balancing_mode'], proxlb_config['vm_balancing_mode_option'], node_statistics, vm_statistics, proxlb_config['vm_balanciness'], app_args, rebalance=False, processed_vms=[])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/bin/proxlb", line 827, in balancing_vm_calculations
node_statistics, vm_statistics = __get_vm_tags_exclude_groups(vm_statistics, node_statistics, balancing_method, balancing_mode)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/bin/proxlb", line 1045, in __get_vm_tags_exclude_groups
random_node, counter = __get_random_node(counter, node_statistics)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/bin/proxlb", line 1072, in __get_random_node
logging.warning(f'{info_warning} Reached limit for random node evaluation. Unable to find a suitable new node.')
^^^^^^^^^^^^
NameError: name 'info_warning' is not defined

[adminsyspro]

I tried to relaunch without any modification and it passed :

6> ProxLB: Info: [balanciness-validation]: Rebalancing for memory is not needed. Highest usage: 93% | Lowest usage: 86%.
<6> ProxLB: Info: [rebalancing-tags-group-exclude]: Rebalancing of VM FW1 is needed due to anti-affinity group policy.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-B evaluated in run 1.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-C evaluated in run 2.
<6> ProxLB: Info: [rebalancing-tags-group-exclude]: Rebalancing VM FW1 to node Proxmox-3AZ-1-C due to anti-affinity group policy.
<6> ProxLB: Info: [rebalancing-tags-group-exclude]: Rebalancing of VM FW2 is needed due to anti-affinity group policy.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-C evaluated in run 1.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-B evaluated in run 2.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-B evaluated in run 3.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-B evaluated in run 4.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-B evaluated in run 5.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-B evaluated in run 6.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-C evaluated in run 7.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 8.
<6> ProxLB: Info: [rebalancing-tags-group-exclude]: Rebalancing VM FW2 to node Proxmox-3AZ-1-A due to anti-affinity group policy.
<6> ProxLB: Info: [rebalancing-vm-calculator]: Balancing calculations done.
<6> ProxLB: Info: [vm-rebalancing-executor]: Rebalancing VM FW2 from node Proxmox-3AZ-1-B to node Proxmox-3AZ-1-A.
<6> ProxLB: Info: [vm-rebalancing-executor]: Rebalancing will be performed parallely.
<6> ProxLB: Info: [cli-output-generator]: Start rebalancing vms to their new nodes.
<6> ProxLB: Info: [cli-output-generator]: Printing cli output of VM rebalancing.
<6> ProxLB: Info: [cli-output-generator-table]: VM Current Node Rebalanced Node Current Storage Rebalanced Storage VM Type
<6> ProxLB: Info: [cli-output-generator-table]: FW2 Proxmox-3AZ-1-B Proxmox-3AZ-1-A N/A (N/A) N/A (N/A) vm
<6> ProxLB: Info: [post-validations]: All post-validations succeeded.
<6> ProxLB: Info: [daemon]: Running in daemon mode. Next run in 24 hours.

[gyptazy]

Thanks, valuable hint. Found an issue which resulted in the first issue. Will fix it later (not sure if I can do it today).

[gyptazy]

Hey @adminsyspro,

I just force-pushed the fix. You may give it a try when you have time for it. Happy to hear if this fixes your issue.

Have a nice weekend,
gyptazy

[adminsyspro]

Hey @gyptazy,

Thanks for the update, I just tried your fix but there is always an issue. I have thousand of line like that : "Reached limit for random node evaluation. Unable to find a suitable new node."

<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-C evaluated in run 28.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-B evaluated in run 29.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<6> ProxLB: Info: [rebalancing-tags-group-exclude]: Rebalancing of VM FW1 is needed due to anti-affinity group policy.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-C evaluated in run 1.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-B evaluated in run 2.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-C evaluated in run 3.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-B evaluated in run 4.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-C evaluated in run 5.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 6.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 7.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 8.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 9.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-C evaluated in run 10.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-B evaluated in run 11.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-C evaluated in run 12.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 13.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-C evaluated in run 14.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-B evaluated in run 15.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 16.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-C evaluated in run 17.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-B evaluated in run 18.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 19.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-B evaluated in run 20.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 21.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 22.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-C evaluated in run 23.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-C evaluated in run 24.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-A evaluated in run 25.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-C evaluated in run 26.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-B evaluated in run 27.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-C evaluated in run 28.
<6> ProxLB: Info: [random-node-getter]: New random node Proxmox-3AZ-1-B evaluated in run 29.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.
<4> ProxLB: Warning: [random-node-getter]: Reached limit for random node evaluation. Unable to find a suitable new node.

[gyptazy]

Hey @adminsyspro,

sorry for that. I pushed a new version but it also shows that new and better unit tests are needed to find such issues faster. However, please give the newly committed version a try. It also prepares a new config options in the vm_balancing where enforce_affinity_groups can be defined. This will now also be turned on as a default.

However, this option is not yet evaluated with this commit version but this should at least fix your issue (hopefully).

Thanks for your time and efforts into testing this!

gyptazy

[adminsyspro]

Hey @gyptazy

It works perfectly now :)

Regards

[adminsyspro]

I have another question about master_only, I defined master_only: 1 but I have this error :

<6> ProxLB: Info: [only-on-master-executor]: Master only rebalancing is defined. Starting validation.
<6> ProxLB: Info: [cluster-master-getter]: Master node: None
<2> ProxLB: Error: [cluster-master-getter]: Could not obtain cluster master. Please check your configuration - stopping.

I have a cluster of 3 node.

[adminsyspro]

I found out why I got this error, I didn't create HA rule yet.

It works now !

[gyptazy]

Hey @adminsyspro,

happy to hear, thanks for testing and confirming. I'll give it some additional tests this evening when back home and if it succeeds all ones it'll be merged.

Cheers,
gyptazy

[gyptazy]

I found out why I got this error, I didn't create HA rule yet.
It works now !

Thanks for the hint. I adjusted the output, hope this makes it more clear:
https://github.com/gyptazy/ProxLB/compare/a8115e1c9da253d656a06a3422db4e729d61c855..1420183be7ff1b53f19a53941bb39bcc102052db#diff-4d47e7584181ff92b3c3f57588b89e4fb11158ac22f3d50066588c07267e5a86R446

[gyptazy]

Validated in different setups and clusters.

LGTM