DB connection is interrupted with multiple management servers

[Luskan777]

problem

Hello, I am using an environment with 2 management servers, using MariaDB Galera as the database server.

A few minutes after starting the management server service, the connection to the database is dropped, and the management server logs show the following record:

ERROR [c.c.u.d.T.Transaction] (AsyncJobMgr-Heartbeat-1:[ctx-c66c8126]) (logid:cfb46db0) Unexpected exception: java.sql.SQLTransientConnectionException: cloud - Connection is not available, request timed out after 80000ms (total=1000, active=1000, idle=0, waiting=22)

My current MariaDB server configuration is as follows:

[mysqld]
binlog_format=ROW
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
query_cache_size=0
query_cache_type=0
bind-address=0.0.0.0
innodb_rollback_on_timeout=1
innodb_lock_wait_timeout=600
max_connections=1000
log-bin=mysql-bin
max_allowed_packet=1024M
net_read_timeout=120
net_write_timeout=120

And the configuration of access to the management servers database (db.properties) are these:

# CloudStack database tuning parameters
db.cloud.connectionPoolLib=hikaricp
db.cloud.maxActive=1000
db.cloud.maxIdle=100
db.cloud.maxWait=900000
db.cloud.minIdleConnections=20
db.cloud.connectionTimeout=80000
db.cloud.keepAliveTime=800000
db.cloud.validationQuery=/* ping */ SELECT 1
db.cloud.testOnBorrow=true
db.cloud.testWhileIdle=true
db.cloud.timeBetweenEvictionRunsMillis=40000
db.cloud.minEvictableIdleTimeMillis=240000
db.cloud.poolPreparedStatements=false
db.cloud.url.params=prepStmtCacheSize=517&cachePrepStmts=true&sessionVariables=sql_mode='STRICT_TRANS_T ABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION'&serverTimezone=UTC

PS: I changed some parameters of both the database server (max_connections, max_allowed_packet and timeout parameters) and the connection pools in the db.properties file (db.cloud.maxActive, db.cloud.connectionTimeout, db.cloud.minIdleConnections), but none of them seem to have solved it, it just takes a little longer to interrupt.

When I check the database server logs I can only get the following error records:

Feb 26 10:19:14 acs-stg-mngt-01 mariadbd[20527]: 2025-02-26 10:19:14 3320 [Warning] Aborted connection 3320 to db: 'cloud_usage' user: 'cloud' host: 'IP_MNGT_SERVER' (Got an error reading communication packets)
Feb 26 10:18:34 acs-stg-mngt-01 mariadbd[20527]: 2025-02-26 10:18:34 2282 [Warning] Aborted connection 2282 to db: 'cloud' user: 'cloud' host: 'IP_MNGT_SERVER' (Got an error reading communication packets)

PS: I noticed that this started happening after installing cloudstack-usage, this same problem already occurred in a Standalone installation (without HA) of Cloudstack and it occurred precisely after installing cloudstack-usage, but I'm still debugging to find out if it is really related to the current problem.

versions

Cloudstack: 4.20
MariaDB: 10.6

The steps to reproduce the bug

1. Configure Cloudstack in HA mode (MariaDB, HA Proxy)
2. Add 3 KVM hosts
3. Enable HA hosts

What to do about it?

No response

[DaanHoogland]

@Luskan777 , since mariadb 10 is mysql 5 compatible and ACS specifies mysql 8.0, this might be your issue. I am not sure if this true but worth a shot.

[weizhouapache]

Hi @DaanHoogland , I updated my MariaDB version to 11.2, but I still have the same problem.

From what I've researched, it seems that the connections are not being closed correctly in some part of the application.

The problem starts to happen mainly when I enable HA on the agent hosts (KVM). However, in another Cloudstack installation where I used Mysql Server (without HA), this happened a few times as soon as I installed cloudstack-usage, but since it didn't happen very often, I ended up not paying attention.

The problem doesn't seem to be on the database server (MariaDB), because the MariaDB database servers are healthy from what I've monitored, there is no excessive memory or CPU consumption. So I believe there is some limitation in the configuration and use of the hikaricp library. I've explored several configurations, but I haven't had any success with any of them. Can you tell me if I'm missing something?

I tested mariadb for pr #9941 on rocky8/9, it worked fine. However the pr is targeted to 4.19 so it does not use hikaricp, I will test with 4.20-snapshot.

Based on your suspicion, can you change some settings in db.properties to dbcp2, then restart management server ?

db.cloud.connectionPoolLib
db.cloud_usage.connectionPoolLib
db.simulator.connectionPoolLib

Refer to https://docs.cloudstack.apache.org/en/4.20.0.0/adminguide/tuning.html#selecting-database-connection-pool-library

[Luskan777]

Hi @DaanHoogland , I updated my MariaDB version to 11.2, but I still have the same problem.
From what I've researched, it seems that the connections are not being closed correctly in some part of the application.
The problem starts to happen mainly when I enable HA on the agent hosts (KVM). However, in another Cloudstack installation where I used Mysql Server (without HA), this happened a few times as soon as I installed cloudstack-usage, but since it didn't happen very often, I ended up not paying attention.
The problem doesn't seem to be on the database server (MariaDB), because the MariaDB database servers are healthy from what I've monitored, there is no excessive memory or CPU consumption. So I believe there is some limitation in the configuration and use of the hikaricp library. I've explored several configurations, but I haven't had any success with any of them. Can you tell me if I'm missing something?

I tested mariadb for pr #9941 on rocky8/9, it worked fine. However the pr is targeted to 4.19 so it does not use hikaricp, I will test with 4.20-snapshot.

Based on your suspicion, can you change some settings in db.properties to dbcp2, then restart management server ?

db.cloud.connectionPoolLib db.cloud_usage.connectionPoolLib db.simulator.connectionPoolLib

Refer to https://docs.cloudstack.apache.org/en/4.20.0.0/adminguide/tuning.html#selecting-database-connection-pool-library

Hi @weizhouapache , thanks for your contribution.

Perfect, I will test the configuration with dbcp2, I will get back to you with the result soon

[weizhouapache]

I tested mariadb for pr #9941 on rocky8/9, it worked fine. However the pr is targeted to 4.19 so it does not use hikaricp, I will test with 4.20-snapshot.
Based on your suspicion, can you change some settings in db.properties to dbcp2, then restart management server ?
db.cloud.connectionPoolLib db.cloud_usage.connectionPoolLib db.simulator.connectionPoolLib
Refer to https://docs.cloudstack.apache.org/en/4.20.0.0/adminguide/tuning.html#selecting-database-connection-pool-library

Hi @weizhouapache , thanks for your contribution.

Perfect, I will test the configuration with dbcp2, I will get back to you with the result soon

I run smoke tests with two OL8 management server, two OL8 hosts and mariadb server 11.7 (not mariadb galera cluster).

[root@pr10006-t12509-kvm-ol8-mgmt1 ~]# mysql -uroot -p cloud
mysql: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb' instead
Enter password: 
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 4374
Server version: 11.7.2-MariaDB-log MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [cloud]> 

it uses default hikaricp library. The test results look good
#10006 (comment)

[Luskan777]

@weizhouapache and @DaanHoogland , it seems that the problem is in my HAproxy configuration. Since it is a MariaDB Galera, I use HAproxy to do load balancing between the database servers, which is common when using MariaDB Galera. It seems that HAproxy was closing the connections unexpectedly, which is why the MariaDB logs recorded the error (Got an error reading communication packets).
So, to solve the problem, I changed some HAproxy configuration parameters, such as maxconn and timeout.
In case someone else has the same problem one day, I will leave my current HAproxy configuration below.

global
    log /dev/log local0
    daemon

defaults
    log global
    mode tcp
    timeout client 1h
    timeout server 1h
    timeout connect 10s

frontend mysql_front
    bind *:33306
    mode tcp
    default_backend mysql_backend

backend mysql_backend
    mode tcp
    balance source
    option tcp-check
    timeout client 1h
    timeout server 1h
    timeout connect 10s
    server db1 192.168.77.10:3306 check
    server db2 192.168.77.11:3306 check
    server db3 192.168.77.12:3306 check backup
    server db4 192.168.77.13:3306 check backup

Anyway, even though the problem is not with Cloudstack, but with HAproxy + MariaDB Galera, I hope I have helped in some way.
Thank you for your time @weizhouapache and @DaanHoogland

[DaanHoogland]

Thanks @Luskan777 , I am sure this is going to help some people

[weizhouapache]

thanks @Luskan777 for the sharing

In my opinion, you can use mariadb galera cluster without haproxy
you can test with the following config in db.properties

db.cloud.uri=jdbc:mariadb:sequential://xx.xx.xx.1:3306,xx.xx.xx.2:3306,xx.xx.xx.3:3306/cloud?autoReconnect=true&prepStmtCacheSize=517&cachePrepStmts=true&sessionVariables=sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION'&serverTimezone=UTC

[Luskan777]

Thanks @weizhouapache , I will test with this configuration, I will get back to you soon.

[weizhouapache]

@Luskan777
you can test different failover modes to see if they meet your expectation
refer to https://mariadb.com/kb/en/failover-and-high-availability-with-mariadb-connector-j/#failover-and-load-balancing-modes

look forward to your test results