Add a python testing framework (pgbouncer#792)

This changes our testing infrastructure to use python instead of bash.
Both `test/test.sh` and `test/ssl/test.sh` and have been completely 
coverted to python. 

The main reason I want to introduce this change is because Bash
is one of the worst languages to write tests in. One of the main goals
of tests is to find unexpected errors. And bash makes it very easy not
to notice such errors. This made it quite hard for me to write correct
tests and review newly introduced tests in PRs for correctness.

Another big reason I want this change is because writing tests 
for pgbouncer#666 requires running multiple pgbouncer processes at the 
same time. The bash infrastructure did not allow for this. The 
new python testing infrastructure.

Finally there's some other advantages that this new test
infrastructure brings:
1. Tests can be run in parallel, which makes them go **much** 
    faster. The duration of the test step in CI is now ~1 minute 
    for non valgrind runs and ~2 minutes for valgrind runs. 
    With the bash testing infrastructure these timings were 
    5.5 minutes and ~8 minutes respectively.
2. Running tests with `valgrind` locally is now as simple as 
    `ENABLE_VALGRIND=1 pytest -n auto`
3. Some tests would not run on some OSes (e.g. ssl tests on
    windows). I made sure almost all are now fully cross platform
    (for the platforms we support in CI).
4. `USE_SUDO=1` tests are now run in CI on MacOS and FreeBSD.
    (I confirmed `USE_SUDO=1` works with Linux on my local machine, 
    but was unable to get it working in CI due to `iptables` not being
    allowed in the CI containers).

How to run tests can be found in the updated `test/README.md` file.

PS. I haven't tested with OpenBSD, since I don't have access to such a
machine. I'm fairly confident that the tests will run fine though, since
they pass on all CI plaftorms (including MacOS and FreeBSD). But if
anyone else has access to an OpenBSD box, it would be great if you
could test that tests pass there as well.

.appveyor.yml

@@ -8,8 +8,12 @@ environment:
 install:
   - git submodule update --init --recursive
   - set PATH=C:/msys64/usr/bin;%PATH%
-  - sh -l -c "pacman --noconfirm -S --needed base-devel ${MINGW_PACKAGE_PREFIX}-toolchain ${MINGW_PACKAGE_PREFIX}-libevent ${MINGW_PACKAGE_PREFIX}-openssl ${MINGW_PACKAGE_PREFIX}-postgresql autoconf automake libtool python zip"
-  - choco install pandoc
+  - choco install -y --no-progress msys2
+  - sh -l -c "pacman --noconfirm -S --needed base-devel ${MINGW_PACKAGE_PREFIX}-gcc ${MINGW_PACKAGE_PREFIX}-libevent ${MINGW_PACKAGE_PREFIX}-openssl ${MINGW_PACKAGE_PREFIX}-postgresql autoconf automake libtool ${MINGW_PACKAGE_PREFIX}-python ${MINGW_PACKAGE_PREFIX}-python-pip zip"
+  - sh -l -c 'pip install -r /c/projects/pgbouncer-pgbouncer/requirements.txt'
+  - echo 127.0.0.1 localhost >> c:\Windows\System32\Drivers\etc\hosts
+  - sh -l -c 'echo "127.0.0.1   localhost" >> /etc/hosts'
+  - choco install -y --no-progress pandoc
 
 build_script:
   - set HOME=.
@@ -26,7 +30,7 @@ artifacts:
 test_script:
   - set HOME=.
   - set PATH=C:/msys64/usr/bin;%PATH%
-  - sh -l -c "make -j4 check tls_support=no"
+  - sh -l -c "make -j4 check CONCURRENCY=4"
   - sh -l -c "windres pgbouncer.exe"
 
 on_failure:

.cirrus.yml

@@ -32,12 +32,12 @@ task:
     - container:
         image: ubuntu:20.04
       env:
-        use_valgrind: yes
+        ENABLE_VALGRIND: yes
         CFLAGS: -O0 -g
     - container:
         image: ubuntu:20.04
       env:
-        use_valgrind: yes
+        ENABLE_VALGRIND: yes
         CFLAGS: -O0 -g
         PGVERSION: 9.6
     - container:
@@ -66,30 +66,22 @@ task:
     - curl https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
     - echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" | tee /etc/apt/sources.list.d/pgdg.list
     - apt-get update
-    - pkgs="autoconf automake ca-certificates cpio libc-ares-dev libevent-dev libssl-dev libsystemd-dev libtool libudns-dev make pandoc postgresql-$PGVERSION pkg-config python3"
+    - pkgs="autoconf automake ca-certificates cpio libc-ares-dev libevent-dev libssl-dev libsystemd-dev libtool libudns-dev make pandoc postgresql-$PGVERSION pkg-config python3 python3-pip sudo iptables"
     - case $CC in clang) pkgs="$pkgs clang";; esac
-    - if [ x"$use_valgrind" = x"yes" ]; then pkgs="$pkgs valgrind"; fi
+    - if [ x"$ENABLE_VALGRIND" = x"yes" ]; then pkgs="$pkgs valgrind"; fi
     - if [ x"$use_scan_build" = x"yes" ]; then pkgs="$pkgs clang-tools"; fi
     - apt-get -y install $pkgs
+    - python3 -m pip install -r requirements.txt
+    - if [ "$(lsb_release -cs)" = "bionic" ]; then python3 -m pip install contextlib2; fi
     - useradd user
     - chown -R user .
+    - echo 'user ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers
   build_script:
     - su user -c "./autogen.sh"
     - su user -c "${use_scan_build:+scan-build} ./configure --prefix=$HOME/install --enable-cassert --enable-werror --without-cares --with-systemd $configure_args"
     - su user -c "${use_scan_build:+scan-build} make -j4"
   test_script:
-    - |
-      if [ x"$use_valgrind" = x"yes" ]; then
-        export BOUNCER_EXE_PREFIX="valgrind --quiet --leak-check=full --show-reachable=no --track-origins=yes --error-markers=VALGRIND-ERROR-BEGIN,VALGRIND-ERROR-END --log-file=/tmp/valgrind.%p.log"
-      fi
-    - su user -c "PATH=/usr/lib/postgresql/${PGVERSION}/bin:$PATH make -j4 check"
-    - |
-      if [ x"$use_valgrind" = x"yes" ]; then
-        if grep -q VALGRIND-ERROR /tmp/valgrind.*.log; then
-          cat /tmp/valgrind.*.log
-          exit 1
-        fi
-      fi
+    - su user -c "PATH=/usr/lib/postgresql/${PGVERSION}/bin:$PATH make -j4 check CONCURRENCY=4"
   install_script:
     - make -j4 install
   dist_script:
@@ -116,18 +108,22 @@ task:
       - image: centos:centos7
   setup_script:
     - yum -y install autoconf automake diffutils file libevent-devel libtool make openssl-devel pkg-config postgresql-server systemd-devel wget
-    - if cat /etc/centos-release | grep -q ' 7'; then yum -y install python; else yum -y install python3; fi
+    - if cat /etc/centos-release | grep -q ' 7'; then yum -y install python python-pip; else yum -y install python3 python3-pip sudo iptables; fi
     - wget -O /tmp/pandoc.tar.gz https://github.com/jgm/pandoc/releases/download/2.10.1/pandoc-2.10.1-linux-amd64.tar.gz
     - tar xvzf /tmp/pandoc.tar.gz --strip-components 1 -C /usr/local/
+# XXX: python too old
+    - if cat /etc/centos-release | grep -q ' 7'; then true; else pip3 install -r requirements.txt; fi
+    - if cat /etc/centos-release | grep -q ' 7'; then true; else pip3 install contextlib2; fi
     - useradd user
     - chown -R user .
+    - echo 'user ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers
   build_script:
     - su user -c "./autogen.sh"
     - su user -c "./configure --prefix=$HOME/install --enable-cassert --enable-werror --with-systemd"
     - su user -c "make -j4"
   test_script:
 # XXX: postgresql too old on centos7
-    - if cat /etc/centos-release | grep -q ' 7'; then true; else su user -c "make -j4 check"; fi
+    - if cat /etc/centos-release | grep -q ' 7'; then true; else su user -c "make -j4 check CONCURRENCY=4"; fi
   install_script:
     - make -j4 install
   always:
@@ -142,17 +138,19 @@ task:
       - image: alpine:latest
   setup_script:
     - apk update
-    - apk add autoconf automake bash build-base libevent-dev libtool openssl openssl-dev pkgconf postgresql python3 wget
+    - apk add autoconf automake bash build-base libevent-dev libtool openssl openssl-dev pkgconf postgresql python3 py3-pip wget sudo iptables
     - wget -O /tmp/pandoc.tar.gz https://github.com/jgm/pandoc/releases/download/2.10.1/pandoc-2.10.1-linux-amd64.tar.gz
     - tar xvzf /tmp/pandoc.tar.gz --strip-components 1 -C /usr/local/
+    - python3 -m pip install -r requirements.txt
     - adduser --disabled-password user
     - chown -R user .
+    - echo 'user ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers
   build_script:
     - su user -c "./autogen.sh"
     - su user -c "./configure --prefix=$HOME/install --enable-cassert --enable-werror"
     - su user -c "make -j4"
   test_script:
-    - su user -c "make -j4 check"
+    - su user -c "make -j4 check CONCURRENCY=4"
   install_script:
     - make -j4 install
   always:
@@ -166,10 +164,17 @@ task:
     image_family: freebsd-13-1
   env:
     HAVE_IPV6_LOCALHOST: yes
+    USE_SUDO: true
   setup_script:
-    - pkg install -y autoconf automake bash gmake hs-pandoc libevent libtool pkgconf postgresql12-server python
+    - pkg install -y autoconf automake bash gmake hs-pandoc libevent libtool pkgconf postgresql12-server python devel/py-pip sudo
+    - pip install -r requirements.txt
+    - kldload pf
+    - echo 'anchor "pgbouncer_test/*"' >> /etc/pf.conf
+    - echo 'pf_enable="YES"' >> /etc/rc.conf
+    - service pf start
     - pw useradd user
     - chown -R user .
+    - echo 'user ALL=(ALL) NOPASSWD: ALL' >> /usr/local/etc/sudoers
   env:
     CPPFLAGS: -I/usr/local/include
     LDFLAGS: -L/usr/local/lib
@@ -179,7 +184,7 @@ task:
     - su user -c "./configure --prefix=$HOME/install --enable-werror"
     - su user -c "gmake -j4"
   test_script:
-    - su user -c "gmake -j4 check"
+    - su user -c "gmake -j4 check CONCURRENCY=4"
   install_script:
     - gmake -j4 install
   always:
@@ -193,8 +198,12 @@ task:
     image: ghcr.io/cirruslabs/macos-ventura-base:latest
   env:
     HAVE_IPV6_LOCALHOST: yes
+    USE_SUDO: true
   setup_script:
     - brew install autoconf automake bash libevent libtool openssl pandoc pkg-config postgresql
+    - python3 -m pip install -r requirements.txt
+    - echo 'anchor "pgbouncer_test/*"' | sudo tee -a /etc/pf.conf
+    - sudo pfctl -f /etc/pf.conf
   env:
     CPPFLAGS: -I/opt/homebrew/opt/openssl@3/include
     LDFLAGS: -L/opt/homebrew/opt/openssl@3/lib
@@ -203,7 +212,7 @@ task:
     - ./configure --prefix=$HOME/install --enable-werror
     - make -j4
   test_script:
-    - make -j4 check
+    - make check -j4 CONCURRENCY=4
   install_script:
     - make -j4 install
   always:
@@ -226,14 +235,17 @@ task:
         MSYSTEM: MINGW32
   setup_script:
     - choco install -y --no-progress msys2
-    - sh -l -c "pacman --noconfirm -S --needed base-devel ${MINGW_PACKAGE_PREFIX}-gcc ${MINGW_PACKAGE_PREFIX}-libevent ${MINGW_PACKAGE_PREFIX}-openssl ${MINGW_PACKAGE_PREFIX}-postgresql autoconf automake libtool python zip"
+    - sh -l -c "pacman --noconfirm -S --needed base-devel ${MINGW_PACKAGE_PREFIX}-gcc ${MINGW_PACKAGE_PREFIX}-libevent ${MINGW_PACKAGE_PREFIX}-openssl ${MINGW_PACKAGE_PREFIX}-postgresql autoconf automake libtool ${MINGW_PACKAGE_PREFIX}-python ${MINGW_PACKAGE_PREFIX}-python-pip zip"
+    - sh -l -c 'pip install -r requirements.txt'
+    - echo 127.0.0.1 localhost >> c:\Windows\System32\Drivers\etc\hosts
+    - sh -l -c 'echo "127.0.0.1   localhost" >> /etc/hosts'
     - choco install -y --no-progress pandoc
   build_script:
     - sh -l -c "./autogen.sh"
     - sh -l -c "./configure --prefix=$HOME/install --enable-werror PANDOC=/c/programdata/chocolatey/bin/pandoc LDFLAGS=-static LIBS=-liphlpapi PKG_CONFIG='pkg-config --static'"
     - sh -l -c "make -j4"
   test_script:
-    - sh -l -c "make -j4 check tls_support=no"
+    - sh -l -c "make -j4 check CONCURRENCY=4"
     - sh -l -c "windres pgbouncer.exe"
   install_script:
     - sh -l -c "make -j4 install"

.gitignore

@@ -29,3 +29,4 @@
 *.exe
 *.gz
 *.swp
+.venv

Makefile

@@ -128,6 +128,10 @@ AM_LANG_RC_LINK = false
 # now load antimake
 #
 
+PYTEST = $(shell command -v pytest || echo 'python3 -m pytest')
+
+CONCURRENCY = auto
+
 USUAL_DIR = lib
 
 abs_top_srcdir ?= $(CURDIR)
@@ -139,6 +143,7 @@ config.mak:
 
 check: all
 	etc/optscan.sh
+	PYTHONIOENCODING=utf8 $(PYTEST) -n $(CONCURRENCY)
 	$(MAKE) -C test check
 
 w32zip = $(PACKAGE_TARNAME)-$(PACKAGE_VERSION)-windows-$(host_cpu).zip

pyproject.toml

@@ -0,0 +1,20 @@
+[tool.pytest.ini_options]
+addopts = [
+    "--import-mode=prepend",
+    "--showlocals",
+    "--tb=short",
+]
+timeout = 30
+
+# The asyncio_mode setting doesn't work on outdated versions of pytest-asyncio.
+# For python 3.6 no version is available that supports the asyncio_mode
+# setting, and for some of the older OSes Python 3.6 is the newest python
+# available. So @pytest.mark.asyncio is used everywhere instead. Once none of
+# our supported OSes require usage we can drop those marks and rely on the
+# asyncio_mode setting instead.
+# On these outdated versions having this setting in the config will throw a
+# warning, but we still add it anyway. The reason is that by adding it async
+# fixtures work on all versions, and are pretty much impossible to make work
+# otherwise without lots of version checks. So having a warning on outdated
+# versions is a small price to pay to not have to worry about that.
+asyncio_mode = 'auto'

requirements.txt

@@ -0,0 +1,6 @@
+pytest
+pytest-asyncio
+pytest-timeout
+pytest-xdist
+psycopg
+filelock

src/main.c

@@ -812,6 +812,16 @@ static void xfree(char **ptr_p)
 _UNUSED
 static void cleanup(void)
 {
+	/*
+	 * We don't want to cleanup when we're the target of a takeover, because
+	 * that would close the sockets that we hand over. There's no real clean
+	 * way to detect that we were the target, so the below check is rather
+	 * crude. But since this cleanup is only happening in builds with asserts
+	 * enabled anyway it seems fine.
+	 */
+	if (cf_pause_mode == P_SUSPEND && cf_shutdown == 2) {
+		return;
+	}
 	adns_free_context(adns);
 	adns = NULL;
 

test/Makefile

@@ -10,7 +10,12 @@ DIST_SUBDIRS = ssl
 
 EXTRA_DIST = conntest.sh ctest6000.ini ctest7000.ini run-conntest.sh \
 	     hba_test.eval hba_test.rules Makefile \
-	     test.ini test.sh stress.py userlist.txt
+	     test.ini stress.py userlist.txt \
+	     __init__.py conftest.py utils.py \
+	     test_admin.py test_auth.py test_cancel.py test_limits.py \
+	     test_misc.py test_no_database.py test_no_user.py test_operations.py \
+	     test_ssl.py test_timeouts.py
+
 
 noinst_PROGRAMS = hba_test
 hba_test_CPPFLAGS = -I../include $(LIBEVENT_CFLAGS)
@@ -33,8 +38,4 @@ include ../config.mak
 include ../lib/mk/antimake.mk
 
 check: all
-	./test.sh
 	./hba_test
-ifeq ($(tls_support),yes)
-	$(MAKE) -C ssl check
-endif

test/README.md

@@ -3,16 +3,22 @@ Tests
 
 Various ways to test PgBouncer:
 
-- `test.sh`
+- `test_xxx.py`
 
     General test of basic functionality and different configuration
     parameters including timeouts, pool size, online restart,
-    pause/resume, etc.  To invoke, just run `./test.sh`.  This needs
-    PostgreSQL server programs (`initdb`, `pg_ctl`) in the path, so if
-    you are on a system that doesn't have those in the normal path
-    (e.g., Debian, Ubuntu), set `PATH` beforehand.
+    pause/resume, etc.
 
-    This test is run by `make check`.
+    To be able to run these tests you need to install a few python test
+    libraries.  To do so, you should run the following from of the root of the
+    repository:
+
+    pip3 install --user -r requirements.txt
+
+    To run the tests after doing that, just run `pytest -n auto` from the root
+    of the repository.  This needs PostgreSQL server programs (`initdb`,
+    `pg_ctl`) in the path, so if you are on a system that doesn't have those in
+    the normal path (e.g., Debian, Ubuntu), set `PATH` beforehand.
 
     Optionally, this test suite can use `iptables`/`pfctl` to simulate
     various network conditions.  To include these tests, set the
@@ -22,11 +28,28 @@ Various ways to test PgBouncer:
     `/etc/sudoers` appropriately at your peril.  Check the source if
     there are any doubts.
 
-- `ssl/test.sh`
+    This test is run by `make check`.
+
+    You can review the pytest docs on how to run tests with pytest, but the most
+    common commands that you'll want to use are:
+
+    ```bash
+    # Run all tests in parallel
+    pytest -n auto
+
+    # Run all tests sequentially
+    pytest
+
+    # Run a specific test
+    pytest test/test_limits.py::test_max_user_connections
+
+    # Run a specific test file in parallel
+    pytest -n auto test/test_limits.py
 
-    Tests SSL/TLS functionality.  Otherwise very similar to `test.sh`.
+    # Run any test that contains a certain string in the name
+    pytest -k ssl
+    ```
 
-    This test is run by `make check` if TLS support is enabled.
 
 - `hba_test`