V1 of django-pgmigrate

``django-pgmigrate`` helps you avoid costly downtime with Postgres migrations
and provides the following features to alleviate problematic locking
scenarios when running migrations:

* Detect blocking queries and terminate them automatically.
* Print blocking queries so that you can inspect
  and terminate them manually.
* Set the lock timeout so that migrations are terminated if they block too long.

Type: api-break

README.rst

@@ -1,11 +1,23 @@
 django-pgmigrate
-########################################################################
+################
 
-Documentation
-=============
+``django-pgmigrate`` helps you avoid costly downtime with Postgres migrations.
 
-`View the django-pgmigrate docs here
-<https://django-pgmigrate.readthedocs.io/>`_.
+Imagine the following happens:
+
+1. A long-running task queries a model in a transaction and keeps the transaction open.
+2. ``python manage.py migrate`` tries to change a field on the model.
+
+Because of how Postgres queues locks, this common scenario causes **every**
+subsequent query on the model to block until the query from 1) has finished.
+
+``django-pgmigrate`` provides the following features to alleviate problematic locking
+scenarios when running migrations:
+
+* Detect blocking queries and terminate them automatically (the default behavior).
+* Print blocking queries so that you can inspect
+  and terminate them manually.
+* Set the lock timeout so that migrations are terminated if they block too long.
 
 Installation
 ============
@@ -14,11 +26,56 @@ Install django-pgmigrate with::
 
     pip3 install django-pgmigrate
 
-After this, add ``pgmigrate`` to the ``INSTALLED_APPS``
+After this, add ``pgactivity``, ``pglock``, and ``pgmigrate`` to the ``INSTALLED_APPS``
 setting of your Django project.
 
+Quick Start
+===========
+
+After following the installation instructions, running
+``python manage.py migrate`` will automatically terminate any blocking
+queries. Here's an example of what it looks like:
+
+.. image:: docs/static/terminate_blocking.png
+
+There are two additional outputs in the ``migrate`` command versus the original:
+
+1. The first output line shows the Postgres process ID. This is useful for
+   querying activity that's blocking the process.
+2. The yellow text shows when a blocking query was detected and terminated.
+   In our case, it was blocking auth migration 12.
+
+You can configure ``django-pgmigrate`` to show blocked queries instead of automatically
+killing them, and you can also set the lock timeout to automatically cancel migrations if
+they block for too long.
+See the documentation section below for more details.
+
+Compatibility
+=============
+
+``django-pgmigrate`` is compatible with Python 3.7 - 3.10, Django 2.2 - 4.1, and Postgres 10 - 15.
+
+
+Documentation
+=============
+
+`View the django-pgmigrate docs here
+<https://django-pgmigrate.readthedocs.io/>`_ to learn more about:
+
+* How blocking queries are automatically terminated.
+* Configuring the command to show blocking activity instead of terminating it, along
+  with instructions on how to manually view and terminate activity.
+* Configuring lock timeouts to automatically stop migrations if they block for too long.
+* Advanced usage such as creating custom actions to run when queries are blocked.
+
 Contributing Guide
 ==================
 
 For information on setting up django-pgmigrate for development and
 contributing changes, view `CONTRIBUTING.rst <CONTRIBUTING.rst>`_.
+
+Primary Authors
+===============
+
+- `Wes Kendall <https://github.com/wesleykendall>`__
+- `Paul Gilmartin <https://github.com/PaulGilmartin>`__

docs/advanced.rst

@@ -0,0 +1,74 @@
+.. _advanced:
+
+Advanced Configuration
+======================
+
+Here cover more advanced configuration scenarios.
+
+Customizing the Blocking Action
+-------------------------------
+
+By default, ``settings.PGMIGRATE_BLOCKING_ACTION`` is set to ``pgmigrate.Terminate``,
+meaning blocking queries are automatically terminated. It can also be
+set to ``pgmigrate.Show`` to show blocking queries or ``None`` to disable any
+action altogether.
+
+You can supply a custom action to ``settings.PGMIGRATE_BLOCKING_ACTION`` to
+further customize what happens when migrations are blocked. Inherit ``pgmigrate.BlockingAction``
+and implement the ``worker`` method, which takes the migrate management command
+instance and a ``pglock.models.BlockedPGLock`` queryset matching
+all blocking locks. The function returns the blocking locks that
+were handled.
+
+Here's what the ``pgmigrate.Terminate`` action looks like:
+
+.. code-block:: python
+
+    import pgmigrate
+
+    class Terminate(pgmigrate.BlockingAction):
+        def worker(self, cmd, blocking_locks):
+            """
+            A periodic background task that terminates blocking locks.
+
+            Args:
+                cmd: The instance of the "migrate" management command
+                blocking_locks: A queryset of matching locks using the
+                    ``BlockedPGLock`` model from the ``django-pglock`` library.
+            """
+            terminated = blocking_locks.terminate_blocking_activity()
+
+            if terminated:  # pragma: no branch
+                pluralize = "ies" if len(terminated) != 1 else "y"
+                if cmd.verbosity:
+                    cmd.stdout.write(
+                        cmd.style.WARNING(
+                            f"\n  Terminated {len(terminated)} blocking quer{pluralize}..."
+                        ),
+                        ending=" ",
+                    )
+
+            return terminated
+
+Remember, the action is ran periodically during migrations. Above we're using ``cmd.stdout``
+to print messages because ``cmd`` is an instance of a management command. See
+`the Django docs <https://docs.djangoproject.com/en/4.1/howto/custom-management-commands/>`__
+for more information on how management commands work.
+
+Consult the `django-pglock docs <https://django-pglock.readthedocs.io>`__ for more information
+on how to use the ``BlockedPGLock`` model and queryset methods.
+
+Configuring the Blocking Action Interval
+----------------------------------------
+
+By default, blocking actions are ran every second. Supply a ``datetime.timedelta`` object
+to ``settings.PGMIGRATE_BLOCKING_ACTION_INTERVAL`` to change this.
+
+Disabling Patching of the Migrate Command
+-----------------------------------------
+
+By default, the ``migrate`` command is patched to use the ``pgmigrate`` command from ``django-pgmigrate``.
+If this isn't desirable, set ``settings.PGMIGRATE_PATCH_MIGRATE`` to ``False``.
+
+If disabled, you'll need to run the ``pgmigrate`` management command to apply migrations
+and use the features of ``django-pgmigrate``.

docs/automatic.rst

@@ -0,0 +1,71 @@
+.. _automatic:
+
+Automatically Terminating Blocking Queries
+==========================================
+
+After :ref:`installation` of ``django-pgmigrate``, queries that
+block migrations will be automatically terminated by default.
+
+Here's an example of what it looks like:
+
+.. image:: static/terminate_blocking.png
+
+There are two main differences in the above output versus the normal
+``migrate`` output:
+
+1. The first output line shows the Postgres process ID. This is useful for
+   manually querying active blocking locks.
+2. The yellow text shows when a blocking query was detected and terminated.
+   In our case, a query was blocking auth migration 12.
+
+.. tip::
+
+   If you'd like to disable automatically terminating blocking queries, consult
+   the :ref:`manual` or :ref:`advanced` sections.
+
+How it Works
+------------
+
+Underneath the hood, the `django-pglock library <https://django-pglock.readthedocs.io>`__
+is used to discover blocking locks and terminate them. Specifically, the
+``pglock.prioritize`` decorator is applied to migrations. This decorator:
+
+* Runs a background thread and periodically checks for blocking locks.
+* Uses the ``pglock.models.BlockedPGLock`` proxy model, which uses
+  Postgres's `pg_blocking_pids function <https://www.postgresql.org/docs/current/functions-info.html>`__ to accurately determine
+  which queries are blocking migrations.
+
+Consult the `django-pglock library docs <https://django-pglock.readthedocs.io>`__
+for more information on how ``pglock.prioritize`` works.
+
+How Queries are Terminated
+--------------------------
+
+Blocking queries are terminated using Postgres's `pg_terminate_backend function <https://www.postgresql.org/docs/9.3/functions-admin.html>`__.
+Calling this Postgres function on a query will
+result in a ``django.db.utils.OperationalError`` being raised in the
+process executing the query. If the process was in a transaction, the
+transaction will be rolled back.
+
+Only Terminate Long-Running Queries
+-----------------------------------
+
+By default, all blocking queries are immediately terminated when discovered.
+You can configure the underlying action to only terminate blocking queries based on their duration.
+Do this in settings.py:
+
+.. code-block:: python
+
+   import pgmigrate
+
+   PGMIGRATE_BLOCKING_ACTION = pgmigrate.Terminate(blocking_activity__duration__gte="5 seconds")
+
+The ``pgmigrate.Terminate`` action takes filters that can be applied to the underlying
+``pglock.models.BlockedPGLock`` queryset. In this case, we are filtering it by any blocking queries
+that have been running longer than five seconds.
+
+.. note::
+
+   Remember, the background worker runs on a periodic interval that defaults to one second. Given
+   our example above, this means blocking queries can run up to six seconds before being
+   terminated.

docs/contributing.rst

@@ -1 +1,3 @@
+.. _contributing:
+
 .. include:: ../CONTRIBUTING.rst

docs/index.rst

@@ -1,7 +1,66 @@
 django-pgmigrate
-=======================================================================
+================
 
-Welcome to the docs for django-pgmigrate! It doesn't appear that
-the author has created any Sphinx docs for their project yet. Try
-viewing the `README <https://github.com/Opus10/django-pgmigrate>`_
-of their project for documentation.
+``django-pgmigrate`` helps you avoid costly downtime with Postgres migrations.
+
+Imagine the following happens:
+
+1. A long-running task queries a model in a transaction and keeps the transaction open.
+2. ``python manage.py migrate`` tries to change a field on the model.
+
+Because of how Postgres queues locks, this common scenario causes **every**
+subsequent query on the model to block until the query from 1) has finished.
+
+``django-pgmigrate`` provides the following features to alleviate problematic locking
+scenarios when running migrations:
+
+* Detect blocking queries and terminate them automatically (the default behavior).
+* Print blocking queries so that you can inspect
+  and terminate them manually.
+* Set the lock timeout so that migrations are terminated if they block too long.
+
+Quick Start
+-----------
+
+After following the :ref:`installation` section, running
+``python manage.py migrate`` will automatically terminate any blocking
+queries. Here's an example of what it looks like:
+
+.. image:: static/terminate_blocking.png
+
+There are two additional outputs in the ``migrate`` command versus the original:
+
+1. The first output line shows the Postgres process ID. This is useful for
+   querying activity that's blocking the process.
+2. The yellow text shows when a blocking query was detected and terminated.
+   In our case, it was blocking auth migration 12.
+
+You can configure ``django-pgmigrate`` to show blocked queries instead of automatically
+killing them, and you can also set the lock timeout to automatically cancel migrations if
+they block for too long.
+See the next steps below for more details.
+
+Compatibility
+-------------
+
+``django-pgmigrate`` is compatible with Python 3.7 - 3.10, Django 2.2 - 4.1, and Postgres 10 - 15.
+
+Next Steps
+----------
+
+We recommend everyone first read:
+
+* :ref:`installation` for how to install the library.
+
+After this, there are several usage guides:
+
+* :ref:`automatic` for more information on how blocking queries are automatically terminated.
+* :ref:`manual` for instructions on how to view blocking activity and manually terminate it.
+* :ref:`timeout` for configuring lock timeouts for migrations.
+* :ref:`advanced` for advanced usage such as creating custom actions to run when queries are blocked.
+
+Core API information exists in these sections:
+
+* :ref:`settings` for all available Django settings.
+* :ref:`release_notes` for information about every release.
+* :ref:`contributing` for details on contributing to the codebase.

docs/installation.rst

@@ -1,9 +1,11 @@
+.. _installation:
+
 Installation
 ============
 
 Install django-pgmigrate with::
 
     pip3 install django-pgmigrate
 
-After this, add ``pgmigrate`` to the ``INSTALLED_APPS``
+After this, add ``pgactivity``, ``pglock``, and ``pgmigrate`` to the ``INSTALLED_APPS``
 setting of your Django project.

docs/manual.rst

@@ -0,0 +1,72 @@
+.. _manual:
+
+Manually Terminating Blocking Queries
+=====================================
+
+If it's not desirable to automatically terminate queries, you
+can configure ``django-pgmigrate`` to show the blocking queries
+so that they can be manually handled.
+
+Set ``settings.PGMIGRATE_BLOCKING_ACTION`` to ``pgmigrate.Show`` to
+configure this behavior. For example, in ``settings.py``:
+
+.. code-block:: python
+
+    import pgmigrate
+
+    PGMIGRATE_BLOCKING_ACTION = pgmigrate.Show
+
+Here's an example of what it looks like:
+
+.. image:: static/show_blocking.png
+
+In the above, a warning message is printed when blocking activities
+are found. The Postgres process IDs of the activities are printed.
+Queries will remain blocked until they finish or are manually terminated.
+
+Manually Inspecting and Terminating Queries
+-------------------------------------------
+
+``django-pgmigrate`` automatically installs the
+`django-pgactivity <https://django-pgactivity.readthedocs.io>`__ library, which
+makes it easy to view and terminate active queries.
+
+Our original example printed a blocking ID of ``38076``.
+We can run the following to show the duration of the query and the SQL::
+
+    python manage.py pgactivity 38076
+
+Output looks like this::
+
+    38076 | 0:00:36 | IDLE_IN_TRANSACTION | None | select * from auth_user;
+
+The second column is the duration of the query. The final column is the SQL.
+You can terminate the query with::
+
+    python manage.py pgactivity 38076 --terminate
+
+Once the query is terminated, migrations will continue.
+
+.. tip::
+
+    The ``pgactivity`` command can take multiple process IDs. These can
+    be directly copied from the output of the ``pgmigrate`` command.
+
+Adding Application Context to Queries
+-------------------------------------
+
+The `django-pgactivity <https://django-pgactivity.readthedocs.io>`__ library
+comes with middleware to automatically annotate the URL that issued the SQL
+statement, adding more information to help you understand where queries originate.
+Add ``pgactivity.middleware.ActivityMiddleware`` to ``settings.MIDDLEWARE``,
+and ``python manage.py pgactivity`` will also show the application context in the
+results.
+
+Once configured, our example output from above would look like this::
+
+   38076 | 0:00:36 | IDLE_IN_TRANSACTION | {'url': '/admin/', 'method': 'GET'} | select * from auth_user;
+
+We recommend `reading the django-pgactivity docs <https://django-pgactivity.readthedocs.io>`__ 
+to learn more about how it works, along with learning how to add context to management commands and
+background tasks. The docs also explain how to configure the ``pgactivity`` command for other
+use cases.