Added silk_request_garbage_collect command for out-of-band garbage co…

…llection. (#541)

* Added silk_request_garbage_collect command for out-of-band garbage collection.

To avoid deadlock issues as mentioned e.g. in #265 #294 #371
Based on #265 (comment)

* Okay, to please @codecov :)

Co-authored-by: Nikolaus Schlemm <[email protected]>
Co-authored-by: Albert Wang <[email protected]>

README.md

@@ -478,6 +478,12 @@ The garbage collection is only run on a percentage of requests to reduce overhea
 SILKY_MAX_RECORDED_REQUESTS_CHECK_PERCENT = 10
 ```
 
+In case you want decouple silk's garbage collection from your webserver's request processing, set SILKY_MAX_RECORDED_REQUESTS_CHECK_PERCENT=0 and trigger it manually, e.g. in a cron job:
+
+```bash
+python manage.py silk_request_garbage_collect
+```
+
 ### Enable query analysis
 
 To enable query analysis when supported by the dbms a config var can be set in order to execute queries with the analyze features.

docs/quickstart.rst

@@ -32,7 +32,7 @@ Add the following to your ``urls.py``:
 
 Run ``migrate`` to create Silk's database tables:
 
-.. code-block:: python
+.. code-block:: bash
 
     python manage.py migrate
 

docs/troubleshooting.rst

@@ -24,3 +24,11 @@ Middleware
 The middleware is placement sensitive. If the middleware before ``silk.middleware.SilkyMiddleware`` returns from ``process_request`` then ``SilkyMiddleware`` will never get the chance to execute. Therefore you must ensure that any middleware placed before never returns anything from ``process_request``. See the `django docs <https://docs.djangoproject.com/en/dev/topics/http/middleware/#process-request>`_ for more information on this.
 
 This `GitHub issue <https://github.com/jazzband/django-silk/issues/12>`_ also has information on dealing with middleware problems.
+
+Garbage Collection
+------------------
+
+To `avoid <https://github.com/jazzband/django-silk/issues/265>`_ `deadlock <https://github.com/jazzband/django-silk/issues/294>`_ `issues <https://github.com/jazzband/django-silk/issues/371>`_, you might want to decouple silk's garbage collection from your webserver's request processing, set ``SILKY_MAX_RECORDED_REQUESTS_CHECK_PERCENT=0`` and trigger it manually, e.g. in a cron job:
+
+.. code-block:: bash
+    python manage.py silk_request_garbage_collect

project/tests/test_command_garbage_collect.py

@@ -0,0 +1,22 @@
+from django.core import management
+from django.test import TestCase
+
+from silk import models
+from silk.config import SilkyConfig
+
+from .factories import RequestMinFactory
+
+
+class TestViewClearDB(TestCase):
+    def test_garbage_collect_command(self):
+        SilkyConfig().SILKY_MAX_RECORDED_REQUESTS = 2
+        RequestMinFactory.create_batch(3)
+        self.assertEqual(models.Request.objects.count(), 3)
+        management.call_command("silk_request_garbage_collect")
+        self.assertEqual(models.Request.objects.count(), 2)
+        management.call_command("silk_request_garbage_collect", max_requests=1)
+        self.assertEqual(models.Request.objects.count(), 1)
+        management.call_command(
+            "silk_request_garbage_collect", max_requests=0, verbosity=2
+        )
+        self.assertEqual(models.Request.objects.count(), 0)

silk/management/commands/silk_request_garbage_collect.py

@@ -0,0 +1,29 @@
+from django.core.management.base import BaseCommand
+
+import silk.models
+from silk.config import SilkyConfig
+
+
+class Command(BaseCommand):
+    help = "Triggers silk's request garbage collect."
+
+    def add_arguments(self, parser):
+        parser.add_argument(
+            "-m",
+            "--max-requests",
+            default=SilkyConfig().SILKY_MAX_RECORDED_REQUESTS,
+            type=int,
+            help="Maximum number of requests to keep after garbage collection.",
+        )
+
+    def handle(self, *args, **options):
+        if "max_requests" in options:
+            max_requests = options["max_requests"]
+            SilkyConfig().SILKY_MAX_RECORDED_REQUESTS = max_requests
+        if options["verbosity"] >= 2:
+            max_requests = SilkyConfig().SILKY_MAX_RECORDED_REQUESTS
+            request_count = silk.models.Request.objects.count()
+            self.stdout.write(
+                f"Keeping up to {max_requests} of {request_count} requests."
+            )
+        silk.models.Request.garbage_collect(force=True)