Add pgxn_consumer with Mastodon & Twitter handlers

Rip out the Twitter notification in the upload controller, which was always a hack; uploads will now be faster without that overhead. Replace it with a trigger on the `distributions` table that calls `pg_notify('release', NEW.meta)`. This will send it to any Postgres clients listening on the `pgxn_release` channel. Implement PGXN::Manager::Consumer to be that listener. It loads any number of event handlers from the configuration file, where the old `twitter` key has been replaced by a `consumers` key, now an array of objects with configurations for handlers. The "type" key corresponds to a sub-package of PGXN::Manager::Consumer, which must exist for it to work, while the "events" key names the events for the handler to handle (currently `release`, `new_user`, and `new_mirror`). The consumer then connects to the database, listens for all event channels of interest (currently `pgxn_release`, `pgxn_new_user`, `pgxn_new_mirror`), and polls on an interval for new messages (5 seconds by default). The connection is resilient: if it loses the connection, it will simply log the error and keep trying every five seconds. PGXN::Manager::Consumer::twitter is largely the same as what was in the controller, but now with tests. PGXN::Manager::Consumer::mastodon is new, and needs a server name and access token, as well as an optional post delay, to do its thing It's a dead simple HTTP Post, so just uses LWP::UserAgent to make the call. This class also has new formatting for the message, including the abstract and a random assortment of Emoji. The trigger calls a new function, `pgxn_notify()`, which currently just dispatches to `pg_notify()`, but could be extended in the future to, say, write to a table so that the consumer can later read from the table if, for some reason, it goes down and misses notifications. Maybe that won't ever be much of a problem, since the manager and consumer should always run together, but it was simple enough to create the new function so that triggers wouldn't have to be changed in the future. While at it, change the UX text so it no longer mentions posting to Twitter. In the future we'll have to add a field for Mastodon and perhaps other likely outlets. Also fix email sending so that Unicode text is properly encoded as UTF-8.
pgxn · Feb 12, 2023 · 2c77379 · 2c77379
1 parent 84b5da2
commit 2c77379
Show file tree

Hide file tree

Showing 41 changed files with 2,016 additions and 238 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -9,7 +9,7 @@ jobs:
   build:
     strategy:
       matrix:
-        pg: [14, 13, 12]
+        pg: [15, 14, 13, 12]
     name: 🐘 PostgreSQL ${{ matrix.pg }}
     runs-on: ubuntu-latest
     container: pgxn/pgxn-tools
@@ -19,7 +19,7 @@ jobs:
       - name: Start PostgreSQL ${{ matrix.pg }}
         run: |-
           pg-start ${{ matrix.pg }} postgresql-plperl-${{ matrix.pg }} libxml-parser-perl libarchive-zip-perl libarchive-extract-perl libaliased-perl libsoftware-license-perl libtap-parser-sourcehandler-pgtap-perl libtest-file-perl libtest-file-contents-perl libtest-harness-perl libtest-mockmodule-perl libtest-nowarnings-perl libtest-xml-perl libtest-xpath-perl libclass-isa-perl libdata-dump-perl libdata-validate-uri-perl libdbi-perl libdbix-connector-perl libemail-valid-perl libemail-mime-perl libemail-address-perl libencode-perl libexception-class-dbi-perl libhttp-body-perl libhttp-negotiate-perl libjson-xs-perl libmoose-perl libmoosex-singleton-perl libnamespace-autoclean-perl libplack-perl libplack-middleware-session-perl libplack-middleware-methodoverride-perl libsemver-perl libtemplate-declare-perl libtry-tiny-perl liburi-template-perl libplack-middleware-debug-perl libplack-middleware-reverseproxy-perl libtest-pod-perl libtest-pod-coverage-perl libtest-spelling-perl cpanminus
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
       - name: Build PGXN Manager
         run: |-
           perl Build.PL
@@ -31,4 +31,5 @@ jobs:
           pg_ctlcluster ${{ matrix.pg }} test restart
           ./Build db
       - name: Test PGXN::Manager
+        env: { PGUSER: postgres }
         run: ./Build test
diff --git a/Build.PL b/Build.PL
@@ -60,6 +60,7 @@ $class->new(
         'TAP::Parser::SourceHandler::pgTAP' => '3.23',
         'Test::File'                        => '1.29',
         'Test::File::Contents'              => '0.05',
+        'Test::Exception'                   => '0.43',
         'Test::Harness'                     => '3.18',
         'Test::MockModule'                  => '0.05',
         'Test::More'                        => '0.70',
@@ -104,6 +105,8 @@ $class->new(
         'I18N::LangTags::Detect'            => '1.03',
         'JSON::XS'                          => '2.3',
         'Locale::Maketext'                  => '1.13',
+        'LWP::Protocol::https'              => 0,
+        'LWP::UserAgent'                    => 0,
         'Moose'                             => '1.15',
         'MooseX::Singleton'                 => '0.25',
         'Moose::Util::TypeConstraints'      => '1.15',
@@ -120,12 +123,14 @@ $class->new(
         'Plack::Session::Store::File'       => '0.12',
         'Plack::Middleware::Session'        => '0.12',
         'Plack::Middleware::MethodOverride' => '0.10',
+        'Proc::Daemon'                      => 0,
         'Pod::Usage'                        => '1.33',
         'namespace::autoclean'              => '0.11',
         'Router::Resource'                  => '0.11',
         'SemVer'                            => '0.10.0',
         'Template::Declare'                 => '0.43',
         'Template::Declare::Tags'           => '0.43',
+        'Time::HiRes'                       => 0,
         'Try::Tiny'                         => '0.06',
         'URI::Template'                     => '0.15',
     },

diff --git a/Changes b/Changes
@@ -1,12 +1,17 @@
 Revision history for Perl extension PGXN::Manager
 
-0.22.1
+0.30.0
        - Upgraded jQuery to v3.6.0 and the jquery.validate plugin to 1.19.3.
        - Switched to serve jQuery and jquery.validate locally rather than
          from CDNs. This eliminates a vector for data collection by the
          CDNs.
        - Fixed an error when a the forgotten password form is submitted
          with an invalid label value.
+       - Replaced the Twitter posting hack with an extensible, event-oriented
+         system based on PostgreSQL LISTEN/NOTIFY queues, and a new client,
+         `pgxn_consumer` to process events and post them to Twitter and
+         Mastodon. Requires a configuration change to enable event handlers.
+       - Fixed character encoding in emails.
 
 0.22.1   2022-01-16T20:32:34Z
        - Re-implemented the basic auth middleware to prevent passwords

diff --git a/README.md b/README.md
@@ -279,7 +279,7 @@ system.
 Copyright and License
 ---------------------
 
-Copyright (c) 2010-2021 David E. Wheeler.
+Copyright (c) 2010-2023 David E. Wheeler.
 
 This module is free software; you can redistribute it and/or modify it under
 the [PostgreSQL License](https://www.opensource.org/licenses/postgresql).

diff --git a/bin/check_mirrors b/bin/check_mirrors
@@ -11,7 +11,7 @@ use File::Basename;
 use File::Spec;
 use Getopt::Long;
 
-BEGIN { our $VERSION = v0.22.1 }
+BEGIN { our $VERSION = v0.30.0 }
 
 Getopt::Long::Configure( qw(bundling) );
 
@@ -278,7 +278,7 @@ David E. Wheeler <[email protected]>
 
 =head1 Copyright and License
 
-Copyright (c) 2010-2021 David E. Wheeler.
+Copyright (c) 2010-2023 David E. Wheeler.
 
 This module is free software; you can redistribute it and/or modify it under
 the L<PostgreSQL License|https://www.opensource.org/licenses/postgresql>.

diff --git a/bin/gendoc b/bin/gendoc
@@ -227,7 +227,7 @@ David E. Wheeler <[email protected]>
 
 =head1 Copyright and License
 
-Copyright (c) 2010-2021 David E. Wheeler. Some Rights Reserved.
+Copyright (c) 2010-2023 David E. Wheeler. Some Rights Reserved.
 
 This module is free software; you can redistribute it and/or modify it under
 the same terms as Perl itself.

diff --git a/bin/get_twitter_token b/bin/get_twitter_token
@@ -114,7 +114,7 @@ David E. Wheeler <[email protected]>
 
 =head1 Copyright and License
 
-Copyright (c) 2010-2021 David E. Wheeler. Some Rights Reserved.
+Copyright (c) 2010-2023 David E. Wheeler. Some Rights Reserved.
 
 This module is free software; you can redistribute it and/or modify it under
 the same terms as Perl itself.

diff --git a/bin/pgxn_consumer b/bin/pgxn_consumer
@@ -0,0 +1,168 @@
+#!/usr/local/bin/perl -w
+
+use 5.10.0;
+use utf8;
+
+use lib 'lib';
+use PGXN::Manager::Consumer;
+exit PGXN::Manager::Consumer->go;
+
+=head1 Name
+
+pgxn_consumer - Consume and handle PGXN event notifications
+
+=head1 Usage
+
+  pgxn_consumer [OPTIONS]
+
+=head1 Description
+
+This program consumes and handles PGXN Manager event notifications published
+on the PostgreSQL LISTEN/NOTIFY queue.
+
+=head1 Options
+
+  -E --env      ENV      Specify the environment in which to run.
+  -p --pid      PIDFILE  PID file path; used with --daemonize.
+  -D --daemonize         Run in the background as a daemon.
+  -i --interval SECONDS  Interval to sleep between consuming events.
+  -V --verbose           Incremental verbosity to STDOUT.
+  -h --help              Print a usage statement and exit.
+  -m --man               Print the complete documentation and exit.
+  -v --version           Print the version number and exit.
+
+=head1 Configuration
+
+In the configuration file that corresponds to the environment (C<-env>
+add a C<consumers> key that contains configuration for each consumer,
+like so:
+
+  "consumers": [
+      {
+          "type": "mastodon",
+          "events": ["release"],
+          "server": "https://mstdn.example.org",
+          "token": "ABCDefgh123456789x0x0x0x0x0x0x0x0x0x0x0"
+      },
+      {
+          "type": "twitter",
+          "events": ["release"],
+          "consumer_key": "",
+          "consumer_secret": "",
+          "access_token": "",
+          "access_token_secret": ""
+      }
+  }
+
+Each consumer object must contain at least two keys: C<type>, identifying
+the type of consumer, and C<events>, an array of the event types for the
+consumer to process. The current list of events is:
+
+=over
+
+=item C<release>
+
+Sent when a a new release is uploaded.
+
+=item C<new_user>
+
+Sent when a new user has been approved by an admin.
+
+=item C<new_mirror>
+
+Sent when a new mirror has been added.
+
+=back
+
+Each object requires additional keys specific to the type of consumer.
+The current list of consumer types and their configurations is:
+
+=over
+
+=item C<mastodon>
+
+Posts messages to Mastodon. This object corresponds to the configuration for
+a registered application on your Mastodon server, which you'll find under
+Preferences -> Development, or the URI C</settings/applications>. The
+supported keys are:
+
+=over
+
+=item C<server>
+
+The base URL for the Mastodon server, e.g., C<https://botsin.space>. Required.
+
+=item C<token>
+
+The Mastodon API access token, something like
+C<ABCDefgh123456789x0x0x0x0x0x0x0x0x0x0x0>. Required.
+
+=item C<delay>
+
+The number of seconds to delay posting the status. Defaults to 0, and otherwise
+must be at least 300. Optional.
+
+=back
+
+=item C<twitter>
+
+Posts messages to Twitter. The object corresponds to the authentication
+tokens and secrets for an app registered on the
+L<Twitter Developer Portal|https://developer.twitter.com/>. The simplest
+way to create this object is to use C<get_twitter_token>. The supported
+keys are:
+
+=over
+
+=item C<consumer_key>
+
+Identifies the Twitter I<application> to connect to Twitter as. Required.
+
+=item C<consumer_secret>
+
+The randomly-generated secret string that authenticates the C<consumer_key>
+application. Required.
+
+=item C<access_token>
+
+An access token is a user-specific credential used to authenticate OAuth 1.0a
+API requests. It specify the Twitter I<account> the request is made on behalf
+of. Required.
+
+=item C<access_token_secret>
+
+The randomly-generated secret that authenticates the C<access_token> account.
+Required.
+
+=back
+
+=back
+
+=head1 Author
+
+David E. Wheeler <[email protected]>
+
+=head1 Copyright and License
+
+Copyright (c) 2011-2023 David E. Wheeler.
+
+This module is free software; you can redistribute it and/or modify it under
+the L<PostgreSQL License|https://www.opensource.org/licenses/postgresql>.
+
+Permission to use, copy, modify, and distribute this software and its
+documentation for any purpose, without fee, and without a written agreement is
+hereby granted, provided that the above copyright notice and this paragraph
+and the following two paragraphs appear in all copies.
+
+In no event shall David E. Wheeler be liable to any party for direct,
+indirect, special, incidental, or consequential damages, including lost
+profits, arising out of the use of this software and its documentation, even
+if David E. Wheeler has been advised of the possibility of such damage.
+
+David E. Wheeler specifically disclaims any warranties, including, but not
+limited to, the implied warranties of merchantability and fitness for a
+particular purpose. The software provided hereunder is on an "as is" basis,
+and David E. Wheeler has no obligations to provide maintenance, support,
+updates, enhancements, or modifications.
+
+=cut
diff --git a/bin/pgxn_maint b/bin/pgxn_maint
@@ -144,7 +144,7 @@ David E. Wheeler <[email protected]>
 
 =head1 Copyright and License
 
-Copyright (c) 2011-2021 David E. Wheeler.
+Copyright (c) 2011-2023 David E. Wheeler.
 
 This module is free software; you can redistribute it and/or modify it under
 the L<PostgreSQL License|https://www.opensource.org/licenses/postgresql>.

diff --git a/conf/dev.json b/conf/dev.json
@@ -28,10 +28,20 @@
         "spec":      "/meta/spec.{format}"
     },
     "release_permalink": "https://pgxn.org/dist/{dist}/",
-    "twitter": {
-        "consumer_key": "",
-        "consumer_secret": "",
-        "access_token": "",
-        "access_token_secret": ""
-    }
+    "consumers": [
+        {
+            "type": "mastodon",
+            "events": ["release"],
+            "server": "https://mstdn.example.org",
+            "token": "ABCDefgh123456789x0x0x0x0x0x0x0x0x0x0x0"
+        },
+        {
+            "type": "twitter",
+            "events": ["release"],
+            "consumer_key": "",
+            "consumer_secret": "",
+            "access_token": "",
+            "access_token_secret": ""
+        }
+    ]
 }
diff --git a/conf/local.json b/conf/local.json
@@ -27,10 +27,20 @@
         "spec":      "/meta/spec.{format}"
     },
     "release_permalink": "https://pgxn.org/dist/{dist}/",
-    "twitter": {
-        "consumer_key": "",
-        "consumer_secret": "",
-        "access_token": "",
-        "access_token_secret": ""
-    }
+    "consumers": [
+        {
+            "type": "mastodon",
+            "events": ["release"],
+            "server": "https://mstdn.example.org",
+            "token": "ABCDefgh123456789x0x0x0x0x0x0x0x0x0x0x0"
+        },
+        {
+            "type": "twitter",
+            "events": ["release"],
+            "consumer_key": "",
+            "consumer_secret": "",
+            "access_token": "",
+            "access_token_secret": ""
+        }
+    ]
 }
diff --git a/conf/proxied.json b/conf/proxied.json
@@ -29,10 +29,20 @@
         "spec":      "/meta/spec.{format}"
     },
     "release_permalink": "https://pgxn.org/dist/{dist}/",
-    "twitter": {
-        "consumer_key": "",
-        "consumer_secret": "",
-        "access_token": "",
-        "access_token_secret": ""
-    }
+    "consumers": [
+        {
+            "type": "mastodon",
+            "events": ["release"],
+            "server": "https://mstdn.example.org",
+            "token": "ABCDefgh123456789x0x0x0x0x0x0x0x0x0x0x0"
+        },
+        {
+            "type": "twitter",
+            "events": ["release"],
+            "consumer_key": "",
+            "consumer_secret": "",
+            "access_token": "",
+            "access_token_secret": ""
+        }
+    ]
 }