Merge pull request #30 from wilburx9/develop

Documentation and UI bug fixes

.github/workflows/backend.yaml

@@ -2,8 +2,10 @@ name: Deploy the AWS Lambdas
 
 on:
   push:
-    tags:
-      - 'v*'
+    branches:
+      - 'live'
+    paths:
+      - 'backend/**'
 
 jobs:
   deploy-backend:

.github/workflows/frontend.yaml

@@ -2,8 +2,10 @@ name: Deploy the Ghost theme
 
 on:
   push:
-    tags:
-      - 'v*'
+    branches:
+      - 'live'
+    paths:
+      - 'frontend/**'
 
 jobs:
   deploy-frontend:
@@ -25,17 +27,17 @@ jobs:
 
       - name: Package the theme into a zip file
         working-directory: frontend
-        run: gulp zip
+        run: yarn zip
 
       - name: Deploy and activate the theme
         working-directory: frontend
         env:
           GHOST_API_URL: ${{ secrets.GHOST_ADMIN_API_URL }}
           GHOST_API_KEY: ${{ secrets.GHOST_ADMIN_API_KEY }}
-        run: gulp deploy
+        run: yarn deploy
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@v2
+        uses: aws-actions/configure-aws-credentials@v4
         with:
           role-to-assume: ${{ secrets.AWS_ROLE }}
           aws-region: ${{ secrets.AWS_REGION }}

README.md

@@ -1 +1,63 @@
 # wilburx9.com
+The source code for wilburx9.com, which is built on [Ghost](https://ghost.org). 
+The code lives in two directories _frontend_ and _backend_; the former is a Ghost theme, and the latter contains Go services 
+that manage newsletters.
+
+## Frontend
+The custom template which was originally cloned from the [Ghost Starter theme ](https://github.com/TryGhost/Starter).
+It's written in handlebars, css and a mixture of JQuery and vanilla JS.
+
+### Routing
+Deviating from the usual Ghost themes, I repurposed "/" into a self-aggrandizing landing page. 
+So the blog lives on "/blog" and the listing pages of individual tags are on "/blog/tag_slug". 
+This was implemented using a [custom route](frontend/routes.yaml).
+Consequently, an empty page was created on Ghost dashboard with the slug "blog" and a custom title and description. 
+See the [Ghost doc](https://ghost.org/docs/themes/routing/#the-default-collection) on this.
+
+### External Articles
+Some of my articles live on Kodeco and Medium, and can't be imported to Ghost. 
+However, I added "external articles" that contain nothing but bookmark cards pointing to the original article. 
+These articles have the "#external" private tag. Special provision has been made such so that such articles are not indexed, and clicking the post-cards from any such article on any article listing section opens the original article.
+
+### Running Locally
+1. Clone this repo
+2. [Install Ghost](https://ghost.org/tutorials/local-ghost/).
+3. Create a symlink in Ghost's installations theme directory that points to this repo's frontend directory by running `ln -s  PROJECT_DIR/frontend GHOST_DIR/content/themes/wilburx9`.
+4. `cd PROJECT_DIR/frontend` and run `yarn dev` to build and watch for new changes.
+5. `cd GHOST_DIR` and run `ghost restart`.
+6. Go to installed themes in the design settings of Ghost dashboard and select "wilburx9".
+
+### Deploying
+Deployment can be done manually or using the [CD workflow](.github/workflows/frontend.yaml).
+* **Manually**
+  1. Clone the repo
+  2. `cd PROJECT_DIR/frontend`.
+  3. `yarn pretest` to build.
+  4. `yarn zip` to package the theme into a zip file.
+  5. Go to Theme settings and upload the generated zip file.
+* **CD Workflow**: The steps for building, deploying and applying the theme are packaged into a CD workflow which runs when there's a push event on the _live_ branch that changed the frontend directory. To take advantage of this:
+  1. Fork the repo
+  2. Create a [Ghost custom integration](https://ghost.org/integrations/custom-integrations/), add Admin API Key and Url to your [projects secrets](https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions#creating-secrets-for-a-repository) and add these as `GHOST_ADMIN_API_KEY` and `GHOST_ADMIN_API_URL` respectively.
+  3. Add `GEN_SOURCEMAPS` to [GitHub variables](https://docs.github.com/en/actions/learn-github-actions/variables) with `true` or `false` depending on if you want to deploy the theme along with the source maps.
+  4. After deployment, the workflow clears the cdn cache to ensure the new theme reflects immediately. And this works on the assumption that your website is deployed on a Lightsail instance behind a Lightsail Distribution. So create a [GitHub OIDC provider AWS on aws IAM](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services) and ensure [the policy has a _Resource_](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_job-functions_create-policies.html) that points to the ARN of the Lightsail instance's Distribution and has a `lightsail:ResetDistributionCache` in the `Actions` array. Then, add the ARN of the role and AWS region to GitHub secrets as `AWS_ROLE` and `AWS_REGION`.
+
+## Backend
+Ghost's newsletter implementation is not very customizable as I wanted readers to choose the kind of newsletter to want to receive. 
+Hence, the backend directory contains two Go Services deployed to AWS Lambda:
+ - **subscribe** service receives the email and newsletter preferences from the frontend, validates the captcha and forwards it to MailerLite.
+ - **broadcast** when a new article is published, Ghost hits this webhook to broadcast to appropriate subscribers.
+
+### Deploying
+Deployment is done by a [CD workflow](.github/workflows/backend.yaml) which is triggered when there's a new push event on the _live_ branch that changed the backend directory.
+* **Secrets**: AWS Systems Manager Parameter Store is used to store the secrets used by both services. These secrets are:
+  1. `WILBURX9_ALLOWED_ORIGINS`: The origins allowed by the Lambdas; should be the site's homepage. Without this, the subscription form will fail because of good old CORS error.
+  2. `WILBURX9_EMAIL_SENDER`: The email for the sender of the newsletter.
+  3. `WILBURX9_MAILER_LITE_TOKEN`: [API token for MailerLite](https://www.mailerlite.com/help/where-to-find-the-mailerlite-api-key-groupid-and-documentation).
+  4. `WILBURX9_TURNSTILE_HOSTNAME`: Your website domain configured on Cloudflare [Tunrnstile](https://developers.cloudflare.com/turnstile/) dashboard.
+  5. `WILBURX9_TURNSTILE_SECRET`: Turnstile site's Secret key.
+* Create Lambdas
+  1. Create two Lambda Functions on AWS using the Go 1.x runtime and x86_64 architecture. Ensure their roles has a statement that allows reading from `ssm:GetParameter`; this is to ensure the services can read the secrets created above.
+  2. Add the function names to [GitHub variables](https://docs.github.com/en/actions/learn-github-actions/variables) using  `LAMBDA_FUNCTION_SUBSCRIBE` and `LAMBDA_FUNCTION_BROADCAST`.
+* IAM Role
+  1. Add `lambda:UpdateFunctionCode` to the `Actions` array of the IAM role created when deploying the frontend. This is so the CLI pipeline can update Lambda function.
+  2. Add the ARNs of the Lambda functions to the  `Resource` array of same IAM role.

backend/broadcast.go

@@ -6,19 +6,20 @@ import (
 	"context"
 	_ "embed"
 	"encoding/json"
-	"errors"
 	"fmt"
 	"github.com/PuerkitoBio/goquery"
 	"github.com/aws/aws-lambda-go/events"
 	"github.com/aws/aws-lambda-go/lambda"
 	"github.com/go-playground/validator/v10"
 	"github.com/mailerlite/mailerlite-go"
+	"github.com/samber/lo"
 	"html"
 	"log"
 	"math"
 	"net/http"
 	"os"
 	"regexp"
+	"slices"
 	"strconv"
 	"strings"
 	"text/template"
@@ -136,22 +137,14 @@ func createCampaign(ctx context.Context, post Post, content string) (string, err
 	if err != nil {
 		return "", err
 	}
-	supportedSegments := []string{
-		fmt.Sprintf("%v: %v", Blog, Photography),
-		fmt.Sprintf("%v: %v", Blog, Programming),
-	}
 
-	// Get the first Segment that matches any of the supported segments
-	var segment string
-	for _, s := range allSegments.Data {
-		for _, tag := range supportedSegments {
-			if strings.EqualFold(s.Name, tag) {
-				segment = s.ID
-			}
-		}
-	}
-	if segment == "" {
-		return "", errors.New("won't send campaigns for non-(programming or photography) articles")
+	primarySegment := fmt.Sprintf("%v: %v", Blog, post.PrimaryTag.Slug)
+	segment, ok := lo.Find(allSegments.Data, func(seg mailerlite.Segment) bool {
+		return strings.EqualFold(seg.Name, primarySegment)
+	})
+
+	if !ok {
+		return "", fmt.Errorf("won't send campaigns for non-(software or photography) articles: %q", primarySegment)
 	}
 
 	sender := AppConfig.EmailSender
@@ -164,10 +157,10 @@ func createCampaign(ctx context.Context, post Post, content string) (string, err
 		},
 	}
 	campaign := &mailerlite.CreateCampaign{
-		Name:     fmt.Sprintf("New Publication: %v", post.Title),
+		Name:     post.Title,
 		Type:     mailerlite.CampaignTypeRegular,
 		Emails:   *emails,
-		Segments: []string{segment},
+		Segments: []string{segment.ID},
 	}
 	c, _, err := MailClient.Campaign.Create(ctx, campaign)
 	if err != nil {
@@ -206,22 +199,15 @@ func (l lambdaReqBody) toPost() Post {
 	featureImage := p.FeatureImage
 
 	// Check if this post is a reference to an external article and retrieve the feature image.
-	doc, err := goquery.NewDocumentFromReader(strings.NewReader(p.HTML))
-	if err == nil {
-		bookmark := doc.Find("figure.kg-bookmark-card")
-
-		// A post which is just a reference to an external article
-		// will contain nothing but the bookmark card and the reading time.
-		if bookmark.Length() > 0 && bookmark.Children().Length() != 2 {
-
-			// Only set the feature image if this post didn't have one
-			if featureImage == "" {
-				img := doc.Find("div.kg-bookmark-thumbnail img")
-				if img.Length() > 0 {
-					featureImage, _ = img.Attr("src")
-				}
+	if featureImage == "" && slices.ContainsFunc(p.Tags, func(item tag) bool {
+		return strings.EqualFold(item.Name, "#external")
+	}) {
+		doc, err := goquery.NewDocumentFromReader(strings.NewReader(p.HTML))
+		if err == nil {
+			img := doc.Find("div.kg-bookmark-thumbnail img")
+			if img.Length() > 0 {
+				featureImage, _ = img.Attr("src")
 			}
-
 		}
 	}
 
@@ -237,7 +223,8 @@ func (l lambdaReqBody) toPost() Post {
 		FeatureImageCaption: featureImageCaption,
 		Excerpt:             p.Excerpt,
 		URL:                 p.URL,
-		Tag:                 p.PrimaryTag.Slug,
+		PrimaryTag:          p.PrimaryTag,
+		Tags:                p.Tags,
 	}
 }
 
@@ -249,33 +236,35 @@ type Post struct {
 	FeatureImageCaption string
 	Excerpt             string
 	URL                 string
-	Tag                 string
+	PrimaryTag          tag
+	Tags                []tag
 }
 
 type lambdaReqBody struct {
 	Post struct {
 		Current struct {
 			Excerpt             string    `json:"excerpt" validate:"required"`
-			FeatureImage        string    `json:"feature_image" validate:"http_url"`
-			FeatureImageCaption string    `json:"feature_image_caption" validate:"required"`
+			FeatureImage        string    `json:"feature_image"`
+			FeatureImageCaption string    `json:"feature_image_caption"`
 			ID                  string    `json:"id" validate:"required"`
 			PublishedAt         time.Time `json:"published_at" validate:"required"`
-			ReadingTime         int64     `json:"reading_time" validate:"required"`
+			ReadingTime         int64     `json:"reading_time"` // Not required of short posts with 0 reading time
 			Status              string    `json:"status" validate:"required"`
 			Title               string    `json:"title" validate:"required"`
 			UpdatedAt           time.Time `json:"updated_at" validate:"required"`
 			URL                 string    `json:"url" validate:"http_url"`
 			Visibility          string    `json:"visibility" validate:"required"`
 			HTML                string    `json:"html" validate:"required"`
-
-			PrimaryAuthor struct {
+			PrimaryTag          tag       `json:"primary_tag" validate:"required"`
+			Tags                []tag     `json:"tags" validate:"required"`
+			PrimaryAuthor       struct {
 				Name string `json:"name" validate:"required"`
 			} `json:"primary_author" validate:"required"`
-
-			PrimaryTag struct {
-				Name string `json:"name" validate:"required"`
-				Slug string `json:"slug" validate:"required"`
-			} `json:"primary_tag" validate:"required"`
 		} `json:"current" validate:"required"`
 	} `json:"post" validate:"required"`
 }
+
+type tag struct {
+	Slug string `json:"slug" validate:"required"`
+	Name string `json:"name" validate:"required"`
+}

backend/common/config.go

@@ -38,7 +38,7 @@ func newConfig() (*Config, error) {
 		"turnstile_secret":   "",
 	}
 
-	// Read all the secrets into the map using the map keys
+	// Read all the secrets into the map
 	for k, v := range m {
 		key := strings.ToUpper(fmt.Sprintf("wilburx9_%v", k))
 		input := &ssm.GetParameterInput{

backend/common/utils.go

@@ -10,11 +10,9 @@ import (
 	"time"
 )
 
-const (
-	Photography = "photography"
-	Programming = "programming"
-	Blog        = "blog"
-)
+var Groups = []string{"photography", "software"}
+
+const Blog = "blog"
 
 func init() {
 	config, err := newConfig()

backend/go.mod

@@ -1,6 +1,6 @@
 module backend
 
-go 1.20
+go 1.21
 
 require (
 	github.com/PuerkitoBio/goquery v1.8.1
@@ -17,7 +17,9 @@ require (
 	github.com/google/go-querystring v1.1.0 // indirect
 	github.com/jmespath/go-jmespath v0.4.0 // indirect
 	github.com/leodido/go-urn v1.2.3 // indirect
+	github.com/samber/lo v1.39.0 // indirect
 	golang.org/x/crypto v0.10.0 // indirect
+	golang.org/x/exp v0.0.0-20220303212507-bbda1eaf7a17 // indirect
 	golang.org/x/net v0.11.0 // indirect
 	golang.org/x/sys v0.9.0 // indirect
 	golang.org/x/text v0.10.0 // indirect

backend/go.sum

@@ -31,6 +31,8 @@ github.com/mailerlite/mailerlite-go v1.0.2/go.mod h1:tJH9ttRYbP1kkb+c8WIPa9C1EvV
 github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/samber/lo v1.39.0 h1:4gTz1wUhNYLhFSKl6O+8peW0v2F4BCY034GRpU9WnuA=
+github.com/samber/lo v1.39.0/go.mod h1:+m/ZKRl6ClXCE2Lgf3MsQlWfh4bn1bz6CXEOxnEXnEA=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
 github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
@@ -43,6 +45,8 @@ golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACk
 golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
 golang.org/x/crypto v0.10.0 h1:LKqV2xt9+kDzSTfOhx4FrkEBcMrAgHSYgzywV9zcGmM=
 golang.org/x/crypto v0.10.0/go.mod h1:o4eNf7Ede1fv+hwOwZsTHl9EsPFO6q6ZvYR8vYfY45I=
+golang.org/x/exp v0.0.0-20220303212507-bbda1eaf7a17 h1:3MTrJm4PyNL9NBqvYDSj3DHl46qQakyfqfWo4jgfaEM=
+golang.org/x/exp v0.0.0-20220303212507-bbda1eaf7a17/go.mod h1:lgLbSvA5ygNOMpwM/9anMpWVlVJ7Z+cHWq/eFuinpGE=
 golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
 golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=

backend/subscribe.go

@@ -9,9 +9,11 @@ import (
 	"github.com/aws/aws-lambda-go/events"
 	"github.com/aws/aws-lambda-go/lambda"
 	"github.com/mailerlite/mailerlite-go"
+	"github.com/samber/lo"
 	"log"
 	"net/http"
 	"net/mail"
+	"slices"
 	"strings"
 )
 
@@ -169,26 +171,22 @@ func validateForm(body string) (requestData, string, error) {
 
 // cleanTags ensures the tags in the request are valid. Also, adds default tags
 func cleanTags(rawTags []string) []string {
-	var tapsMap = make(map[string]bool, 0) // Use a map to prevent duplicates
+	var tagsMap = make(map[string]bool, 0) // Use a map to prevent duplicates
 
 	for _, tag := range rawTags {
 		trimmed := strings.ToLower(strings.TrimSpace(tag))
 		// Only take the tag if it's valid
-		if trimmed == Photography || trimmed == Programming {
-			tapsMap[trimmed] = true
+		if slices.Contains(Groups, trimmed) {
+			tagsMap[trimmed] = true
 		}
 	}
 
 	var tags = []string{Blog} // Every subscriber belongs to the "blog" tag
-
-	// Convert the map to a list
-	for v := range tapsMap {
-		tags = append(tags, v)
-	}
+	tags = append(tags, lo.Keys(tagsMap)...)
 
 	// If tags wasn't sent in the request, add all supported tags
 	if len(tags) == 1 {
-		tags = append(tags, Photography, Programming)
+		tags = append(tags, Groups...)
 	}
 
 	return tags