feat: full text search for pubs #919

tefkah · 2025-02-03T17:33:07Z

Issue(s) Resolved

High-level Explanation of PR

This PR adds, in order

A searchVector TSVECTOR column on the pubs database
A trigger which sets this column to a combination of the title and the pubvalues anytime the pubvalues are updated
A function which uses this to look for pubs using ts_query
An API endpoint which allows any logged in user to call this function
A client component which is callable with CMD+K which calls this function on search.

Rough explanation of postgres FTS

I'd recommend this excellent video of the great Ben Awad, will do a much better job of explaining how it works than I or the postgres docs can: https://www.youtube.com/watch?v=szfUbzsKvtE&t=362s

But here's a shot at it:

Postgres can create a vectorized representation of a bunch of stuff with it's to_tsvector function/ TSVECTOR column.
It will basically create small, weighted "stubs" of words, which are easy to compare. eg it will unconjugate verbs and unpluralize nouns, so running->run: and ducks->duck:, roughly. You can then assign weights to certain parts of those vectors, which helps in ranking.

Then postgres does some magic and you can search it with this syntax

SELECT * FROM 'pubs'
WHERE to_tsvector('english', title, 'A') @@ $1 
-- 'some query'

very weird syntax, but it works really well.

Doing to_tsvector every query is very slow though, so I created a column on the pubs table called searchVector, and added an index and a trigger that updates it. This makes like a 500ms -> 1ms diff in perf.

Then I made some search thingy using shadcn's Command thing, which uses it.

That's basically it! Much easier than I thought it would be.

Downsides

Absolutely no permission filtering!
Should definitely be done!
But even if we filter permissions, we won't be find related pubs the user has permissions to through the parent pub!

Maybe kind of slow in the long term, but the query can definitely be optimized a bunch.

No typed return responses (should do that)

Test Plan

Go to arcadia
CMD+K
Look up something latin-y, like vilis.
See a lot of search results!
Navigate to search result on keyboard
Enter
Be taken to pub!
Modify a pub, ideally it's title
Check you are able to find it with this new title

Screenshots (if applicable)

Screenshare.-.2025-02-03.7_00_20.PM.mp4

Screenshare.-.2025-02-03.7_05_33.PM.mp4

Notes

… data while fetching

like, really slightly

tefkah · 2025-02-03T18:20:38Z

core/app/components/SearchDialog.tsx

+	useEffect(() => {
+		const down = (e: KeyboardEvent) => {
+			if (e.key === "k" && (e.metaKey || e.ctrlKey)) {
+				e.preventDefault();
+				onOpenChange?.(true);
+				setOpen(true);
+			}
+		};
+
+		document.addEventListener("keydown", down);
+		return () => document.removeEventListener("keydown", down);
+	}, [onOpenChange]);


this may be done more elegantly, we should probably have some global thing for this, but for now this is fine i think

tefkah · 2025-02-03T18:20:55Z

core/app/components/SearchDialog.tsx

+						<CommandEmpty className="flex w-full flex-col items-center p-6">
+							{isFetching ? (
+								<Loader2 className="h-4 w-4 animate-spin" />
+							) : isError ? (
+								<AlertCircle className="h-4 w-4" />
+							) : (
+								"No results found"
+							)}
+						</CommandEmpty>


not very elegant, lots of states not well represented

tefkah · 2025-02-03T18:21:24Z

core/lib/pubs.ts

@@ -5,7 +5,7 @@ import type { GetPubsResult } from "./server";
 export type PubTitleProps = {
 	title?: string | null;
 	createdAt: Date;
-	values:
+	values?:


we don't need the values for this technically

tefkah · 2025-02-03T18:21:46Z

core/lib/server/pub.fts.db.test.ts

+				Title: "Machine Learning Fundamentals",
+				Description: "An introduction to basic ML concepts",
+				Content: "This article covers neural networks and deep learning basics.",
+				Tags: "AI, ML, Technology",
+			},
+		},
+		{
+			pubType: "Basic Pub",
+			values: {
+				Title: "Deep Learning Applications",
+				Description: "Advanced applications of deep learning",
+				Content: "Exploring real-world uses of neural networks in computer vision.",
+				Tags: "AI, Deep Learning, CV",
+			},
+		},
+		{
+			pubType: "Basic Pub",
+			values: {
+				Title: "Web Development Guide",
+				Description: "Complete guide to modern web development",
+				Content: "Learn about React, TypeScript, and Node.js fundamentals.",
+				Tags: "Web, JavaScript, Programming",
+			},
+		},
+		{
+			pubType: "Basic Pub",
+			values: {
+				Title: "Database Design Patterns",
+				Description: "Essential patterns for database architecture",
+				Content: "Understanding SQL and NoSQL database design principles.",
+				Tags: "Database, SQL, Architecture",


definitely wrote these myself

tefkah · 2025-02-03T18:22:13Z

core/prisma/migrations/20250130165541_add_ts_vector_to_pub_values/migration.sql

+ALTER TABLE "pubs" ADD COLUMN "searchVector" tsvector;
+
+-- CreateIndex
+CREATE INDEX "pubs_searchVector_idx" ON "pubs" USING GIN ("searchVector");


this adds the column and the idx (note: we should add more indices)

tefkah · 2025-02-03T18:23:17Z

core/prisma/migrations/20250130165541_add_ts_vector_to_pub_values/migration.sql

+    WITH updates AS (
+        SELECT 
+            affected."pubId",
+            CASE 
+                WHEN tmp."pubId" IS NULL THEN pubs."title"
+                WHEN TG_OP = 'DELETE' OR tmp."value" IS NULL THEN NULL
+                ELSE tmp."value"
+            END AS new_title
+        FROM (
+            SELECT DISTINCT "pubId"
+            FROM inserted_updated_deleted_rows
+        ) AS affected
+        LEFT JOIN tmp_affected_pubs tmp ON tmp."pubId" = affected."pubId"
+        JOIN pubs ON pubs.id = affected."pubId"
+    )


this is just slightly rewriting this as a CTE, st i can use that title in the search vector as well

platform/core/prisma/migrations/20241219131700_fix_title_update_trigger/migration.sql

Lines 46 to 67 in 78aab11

UPDATE

"pubs"

SET

"updatedAt" = CURRENT_TIMESTAMP,

"title" = CASE

-- this is what's changed from the earlier function

WHEN tmp."pubId" IS NULL THEN

pubs."title"

WHEN TG_OP = 'DELETE' OR tmp."value" IS NULL THEN

NULL

ELSE

tmp."value"

END

FROM ( SELECT DISTINCT

"pubId"

FROM

inserted_updated_deleted_rows

) AS affected

LEFT JOIN tmp_affected_pubs tmp ON tmp."pubId" = affected."pubId"

WHERE

"pubs"."id" = affected."pubId";

RETURN NULL;

tefkah · 2025-02-03T18:25:57Z

core/prisma/migrations/20250130165541_add_ts_vector_to_pub_values/migration.sql

+    UPDATE "pubs"
+    SET
+        "updatedAt" = CURRENT_TIMESTAMP,
+        "title" = updates.new_title,
+        -- we weight the searchVector based on the title and its values
+        "searchVector" = (
+            SELECT 
+                setweight(to_tsvector('english', COALESCE(updates.new_title, '')), 'A') ||
+                setweight(to_tsvector('english', COALESCE(
+                    (SELECT string_agg(CAST(value #>> '{}' AS TEXT), ' ')
+                    FROM pub_values
+                    WHERE "pubId" = updates."pubId"), 
+                '')), 'B')
+        )
+    FROM updates
+    WHERE "pubs"."id" = updates."pubId";


this, very naively, just sets the search vector as

(with A priority) title (with B priority) `${pub.Values.join(' ')}`

we could do this more sophisticatedly, ignore all the values we don't want to look up (maybe only add stringlike fields), join with values we do want to look up (add the user info instead of the memberId for instance), or give certain fields different priorities (rich text fields should probably have different priorities in the future)

but for now, i think this is fiiine

packages/contracts/src/resources/site.ts

packages/db/src/kanel/kanel-kysely-zod-compatibility-hook.cjs

tefkah · 2025-02-03T18:28:01Z

packages/ui/src/command.tsx

-			"flex h-full w-full flex-col overflow-hidden rounded-md bg-white text-gray-950",
+			"flex h-full w-full flex-col overflow-hidden bg-white text-gray-950",


interesting: for some reason this "rounded" caused the whole command list to be blurry

hngngn/shadcn-solid#100

tefkah · 2025-02-03T18:28:27Z

packages/ui/src/command.tsx

+				<DialogTitle className={cn(showTitle || "sr-only")}>{title}</DialogTitle>
+				<DialogDescription className={cn(showDescription || "sr-only")}>
+					{description}
+				</DialogDescription>


these are necessary for accessibilty, otherwise you get a bunch of errors in the console/in dev

3mcd

Feels great to use. Thanks for all the code comments and great job on those tests too!

tefkah added 9 commits January 30, 2025 19:07

feat: rough fts

2ad0e72

feat: add search to api

7a6aa9d

feat: search dialog

828b6bf

fix: allow optional pubValues for getPubTitle

f2cdf15

fix: only log out explain of search query when debugging

be544e7

fix: add better tests

b6651a4

fix: add title and description for command dialogue and keep previous…

839f331

… data while fetching

feat: add more sophisticated searching, slower though

6244104

fix: slightly more efficient auth check for api routes

81f523b

like, really slightly

tefkah requested review from kalilsn, allisonking and 3mcd February 3, 2025 18:19

tefkah marked this pull request as ready for review February 3, 2025 18:19

tefkah commented Feb 3, 2025

View reviewed changes

packages/contracts/src/resources/site.ts Outdated Show resolved Hide resolved

tefkah commented Feb 3, 2025

View reviewed changes

packages/db/src/kanel/kanel-kysely-zod-compatibility-hook.cjs Show resolved Hide resolved

tefkah commented Feb 3, 2025

View reviewed changes

packages/db/src/kanel/kanel-kysely-zod-compatibility-hook.cjs Outdated Show resolved Hide resolved

tefkah commented Feb 3, 2025

View reviewed changes

tefkah added 5 commits February 4, 2025 14:08

chore: clean up some console.logs

7342fca

fix: add types and stage info to return

1836c39

fix: make debug less likely

326e65c

Merge branch 'main' into tfk/fts

3957135

merge main

81987af

Merge branch 'main' into tfk/fts

730a536

3mcd approved these changes Feb 6, 2025

View reviewed changes

3mcd merged commit 1cbb099 into main Feb 6, 2025
6 checks passed

3mcd deleted the tfk/fts branch February 6, 2025 15:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: full text search for pubs #919

feat: full text search for pubs #919

tefkah commented Feb 3, 2025 •

edited

Loading

tefkah Feb 3, 2025

tefkah Feb 3, 2025

tefkah Feb 3, 2025

tefkah Feb 3, 2025

tefkah Feb 3, 2025

tefkah Feb 3, 2025

tefkah Feb 3, 2025

tefkah Feb 3, 2025

tefkah Feb 3, 2025

3mcd left a comment

	UPDATE
	"pubs"
	SET
	"updatedAt" = CURRENT_TIMESTAMP,
	"title" = CASE
	-- this is what's changed from the earlier function
	WHEN tmp."pubId" IS NULL THEN
	pubs."title"
	WHEN TG_OP = 'DELETE' OR tmp."value" IS NULL THEN
	NULL
	ELSE
	tmp."value"
	END
	FROM ( SELECT DISTINCT
	"pubId"
	FROM
	inserted_updated_deleted_rows
	) AS affected
	LEFT JOIN tmp_affected_pubs tmp ON tmp."pubId" = affected."pubId"
	WHERE
	"pubs"."id" = affected."pubId";
	RETURN NULL;

		"flex h-full w-full flex-col overflow-hidden rounded-md bg-white text-gray-950",
		"flex h-full w-full flex-col overflow-hidden bg-white text-gray-950",

feat: full text search for pubs #919

feat: full text search for pubs #919

Conversation

tefkah commented Feb 3, 2025 • edited Loading

Issue(s) Resolved

High-level Explanation of PR

Rough explanation of postgres FTS

Downsides

Test Plan

Screenshots (if applicable)

Notes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

3mcd left a comment

Choose a reason for hiding this comment

tefkah commented Feb 3, 2025 •

edited

Loading