Merge branch 'main' into computed-insert

PostgREST · Apr 3, 2024 · d7c3354 · d7c3354
2 parents 15568e4 + d7c64a9
commit d7c3354
Show file tree

Hide file tree

Showing 12 changed files with 126 additions and 123 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -27,6 +27,7 @@ This project adheres to [Semantic Versioning](http://semver.org/).
  - #3327, Fix slow responses on schema cache reloads - @steve-chavez
  - #3340, Log when the LISTEN channel gets a notification - @steve-chavez
  - #3345, Fix in-database configuration values not loading for `pgrst.server_trace_header` and `pgrst.server_cors_allowed_origins` - @laurenceisla
+ - #3361, Clarify PGRST204(column not found) error message - @steve-chavez
 
 ### Deprecated
 

diff --git a/docs/references/errors.rst b/docs/references/errors.rst
@@ -259,7 +259,7 @@ Related to the HTTP request elements.
 Group 2 - Schema Cache
 ~~~~~~~~~~~~~~~~~~~~~~
 
-Related to a :ref:`stale schema cache <stale_schema>`. Most of the time, these errors are solved by :ref:`reloading the schema cache <schema_reloading>`.
+Related to a :ref:`schema_cache`. Most of the time, these errors are solved by :ref:`schema_reloading`.
 
 +---------------+-------------+-------------------------------------------------------------+
 | Code          | HTTP status | Description                                                 |

diff --git a/docs/references/schema_cache.rst b/docs/references/schema_cache.rst
@@ -3,46 +3,29 @@
 Schema Cache
 ============
 
-Some PostgREST features need metadata from the database schema. Getting this metadata requires expensive queries. To avoid repeating this work, PostgREST uses a schema cache.
-
-+--------------------------------------------+-------------------------------------------------------------------------------+
-| Feature                                    | Required Metadata                                                             |
-+============================================+===============================================================================+
-| :ref:`resource_embedding`                  | Foreign key constraints                                                       |
-+--------------------------------------------+-------------------------------------------------------------------------------+
-| :ref:`Functions <functions>`               | Function signature (parameters, return type, volatility and                   |
-|                                            | `overloading <https://www.postgresql.org/docs/current/xfunc-overload.html>`_) |
-+--------------------------------------------+-------------------------------------------------------------------------------+
-| :ref:`Upserts <upsert>`                    | Primary keys                                                                  |
-+--------------------------------------------+-------------------------------------------------------------------------------+
-| :ref:`Insertions <insert>`                 | Primary keys (optional: only if the Location header is requested)             |
-+--------------------------------------------+-------------------------------------------------------------------------------+
-| :ref:`OPTIONS requests <options_requests>` | View INSTEAD OF TRIGGERS and primary keys                                     |
-+--------------------------------------------+-------------------------------------------------------------------------------+
-| :ref:`open-api`                            | Table columns, primary keys and foreign keys                                  |
-+                                            +-------------------------------------------------------------------------------+
-|                                            | View columns and INSTEAD OF TRIGGERS                                          |
-+                                            +-------------------------------------------------------------------------------+
-|                                            | Function signature                                                            |
-+--------------------------------------------+-------------------------------------------------------------------------------+
-
-.. _stale_schema:
-
-Stale Schema Cache
-------------------
-
-One operational problem that comes with a cache is that it can go stale. This can happen for PostgREST when you make changes to the metadata before mentioned. Requests that depend on the metadata will fail.
-
-You can solve this by reloading the cache manually or automatically.
+PostgREST requires metadata from the database schema to provide a REST API that abstracts SQL details. One example of this is the interface for :ref:`resource_embedding`.
 
-.. note::
-  If you are using :ref:`in_db_config`, a schema reload will always :ref:`reload the configuration<config_reloading>` as well.
+Getting this metadata requires expensive queries. To avoid repeating this work, PostgREST uses a schema cache.
 
 .. _schema_reloading:
 
 Schema Cache Reloading
 ----------------------
 
+To not let the schema cache go stale (happens when you make changes to the database), you need to reload it.
+
+You can do this with UNIX signals or with PostgreSQL notifications. It's also possible to do this automatically using `event triggers <https://www.postgresql.org/docs/current/event-trigger-definition.html>`_.
+
+.. note::
+
+  - If you are using the :ref:`in_db_config`, a schema cache reload will :ref:`reload the configuration<config_reloading>` as well.
+  - There’s no downtime when reloading the schema cache. The reloading will happen on a background thread while serving requests.
+
+.. _schema_reloading_signals:
+
+Schema Cache Reloading with Unix Signals
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
 To manually reload the cache without restarting the PostgREST server, send a SIGUSR1 signal to the server process.
 
 .. code:: bash
@@ -59,8 +42,6 @@ For docker you can do:
   # or in docker-compose
   docker-compose kill -s SIGUSR1 <service>
 
-There’s no downtime when reloading the schema cache. The reloading will happen on a background thread while serving requests.
-
 .. _schema_reloading_notify:
 
 Schema Cache Reloading with NOTIFY
@@ -81,7 +62,7 @@ The ``pgrst`` notification channel is enabled by default. For configuring the ch
 Automatic Schema Cache Reloading
 --------------------------------
 
-You can do automatic schema cache reloading in a pure SQL way and forget about stale schema cache errors. For this use an `event trigger <https://www.postgresql.org/docs/current/event-trigger-definition.html>`_ and ``NOTIFY``.
+You can do automatic reloading and forget there is a schema cache. For this use an `event trigger <https://www.postgresql.org/docs/current/event-trigger-definition.html>`_ and ``NOTIFY``.
 
 .. code-block:: postgres
 

diff --git a/nix/tools/docs.nix b/nix/tools/docs.nix
@@ -59,6 +59,24 @@ let
       '';
 
   render =
+    let
+      pdflatex = texlive.combine {
+        inherit (texlive)
+          amsmath
+          booktabs
+          cancel
+          gensymb
+          mathdots
+          multirow
+          pgf
+          pgf-blur
+          scheme-basic
+          siunitx
+          standalone
+          yhmath
+          ;
+      };
+    in
     checkedShellScript
       {
         name = "postgrest-docs-render";
@@ -67,7 +85,7 @@ let
         withTmpDir = true;
       }
       ''
-        ${texlive.combined.scheme-full}/bin/pdflatex -halt-on-error -output-directory="$tmpdir" db.tex
+        ${pdflatex}/bin/pdflatex -halt-on-error -output-directory="$tmpdir" db.tex
         ${imagemagick}/bin/convert -density 300 "$tmpdir/db.pdf" ../_static/db.png
       '';
 

diff --git a/src/PostgREST/ApiRequest.hs b/src/PostgREST/ApiRequest.hs
@@ -95,10 +95,10 @@ data DbAction
   = ActRelationRead {dbActQi :: QualifiedIdentifier, actHeadersOnly :: Bool}
   | ActRelationMut  {dbActQi :: QualifiedIdentifier, actMutation :: Mutation}
   | ActRoutine      {dbActQi :: QualifiedIdentifier, actInvMethod :: InvokeMethod}
+  | ActSchemaRead   Schema Bool
 
 data Action
   = ActDb           DbAction
-  | ActSchemaRead   Schema Bool
   | ActRelationInfo QualifiedIdentifier
   | ActRoutineInfo  QualifiedIdentifier
   | ActSchemaInfo
@@ -190,8 +190,8 @@ getAction resource schema method =
     (ResourceRelation rel, "DELETE")  -> Right . ActDb $ ActRelationMut  (qi rel) MutationDelete
     (ResourceRelation rel, "OPTIONS") -> Right $ ActRelationInfo (qi rel)
 
-    (ResourceSchema, "HEAD")          -> Right $ ActSchemaRead schema True
-    (ResourceSchema, "GET")           -> Right $ ActSchemaRead schema False
+    (ResourceSchema, "HEAD")          -> Right . ActDb $ ActSchemaRead schema True
+    (ResourceSchema, "GET")           -> Right . ActDb $ ActSchemaRead schema False
     (ResourceSchema, "OPTIONS")       -> Right ActSchemaInfo
 
     _                                 -> Left $ UnsupportedMethod method

diff --git a/src/PostgREST/App.hs b/src/PostgREST/App.hs
@@ -9,7 +9,6 @@ Some of its functionality includes:
 - Producing HTTP Headers according to RFCs.
 - Content Negotiation
 -}
-{-# LANGUAGE NamedFieldPuns  #-}
 {-# LANGUAGE RecordWildCards #-}
 module PostgREST.App
   ( postgrest
@@ -24,7 +23,6 @@ import Data.String              (IsString (..))
 import Network.Wai.Handler.Warp (defaultSettings, setHost, setPort,
                                  setServerName)
 
-import qualified Data.HashMap.Strict        as HM
 import qualified Data.Text.Encoding         as T
 import qualified Hasql.Transaction.Sessions as SQL
 import qualified Network.Wai                as Wai
@@ -54,7 +52,6 @@ import PostgREST.Query                (DbHandler)
 import PostgREST.Response.Performance (ServerTiming (..),
                                        serverTimingHeader)
 import PostgREST.SchemaCache          (SchemaCache (..))
-import PostgREST.SchemaCache.Routine  (Routine (..))
 import PostgREST.Version              (docsVersion, prettyVersion)
 
 import qualified Data.ByteString.Char8 as BS
@@ -172,14 +169,11 @@ handleRequest AuthResult{..} conf appState authenticated prepared pgVer apiReq@A
   case iAction of
     ActDb dbAct -> do
       (planTime', plan) <- withTiming $ liftEither $ Plan.actionPlan dbAct conf apiReq sCache
-      (txTime', resultSet) <- withTiming $ runQuery (planIsoLvl plan) (planFunSettings plan) (Plan.pTxMode plan) $ Query.actionQuery plan conf apiReq pgVer
-      (respTime', pgrst) <- withTiming $ liftEither $ Response.actionResponse plan (dbActQi dbAct) apiReq resultSet
-      return $ pgrstResponse (ServerTiming jwtTime parseTime planTime' txTime' respTime') pgrst
-
-    ActSchemaRead tSchema headersOnly -> do
-      (planTime', iPlan) <- withTiming $ liftEither $ Plan.inspectPlan apiReq headersOnly tSchema
-      (txTime', oaiResult) <- withTiming $ runQuery roleIsoLvl mempty (Plan.ipTxmode iPlan) $ Query.openApiQuery iPlan conf sCache pgVer
-      (respTime', pgrst) <- withTiming $ liftEither $ Response.openApiResponse iPlan (T.decodeUtf8 prettyVersion, docsVersion) oaiResult conf sCache iSchema iNegotiatedByProfile
+      (txTime', queryResult) <- withTiming $ runDbHandler appState conf (Plan.planIsoLvl conf authRole plan) (Plan.planTxMode plan) authenticated prepared observer $ do
+        Query.setPgLocals plan conf authClaims authRole apiReq
+        Query.runPreReq conf
+        Query.actionQuery plan conf apiReq pgVer sCache
+      (respTime', pgrst) <- withTiming $ liftEither $ Response.actionResponse queryResult (dbActQi dbAct) apiReq (T.decodeUtf8 prettyVersion, docsVersion) conf sCache iSchema iNegotiatedByProfile
       return $ pgrstResponse (ServerTiming jwtTime parseTime planTime' txTime' respTime') pgrst
 
     ActRelationInfo identifier -> do
@@ -196,20 +190,6 @@ handleRequest AuthResult{..} conf appState authenticated prepared pgVer apiReq@A
       return $ pgrstResponse (ServerTiming jwtTime parseTime Nothing Nothing respTime') pgrst
 
   where
-    roleSettings = fromMaybe mempty (HM.lookup authRole $ configRoleSettings conf)
-    roleIsoLvl = HM.findWithDefault SQL.ReadCommitted authRole $ configRoleIsoLvl conf
-    runQuery isoLvl funcSets mode query =
-      runDbHandler appState conf isoLvl mode authenticated prepared observer $ do
-        Query.setPgLocals conf authClaims authRole (HM.toList roleSettings) funcSets apiReq
-        Query.runPreReq conf
-        query
-
-    planIsoLvl (Plan.CallReadPlan{crProc}) = fromMaybe roleIsoLvl $ pdIsoLvl crProc
-    planIsoLvl _ = roleIsoLvl
-
-    planFunSettings (Plan.CallReadPlan{crProc}) = pdFuncSettings crProc
-    planFunSettings _ = mempty
-
     pgrstResponse :: ServerTiming -> Response.PgrstResponse -> Wai.Response
     pgrstResponse timing (Response.PgrstResponse st hdrs bod) = Wai.responseLBS st (hdrs ++ ([serverTimingHeader timing | configServerTimingEnabled conf])) bod
 

diff --git a/src/PostgREST/Error.hs b/src/PostgREST/Error.hs
@@ -249,7 +249,7 @@ instance JSON.ToJSON ApiRequestError where
     (Just "Try renaming the parameters or the function itself in the database so function overloading can be resolved")
 
   toJSON (ColumnNotFound relName colName) = toJsonPgrstError
-    SchemaCacheErrorCode04 ("Column '" <> colName <> "' of relation '" <> relName <> "' does not exist") Nothing Nothing
+    SchemaCacheErrorCode04 ("Could not find the '" <> colName <> "' column of '" <> relName <> "' in the schema cache") Nothing Nothing
 
 -- |
 -- If no relationship is found then:

diff --git a/src/PostgREST/Plan.hs b/src/PostgREST/Plan.hs
@@ -18,9 +18,12 @@ resource.
 module PostgREST.Plan
   ( actionPlan
   , ActionPlan(..)
+  , DbActionPlan(..)
   , InspectPlan(..)
   , inspectPlan
   , callReadPlan
+  , planTxMode
+  , planIsoLvl
   ) where
 
 import qualified Data.ByteString.Lazy          as LBS
@@ -90,7 +93,7 @@ import Protolude hiding (from)
 -- Setup for doctests
 -- >>> import Data.Ranged.Ranges (fullRange)
 
-data ActionPlan
+data DbActionPlan
   = WrappedReadPlan
   { wrReadPlan :: ReadPlanTree
   , pTxMode    :: SQL.Mode
@@ -123,31 +126,46 @@ data InspectPlan = InspectPlan {
   , ipSchema   :: Schema
   }
 
+data ActionPlan = Db DbActionPlan | MaybeDb InspectPlan
+
+planTxMode :: ActionPlan -> SQL.Mode
+planTxMode (Db x)      = pTxMode x
+planTxMode (MaybeDb x) = ipTxmode x
+
+planIsoLvl :: AppConfig -> ByteString -> ActionPlan -> SQL.IsolationLevel
+planIsoLvl AppConfig{configRoleIsoLvl} role actPlan = case actPlan of
+  Db CallReadPlan{crProc} -> fromMaybe roleIsoLvl $ pdIsoLvl crProc
+  _                       -> roleIsoLvl
+  where
+    roleIsoLvl = HM.findWithDefault SQL.ReadCommitted role configRoleIsoLvl
+
 actionPlan :: DbAction -> AppConfig -> ApiRequest -> SchemaCache -> Either Error ActionPlan
 actionPlan dbAct conf apiReq sCache = case dbAct of
   ActRelationRead identifier headersOnly ->
-    wrappedReadPlan identifier conf sCache apiReq headersOnly
+    Db <$> wrappedReadPlan identifier conf sCache apiReq headersOnly
   ActRelationMut identifier mut ->
-    mutateReadPlan mut apiReq identifier conf sCache
+    Db <$> mutateReadPlan mut apiReq identifier conf sCache
   ActRoutine identifier invMethod ->
-    callReadPlan identifier conf sCache apiReq invMethod
+    Db <$> callReadPlan identifier conf sCache apiReq invMethod
+  ActSchemaRead tSchema headersOnly ->
+    MaybeDb <$> inspectPlan apiReq headersOnly tSchema
 
-wrappedReadPlan :: QualifiedIdentifier -> AppConfig -> SchemaCache -> ApiRequest -> Bool -> Either Error ActionPlan
+wrappedReadPlan :: QualifiedIdentifier -> AppConfig -> SchemaCache -> ApiRequest -> Bool -> Either Error DbActionPlan
 wrappedReadPlan  identifier conf sCache apiRequest@ApiRequest{iPreferences=Preferences{..},..} headersOnly = do
   rPlan <- readPlan identifier conf sCache apiRequest
   (handler, mediaType)  <- mapLeft ApiRequestError $ negotiateContent conf apiRequest identifier iAcceptMediaType (dbMediaHandlers sCache) (hasDefaultSelect rPlan)
   if not (null invalidPrefs) && preferHandling == Just Strict then Left $ ApiRequestError $ InvalidPreferences invalidPrefs else Right ()
   return $ WrappedReadPlan rPlan SQL.Read handler mediaType headersOnly
 
-mutateReadPlan :: Mutation -> ApiRequest -> QualifiedIdentifier -> AppConfig -> SchemaCache -> Either Error ActionPlan
+mutateReadPlan :: Mutation -> ApiRequest -> QualifiedIdentifier -> AppConfig -> SchemaCache -> Either Error DbActionPlan
 mutateReadPlan  mutation apiRequest@ApiRequest{iPreferences=Preferences{..},..} identifier conf sCache = do
   rPlan <- readPlan identifier conf sCache apiRequest
   mPlan <- mutatePlan mutation identifier apiRequest sCache rPlan
   if not (null invalidPrefs) && preferHandling == Just Strict then Left $ ApiRequestError $ InvalidPreferences invalidPrefs else Right ()
   (handler, mediaType)  <- mapLeft ApiRequestError $ negotiateContent conf apiRequest identifier iAcceptMediaType (dbMediaHandlers sCache) (hasDefaultSelect rPlan)
   return $ MutateReadPlan rPlan mPlan SQL.Write handler mediaType mutation
 
-callReadPlan :: QualifiedIdentifier -> AppConfig -> SchemaCache -> ApiRequest -> InvokeMethod -> Either Error ActionPlan
+callReadPlan :: QualifiedIdentifier -> AppConfig -> SchemaCache -> ApiRequest -> InvokeMethod -> Either Error DbActionPlan
 callReadPlan identifier conf sCache apiRequest@ApiRequest{iPreferences=Preferences{..},..} invMethod = do
   let paramKeys = case invMethod of
         InvRead _ -> S.fromList $ fst <$> qsParams'