Common Sense Software Design
This diagram is a representation of a design approach I refer to as the Common Sense Software Design for RDBMS-backed type of software.
1) Service Class
Service class does stuff.
It is the complete feature implementation that fulfills functional requirements:
- Use parameters or requests to:
- Create database queries, either with SQL or ORM, and execute and retrieve the data.
- Call stored procedures or database functions, and retrieve the data.
- Process the database results and form results or responses.
Pseudo example:
class MyFeature(Database db /*, other dependencies*/)
+Common Sense Software Design Common Sense Software Design
This diagram is a representation of a design approach I refer to as the Common Sense Software Design for RDBMS-backed type of software.
1) Service Class
Service class does stuff.
It is the complete feature implementation that fulfills functional requirements:
- Use parameters or requests to:
- Create database queries, either with SQL or ORM, and execute and retrieve the data.
- Call stored procedures or database functions, and retrieve the data.
- Process the database results and form results or responses.
Pseudo example:
class MyFeature(Database db /*, other dependencies*/)
{
int ServiceMethod1(int p1, int p2)
{
diff --git a/blog/haters-build-software/index.html b/blog/haters-build-software/index.html
index e245c528..806948dc 100644
--- a/blog/haters-build-software/index.html
+++ b/blog/haters-build-software/index.html
@@ -1 +1 @@
-Haters Build Software Haters Build Software What is Micro ORM? What is Micro ORM?
In software development, we all know very well what an ORM is:
A tool or usually a program library that implements object-relational mapping (ORM, O/RM), or O/R mapping.
This mapping is a two-way or bi-directional mapping:
- From the database to the database client (your application).
DB -> APP
Or, to be more specific: from the query results - to your application memory structures, such as object instances and other structures.
- From the database client (your application) to the database.
APP -> DB
Or, to be more specific: from the query and command programming constructs in your application language (such as C# Linq command and queries) - to query that your database understands and can run such as SQL typically.
The best-known and most widely used ORM in the .NET ecosystem is, of course, the Microsoft Entity Framework, which is, of course, the full-blown ORM that implements wo-way (bi-directional) mappings.
On the other hand, the Micro ORM means it's just a lightweight mapper that maps only one way and one way only. From the query results - to your application memory structures, such as object instances and other structures.
Micro ORM is a term invented and popularized by Dapper, the best-known MicroORM built by Stackoverflow. In practice, that means that when using Micro ORM - all SQL has to be in a raw string command (a raw SQL). For example:
public class Film
+What is Micro ORM? What is Micro ORM?
In software development, we all know very well what an ORM is:
A tool or usually a program library that implements object-relational mapping (ORM, O/RM), or O/R mapping.
This mapping is a two-way or bi-directional mapping:
- From the database to the database client (your application).
DB -> APP
Or, to be more specific: from the query results - to your application memory structures, such as object instances and other structures.
- From the database client (your application) to the database.
APP -> DB
Or, to be more specific: from the query and command programming constructs in your application language (such as C# Linq command and queries) - to query that your database understands and can run such as SQL typically.
The best-known and most widely used ORM in the .NET ecosystem is, of course, the Microsoft Entity Framework, which is, of course, the full-blown ORM that implements wo-way (bi-directional) mappings.
On the other hand, the Micro ORM means it's just a lightweight mapper that maps only one way and one way only. From the query results - to your application memory structures, such as object instances and other structures.
Micro ORM is a term invented and popularized by Dapper, the best-known MicroORM built by Stackoverflow. In practice, that means that when using Micro ORM - all SQL has to be in a raw string command (a raw SQL). For example:
public class Film
{
public int FilmId { get; set; }
public string Title { get; set; }
diff --git a/blog/npgsqlrest-update1.6.2/index.html b/blog/npgsqlrest-update1.6.2/index.html
index 03f1705e..bbfadbd2 100644
--- a/blog/npgsqlrest-update1.6.2/index.html
+++ b/blog/npgsqlrest-update1.6.2/index.html
@@ -1,4 +1,4 @@
-NpgsqlRest Update 1.6.2 NpgsqlRest 1.6.2 Update
I'm glad to see that the NpgsqlRest Nuget library is getting matured and refined. Here are some highlights from the changelog since version 1.4.0:
Strict Function Support
Functions in a strict mode (labeled as STRICT or RETURNS NULL ON NULL INPUT) are the functions that will always return NULL if at least one of the parameters is NULL. In that case, the function isn't even executed.
NpgsqlRest will not even try to execute strict functions when at least one of the parameters is NULL. Instead, the call will return HTTP 204 NoContent and bypass talking to the database altogether.
Command Callback
You can add a lambda callback that will receive a created data reader for the PostgreSQL function and the current HTTP context. If you choose to modify the HTTP response, the default behavior will be skipped in favor of your customized response.
For example, if we have some function that returns data, we can add a handler to render CSV instead of the default JSON response:
app.UseNpgsqlRest(new(connectionString)
+NpgsqlRest Update 1.6.2 NpgsqlRest 1.6.2 Update
I'm glad to see that the NpgsqlRest Nuget library is getting matured and refined. Here are some highlights from the changelog since version 1.4.0:
Strict Function Support
Functions in a strict mode (labeled as STRICT or RETURNS NULL ON NULL INPUT) are the functions that will always return NULL if at least one of the parameters is NULL. In that case, the function isn't even executed.
NpgsqlRest will not even try to execute strict functions when at least one of the parameters is NULL. Instead, the call will return HTTP 204 NoContent and bypass talking to the database altogether.
Command Callback
You can add a lambda callback that will receive a created data reader for the PostgreSQL function and the current HTTP context. If you choose to modify the HTTP response, the default behavior will be skipped in favor of your customized response.
For example, if we have some function that returns data, we can add a handler to render CSV instead of the default JSON response:
app.UseNpgsqlRest(new(connectionString)
{
CommandCallbackAsync = async p =>
{
diff --git a/blog/npgsqlrest-v2/index.html b/blog/npgsqlrest-v2/index.html
index 080d6dd4..c0a459b8 100644
--- a/blog/npgsqlrest-v2/index.html
+++ b/blog/npgsqlrest-v2/index.html
@@ -1,4 +1,4 @@
-NpgsqlRest v2 Update 🐘🔥 NpgsqlRest Version 2 Update 🐘🔥
NpgsqlRest is .NET8 Nuget library for automatic REST API from PostgreSQL databases, implemented as .NET8 Middleware.- See the GitHub Readme file.
NpgsqlRest Version 2 is a major upgrade that introduces extendiblity with the concept of plugins..
There are two types of plugins:
- Routine Source Plugins
- Code Generation Plugins
Routine Source Plugins
Routine Source Plugins are plugins that define various sources - from which automatic REST API can be built.
For example, functions are stored procedures source is already built-in into the library and it has been since version 1.
With version 2, you can add the CRUD source plugin, published as a separate, plugin library NpgsqlRest.CrudSource. This plugin can generate automatic REST API for PostgreSQL tables and views, that can:
- Insert into tables and insertable views with or without conflict resolutions and with or without returning the inserted record (Create).
- Read data from tables and views using select operation and filtering by provided parameters (Read).
- Update data in tables and updateable views with or without returning updated data (Update).
- Delete data in tables and deletable views with or without returning deleted data (Delete).
See NpgsqlRest.CrudSource for more information.
Code Generation Plugins
Code Generation Plugins are plugins that are executed after the REST API has been defined and generated from the source. As such, they are used mostly for automatic generations of, well, more code.
Code Generation Plugins automatically generate code.
From the first version - there was always a feature that, based on generated REST APIs - could generate the HTTP Code File.
Those HTTP Code files are the REST Client files, with syntax based on the Visual Studio Code REST Client extension. These files can be used with Visual Studio Code or with the latest Visual Studio to test and run easily REST endpoints.
With version 2, the code generation interface is standardized and the HTTP Code Files functionality is moved to a separate plugin: NpgsqlRest.HttpFiles
Now, there are no more default code generators in the core library - they are all added through plugins as additional libraries.
Besides HTTP Code Files - there is also a Typescript Client Code Generator Plugin: NpgsqlRest.TsClient.
This plugin was created and developed to avoid writing tedious and boring Typescript code. Since REST API endpoints are now automatically generated - there is no reason not to generate API calls for the frontend in Typescript.
Short example of the part of the generated Typescript module:
interface IGetDuplicateEmailsResponse {
+NpgsqlRest v2 Update 🐘🔥 NpgsqlRest Version 2 Update 🐘🔥
NpgsqlRest is .NET8 Nuget library for automatic REST API from PostgreSQL databases, implemented as .NET8 Middleware.- See the GitHub Readme file.
NpgsqlRest Version 2 is a major upgrade that introduces extendiblity with the concept of plugins..
There are two types of plugins:
- Routine Source Plugins
- Code Generation Plugins
Routine Source Plugins
Routine Source Plugins are plugins that define various sources - from which automatic REST API can be built.
For example, functions are stored procedures source is already built-in into the library and it has been since version 1.
With version 2, you can add the CRUD source plugin, published as a separate, plugin library NpgsqlRest.CrudSource. This plugin can generate automatic REST API for PostgreSQL tables and views, that can:
- Insert into tables and insertable views with or without conflict resolutions and with or without returning the inserted record (Create).
- Read data from tables and views using select operation and filtering by provided parameters (Read).
- Update data in tables and updateable views with or without returning updated data (Update).
- Delete data in tables and deletable views with or without returning deleted data (Delete).
See NpgsqlRest.CrudSource for more information.
Code Generation Plugins
Code Generation Plugins are plugins that are executed after the REST API has been defined and generated from the source. As such, they are used mostly for automatic generations of, well, more code.
Code Generation Plugins automatically generate code.
From the first version - there was always a feature that, based on generated REST APIs - could generate the HTTP Code File.
Those HTTP Code files are the REST Client files, with syntax based on the Visual Studio Code REST Client extension. These files can be used with Visual Studio Code or with the latest Visual Studio to test and run easily REST endpoints.
With version 2, the code generation interface is standardized and the HTTP Code Files functionality is moved to a separate plugin: NpgsqlRest.HttpFiles
Now, there are no more default code generators in the core library - they are all added through plugins as additional libraries.
Besides HTTP Code Files - there is also a Typescript Client Code Generator Plugin: NpgsqlRest.TsClient.
This plugin was created and developed to avoid writing tedious and boring Typescript code. Since REST API endpoints are now automatically generated - there is no reason not to generate API calls for the frontend in Typescript.
Short example of the part of the generated Typescript module:
interface IGetDuplicateEmailsResponse {
email: string | null;
count: number | null;
}
diff --git a/blog/npgsqlrest/index.html b/blog/npgsqlrest/index.html
index 7fa88c8c..2a8d754f 100644
--- a/blog/npgsqlrest/index.html
+++ b/blog/npgsqlrest/index.html
@@ -1,4 +1,4 @@
-NpgsqlRest NuGet Library NpgsqlRest NuGet Library
NpgsqlRest is a .NET8 NuGet library that can build a RESTful API from the existing PostgreSQL routines (functions or procedures) with a single extension on a web application builder.
You can think of it as PostgREST, but only for routines (functions or procedures), since PostgREST is doing a lot more. It is implemented as .NET8 middleware so you can create custom builds, or integrate with your .NET8 projects, and it is highly customizable, with some unique features.
The library is AOT-ready, which means it can utilize .NET AOT feature to create self-contained, native binary executables, but more on that later.
Motivation
This is built to be a crucial part of the future low-code data platform based on PostgreSQL. And, as such it was built to satisfy very specific requirements (as opposed to catering to a wider audience). For example, it has two dependencies 1) the latest Microsoft.NET.Sdk.Web that targets net8.0 framework. 2) the latest Npgsql ADO.NET Data Provider for PostgreSQL (current version 8.0.1). Those dependencies will be updated with each new release. If you need to work with older versions, you need to either 1) update them in your project - or - 2) clone the NpgsqlRest repository and make a custom build with downgraded dependencies.
Furthermore, future developments such as new features highly depend on the main project that will be using this library, but with the right sponsorship, anything is possible.
Features and Usage
Most basic usage in a standard .NET8 web application looks like this:
using NpgsqlRest;
+NpgsqlRest NuGet Library NpgsqlRest NuGet Library
NpgsqlRest is a .NET8 NuGet library that can build a RESTful API from the existing PostgreSQL routines (functions or procedures) with a single extension on a web application builder.
You can think of it as PostgREST, but only for routines (functions or procedures), since PostgREST is doing a lot more. It is implemented as .NET8 middleware so you can create custom builds, or integrate with your .NET8 projects, and it is highly customizable, with some unique features.
The library is AOT-ready, which means it can utilize .NET AOT feature to create self-contained, native binary executables, but more on that later.
Motivation
This is built to be a crucial part of the future low-code data platform based on PostgreSQL. And, as such it was built to satisfy very specific requirements (as opposed to catering to a wider audience). For example, it has two dependencies 1) the latest Microsoft.NET.Sdk.Web that targets net8.0 framework. 2) the latest Npgsql ADO.NET Data Provider for PostgreSQL (current version 8.0.1). Those dependencies will be updated with each new release. If you need to work with older versions, you need to either 1) update them in your project - or - 2) clone the NpgsqlRest repository and make a custom build with downgraded dependencies.
Furthermore, future developments such as new features highly depend on the main project that will be using this library, but with the right sponsorship, anything is possible.
Features and Usage
Most basic usage in a standard .NET8 web application looks like this:
using NpgsqlRest;
var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();
diff --git a/blog/pgroutiner/code-gen/index.html b/blog/pgroutiner/code-gen/index.html
index 48edf264..52f0c667 100644
--- a/blog/pgroutiner/code-gen/index.html
+++ b/blog/pgroutiner/code-gen/index.html
@@ -1,4 +1,4 @@
-PgRoutiner Code-Gen PgRoutiner Code-Gen
Data-Access Extensions
pgroutiner -r | --r | -rout | --rout | -routines | --routines
pgroutiner -r | --r | -rout | --rout | -routines | --routines
// pgroutiner auto-generated code
+PgRoutiner Code-Gen PgRoutiner Code-Gen
Data-Access Extensions
pgroutiner -r | --r | -rout | --rout | -routines | --routines
pgroutiner -r | --r | -rout | --rout | -routines | --routines
// pgroutiner auto-generated code
#pragma warning disable CS8632
#pragma warning disable CS8618
using System.Threading.Tasks;
diff --git a/blog/pgroutiner/concept/index.html b/blog/pgroutiner/concept/index.html
index 651e4047..475a935a 100644
--- a/blog/pgroutiner/concept/index.html
+++ b/blog/pgroutiner/concept/index.html
@@ -1,4 +1,4 @@
-PgRoutiner Concept PgRoutiner High-Level Concept
PgRoutiner is a command-line set of tools that work seamlessly with your NET project configuration in JSON files.
.NET projects usually have some configuration files, for example, appsettings.json or appsettings.Development.json or both. And usually, those files will contain a workable connection string. For example, they might look something like this:
{
+PgRoutiner Concept PgRoutiner High-Level Concept
PgRoutiner is a command-line set of tools that work seamlessly with your NET project configuration in JSON files.
.NET projects usually have some configuration files, for example, appsettings.json or appsettings.Development.json or both. And usually, those files will contain a workable connection string. For example, they might look something like this:
{
"ConnectionStrings": {
"MyConnection": "Server=localhost;Db=dvdrental;Port=5432;User Id=postgres;Password=postgres;"
},
diff --git a/blog/pgroutiner/database/index.html b/blog/pgroutiner/database/index.html
index 56206554..f443fad8 100644
--- a/blog/pgroutiner/database/index.html
+++ b/blog/pgroutiner/database/index.html
@@ -1,4 +1,4 @@
-PgRoutiner Database Management PgRoutiner Database Management
List Database Objects
pgroutiner -l | -ls | --ls | --list [ term ]
pgroutiner -l | -ls | --ls | --list [ term ]
Search Database Objects
pgroutiner -s | --s | --search [ term ]
pgroutiner -s | --s | --search [ term ]
View Database Object Definitions
pgroutiner -def | -ddl | --ddl | -definition | --definition [ name|names ]
pgroutiner -def | -ddl | --ddl | -definition | --definition [ name|names ]
PSQL Tool
pgroutiner -sql | -psql | --sql | --psql
pgroutiner -sql | -psql | --sql | --psql
Execution
pgroutiner -x | --x | -exec | --exec | -execute | --execute | --execute [ command|commands|script|dir ]
pgroutiner -x | --x | -exec | --exec | -execute | --execute | --execute [ command|commands|script|dir ]
Insert Scripts
pgroutiner -i | -ins | -inserts | --inserts [ table|tables|view|views|query|queries ]
pgroutiner -i | -ins | -inserts | --inserts [ table|tables|view|views|query|queries ]
Backup and Restore Helpers
pgroutiner --backup [ filename|name_pattern ]
pgroutiner --backup [ filename|name_pattern ]
pgroutiner --restore [ filename|name_pattern ]
pgroutiner --restore [ filename|name_pattern ]
Schema Files and Data Files
pgroutiner -sd | --schema-dump
+PgRoutiner Database Management PgRoutiner Database Management
List Database Objects
pgroutiner -l | -ls | --ls | --list [ term ]
pgroutiner -l | -ls | --ls | --list [ term ]
Search Database Objects
pgroutiner -s | --s | --search [ term ]
pgroutiner -s | --s | --search [ term ]
View Database Object Definitions
pgroutiner -def | -ddl | --ddl | -definition | --definition [ name|names ]
pgroutiner -def | -ddl | --ddl | -definition | --definition [ name|names ]
PSQL Tool
pgroutiner -sql | -psql | --sql | --psql
pgroutiner -sql | -psql | --sql | --psql
Execution
pgroutiner -x | --x | -exec | --exec | -execute | --execute | --execute [ command|commands|script|dir ]
pgroutiner -x | --x | -exec | --exec | -execute | --execute | --execute [ command|commands|script|dir ]
Insert Scripts
pgroutiner -i | -ins | -inserts | --inserts [ table|tables|view|views|query|queries ]
pgroutiner -i | -ins | -inserts | --inserts [ table|tables|view|views|query|queries ]
Backup and Restore Helpers
pgroutiner --backup [ filename|name_pattern ]
pgroutiner --backup [ filename|name_pattern ]
pgroutiner --restore [ filename|name_pattern ]
pgroutiner --restore [ filename|name_pattern ]
Schema Files and Data Files
pgroutiner -sd | --schema-dump
pgroutiner -sd | --schema-dump -sdf | --schema-dump-file [ schema filename ]
pgroutiner -sd | --schema-dump
pgroutiner -sd | --schema-dump -sdf | --schema-dump-file [ schema filename ]
pgroutiner -dd | --data-dump -dumplist [ object list ]
pgroutiner -dd | --data-dump -dumplist [ object list ] -ddf | --data-dump-file [ data dump filename ]
pgroutiner -dd | --data-dump -dumplist [ object list ]
diff --git a/blog/pgroutiner/documentation/index.html b/blog/pgroutiner/documentation/index.html
index e846d59e..87310452 100644
--- a/blog/pgroutiner/documentation/index.html
+++ b/blog/pgroutiner/documentation/index.html
@@ -1,3 +1,3 @@
-PgRoutiner Documentation PgRoutiner Documentation
pgroutiner -md --markdown
+PgRoutiner Documentation PgRoutiner Documentation
pgroutiner -md --markdown
pgroutiner -md --markdown -mdf --md-file [ md filename ]
pgroutiner -md --markdown
pgroutiner -md --markdown -mdf --md-file [ md filename ]
Example
PREVIOUSPgRoutiner Code-GenNEXTMORE ON PGROUTINERPgRoutiner Readme PgRoutiner Default Config PgRoutiner Blog PgRoutiner Concept PgRoutiner Database ManagementTo receive notifications about new posts and updates, consider subscribing to my LinkdIn page:
vb-software linkedin
You will receive notifications about new posts on your LinkedIn feed.CommentsPgRoutiner Blog PgRoutiner Blog
PgRoutiner is a set of command-line tools I created for fun and for my development needs while working on various .NET and PostgreSQL projects.
It integrates with .NET project configuration and allows you to do various stuff out of the box, no configuration needed. For example:
This is just one small example. PgRoutiner is incredibly feature-rich. As it happened, I was constantly adding new features as I was working on other projects. Initially, it was intended to be my personal code generator to relieve me of writing boring data-access code (calling a function and mapping the results, for example).
It was first published as the .NET Global Tool because that was, and still is - the most convenient and easiest way of managing different versions and updates, at least for me. I was really surprised to find out that, at the time of writing this blog, it had over 135K. That's, well, decent, but still not much. Note, while I was working on this blog series, it grew over 140K already.
However, I really didn't think anyone else would want to use it.
It was more of a fun experiment for me. It changed concepts a couple of times; it is loaded with features, and I thought it would be too complicated to be used by someone other than me. I guess I was wrong. I didn't even have time to properly finish the entire documentation, as you can see on the main readme file.
But I did create a nice presentation at one point. In any case, that won't stop me from blogging about it.
A few days ago, I did a migration of PgRoutiner to the new .NET8, and I wanted to try out how it works with the new NativeAOT Feature of the NET8. Native AOT is when we take our normal NET8 project, and we compile it to a native binary executable for the target OS.
AOT builds have very short, almost instantaneous startup times and are, in general, very optimized and fast.
It required some reconfiguration and some tweaking, and some parts had to be rewritten completely - but the final result blew me away: AOT build runs so fast that it is a completely different user experience. I can't go back to the normal .NET Tool after this.
If anyone wants to try it out, builds are on the GitHub release page.
So, what is it, and how it works?
See the PgRoutiner Concept next.
PREVIOUSMORE ON PGROUTINERPgRoutiner Readme PgRoutiner Default Config PgRoutiner Database Management PgRoutiner Code-Gen PgRoutiner DocumentationTo receive notifications about new posts and updates, consider subscribing to my LinkdIn page:
vb-software linkedin
You will receive notifications about new posts on your LinkedIn feed.CommentsPgRoutiner Blog PgRoutiner Blog
PgRoutiner is a set of command-line tools I created for fun and for my development needs while working on various .NET and PostgreSQL projects.
It integrates with .NET project configuration and allows you to do various stuff out of the box, no configuration needed. For example:
This is just one small example. PgRoutiner is incredibly feature-rich. As it happened, I was constantly adding new features as I was working on other projects. Initially, it was intended to be my personal code generator to relieve me of writing boring data-access code (calling a function and mapping the results, for example).
It was first published as the .NET Global Tool because that was, and still is - the most convenient and easiest way of managing different versions and updates, at least for me. I was really surprised to find out that, at the time of writing this blog, it had over 135K. That's, well, decent, but still not much. Note, while I was working on this blog series, it grew over 140K already.
However, I really didn't think anyone else would want to use it.
It was more of a fun experiment for me. It changed concepts a couple of times; it is loaded with features, and I thought it would be too complicated to be used by someone other than me. I guess I was wrong. I didn't even have time to properly finish the entire documentation, as you can see on the main readme file.
But I did create a nice presentation at one point. In any case, that won't stop me from blogging about it.
A few days ago, I did a migration of PgRoutiner to the new .NET8, and I wanted to try out how it works with the new NativeAOT Feature of the NET8. Native AOT is when we take our normal NET8 project, and we compile it to a native binary executable for the target OS.
AOT builds have very short, almost instantaneous startup times and are, in general, very optimized and fast.
It required some reconfiguration and some tweaking, and some parts had to be rewritten completely - but the final result blew me away: AOT build runs so fast that it is a completely different user experience. I can't go back to the normal .NET Tool after this.
If anyone wants to try it out, builds are on the GitHub release page.
So, what is it, and how it works?
See the PgRoutiner Concept next.
PREVIOUSMORE ON PGROUTINERPgRoutiner Readme PgRoutiner Default Config PgRoutiner Database Management PgRoutiner Code-Gen PgRoutiner DocumentationTo receive notifications about new posts and updates, consider subscribing to my LinkdIn page:
vb-software linkedin
You will receive notifications about new posts on your LinkedIn feed.CommentsPiko ORM Piko ORM
Piko ORM is so, so tiny. It's Piko. The world's tiniest, smallest ORM for .NET.
Works best with PostgreSQL.
using System.Data;
+Piko ORM Piko ORM
Piko ORM is so, so tiny. It's Piko. The world's tiniest, smallest ORM for .NET.
Works best with PostgreSQL.
using System.Data;
using NpgsqlTypes;
namespace DataAccess;
diff --git a/blog/postgresql-array/index.html b/blog/postgresql-array/index.html
index 70df43ec..b0eebe71 100644
--- a/blog/postgresql-array/index.html
+++ b/blog/postgresql-array/index.html
@@ -1,4 +1,4 @@
-PostgreSQL Arrays 🐘 PostgreSQL Arrays 🐘
Many-To-Many
Traditionally in relational databases, many-to-many data design is implemented using three tables.
Classic examples are users and roles: One user can be in many roles and one role can be on many users. For example, most developers are familiar with something like this (or, a variation of this approach):
create table users (
+PostgreSQL Arrays 🐘 PostgreSQL Arrays 🐘
Many-To-Many
Traditionally in relational databases, many-to-many data design is implemented using three tables.
Classic examples are users and roles: One user can be in many roles and one role can be on many users. For example, most developers are familiar with something like this (or, a variation of this approach):
create table users (
user_id int not null primary key generated always as identity,
name text not null unique
);
diff --git a/blog/postgresql-paging/index.html b/blog/postgresql-paging/index.html
index 7eb69aa5..fa755ca9 100644
--- a/blog/postgresql-paging/index.html
+++ b/blog/postgresql-paging/index.html
@@ -1,4 +1,4 @@
-PostgreSQL Paging Benchmarks PostgreSQL Paging Benchmarks
This is a performance benchmark project that tests different data paging approaches in PostgreSQL. You can find the source code in the PgPagingBenchmarks repository.
Setup
PostgreSQL 16 instance on my laptop:
select version();
+PostgreSQL Paging Benchmarks PostgreSQL Paging Benchmarks
This is a performance benchmark project that tests different data paging approaches in PostgreSQL. You can find the source code in the PgPagingBenchmarks repository.
Setup
PostgreSQL 16 instance on my laptop:
select version();
PostgreSQL 16.0 (Ubuntu 16.0-1.pgdg20.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0, 64-bit
select version();
PostgreSQL 16.0 (Ubuntu 16.0-1.pgdg20.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0, 64-bit
Initial schema (source):
begin;
diff --git a/blog/postgresql-record-type/index.html b/blog/postgresql-record-type/index.html
index dd08c3b2..825c7c14 100644
--- a/blog/postgresql-record-type/index.html
+++ b/blog/postgresql-record-type/index.html
@@ -1,4 +1,4 @@
-Execute Custom Function For Each Record Execute Custom Function For Each Record
Start with a query that returns a series of numbers: select q.i from generate_series(1, 10) q(i)
The goal is to execute a custom function for each record in this select (this is just an example; query could be anything).
That something will always return a previous value for the field i in the q(i) test table.
That is essentially a custom implementation of the LAG window function, but this serves as a proof of concept to demonstrate that we can do a custom function call for each value in any SELECT.
We will do this in a function that test_rec() that returns table (i, j), where i is a value from generate_series and j is the result from a custom function executed for each result that represents a previous (lag) value.
First, we need a wrap-up that generate-series in a subquery because generate_series returns a number, not a table record, and we need a table record:
select *
+Execute Custom Function For Each Record Execute Custom Function For Each Record
Start with a query that returns a series of numbers: select q.i from generate_series(1, 10) q(i)
The goal is to execute a custom function for each record in this select (this is just an example; query could be anything).
That something will always return a previous value for the field i in the q(i) test table.
That is essentially a custom implementation of the LAG window function, but this serves as a proof of concept to demonstrate that we can do a custom function call for each value in any SELECT.
We will do this in a function that test_rec() that returns table (i, j), where i is a value from generate_series and j is the result from a custom function executed for each result that represents a previous (lag) value.
First, we need a wrap-up that generate-series in a subquery because generate_series returns a number, not a table record, and we need a table record:
select *
from (
select q.i
from generate_series(1, 10) q(i)
diff --git a/blog/postgresql-temporal-tables/index.html b/blog/postgresql-temporal-tables/index.html
index d4d3fcae..bdbd4cd4 100644
--- a/blog/postgresql-temporal-tables/index.html
+++ b/blog/postgresql-temporal-tables/index.html
@@ -1,4 +1,4 @@
-Custom Temporal Tables in PostgreSQL Custom Temporal Tables in PostgreSQL
The following script will add support for temporal tables in PostgreSQL.
Description
It will add two new functions in the public schema:
1) function temporal_table_trigger
Trigger function that inserts temporal data.
This function will create a new schema (source schema plus __history) and a new table with temporal data in it.
The new table will have the exact same schema without keys and constraints with one extra field - data_valid_to timestamptz not null default now();
2) function init_temporal_tables
This function initiates the entire temporal tables system by creating trigger temporal_table_trigger on each tables It accepts an array of schema names we want to have temporal tables.
To call this function for the public schema:
select init_temporal_tables(array['public']);
select init_temporal_tables(array['public']);
Script
create or replace function temporal_table_trigger()
+Custom Temporal Tables in PostgreSQL Custom Temporal Tables in PostgreSQL
The following script will add support for temporal tables in PostgreSQL.
Description
It will add two new functions in the public schema:
1) function temporal_table_trigger
Trigger function that inserts temporal data.
This function will create a new schema (source schema plus __history) and a new table with temporal data in it.
The new table will have the exact same schema without keys and constraints with one extra field - data_valid_to timestamptz not null default now();
2) function init_temporal_tables
This function initiates the entire temporal tables system by creating trigger temporal_table_trigger on each tables It accepts an array of schema names we want to have temporal tables.
To call this function for the public schema:
select init_temporal_tables(array['public']);
select init_temporal_tables(array['public']);
Script
create or replace function temporal_table_trigger()
returns trigger
language plpgsql
as
diff --git a/blog/recursion-postgresql/part1-different-type-of-recursion/index.html b/blog/recursion-postgresql/part1-different-type-of-recursion/index.html
index e1227fc6..4f54c752 100644
--- a/blog/recursion-postgresql/part1-different-type-of-recursion/index.html
+++ b/blog/recursion-postgresql/part1-different-type-of-recursion/index.html
@@ -1,4 +1,4 @@
-Recursion with PostgreSQL Part 1 - A Different Type of Recursion Recursion with PostgreSQL Part 1 - A Different Type of Recursion
PostgreSQL offers a powerful procedural programming model out of the box (in addition to the standard SQL).
You can combine that with the standard SQL approach to overcome almost any issue and solve any programming task. Sometimes, the good old procedural approach may be a more straightforward way out of the complex data problems than standard SQL.
This article will try to demonstrate that approach to a complex problem example.
The Problem
Let's say we want to write a function that will return all related tables for a table name in a parameter.
We will start with a made-up schema:
create table division_status (
+Recursion with PostgreSQL Part 1 - A Different Type of Recursion Recursion with PostgreSQL Part 1 - A Different Type of Recursion
PostgreSQL offers a powerful procedural programming model out of the box (in addition to the standard SQL).
You can combine that with the standard SQL approach to overcome almost any issue and solve any programming task. Sometimes, the good old procedural approach may be a more straightforward way out of the complex data problems than standard SQL.
This article will try to demonstrate that approach to a complex problem example.
The Problem
Let's say we want to write a function that will return all related tables for a table name in a parameter.
We will start with a made-up schema:
create table division_status (
division_status_id int primary key,
name text
);
diff --git a/blog/recursion-postgresql/part2-performances/index.html b/blog/recursion-postgresql/part2-performances/index.html
index f139bca5..34ff04da 100644
--- a/blog/recursion-postgresql/part2-performances/index.html
+++ b/blog/recursion-postgresql/part2-performances/index.html
@@ -1,4 +1,4 @@
-Recursion with PostgreSQL Part 2 - Performances Recursion with PostgreSQL Part 2 - Performances
This is a follow-up article on my previous blog post A Different Type of SQL Recursion with PostgreSQL.
Previously, I explored two different approaches to recursion with PostgreSQL.
Now, it's time to do some testing.
Testing
Instead of a view on system tables that enables us to traverse table relationship tree, I've decided to create a simple test table:
create table big_tree_test (
+Recursion with PostgreSQL Part 2 - Performances Recursion with PostgreSQL Part 2 - Performances
This is a follow-up article on my previous blog post A Different Type of SQL Recursion with PostgreSQL.
Previously, I explored two different approaches to recursion with PostgreSQL.
Now, it's time to do some testing.
Testing
Instead of a view on system tables that enables us to traverse table relationship tree, I've decided to create a simple test table:
create table big_tree_test (
id text not null,
parent_id text not null
);
create table big_tree_test (
diff --git a/blog/recursion-postgresql/part3-cycle-detection/index.html b/blog/recursion-postgresql/part3-cycle-detection/index.html
index 802bfb42..f4bb7e56 100644
--- a/blog/recursion-postgresql/part3-cycle-detection/index.html
+++ b/blog/recursion-postgresql/part3-cycle-detection/index.html
@@ -1,4 +1,4 @@
-Recursion with PostgreSQL Part 3 - Cycle Detection Recursion with PostgreSQL Part 3 - Cycle Detection
Yesterday, I wrote another article for my Recursion with PostgreSQL series: Recursion with PostgreSQL Follow-Up 1 - Performances .
My conclusion was that the CTE recursion method was severely lacking a way out of the infinite loops - and then I proceeded to write a version that defined a maximum recursion depth as a parameter:
create or replace function select_nodes_recursive_cte(
+Recursion with PostgreSQL Part 3 - Cycle Detection Recursion with PostgreSQL Part 3 - Cycle Detection
Yesterday, I wrote another article for my Recursion with PostgreSQL series: Recursion with PostgreSQL Follow-Up 1 - Performances .
My conclusion was that the CTE recursion method was severely lacking a way out of the infinite loops - and then I proceeded to write a version that defined a maximum recursion depth as a parameter:
create or replace function select_nodes_recursive_cte(
_id text,
_max_recursion int = 100
)
diff --git a/blog/recursion-postgresql/part4-right-path/index.html b/blog/recursion-postgresql/part4-right-path/index.html
index 3e70346a..bd37c87f 100644
--- a/blog/recursion-postgresql/part4-right-path/index.html
+++ b/blog/recursion-postgresql/part4-right-path/index.html
@@ -1,4 +1,4 @@
-Recursion with PostgreSQL Part 4 - Finding the Right Path Recursion with PostgreSQL Part 4 - Finding the Right Path
The mystery has finally been resolved, and I have learned something today.
In my previous article, I managed to get some really crazy performances in tree processing by using PostgreSQL recursive procedural-style function.
While recursive CTE was getting some severe performance degradation - even on a smaller scale: It usually would eat all of my available space.
Thanks to @fatalmind (Markus Winand) comment, I got better clarification on what path and cycle really mean.
Example
So we have this test table:
create table big_tree_test (id text, parent_id text);
create table big_tree_test (id text, parent_id text);
id | parent_id |
+Recursion with PostgreSQL Part 4 - Finding the Right Path Recursion with PostgreSQL Part 4 - Finding the Right Path
The mystery has finally been resolved, and I have learned something today.
In my previous article, I managed to get some really crazy performances in tree processing by using PostgreSQL recursive procedural-style function.
While recursive CTE was getting some severe performance degradation - even on a smaller scale: It usually would eat all of my available space.
Thanks to @fatalmind (Markus Winand) comment, I got better clarification on what path and cycle really mean.
Example
So we have this test table:
create table big_tree_test (id text, parent_id text);
create table big_tree_test (id text, parent_id text);
id | parent_id |
---------+-----------+
node_1 | node_3 |
node_1 | node_8 |
diff --git a/blog/sql-vs-ddd/index.html b/blog/sql-vs-ddd/index.html
index 1f28bcd2..d70d3d77 100644
--- a/blog/sql-vs-ddd/index.html
+++ b/blog/sql-vs-ddd/index.html
@@ -1,4 +1,4 @@
-How DDD is screwing up your SQL How DDD is screwing up your SQL
This morning, I came across an interesting article about DDD and SQL on the internet named How SQL is screwing up your DDD. Normally I would post this article on my LinkedIn page with my own thoughts, but the subject matter is way too interesting, so, I'll just make my own blogpost.
The main premise of the article is an exercise, a code kata, in data modeling. Quote:
The goal is to model a collection of value object items, where the collection maintains a “default” item. It forces you to consider things like
How do you handle selecting which item is the default? How do you avoid the possibility of the collection ending up with no default item? How do you handle the case where the default item is removed from the collection? How do you avoid the possibility of the collection having no items at all?
A hypothetical example given is the Customer entity, which possesses a collection of Address.
It appears that the ideal domain model in a domain-driven world for this problem would be something like this:
public class Address
+How DDD is screwing up your SQL How DDD is screwing up your SQL
This morning, I came across an interesting article about DDD and SQL on the internet named How SQL is screwing up your DDD. Normally I would post this article on my LinkedIn page with my own thoughts, but the subject matter is way too interesting, so, I'll just make my own blogpost.
The main premise of the article is an exercise, a code kata, in data modeling. Quote:
The goal is to model a collection of value object items, where the collection maintains a “default” item. It forces you to consider things like
How do you handle selecting which item is the default? How do you avoid the possibility of the collection ending up with no default item? How do you handle the case where the default item is removed from the collection? How do you avoid the possibility of the collection having no items at all?
A hypothetical example given is the Customer entity, which possesses a collection of Address.
It appears that the ideal domain model in a domain-driven world for this problem would be something like this:
public class Address
{
public int Id { get; private set; }
public string Street { get; private set; }
diff --git a/blog/transaction-script/index.html b/blog/transaction-script/index.html
index c1cdd040..d4521784 100644
--- a/blog/transaction-script/index.html
+++ b/blog/transaction-script/index.html
@@ -1,4 +1,4 @@
-Transaction Script DDD vs SQL Which is the Better Approach?
And why?
- Readability?
- Maintainability?
- Testability?
- Debugging?
- Performances?
Transaction Script in Java
import java.sql.Connection;
+Transaction Script DDD vs SQL Which is the Better Approach?
And why?
- Readability?
- Maintainability?
- Testability?
- Debugging?
- Performances?
Transaction Script in Java
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
diff --git a/blog/what-have-stored-procedures-ever-done/index.html b/blog/what-have-stored-procedures-ever-done/index.html
index ce459b15..84ffb3e5 100644
--- a/blog/what-have-stored-procedures-ever-done/index.html
+++ b/blog/what-have-stored-procedures-ever-done/index.html
@@ -1 +1 @@
-What have the STORED PROCEDURES ever done for us? What have the STORED PROCEDURES ever done for us?
So, in all seriousness - what have the STORED PROCEDURES ever done for us?
Security?
You can very easily set up any database that has support for SP in a way that:
Your application is only aware of that one database account that can only execute the SP required by your application and nothing more.
That application account can't access data at all directly.
That application account is created and managed by another database account with a higher level of privileges and that is stored and protected much more securely.
That system is usually called the least-privilege principle and it can help you to:
Protect your data from theft way more securely because the account exposed to the application doesn't have direct access to data.
Guard your system against SQL injection attacks. The SQL injection is an impossibility (unless you decide to use dynamic SQL).
Prevents well-meaning junior developers from writing horribly inefficient queries. Well, that's not security per se, but it is a very useful side-effect.
Militaries around the world use this principle also called - "need-to-know-basis".
And the military takes security very, very seriously.
Do you?
Performances?
Well, this one is easy, everyone knows that at least - STORED PROCEDURES will give you performance gain.
Database code needs to be closer to the actual data and usually, there is much less network utilization between database and application.
Besides that, STORED PROCEDURE can be heavily optimized by the database engine and execution plans cached and ready.
Maintainability?
Imagine this scenario:
Users are seeing data on their screens that shouldn't be there. So, it's a bug, it needs a bit of maintenance obviously.
What do you do?
You can, of course, just create a Jira ticket ... or, or you can do something like this:
Connect to the database server with the query tool Execute the STORED PROCEDURE that serves that screen and examine the data. If data contains the bug, then you know that the problem is within the STORED PROCEDURE, if it doesn't then the bug is somewhere within the application layer. Now, within seconds you narrowed a search for a bug to at least half of the system.
If it is in your database layer, you can fix it immediately with one ALTER PROCEDURE (or REPLACE FUNCTION) call ... and it's fixed within minutes or even less - with zero downtime.
If it is not, then at least you know that it doesn't have anything to do with the database layer, and you can focus your investigation on your classes, designed domain models with patterns, abstract factories, reducers API controllers, injected services and whatnot you name it. Good luck with that, it will take some time for sure, and the application needs to be patched, redeployed, and all that jazz.
Does your user care about that?
Or, do they usually want to see bugs fixed right here and right now?
But that is not the only factor in this story that affects maintainability.
Your database is a system of course, and if you surround your database with STORED PROCEDURES, which is essentially an API layer - you got yourself a black box.
A black box is a system with known inputs and known outputs - without any knowledge of inner workings.
Your database may serve dozens and dozens of applications, services, micro-services, reporting systems, integrations, etc, etc, and, by having a black box concept you can change the schema and inner workings while respecting known outputs for know inputs with supreme confidence and efficiency.
How's that for maintainability?
Availability?
You want to have a system with maximum availability, don't you? I mean, downtimes are bad for business. Very bad.
I may have already mentioned above that you can patch a bug in STORED PROCEDURE without having to stop anything. Hence, availability.
No downtime.
Simply run "ALTER PROCEDURE" or "REPLACE FUNCTION" and there you go.
But it gets better. Imagine this. You have a huge table with a gazillion records. And for some reason, you have to change the data type of one field in that table. What do you do? Well, of course, you rush to create a migration that alters the table and alters the field type and then you deploy your update and ... nothing happens. For hours. The table is locked, and the entire system is unresponsive because everybody has to wait for your alter to finish and that may take hours. Many, many expensive hours of downtime.
But not if you use STORED PROCEDURES!
You could also write a script that will do this.
Create an auxiliary table from your main table without relations Alter procedures to insert data in that auxiliary table Alter the main table to change the data type in your field, and wait to finish After it finishes, insert data from the auxiliary table and revert procedure to original version No downtime.
Availability.
Apart from security, performance, maintainability, and availability, what STORED PROCEDURES have ever really done for us?
I really don't know.
To receive notifications about new posts and updates, consider subscribing to my LinkdIn page:
vb-software linkedin
You will receive notifications about new posts on your LinkedIn feed.CommentsWhat have the STORED PROCEDURES ever done for us? What have the STORED PROCEDURES ever done for us?
So, in all seriousness - what have the STORED PROCEDURES ever done for us?
Security?
You can very easily set up any database that has support for SP in a way that:
Your application is only aware of that one database account that can only execute the SP required by your application and nothing more.
That application account can't access data at all directly.
That application account is created and managed by another database account with a higher level of privileges and that is stored and protected much more securely.
That system is usually called the least-privilege principle and it can help you to:
Protect your data from theft way more securely because the account exposed to the application doesn't have direct access to data.
Guard your system against SQL injection attacks. The SQL injection is an impossibility (unless you decide to use dynamic SQL).
Prevents well-meaning junior developers from writing horribly inefficient queries. Well, that's not security per se, but it is a very useful side-effect.
Militaries around the world use this principle also called - "need-to-know-basis".
And the military takes security very, very seriously.
Do you?
Performances?
Well, this one is easy, everyone knows that at least - STORED PROCEDURES will give you performance gain.
Database code needs to be closer to the actual data and usually, there is much less network utilization between database and application.
Besides that, STORED PROCEDURE can be heavily optimized by the database engine and execution plans cached and ready.
Maintainability?
Imagine this scenario:
Users are seeing data on their screens that shouldn't be there. So, it's a bug, it needs a bit of maintenance obviously.
What do you do?
You can, of course, just create a Jira ticket ... or, or you can do something like this:
Connect to the database server with the query tool Execute the STORED PROCEDURE that serves that screen and examine the data. If data contains the bug, then you know that the problem is within the STORED PROCEDURE, if it doesn't then the bug is somewhere within the application layer. Now, within seconds you narrowed a search for a bug to at least half of the system.
If it is in your database layer, you can fix it immediately with one ALTER PROCEDURE (or REPLACE FUNCTION) call ... and it's fixed within minutes or even less - with zero downtime.
If it is not, then at least you know that it doesn't have anything to do with the database layer, and you can focus your investigation on your classes, designed domain models with patterns, abstract factories, reducers API controllers, injected services and whatnot you name it. Good luck with that, it will take some time for sure, and the application needs to be patched, redeployed, and all that jazz.
Does your user care about that?
Or, do they usually want to see bugs fixed right here and right now?
But that is not the only factor in this story that affects maintainability.
Your database is a system of course, and if you surround your database with STORED PROCEDURES, which is essentially an API layer - you got yourself a black box.
A black box is a system with known inputs and known outputs - without any knowledge of inner workings.
Your database may serve dozens and dozens of applications, services, micro-services, reporting systems, integrations, etc, etc, and, by having a black box concept you can change the schema and inner workings while respecting known outputs for know inputs with supreme confidence and efficiency.
How's that for maintainability?
Availability?
You want to have a system with maximum availability, don't you? I mean, downtimes are bad for business. Very bad.
I may have already mentioned above that you can patch a bug in STORED PROCEDURE without having to stop anything. Hence, availability.
No downtime.
Simply run "ALTER PROCEDURE" or "REPLACE FUNCTION" and there you go.
But it gets better. Imagine this. You have a huge table with a gazillion records. And for some reason, you have to change the data type of one field in that table. What do you do? Well, of course, you rush to create a migration that alters the table and alters the field type and then you deploy your update and ... nothing happens. For hours. The table is locked, and the entire system is unresponsive because everybody has to wait for your alter to finish and that may take hours. Many, many expensive hours of downtime.
But not if you use STORED PROCEDURES!
You could also write a script that will do this.
Create an auxiliary table from your main table without relations Alter procedures to insert data in that auxiliary table Alter the main table to change the data type in your field, and wait to finish After it finishes, insert data from the auxiliary table and revert procedure to original version No downtime.
Availability.
Apart from security, performance, maintainability, and availability, what STORED PROCEDURES have ever really done for us?
I really don't know.
To receive notifications about new posts and updates, consider subscribing to my LinkdIn page:
vb-software linkedin
You will receive notifications about new posts on your LinkedIn feed.CommentsWhich Way .NET Developer?