Pipelining mode

[nikita-volkov]

This is a draft of integration with the pipelining mode provided in libpq >14. So far I haven't done any testing at all, so it might just not work at all yet :)

The purpose of this PR is to provide a basis for all the other necessary stuff for the release to be added to. I've managed to make the API backwards compatible.

What's lacking of the necessary stuff:

• Tests
• Docs
• Solution for distributing the libpq-wrapper lib

What would be nice to have for the release:

• Benchmarks
• Profiling

Everything is up for discussion here including the current design and implementation. In particular I think there is some space for performance improvement in the way the Pipeline abstraction is implemented, but it might do okay for the initial release.

Any contribution is welcome. If you decide to jump in on any of the mentioned stuff please make PRs to this branch. Also prior to jumping on any work please notify in this thread to avoid stepping on each others toes.

FYI @GulinSS, @steve-chavez, @robx, @wolfgangwalther, @domenkozar, @avanov

[nikita-volkov]

Made a bunch of tests. It doesn't work yet. Has to do with the processing of results. I'll work on this further.

[nikita-volkov]

Okay. I've managed to resolve this. However the error processing is now a bit dirty during pipeline execution. The error model needs revision unfortunately, so the changes will likely be backward incompatible. Working on it.

[avanov]

@nikita-volkov thanks for looking into this! I personally anticipate it a lot, been hoping for it since your old experiments with the binary protocol, as I've seen a few cases in a commercial setting where Haskell data processors were losing perf to Python because the latter had support for pipelining, so this can level the playing field in future debates over picking a tech stack for a service :) In fact, I haven't heard of any other Haskell PG driver that would provide support for pipelining, so this could be the first one. 🚀

Regarding the error handling, FWIW, the Python one essentially has two different interfaces: the plain old regular interface for .execute() and the pipelining mode via .executemany(), and the error/exception recovery assumptions for these modes are different too. Have you considered keeping the pipelining interface as a separate entity with a minimal level of primitives sharing between the current hasql-transaction statements and the pipelining statements (let's say, sharing only at a level of Connection)?

[nikita-volkov]

Alright guys, It seems to be fully working now! It took some revision of the error model unfortunately. I think I'll polish it in the coming days.

I'll now extend the task list on this feature with the following requirement:

• Provide extensive instructions on the migration of the error model in the changelog

Another good news is that it's now covered with a bunch of tests. Following are the results. PRs with more tests are welcome!

Hasql.Pipeline
  Single-statement
    Unprepared
      Collects results and sends params [✔]
    Prepared
      Collects results and sends params [✔]
  Multi-statement
    On unprepared statements
      Collects results and sends params [✔]
    On prepared statements
      Collects results and sends params [✔]
    When a part in the middle fails
      With query error
        Captures the error [✔]
        Leaves the connection usable [✔]
      With decoding error
        Captures the error [✔]
        Leaves the connection usable [✔]

@avanov Unfortunately looking for ways to isolate it completely would have caused complications in both the API and the implementation.

[nikita-volkov]

Alright. Another milestone has been reached! I've bundled the necessary updates to libpq into Hasql, so that we neither get blocked by having them merged into postgresql-libpq or having somebody maintain a fork just because of that.

Updated the changelog too.

[nikita-volkov]

Docs are now there. Changelog is updated. I consider the absence of criticism during this PR a positive feedback :) Merging!

[nikita-volkov]

Released in 1.7