diff --git a/README.md b/README.md index ea06a3e3..cd705a66 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ My experience with some of used technologies is limited. The scores might not re ## Background -I primarily used nodejs and express to serve data to my frontends in the past. Over the time I have seen alternative solutions my clients used for developing the api's for web applications. I see a number of interesting alternatives to nodejs/express in 2020. My goal is to test populair and performant approaches for building api's in the open source world, covering programming languages I already know (JavaScript, Python) and 2 new languages. I selected Go and Rust because these are modern languages and web oriented. I discarded Java, C++, PHP and many others based on my personal preference. During my investigation I came across discussions about the performance of various api libraries. That brought me to idea to create this repo with all api solutions I am interested in and to perform load tests on each of them. For load tests I used [autocannon](https://github.com/mcollina/autocannon). +I primarily used nodejs and express to serve data to my frontends in the past. Over the time I have seen alternative solutions my clients used for developing the api's for web applications. I see a number of interesting alternatives to nodejs/express in 2020. My goal is to test populair and performant approaches for building api's in the open source world, covering programming languages I already know (JavaScript, Python) and 2 new languages. I selected Go and Rust because these are modern languages and web oriented. During my investigation I came across discussions about the performance of various api libraries. That brought me to idea to create this repo with all api solutions I am interested in and to perform load tests on each of them. For load tests I used [autocannon](https://github.com/mcollina/autocannon). Api server is an important part of the backend solution. Another important part is the database. I use identical PostgreSQL docker container with all api's. Postgres is well supported in all technologies I want to test and quite popular and performant. @@ -18,19 +18,23 @@ Below it the overview of tested techologies and my personal opinion. The perform | Api | Langauge | Library | Performance | Ease | Size MB\*\* | My Rank | | ----------------- | -------------- | --------- | ----------- | ---- | ----------- | ------- | -| todo-actix-api | Rust | actix-web | excellent | hard | 10 - 80 | 1 | -| todo-dotnet-api | C# dotnet MS | Entity | fair | hard | 215 - 1000 | 5 | +| todo-actix-api | Rust | actix-web | excellent | hard | 10 - 80 | 3 | +| todo-bun-elysia | Bun | elysia | very good | easy | 100 - 150 | 2 | +| todo-bun-pgjs | Bun | elysia | excellent | easy | 100 - 150 | 2 | | todo-express-api | NodeJS | express | good | easy | 40 - 200 | 3 | | todo-fast-api | Python | fastapi | good | fair | 300 | 3 | -| todo-fastify-api | NodeJS | fastify | very good | easy | 40 - 200 | 1 | -| todo-fiber-api | Golang | fiber | good | fair | 16 | 3 | +| todo-fastify-api | NodeJS | fastify | very good | easy | 40 - 200 | 2 | +| todo-fiber1-api | Golang | fiber v1 | good | easy | 16 | 3 | +| todo-fiber2-pg | Golang | fiber v2 | very good | easy | 16 | 2 | +| todo-fiber2-pgx | Golang | fiber v2 | excellent | easy | 10 | 1 | | todo-flask-api | Python | flask | fair | easy | 70 | 4 | -| todo-hasura-api | Haskel/GraphQL | hasura | fair | easy | ?? | 4 | +| todo-h3-api | NodeJS | h3, pg.js | good | fair | 40 - 200 | 2 | +| todo-h3-pgjs | NodeJS | h3 | very good | fair | 40 - 200 | 2 | +| todo-hasura-api | Haskel/GraphQL | hasura | fair | fair | ?? | 4 | | todo-mux-api | Golang | net/http | good | hard | 14 | 4 | -| todo-h3-api | NodeJS | h3 | very good | fair | 40 - 200 | 3 | -| todo-oak-api | Deno | oak | good | fair | 131 | 2 | +| todo-oak-api | Deno | oak | good | fair | 131 | 3 | | todo-polka-api | NodeJS | polka | very good | fair | 40 | 2 | -| todo-supabase-api | Haskel | supabase | fair | easy | ?? | 3 | +| todo-postgrest-api| Haskel | postgrest | good | easy | 20 | 3 | \*\* Docker image size produced by Dockerfile used for the benchmark run. Minimal image size is achieved using alpine but it has impact on the maximum performance of node libraries (fastify, express, polka and h3). It seems to me that maximal performance with node libraries and the reasonable image size is achieved using node-debian-slim as base image. @@ -38,11 +42,11 @@ Below it the overview of tested techologies and my personal opinion. The perform - `Golang`: Default [net/http](https://golang.org/pkg/net/http/) library is used and [fiber](https://github.com/gofiber/fiber) which advertise itself to be very fast and uses kind-of-express-way approach (easy to switch from NodeJS/Express). - `Rust`: [Actix-web](https://github.com/actix/actix-web) is popular in Rust world and achieves the highest performance scores in the [benchmark](https://www.techempower.com/benchmarks/#section=data-r0&hw=ph&test=composite&a=2). In my load tests too it is the fastest api library. -- `NodeJS`: [Polka](https://github.com/lukeed/polka) seem to be advertised as the fastest NodeJS web server. [Express](https://expressjs.com/) is used as a benchmark to Polka and as most popular node api server. Later is [nanoexpress](https://github.com/nanoexpress/nanoexpress) added as new fast solution. Due to some problems I removed nanoexpress in october 2021. The node library [fastify](https://www.fastify.io/) is also added after becoming more popular. The last node library added is [h3](https://github.com/unjs/h3), which is very light solution used by Nuxt. The libraries polka, fastify and h3 achieved very good results! -- `Deno`: It is new technology recently moved to version 1. Most popular choice medio 2020 seem to be [Oak](https://github.com/oakserver/oak) http server. +- `NodeJS`: [Polka](https://github.com/lukeed/polka) seem to be advertised as the fastest NodeJS web server. [Express](https://expressjs.com/) is used as a benchmark to Polka and as most popular node api server. The node library [fastify](https://www.fastify.io/) is also added after becoming more popular. The last node library added is [h3](https://github.com/unjs/h3), which is very light solution used by Nuxt. The libraries polka, fastify and h3 achieved very good results! +- `Deno`: It is new technology recently moved to version 1. Most popular choice medio 2020 seem to be [Oak](https://github.com/oakserver/oak) http server. In 2023 I excluded it from benchmark test because updates are required and I have not found time yet to do that. - `Python`: [Flask](https://flask.palletsprojects.com/en/1.1.x/) is popular basic web server widely used. [FastApi](https://github.com/tiangolo/fastapi) is marked as the fastest python library for api's. My tests confirm that FastApi is significantly faster than flask. - `GraphQL`: is alternative approach to standard REST api architecture. All other api's use REST approach. [Hasura](https://hasura.io/docs/1.0/graphql/manual/index.html) api, which is Haskel/GraphQL/Postgres implementation, implements the GraphQL endpoint and offers basic CRUD operations out of the box. It was quite easy to implement basic CRUD operations with Hasura. The performance is lower, which I expected, and the amount of traffic is significantly higher, which was surpring to me. -- `dotnet core and C#`: I created dotnet core api using Udemy training. It uses modern async approach, the Entity framework and MSSQL instead of Postgres. The performance of this stack, in Docker continers, is bellow my expectations. Maybe the performance bottleneck is MSSQL or maybe running on Linux/Docker stack. The mainstram approach with C# is to use MSSQL and run the stack on the windows server. Anyway, I expected better results from compiled, strongly typed language like C# and dotnet core on Linux stack. +- `Bun`: I created two api using different postgress drivers. Bun seem to be perform better than node api's. The scores are high especially on Mac M2 machine. ## What these load test results mean actually (?) @@ -56,47 +60,24 @@ Load tests of each solution give the `combined performance result` which include ## Conclusion -I runned load tests on 4 machines (2 laptops and 2 desktop) for all api's. All machines use Linux OS (Ubuntu/Linux Mint). The results are saved in the separate branches with the name of the machine (eg. dell-xps-2018...). I noticed slight differences in the ranking between used hardware/machine. This is a bit surprising. It looks to me that different programming languages and libraries utilize specific hardware better. The biggest difference I noticed is performance difference between Intel machines and AMD Ryzen machine. Actix and Dotnet show better performance on the Intel processors. +I runned load tests on multiple machines (4 laptops and 2 desktops). The results are saved in the separate json files. There are significant differences in the ranking between used hardware/machines. This is a bit surprising. It seems that different programming languages and libraries utilize specific hardware better. The difference between Intel, AMD and Apple M2 processors is significant. Actix shows better performance on the Intel processors and node/bun on the Mac M2 processor. -In addition, my knowledge of specific library is limited and can influence the scores. As an example the performance of FastApi significantly improved (from 50k to 90k) after tweaking api for the number of workers used on a specific machine (dell-xps-2018). Similair fluctuations in actix-api were also caused by experimenting with the number of used workers. I also noticed that the different number of workers produces the highest score on different machines/hardware. Setting number of workers to higher number does not yield the better score all the time. +Note that my knowledge of specific libraries is limited and has influence on the scores. For example the performance of Fiber v2 api significantly improved after tweaking the api for the number of processes used. Significant fluctuations in the actix-api performance were also caused by experimenting with the number of used workers. I also noticed that different potgress libraries (drivers) have a significant influence on the overall api performance. See the differences in scores between todo-fiber2-pg and todo-fiber2-pgx, todo-bun-elysia and todo-bun-pgjs, todo-h3-api and todo-h3-pgjs. These api's are identical but use different postgress libraries. -The absolute scores/numbers per machine are different, but `rust api using actix-web is clearly one of the fastest and python/flask, hasura and dotnet are the slowest`. NodeJS (polka, express), Deno (oak) and Golang api's (fiber and standard http/mux) are in the middle of the chart. Surprisingly Python FastAPI seem to be performing very close to Golang and NodeJS/Deno api's after I optimized the number of workers. There might be some room to improve performance of Golang api's too but my knowledge of Golang at the moment is fairly limited. +The absolute scores/numbers per machine are different, but `golang-fiber2 and rust actix-web are clearly one of the fastest and python/flask and hasura are the slowest`. NodeJS (h3, polka, express), Deno (oak) and Golang api's (fiber v1 and the standard http/mux) are usually in the middle of the chart. In the past the Python FastAPI was performing very close to Golang and NodeJS/Deno api's after I optimized number of workers. Unfortunatelly I decided to exclude it in 2023 because the api was not able to run on Mac M2 chip without an upgrade and currently I have no time to upgrade it. -The scores from my dell-xps-2018 laptop are shown in the image below. An interactive version of this chart with more scores is available on `http://localhost:3000` (NextJS app) after runing `npm run dev` in the tests folder. Of course this means that you first need to clone this repo locally on your machine :-). +Based on the load tests and my experience trying new languages Golang and Rust I decided to invest more time in learning Golang rather than Rust. Rust is powerfull language but requires (lot) more time untill sufficient level is reached. -

-Benchmark results on Dell XPS i7-8550U (Linux Mint 19) - -

+New surprises in 2023 are bun-postgres.js and h3-postgres.js. They performs very well, especially on Mac M2 machines. Definetly word checking them if you preffer using node ecosystem. -Based on the load tests outcomes and my experience trying the new languages Golang and Rust I decided to invest more time in learning Rust rather than Golang. But I must admit that Golang seem to be easier to learn and faster in compiling than Rust. +## Run load tests -Another pleasent surprise is Fastify. It performs very well, on AMD Ryzen machine it maches the rust-actix api performance and on Intel seem to be the second best performer in most cases. It is becoming very populair in the Node community and with good reason. Definetly word checking it if you are using node ecosystem. - -

-Benchmark results on AMD Ryzen 9 3900X machine (Linux Mint 20) - -

- -## Development - -This repo requires `docker and docker-compose` to run todo api's. For running load test and viewing simple table results you need `nodejs and npm`. +This repo requires `docker and docker-compose` to run todo api's. For running load test and viewing all results you need `nodejs and npm`. Running the load tests for the first time will require more time than the subsequent runs because all docker images need to be downloaded to your local machine. The easiest way to run load tests on Linux/Mac is to use test-round.sh bash script. The script will create required containers in the background, run the test and then cleanup containers and the volumes used. You might need to make the script executable first. -```bash -# make executable (if needed) -sudo chmod +x test-round.sh -``` - -## How this repo works - -Each todo\* folder contains complete api solution. All api's are functionally identical. They perform simple CRUD operations on Postgres database (todo_db). Each api folder has readme file where you can read how to start the api and run load test. - -1. install npm dependencies for load tests and test report webpage. - ```bash # go to tests folder cd tests @@ -104,11 +85,8 @@ cd tests npm install # go back to root cd ../ -``` - -2. run tests using test-round.sh bash script (`Linux/MacOS only`): this script will run one round of 30 sec. load tests for all api's. You should run at least 3-5 rounds to have more reliable results for your machine. - -```bash +# make executable (if needed) +sudo chmod +x test-round.sh # run test-round shell script (linux/MacOS) ./test-round.sh ``` @@ -117,4 +95,19 @@ cd ../ All contributions are more than welcome as I am interested in bringing all of the used technologies to maximum performance and increase my knowledge. -If you want to contribute the scores of your machine please do so. I can create a new branch with your machine name. Based on my experience with running the api's on three different machines I expect that results on some other machines (and OS-es) could be quite different. +If you want to contribute the scores of your machine please do so. Based on my experience with running the api's on three different machines I expect that results on some other machines (and OS-es) could be quite different. + +### Api requirements + +To run benchamrk you should implement api [according to this readme.md file](./tests/autocannon/README.md). + +### Include your api to tests + +1. Your contribution should be created in a new todo-[language]-[framework] folder. In this folder you can place your source code however you like. Only requirement is that docker-compose.yml file should be in the root of your folder. We use docker compose to run your solution. +2. Add new load test script in the tests/autocannon folder and include it package.json. + +For more information you can look at following files: +- `loadtest.sh`: running load test of single api solution +- `docker-compose.yml`: env variables used to ensure similar setup +- `load_test_[solution].js`: autocannon load test script. For example have a look at './tests/autocannon/load_test_bun_pgjs.js' +- `./tests/package.json`: see scripts section for calling a load tests for a solition diff --git a/tests/README.md b/tests/README.md index 5806b4d9..aa6d482f 100644 --- a/tests/README.md +++ b/tests/README.md @@ -49,7 +49,7 @@ npm run dev ## Endpoints tested -In the 2021 update we want to test identical endpoints and achieve idential load as much as possible. There might be slight differences but the idea is to align (at least) all REST api to use identical tests. GraphQL is different concerning the endpoint approach but the comparable load and operations will be tested. +In the 2023 update we want to test identical endpoints and achieve idential load as much as possible. There might be slight differences but the idea is to align (at least) all REST api to use identical tests. GraphQL is different concerning the endpoint approach but the comparable load and operations will be tested. [Detailed description of api endpoints is here](./tests/autocannon/README.md) - `GET /` homepage is simple json return with message:"OK" or something like that - `POST /list` create new todo list