Redis performance is poor in Docker network compared to unix sockets #30
Comments
|
If I dont use any pipelining and just 1 client, heres the results. Interesting that docker is only slightly slower than host tcp here. Both much slower than unix sockets
|
|
Here's a real-world test - using it with a Wordpress site.
This is a very bare bones site compared to what I normally run, and the average time is usually something like 5-10ms. So this is more than 10x slower. So there's surely something else going on than just Unix sockets and docker networking. (though, I see now that it's all more than 10x slower than the initial benchmark shared above, so could be an explanation). I have to go right now, but I'll post the same metrics if/when I can get WP to connect to Redis on the host machine, over tcp and unix. I don't see any reason why those WP metrics couldn't/shouldn't be the same sub-10ms |
|
We can also try to get the Relay redis client/protocol working with DDEV. I use it on normal non-dockerized Ubuntu servers. It's inherently faster than phpredis, and also has its own caching layer (which is limited with the free version, and the full version isn't at all worthwhile if you can get the average times down to 10ms - it would only make sense if they were 100ms+) |
rfay
changed the title
|
Heres the metrics with redis on host machine with tcp. Not a whole lot of difference, which was to be expected given the single-client benchmark results above.
And I just got it connected to redis on host machine via unix socket. It seems to be somewhat faster after some similar browsing to prime it - probably within the 20-40% difference range that was seen in the redis-benchmarks above.
So, implementing UNIX sockets in the docker container is perhaps worthwhile. But its still a fair bit slower than expected (on a relative basis - obviously 0.05 seconds is immaterial...). Though, my main basis for comparison is on much faster latest-gen dedicated servers hardware at Hetzner. This could very well be just the expected performance for my laptop. I suppose I'd have to run the site on the host machine to see if docker is adding any extra overhead. I will update this issue if I end up doing that. |
|
Here is it with Unix Sockets to the Host Machine Redis Server and Relay
Perhaps Relay is to account for what I remember! To check, here it is with the memory/cache feature turned off (the lump in the middle is when I reloaded like 10 pages all at once):
So, it does what it says! The cache helps - though, in this case, its maybe 0.03 seconds improvement, so whether it is worth $150/month is up to you. And even without the cache (which I simulated by turning it off completely, because it drops to a low RAM amount after an hour) it is faster than phpredis. Anyway, here's the Relay Dockerfile. The sed commands for the variables could probably be set via DDEV This Wordpress Redis plugin can be referenced for how to make use of Relay in your application. And here's the relay-specific code I can share what I did for Unix sockets if there is interest. Though I think someone else who is more familiar with docker might need to implement it in this addon. |
|
I'm not optimistic that unix sockets can be used either in Docker or in a typical production setup (as the server would usually be on a different server, just as it is in Docker). But maybe a maintainer/user of this add-on will want to follow up. (I'm not surprised that unix sockets is faster) |
This is all irrelevant if someone is running Redis on a separate server - network latency would be 100-1000x more than the latency for a query to a local Redis server. But, as shown, it can be helpful when using DDEV locally (which is essentially officially advised not to use in production). But if you are running Redis on the same production server as your application (which I absolutely intend to do, along with mariadb for the same latency reasons) - docker or not - then you could benefit from of all this as well. I also don't suppose that Redis developed and documented support for unix sockets just for use in local development... I'd even go so far as to say that if someone with a wordpress site is running Redis and/or mariadb on a separate server, then they don't have a clue what they're doing, because WP's architecture often results in many hundreds of queries per request. Best to scale vertically - which you can get up to nearly 200 physical cores in a server nowadays. And/or replicate the db and redis to multiple servers so that they can be colocated with the application. Also, the links I shared in the OP seem to show that it is possible to do Unix Sockets in Docker. And Ive partially shown it - I created a socket in the host machine and then added it to the ddev web container via a bind volume.
services:
web:
volumes:
- /var/run/redis:/var/run/redis |
|
I got both redis and mariadb unix sockets working with wordpress. Its dead-simple with the first link in OP. Its all just about making a socket path available to all containers via a common volume (just as we do for other paths, like the web app directory) services:
tmp:
image: busybox
command: chmod -R 777 /tmp/docker
volumes:
- /tmp/docker/
web:
volumes_from:
- tmp
db:
volumes_from:
- tmp
redis:
volumes_from:
- tmpIt seems it doesn't even need to be a bind volume to the host system such as I used in the previous comment... Might be worth testing that - although, like this every project would have its own isolated socket within its own docker network, rather than all using the same host-system
[mysqld]
socket = /tmp/docker/mysql.sock
I havent done any further benchmarking yet, but I dont really need to - its well known that sockets are 20-50% faster than TCP, and I showed that Redis Relay provides a considerable performance increase over even Unix sockets. All the necessary snippets are shared in this thread, and they really wouldnt affect anything to add - even if people dont use them (which seems to be the general DDEV ethos for all of its images/containers). I'm happy to discuss any changes if you want a PR here or elsewhere, or feel free to incorporate them yourselves as you see fit. |
|
Congratulations on the excellent experimentation and analysis! Maybe maintainer @seebeen will pick it up and run with it. I barely know what redis is. |
|
Hello Nick, Thanks for the immense effort in benchmarking and wrangling with configs. You are 100% correct:
However - there are additional factors which slow redis server, but it mostly boils down to sysctl config and overrides. I have a local branch with a similar setup as yours - socket communication with tmp dirs and bind mounts. But it has compatibility problems with colima and docker desktop on MacOS (M1) - so I disregarded it. All in all - I believe the compabitility issues outweigh the performance benefits. Especially since DDEV is primarily a development environment. If you have the time - I'd love to see your fork with socket implementation which passes the tests on MacOS, WSL, Linux - with both Docker and Colima. If all works, We can work together to create a config script which will give users the implementation choice - default network stack or socket connection. Again - thank you for spending your time in order to make this extension better - and I hope you aren't dissuaded on working on it further. P.S. If you'd like to chat / talk about finer points I'm available on DDEV discord. You can't miss me 😅 |
Its the industry-standard (though they somewhat committed suicide recently with changes to their open source licensing, so there's a lot of forks and new api-compatible competitors - perhaps I'll open a separate issue to add support for using those instead of Redis, since the same clients can connect to them seamlessly) in-memory KV store, generally used for caching of various sorts. If not used on same server, it seems somewhat pointless to use it at all as the latency gains from in-memory store would be vastly outweighed by network latency. Might as well use disk storage at that point to save money - Rails' new Solid Cache does so with Mysql/postgres/sqlite/etc... to great effect. So, if you have Redis (and Mysql/mariadb for that matter, as I easily incorporated in my previous comment!) in use on the same server as the application, might as well use unix sockets to improve query speed - particularly with a rats nest like Wordpress that makes such an unconscionable amount of db/redis requests per php page request. And Relay is a cherry on top, regardless of TCP/Unix socket choice - its in-process cache prevents many requests from even leaving PHP, let alone the web container. |
|
@seebeen Yeah, I had seen some documentation and discussions about how docker bridge networks are slower than using the host network, and assume that's what mostly accounts for the unexpectedly slow redis response times that started this issue. But I figured that host network simply wouldnt work at all in the context of DDEV. Though, using Relay essentially solves this problem because not only does it communicate with Redis more quickly from PHP, but it also has its own RAM cache that is in-process with PHP itself, so the request doesn't even leave PHP at all, let alone the webserver container. In that respect, it sort of makes unix sockets irrelevant as well - though there's no reason not to use them anyway, and they're helpful for when a request misses Relay's cache. As mentioned, the RAM limit drops from unlimited to 16MB after an hour, which significantly lessens its utility as it will be constantly purging items from its cache on a LRU basis. Yet, surely this isn't a problem in a dev environment, where there's minimal requests, one (or few) users, and therefore not a whole lot of variety in cahed data. (there's also a... trick... that can be used to completely sidestep this 1-hour limit, but I'm hesitant to share it publicly out of respect for the dev's generosity in sharing Relay as a free version. It can remain a hacker's secret) Anyway, that's unfortunate that there's compatibility issues with unix sockets on colima and mac docker desktop! But, as you suggested, perhaps there's a way we can make it work such that its an easy opt-in. I'm happy to collaborate on that. Probably best to keep discussion here rather than Discord, so as to make it easier for others to discover, as well as force us (me) to be more concise - I tend to ramble even more on Discord, so am generally avoiding it these days. When you say "fork", you mean of this add-on? I didn't actually fork anything - just installed it and then, in the end, manually added 4 files/snippets. I'll re-list slightly modified versions here, so as to consolidate all my prevoius ramblings into one place.
services:
unixsockets:
image: busybox
command: chmod -R 777 /tmp/unixsockets
volumes:
- /tmp/unixsockets/
web:
volumes_from:
- unixsockets
db:
volumes_from:
- unixsockets
redis:
volumes_from:
- unixsockets
[mysqld]
socket = /tmp/unixsockets/mysql.sockI suppose that the mysql stuff would be better placed elsewhere, but I dont think it merits an entire add-on just for it... Likewise the Let me know how you'd like to proceed! Any suggestions on how to include/activate these things - be it in this addon, a separate addon, or perhaps even ddev core (though I think we'd all prefer to avoid that!) - would be appreciated. I dont see much issue with installing and activating it all by default, unless it literally breaks the If you want me to put those 4 files in a fork of this addon, i can do that. Let me know! |
|
Another thing you might want to consider if you're having trouble with redis performance is to install it in the webserver instead of as a separate service/container. Just use a Dockerfile.redis to install it there with https://redis.io/docs/latest/operate/oss_and_stack/install/install-redis/install-redis-on-linux/ And despite the suggestion there, just You could start it with |
|
Thanks, nice idea. I just tried for a bit but can't (be bothered to) figure it out. Its having trouble with the config files etc... Much easier with docker and this addon. With the unix sockets and Relay, its performing extremely well. |
|
Turns out that DDEV needs the Its all for the best though, because here's a simpler way to get all of this working. No
|
|
I see that there's an active pr for making ddev get more extensible/flexible. Best to wait for that to get merged before proceeding with anything here |
|
@unfixa1 yes, using Unix sockets is the point of this issue and was detailed in my various comments above... |
and others
I just ran a simple benchmark (
redis-benchmark -q -n 1000000 -s /var/run/redis/redis.sock -t set,get,mset -P 16 -c 100) with 3 networking configurationsYou can see that unix sockets are considerably faster than tcp. And there seems to be a signficant tax with the Docker Container.
I don't know much about docker networking, so I dont know what to look at or change.
Here's some links that I've found that might be helpful
https://medium.com/@jonbaldie/how-to-connect-to-redis-with-unix-sockets-in-docker-9e94e01b7acd
https://www.reddit.com/r/selfhosted/comments/vf6jeg/i_used_unix_sockets_to_improve_the_performance_of/
https://help.nextcloud.com/t/solved-cant-get-redis-to-work-in-socket-mode/122830/13
https://serverfault.com/questions/1156554/bind-a-host-unix-socket-to-a-container-port
The text was updated successfully, but these errors were encountered: