Feature/shdict zset #1272

ZigzagAK · 2018-03-08T17:54:40Z

I hereby granted the copyright of the changes in this pull request
to the authors of this lua-nginx-module project.

Hello!
May be interresting DICT.zset feature (but without ffi)?

spacewander · 2018-03-09T06:25:43Z

Thank you for your pull request.
But there are some problems I have to point out:

We are trying hard to migrate Lua C API to FFI, so adding more non-FFI API is not desiring. Unless there is not FFI infrastructure like cosocket stuff, but shdict is not. Not even mentioning that the current shdict's Lua C API implementation is unsafe, discussed in Deadlock in ngx_http_lua_shdict_set_helper #1207.
Implement zset with doubly linked list doesn't work well when people need to do zrange (zrange is not implemented yet, but we need to do it one day if we add zset to shdict type). Redis implements zset with skip list.

spacewander · 2018-03-09T15:39:35Z

@ZigzagAK
Apologize for my mistake. The zset is actually built on top of rbtree, so my second problem doesn't make sense at all.

ZigzagAK · 2018-03-09T16:37:59Z

@spacewander
I don't use ffi (lua-resty-core) api. In my use case i can't found performance improvements with FFI based api. Integral performance of my application with FFI looks like a performance with standart LUA C API (luajit).

I checked my application with https://github.com/openresty/stapxx#lj-vm-states and https://github.com/openresty/stapxx#ngx-lj-trace-exits. When i use FFI api, application spend all the time in JIT code, but performance was not greather when i used standart LUA C API (luajit).

spacewander · 2018-03-10T02:54:47Z

@ZigzagAK
Maybe the bottleneck of the application is not JIT.
Besides the performance, the shdict API implemented in Lua C API is unsafe. They hold shm lock when calling the Lua C API. When LuaJIT runs the Lua C API, it may do GC check and trigger the GC. If the shdict is accessed again via the __gc metamethod, there will be a deadlock.

ZigzagAK · 2018-03-10T08:52:16Z

@spacewander

If the shdict is accessed again via the __gc metamethod, there will be a deadlock.

Yes, I understad it. Workaround: don't use lua objects with __gc metamethods, which may call shdict methods on the same shm.

Maybe the bottleneck of the application is not JIT.

Bootleneck was in the CPU. Most of the time CPU spent in strings and tables allocation/deallocation, shm mutex concurency. But in that moment i couldn't decrease strings and tables usage. And performance without ffi is acceptable and increase it on 10-20% is not nessessary. The performance on 6 core CPU is around 15k/sec with all lua code. We can simple increase number of virtual nodes.

mergify · 2020-06-26T00:39:30Z