Automated regression testing

[katlogic]

Currently, hacking on LuaJIT code is difficult because there seems to be no way to test it.

Some thoughts:

• Adapt code from http://www.lua.org/tests/. Sadly LuaJIT severely deviates from Lua in some regards. Either fix LuaJIT to match Rio Lua, or tailor the tests to LuaJIT.
• https://github.com/franko/luajit-lang-toolkit/tree/master/tests might be useful too, though it tests only very basic language features.
• Build infrastructure to test regressions in JIT. @fsfod 's https://github.com/fsfod/LuaJIT/blob/stringbuffer/tests/tracetracker.lua seems interesting
• Collect repros of past bugs into the suite. Both because it's often exotic code which might trigger other buggy behavior in the future or to avoid bugs resurfacing.
• Integrate the tests into CI, so introduced breakage could be discovered early on.

Other ideas?

[DemiMarie]

What are some of the ways in which LuaJIT deviates from Lua?

[refi64]

Wasn't @MikePall going to clean up his test suite and upload it online?

[pmatos]

Truthfully, if @MikePall has no time to clean it up, it might be better to provide it in 'raw' form such that somehow might clean it up than to have nothing at all.

I am new to LuaJIT but I am certainly surprised that such a successful project has survived so long without a proper regression suite.

[refi64]

@pmatos It has a regression suite. We just can't see it. :)

[katlogic]

@drbo: From top of my head:

• error()s thrown messages format and wording in a lot of places
• Underdefined/underspecified behavior, especially regarding gc. Lua test suite makes assumption that gc will trigger eventually because of repeated allocations, this is not true in LuaJIT
• Straight bugs triggered by the test suite, such as function x() coroutine.wrap(x)() end x()

[agladysh]

NB: This can be used as test suite as well: https://github.com/fperrad/lua-TestMore/tree/master/test_lua51

[katlogic]

@agladysh This looks pretty good, thanks!