Add support for partial input

[winterland1989]

This issue tracks recently discussion on haskell cafe. maybe we can add something like:

data Result a = Partail (ByteString -> Result a) | Done a ByteString | Failure

readDecimalPartial :: Integral a => ByteString -> Result a

But this seems too complex for a lexer, i will try to improve binary-parsers to see if i can manage a lexer loop there first.

[BebeSparkelSparkel]

@winterland1989 Can you provide a link?

[winterland1989]

This is 7years ago issue, I can't really find it any more lol.

[winterland1989]

The main idea is in the type, feel free to close it if it's not useful.

[wrengr]

The only case I can see for introducing partial results (in a lexer) is if the token spans the boundaries between input buffers. For integral lexing, it's quite easy (albeit annoying) to patch things up after the fact; whereas for non-integrals, although it can be patched up after the fact, it's a lot more involved (since you'd need to copy stuff over into a new buffer and try again from the beginning).

Since the behavior only differs at end-of-buffer, it should be doable to add support for partial inputs without introducing any slowdown in the main loop. I'll take a look, though I'll also have to figure out some decent benchmarks in order to ensure that the different return type doesn't introduce some unexpected overhead

[wrengr]

I started working on this. It turns out to be a lot more invasive than I was hoping; so it'll have to be given its own separate implementation rather than replacing the current one.