Draft & Verify

[Ryu1845]

Does this repository implement Draft & Verify?

[lucidrains]

@Ryu1845 hey! thanks for sharing that paper!

that looks quite close if not better than the naive early exit strategy (they predict which layers to skip through some heuristic) - but using the same model for speculating / drafting is definitely what i was going for.

i think my prophet transformer idea should be the best though (although i'm biased and still haven't ran any head to head 😆)

[lucidrains]

@Ryu1845 really think we are going to see a resurgence in adaptive computation research over the next year, like actually made practical

[Ryu1845]

I think so too, thanks again for your work.
it looks like the official code for the paper will be uploaded here, but I'll keep an eye on this repo too 😉

[lucidrains]

@Ryu1845 sounds good!

yea i think the main idea from the prophet idea is to take advantage of the cached last layer embedding from the large model, which should be superior to any early exit stuff. if you find me another paper that did that, would definitely read and implement

i'm also using a transformer on top, borrowing working ideas from hierarchical transformer line of research

[Ryu1845]

yea i think the main idea from the prophet idea is to take advantage of the cached last layer embedding from the large model, which should be superior to any early exit stuff.

I don't know of any paper that does this but the medusa project aims to do just that I think.
https://together.ai/blog/medusa
https://github.com/FasterDecoding/Medusa

[lucidrains]

@Ryu1845 ohh yes, they totally did. so the only difference is i use a small transformer as the medusa / prophet heads

ok let me cite them as well

[lucidrains]

@Ryu1845 oh haha, they don't have a paper, just a github repo. may be the new trend

[Ryu1845]

I'm guessing they'll release a paper once they've got a working prototype 😄
It looks like it's still a WIP FasterDecoding/Medusa#3
I actually don't know if it's running yet :/

[lucidrains]

@Ryu1845 ohh, so it isn't functional yet? maybe i'll send their group a message. solving batched spec decoding is a bit tricky with kv cache, but i found a solution (not sure if optimal)

[lucidrains]

~~I'm guessing they'll release a paper once they've got a working prototype 😄 ~~ It looks like it's still a WIP FasterDecoding/Medusa#3 I actually don't know if it's running yet :/

so it works or doesn't work?

[Ryu1845]

it looks like it works, I'm sorry for the misunderstanding on my side

[lucidrains]

nice! that's amazing, i believe in that approach

[jmamou]

@lucidrains
Amazing work!
Do you plan to release your results with early exit?
Thanks