refactor EAGLE 2 (sgl-project#3269)

Co-authored-by: Ying Sheng <[email protected]> Co-authored-by: merrymercy <[email protected]> Co-authored-by: Ying1123 <[email protected]>
igor-pechersky · Feb 3, 2025 · 013021b · 013021b
1 parent 3c8ac78
commit 013021b
Show file tree

Hide file tree

Showing 9 changed files with 1,180 additions and 596 deletions.
diff --git a/examples/runtime/engine/EAGLE_offline_batch_inference.py b/examples/runtime/engine/EAGLE_offline_batch_inference.py
@@ -21,6 +21,7 @@ def main():
         speculative_num_steps=3,
         speculative_eagle_topk=4,
         speculative_num_draft_tokens=16,
+        cuda_graph_max_bs=8,
     )
 
     outputs = llm.generate(prompts, sampling_params)