Skip to content

Commit

Permalink
chore: Add Qwen2.5 and Phi3.5 to Readme (#265)
Browse files Browse the repository at this point in the history
## Summary
Qwen2.5 was released recently - it uses the same model architecture as
Qwen2 (see:
https://huggingface.co/Qwen/Qwen2.5-7B/blob/main/config.json#L3).
Likewise for Phi3.5 (see:
https://huggingface.co/microsoft/Phi-3.5-mini-instruct/blob/main/config.json#L4).
Also, the `model_type`s in the above configs will allow
AutoLigerKernelForCausalLM to work correctly for these models out of the
box.

Adding them to the readme for clarity / marketing reasons :)

- Hardware Type: <BLANK>
- [ x ] run `make test` to ensure correctness
- [ x ] run `make checkstyle` to ensure code style
- [ x ] run `make test-convergence` to ensure convergence
  • Loading branch information
tyler-romero authored Sep 22, 2024
1 parent dd86cbd commit ed4e60c
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -226,9 +226,9 @@ loss.backward()
| Mixtral | `liger_kernel.transformers.apply_liger_kernel_to_mixtral` | RoPE, RMSNorm, SwiGLU, CrossEntropyLoss, FusedLinearCrossEntropy |
| Gemma1 | `liger_kernel.transformers.apply_liger_kernel_to_gemma` | RoPE, RMSNorm, GeGLU, CrossEntropyLoss, FusedLinearCrossEntropy |
| Gemma2 | `liger_kernel.transformers.apply_liger_kernel_to_gemma2` | RoPE, RMSNorm, GeGLU, CrossEntropyLoss |
| Qwen2 | `liger_kernel.transformers.apply_liger_kernel_to_qwen2` | RoPE, RMSNorm, SwiGLU, CrossEntropyLoss, FusedLinearCrossEntropy |
| Qwen2 & Qwen2.5 | `liger_kernel.transformers.apply_liger_kernel_to_qwen2` | RoPE, RMSNorm, SwiGLU, CrossEntropyLoss, FusedLinearCrossEntropy |
| Qwen2-VL | `liger_kernel.transformers.apply_liger_kernel_to_qwen2_vl` | RMSNorm, LayerNorm, SwiGLU, CrossEntropyLoss, FusedLinearCrossEntropy |
| Phi3 | `liger_kernel.transformers.apply_liger_kernel_to_phi3` | RoPE, RMSNorm, SwiGLU, CrossEntropyLoss, FusedLinearCrossEntropy |
| Phi3 & Phi3.5 | `liger_kernel.transformers.apply_liger_kernel_to_phi3` | RoPE, RMSNorm, SwiGLU, CrossEntropyLoss, FusedLinearCrossEntropy |



Expand Down

0 comments on commit ed4e60c

Please sign in to comment.