Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RoPE enhancements #1478

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

sudhakarsingh27
Copy link
Collaborator

Description

TLDR;
Enable application of staggered rope embeddings to different sequences within the same batch.

During generation tasks, different sequences in a batch might have different start positions (technically different end positions as well but that's bounded by the max sequence length in the batch so something we can afford to ignore for now). This change simply modifies the rope kernel to apply the rope embeddings in a staggered manner to different sequences in the batch using an argument start_positions.

(The start_positions and related changes are directly adapted from #829 which was authored by @pggPL)

Fixes # (issue)

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

  • Non breaking changes to apply_rotary_pos_emb function but this is non breaking since start_positions is a default kwarg here.
  • Breaking changes to FusedRoPEFunc and all the extensions/kernels that are called internally by this function.

@sudhakarsingh27 sudhakarsingh27 changed the title RoPE functionality enhancements RoPE enhancements Feb 11, 2025
@sudhakarsingh27 sudhakarsingh27 self-assigned this Feb 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant