Skip to content

Commit

Permalink
return correct context length for text-embedding-ada-002
Browse files Browse the repository at this point in the history
from https://openai.com/blog/new-and-improved-embedding-model:
```
Longer context. The context length of the new model is increased by a factor of four, from 2048 to 8192, making it more convenient to work with long documents.
```
  • Loading branch information
ursachec authored and zurawiki committed Oct 16, 2023
1 parent 8b5dc54 commit e2e54f2
Showing 1 changed file with 3 additions and 0 deletions.
3 changes: 3 additions & 0 deletions tiktoken-rs/src/model.rs
Original file line number Diff line number Diff line change
Expand Up @@ -62,5 +62,8 @@ pub fn get_context_size(model: &str) -> usize {
if starts_with_any!(model, "text-ada-001", "text-babbage-001", "text-curie-001") {
return 2049;
}
if starts_with_any!(model, "text-embedding-ada-002") {
return 8192;
}
4096
}

0 comments on commit e2e54f2

Please sign in to comment.