Added improved token count calculations

Should be more accurate in different languages than just using character count as reference. Using a modified Node.js package gpt-3-encoder to work client-side.
patrikzudel · Mar 8, 2023 · cb1331c · cb1331c
1 parent d279195
commit cb1331c
Show file tree

Hide file tree

Showing 5 changed files with 50,220 additions and 21 deletions.
diff --git a/README.md b/README.md
@@ -19,8 +19,9 @@
 - Ability to use your own OpenAI API key.
 - No minimum cost, great for students!
   - Cheaper than ChatGPT Premium for most users.
-- Usage estimator for pricing - See how much you've spent so far!
-  - Estimating token count (4chars = 1token)
+- Usage calculator for pricing - See how much you've spent so far!
+  - Calculating token count with OpenAI's Tokenizer.
+    - Could be not 100% accurate, but its pretty good!
 - Token / cost saving features:
   - Send message without history. 
     - When you are asking a sequence of unrelated questions, don't bother sending the whole history!

diff --git a/public/encoder.json b/public/encoder.json