tiktoken-lite

一个专门用来计算gpt的tokens的轻量级库

仅支持以下模型

type TiktokenModel = "gpt-3.5-turbo" | "gpt-3.5-turbo-0301" | "gpt-4" | "gpt-4-0314" | "gpt-4-32k" | "gpt-4-32k-0314";

fork from tiktoken

Get Started

npm install tiktoken-lite

Usage

import { get_encoding } from "tiktoken-lite";

const encoding = get_encoding();

console.log(encoding.encode('hello world'));