Token Counter 21.3.2024.

A web application for token counting within OpenAI framework. It can be used by uploading TXT files or directly writing text into the provided text area field. It is based on Streamlit and TikToken. Additionaly, offers the flexibility to select from a variery of additional models through radio buttons.

REQUIREMENTS: pip install tiktoken pip install streamlit

RUN: streamlit run tokenizer.py