Skip to content

A C++ program that will to convert input file(s) into a series of tokens.

Notifications You must be signed in to change notification settings

Pip-Install-Party/Tokenization

Repository files navigation

Component 2 - Tokenization

This program will identify and remove C-style comments from an input test file using a deterministic finite state automoton (DFA) before using another DFA to convert the input file into a series of tokens.

Authored by: Blake Marshall, Brandon Robinson, Holden Ea, Rolando Yax, and Jacob Sellers.

Running the program:

This project can be run via make.

make

./tokenize.x

If you have a Windows based machine, you will need to adjust the Makefile to generate a .exe executable, rather than .x.

About

A C++ program that will to convert input file(s) into a series of tokens.

Resources

Stars

Watchers

Forks