Add try/except to test-tokenizer-random.py

[rmusser01]

Added try/except to main block;
Added (optional) additional cli args:
--max-errors, default=10, "Maximum number of errors before stopping"
--iterations, default=100, help="Number of iterations for random generators"
--tokenizers, nargs="+", help="List of tokenizers to test", default=tokenizers

refactored find_first_mismatch function;
Added self.free() to the LibLlama class

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

Copilot reviewed 1 out of 1 changed files in this pull request and generated no suggestions.

[jaime-m-p]

It's not equivalent with different list sizes.
Check if v3 is a good alternative:

def find_first_mismatch_v1(ids1, ids2):
    for i, (a, b) in enumerate(zip(ids1, ids2)):
        if a != b:
            return i
    if len(ids1) == len(ids2):
        return -1
    return min(len(ids1), len(ids2))

def find_first_mismatch_v2(ids1, ids2) -> int:
    return next((i for i, (a, b) in enumerate(zip(ids1, ids2)) if a != b), -1)

def find_first_mismatch_v3(ids1, ids2) -> int:
    index = next((i for i, (a, b) in enumerate(zip(ids1, ids2)) if a != b), -1)
    if index < 0 and len(ids1) != len(ids2):
        index = min(len(ids1), len(ids2))
    return index

print( find_first_mismatch_v1( [1,2,3], [1,2,3,4] ) )  # output: 3
print( find_first_mismatch_v2( [1,2,3], [1,2,3,4] ) )  # output: -1
print( find_first_mismatch_v3( [1,2,3], [1,2,3,4] ) )  # output: 3

[rmusser01]

Thanks for catching that/sorry for not catching it myself, will fix and push a change to my PR later today.

[rmusser01]

Just replaced it with your modified method.

[jaime-m-p]

Oh. It was not my intention to suggest removing the types:

- def find_first_mismatch(ids1, ids2) -> int:
+ def find_first_mismatch(ids1: Sequence[Any], ids2: Sequence[Any]) -> int:

[rmusser01]

Re-added the types.

[jaime-m-p]

What are you planning to do with --tokenizers, vocab_file and dir_tokenizer?

The main was intended to test one tokenizer. I was procrastinatingusing __main__ as an external script to loop over tokenizers.

[rmusser01]

Ah, I added them as I wasn't exactly sure of all intended use-cases, and figured adding them as optional args (with defaults that matched the initial settings) would allow for greater customizability down the line while not interfering with existing operations.