Check that processors add the number of tokens they say they will

[boyleconnor]

Fixes #1314

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[Narsil]

No we cannot panic like that it's not OK.

I'm very fine if the reported numbers is not the perfect one, panicking randomly is not acceptable for it

[boyleconnor]

@Narsil, I'm very surprised to read this:

I'm very fine if the reported numbers is not the perfect one

If there is a mismatch between added_tokens() and the number of tokens actually added by processor.process_encodings(), then (if truncation is enabled) Tokenizer.encode() will truncate to the wrong length, which I think most Hugging Face users would consider completely unacceptable behavior (e.g. imagine if you are truncating the tokens on some input before sending it to an expensive API).

IIUC the only way that this failure (added_tokens() != final_length - original_length) would be possible is if a Hugging Face contributor makes a mistake when implementing a Processor for a new tokenization system and the value of added_tokens() doesn't match the actual number of added tokens. I don't see how this could be "randomly" triggered without a significant systemic error being present in the tokenizers code base.

[boyleconnor]

@ArthurZucker do you want to share thoughts on this PR directly? I believe you mentioned on the issue that you would check it out

[ArthurZucker]

Hey! I don't think a new processor is planned, which is why it's not really worth the effort for now, but thanks a lot for wanting to contribute 🤗 I'm sorry in the delayed answer

[boyleconnor]

Okay, I will close this PR, then