Startup trouble with Google-madlad400-10b-mt.

[BradHutchings]

I'm getting a startup error trying to use Google-madlad400-10b-mt, which is allegedly good at 450+ languages.

Here is the error:

warming up the model with an empty run
llama.cpp/server/server.cpp:481: GGML_ASSERT(llama_add_eos_token(model) != 1) failed

error: Uncaught SIGABRT (SI_TKILL) at 0 on Seventeen pid 31640 tid 9544
  /D/hyperv/llamafiles-windows/Brads-LLMs-llamafile.exe
  Math result not representable
  Windows Cosmopolitan 3.9.6 MODE=x86_64 Seventeen 10.0

This feels like it should be quicker to fix than say IBM Granite support. Anyone have a hint for me?

[BradHutchings]

I tried making my own gguf from the official Google repo:

https://huggingface.co/google/madlad400-10b-mt/tree/main

I also tried this gguf:

https://huggingface.co/NikolayKozloff/madlad400-10b-mt-Q8_0-GGUF

I think there's something goofy with the tokenizer.

I haven't tried with llama.cpp yet. That's my next task. It looks like it might not be supported in llama-server there yet.

ggerganov/llama.cpp#9030