Gemini 1.5 Pro results in 500 Internal error with as many tools as Flash

[MrCsabaToth]

Even though I disabled some tools for gemini-1.5-flash (the Exchange Tool and either the Web Search (Duck Duck Go) or the Tavily Web research tool, and also the Alpha Vantage), but Flash was working with several tools.
It seems Pro cannot take as many, or we need to tune some function descriptions?
The 500 error happens during the model initialization.

[MrCsabaToth]

For MVP sprint we can neglect this IMHO and roll with the Flash. We can figure it out after the hackathon deadline.

[MrCsabaToth]

There's an idea to stuff the prompt with the list of the available tools and functions. Even though passing the available tools in a native way should give the information to the model, but maybe reinforcing that with a repeated own version would help. Joan Boronat Ruiz advised that and I'll try it https://medium.com/@tocsa2/i-dont-think-it-s-a-good-idea-to-repeat-the-tools-in-the-system-prompt-because-the-model-receives-e90a7df72859

[MrCsabaToth]

I'm using the extra function stuffing. It didn't prevent some internal errors, however it seems to not interfere with the native tool list because I was able to make queries which require two rounds of function calls (weather prompt uses the location function first and then the weather function).

[MrCsabaToth]

Another related issue google-gemini/generative-ai-dart#194

[MrCsabaToth]

We are now converting over to https://pub.dev/packages/firebase_vertexai (#53), let's see how that deals with functions. So far not good (Unable to submit request because function parameters schema should be of type OBJECT. Learn more: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling), the functions worked with the other package already, we are a step back again.

[MrCsabaToth]

Not 500, but after transition we are dealing with other issues, but not 500s