Function Calling with generateContent() causes error: "ensure that function response turn comes immediately after a function call turn"

[maxbeech]

Description of the bug:

I have followed this tutorial and have got it working, however as soon as I switch from using chat.sendMessage to generateContent, I get the error:

GoogleGenerativeAIFetchError: [GoogleGenerativeAI Error]: Error fetching from https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash:generateContent: [400 Bad Request] Please ensure that function response turn comes immediately after a function call turn.

at the point:

const result2 = await generativeModel.generateContent([{functionResponse: {
        name: "controlLight",
        response: apiResponse,
}}]);

My environment:

• Running using a NodeJS Firebase function
• Using the package: const {GoogleGenerativeAI} = require("@google/generative-ai");

Many thanks for your help!

Actual vs expected behavior:

Actual:
Get the error: GoogleGenerativeAIFetchError: [GoogleGenerativeAI Error]: Error fetching from https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash:generateContent: [400 Bad Request] Please ensure that function response turn comes immediately after a function call turn.

Expected:
Execute the function correctly. (eg. return OK. I've dimmed the lights to 50% and set the color temperature to warm. Let me know if you want to adjust the lighting further.)

Any other information you'd like to share?

Code comparison

When using chat.sendMessage

  async processMessage(event, platformSource) {
    const controlLightFunctionDeclaration = {
      name: "controlLight",
      parameters: {
        type: "OBJECT",
        description: "Set the brightness and color temperature of a room light.",
        properties: {
          brightness: {
            type: "NUMBER",
            description: "Light level from 0 to 100. Zero is off and 100 is full brightness.",
          },
          colorTemperature: {
            type: "STRING",
            description: "Color temperature of the light fixture which can be `daylight`, `cool` or `warm`.",
          },
        },
        required: ["brightness", "colorTemperature"],
      },
    };

    // Executable function code. Put it in a map keyed by the function name
    // so that you can call it once you get the name string from the model.
    const functions = {
      controlLight: ({brightness, colorTemperature}) => {
        return this.setLightValues( brightness, colorTemperature);
      },
    };


    // Access your API key as an environment variable (see "Set up your API key" above)
    const genAI = new GoogleGenerativeAI(geminiApiKeySecret.value());

    const generativeModel = genAI.getGenerativeModel({
      // Use a model that supports function calling, like a Gemini 1.5 model
      model: "gemini-1.5-flash",

      // Specify the function declaration.
      tools: {
        functionDeclarations: [controlLightFunctionDeclaration],
      },
    });


    const chat = generativeModel.startChat();
    const prompt = event.text;
    // const prompt = "Dim the lights so the room feels cozy and warm.";

    // Send the message to the model.
    const result = await chat.sendMessage(prompt);

    // For simplicity, this uses the first function call found.
    const call = result.response.functionCalls()[0];

    if (call) {
    // Call the executable function named in the function call
    // with the arguments specified in the function call and
    // let it call the hypothetical API.
      const apiResponse = await functions[call.name](call.args);

      // Send the API response back to the model so it can generate
      // a text response that can be displayed to the user.
      const result2 = await chat.sendMessage([{functionResponse: {
        name: "controlLight",
        response: apiResponse,
      }}]);

      // Log the text response.
      console.log(result2.response.text());
    }
  }

When using generateContent

  async processMessage(event, platformSource) {
    const controlLightFunctionDeclaration = {
      name: "controlLight",
      parameters: {
        type: "OBJECT",
        description: "Set the brightness and color temperature of a room light.",
        properties: {
          brightness: {
            type: "NUMBER",
            description: "Light level from 0 to 100. Zero is off and 100 is full brightness.",
          },
          colorTemperature: {
            type: "STRING",
            description: "Color temperature of the light fixture which can be `daylight`, `cool` or `warm`.",
          },
        },
        required: ["brightness", "colorTemperature"],
      },
    };

    // Executable function code. Put it in a map keyed by the function name
    // so that you can call it once you get the name string from the model.
    const functions = {
      controlLight: ({brightness, colorTemperature}) => {
        return this.setLightValues( brightness, colorTemperature);
      },
    };


    // Access your API key as an environment variable (see "Set up your API key" above)
    const genAI = new GoogleGenerativeAI(geminiApiKeySecret.value());

    const generativeModel = genAI.getGenerativeModel({
      // Use a model that supports function calling, like a Gemini 1.5 model
      model: "gemini-1.5-flash",

      // Specify the function declaration.
      tools: {
        functionDeclarations: [controlLightFunctionDeclaration],
      },
    });

    const prompt = event.text;
    // const prompt = "Dim the lights so the room feels cozy and warm.";

    // Send the message to the model.
    const result = await generativeModel.generateContent([prompt]);

    // For simplicity, this uses the first function call found.
    const call = result.response.functionCalls()[0];

    if (call) {
    // Call the executable function named in the function call
    // with the arguments specified in the function call and
    // let it call the hypothetical API.
      const apiResponse = await functions[call.name](call.args);

      // Send the API response back to the model so it can generate
      // a text response that can be displayed to the user.
      const result2 = await generativeModel.generateContent([{functionResponse: {
        name: "controlLight",
        response: apiResponse,
      }}]);

      // Log the text response.
      console.log(result2.response.text());
    }
  }
}

I have also tried specifying the version to be v1beta to no benefit as follows:

    const generativeModel = genAI.getGenerativeModel({
      // Use a model that supports function calling, like a Gemini 1.5 model
      model: "gemini-1.5-flash",

      // Specify the function declaration.
      tools: {
        functionDeclarations: [controlLightFunctionDeclaration],
      },
    },
    {
      apiVersion: "v1beta",
    });