Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Message Concatenation Issue with Multiple Roundtrips #4654

Open
tjazsilovsek opened this issue Feb 1, 2025 · 4 comments
Open

Message Concatenation Issue with Multiple Roundtrips #4654

tjazsilovsek opened this issue Feb 1, 2025 · 4 comments
Labels
bug Something isn't working

Comments

@tjazsilovsek
Copy link

tjazsilovsek commented Feb 1, 2025

Description

Version Information:

  • Previous version: 4.0.33
  • Current version: 4.1.16

Components Affected:

  • Frontend: useChat hook in react (nextjs)

Description:
After upgrading from version 4.0.33 to 4.1.16, there's an issue with how multiple roundtrip messages are handled. Instead of creating new messages for each roundtrip (as in 4.0.33), the SDK is now concatenating all content into a single message. This affects both streamed text and tool calls from subsequent roundtrips.

Current Behavior:

  1. Initial roundtrip:
    • Model streams text and/or makes tool calls
    • Content appears correctly in the first message
  2. Subsequent roundtrips:
    • New streamed text is appended to the existing text
    • New tool calls/results are appended to existing tools
    • Everything remains part of the first message instead of creating new messages

Expected Behavior:

  • Each new roundtrip should create a new message (as in version 4.0.33)
  • Tool calls and results from different roundtrips should be in separate messages
  • Streamed text from different roundtrips should appear in separate messages

Impact:

  • Breaks conversation flow and readability
  • Makes it difficult to distinguish between different rounds of interaction
  • Complicates UI implementations that expect separate messages
  • Tool results from different rounds get mixed together

Code example

// app/api/chat/route.ts
import { openai } from "@ai-sdk/openai";
import { streamText, tool } from "ai";
import { z } from "zod";
export const maxDuration = 30;

export async function POST(req: Request) {
	const { messages } = await req.json();

	const result = streamText({
		model: openai("gpt-4o"),
		messages,
		tools: {
			calculator: tool({
				description: "A tool to calculate the sum of two numbers",
				parameters: z.object({
					num1: z.number(),
					num2: z.number(),
				}),
				execute: async ({ num1, num2 }) => {
					return num1 + num2;
				},
			}),
		},
		maxSteps: 5,
	});

	return result.toDataStreamResponse();
}

// app/page.tsx
'use client';

import { useChat } from 'ai/react';

export default function Chat() {
  const { messages, input, handleInputChange, handleSubmit } = useChat();
  return (
    <div className="flex flex-col w-full max-w-md py-24 mx-auto stretch">
      {messages.map(m => (
        <div key={m.id} className="whitespace-pre-wrap">
					{m.role === "user" ? "User: " : "AI: "}
					{m.content}
					{m.toolInvocations?.map((t) => (
						<div key={t.toolCallId}>
							{t.toolCallId}
							{t.toolName}
							{JSON.stringify(t.args)}
              {t.state === 'result' ? JSON.stringify(t.result) : 'pending'}
						</div>
					))}
				</div>
      ))}

      <form onSubmit={handleSubmit}>
        <div className="fixed bottom-0 w-full max-w-md flex gap-2 mb-8">
          <input
            className="flex-1 dark:bg-zinc-900 p-2 border border-zinc-300 dark:border-zinc-800 rounded shadow-xl"
            value={input}
            placeholder="Say something..."
            onChange={handleInputChange}
          />
        </div>
      </form>
    </div>
  );
}

AI provider

@ai-sdk/openai v1.1.9

Additional context

No response

@tjazsilovsek tjazsilovsek added the bug Something isn't working label Feb 1, 2025
@Nishkalkashyap
Copy link

+1

This is a breaking change and ideally should've been part of a major release. Also, version 4.1 release notes make NO mention of this change.

As a workaround, I'm now adding annotations after every step completion in the onStepFinish method of streamText/generateText function. These annotations are then available in the assistant message as an array, which can be used to implement chat behaviour as in SDK version 4.0.*

// api/chat/route.ts
export async function POST(req: Request) {
    return createDataStreamResponse({
      execute: async (dataStream) => {
        const result = streamText({
          model,
          system: `
              You are a friendly assistant! Keep your responses concise and helpful.
              `.trim(),
          maxSteps: 10,
          messages: messages,
          tools: tools,
          onStepFinish(event) {
            dataStream.writeMessageAnnotation({
              id: generateId(),
              finishReason: event.finishReason,
              isContinued: event.isContinued,
              stepType: event.stepType,
              text: event.text,
              toolCalls: event.toolCalls,
              toolResults: event.toolResults,
            });
          },
        });

        result.mergeIntoDataStream(dataStream);
      },
    });
}
// /chat/page.tsx

// version 4.0.* compatible implementation
{/* {message.role === "assistant" && (
  <AssistantMessage message={message} />
)} */}

// version 4.1.* compatible workaround
{message.role === "assistant" && (
  <>
    {message.annotations?.map((annotation) => (
      <AssistantMessage key={annotation.id} message={annotation} />
    ))}
  </>
)}

@user7234
Copy link

user7234 commented Feb 2, 2025

same issue, this feature is better implemented as a param in useChat, allowing users to decide whether to merge assistant messages into a single message.

@maxbaines
Copy link

+1

@peterje
Copy link

peterje commented Feb 5, 2025

+1, just spent my day tracking this down 😓

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

5 participants