Non-streamed responses aren't reflected in the dashboard

[manuelmorales]

Describe the issue

When chat completions requests are sent in a non-streaming way, they are not reflected in the dashboard whatsoever. The issue was found in this Discord thread.

Steps to Reproduce

You can use the script below to reproduce the issue.

#! /bin/bash
set -e
source .env
set -x

curl "$OPENAI_API_BASE/chat/completions" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $OPENAI_API_KEY" \
    -d '{
        "model": "gpt-4o-mini",
        "messages": [
            {
                "role": "system",
                "content": "You are a helpful assistant."
            },
            {
                "role": "user",
                "content": "Write a haiku that explains the concept of recursion."
            }
        ]
    }'

The request above returns 200 OK and returns correct chat completions. But it doesn't show in the dashboard. I don't know if it's scanned for secrets.

It happens with Anthropic too.

Just by adding "stream": true to the body of the request is enough to make the problem go away.

Operating System

Linux (Intel)

IDE and Version

curl

Extension and Version

No extension was used.

Provider

OpenAI

Model

Tested with 4o-mini and sonnet. But I suspect it applies to all models

Codegate version

v0.1.16 (docker image ghcr.io/stacklok/codegate latest a20bef562ed0)

Logs

No response

Additional Context

No response

[kantord]

I can confirm the same problem

[dussab]

@kantord @manuelmorales I think PR #967 should fix this.

[aponcedeleonch]

@dussab I just tested this and the prompt and alerts are getting recorded correctly but not the outputs. We still cannot close the issue :(

[lukehinds]

thanks @aponcedeleonch