Kong admin fails (500) to handle HTTP-request with duplicated header keys Content-Type

[Toliak]

Is there an existing issue for this?

• I have searched the existing issues (queries: "Content-Type" 500, "bad argument #1 to 'sub'")

Can be related: #8734

Kong version ($ kong version)

Kong Enterprise 3.9.0.0

Current Behavior

I'm using kong:3.9.0-ubuntu docker image.
When I send crafted HTTP-packet with multiple Content-Type keys in the header, Kong fails with Internal Server Error 500.

Expected Behavior

Maybe it should return Bad request (400).

Steps To Reproduce

The start.sh file (based on the docker image documentation):

#! /bin/bash
set -ue
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
MOUNT_DIR="$SCRIPT_DIR/kong-vol"

docker run -it --name kong-dbless \
 -v "$MOUNT_DIR:/kong/declarative/" \
 -e "KONG_DATABASE=off" \
 -e "KONG_DECLARATIVE_CONFIG=/kong/declarative/kong.yml" \
 -e "KONG_PROXY_ACCESS_LOG=/dev/stdout" \
 -e "KONG_ADMIN_ACCESS_LOG=/dev/stdout" \
 -e "KONG_PROXY_ERROR_LOG=/dev/stderr" \
 -e "KONG_ADMIN_ERROR_LOG=/dev/stderr" \
 -e "KONG_ADMIN_LISTEN=0.0.0.0:8001" \
 -e "KONG_ADMIN_GUI_URL=http://localhost:8002" \
 -e KONG_LICENSE_DATA \
 -p 9000:8000 \
 -p 9001:8001 \
 -p 9002:8002 \
 -p 9003:8003 \
 -p 9004:8004 \
 kong/kong-gateway:3.9.0.0

The Packet packet.txt (two empty lines at the end. The line-endings must be CRLF):

POST / HTTP/1.1
Accept: */*
Host: 127.0.0.0
Content-Type: application/x-www-form-urlencoded
Content-Type: application/x-www-form-urlencoded

The send.sh script:

#! /bin/bash
set -ue
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
PACKETS_DIR="$SCRIPT_DIR/packets"

cat "$PACKETS_DIR/$1" | netcat 127.0.0.1 9001 &
sleep 2
kill %1

Steps:

1. Run kong using bash start.sh
2. Run bash send.sh packet.txt

Anything else?

Logs from netcat:

HTTP/1.1 500 Internal Server Error
Date: Fri, 17 Jan 2025 13:12:41 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
Access-Control-Allow-Origin: *
Content-Length: 42
X-Kong-Admin-Latency: 7
Server: kong/3.9.0.0-enterprise-edition

{"message":"An unexpected error occurred"}

Logs from server:

2025/01/17 13:12:41 [error] 2550#0: *7991 [lua] api_helpers.lua:541: handle_error(): /usr/local/share/lua/5.1/kong/api/api_helpers.lua:267: bad argument #1 to 'sub' (string expected, got table)

stack traceback:
        [C]: in function 'sub'
        /usr/local/share/lua/5.1/kong/api/api_helpers.lua:267: in function 'filter'
        /usr/local/share/lua/5.1/lapis/application.lua:22: in function 'run_before_filter'
        /usr/local/share/lua/5.1/lapis/application.lua:177: in function 'resolve'
        /usr/local/share/lua/5.1/lapis/application.lua:217: in function </usr/local/share/lua/5.1/lapis/application.lua:215>
        [C]: in function 'xpcall'
        /usr/local/share/lua/5.1/lapis/application.lua:215: in function 'dispatch'
        /usr/local/share/lua/5.1/lapis/nginx.lua:231: in function 'serve'
        /usr/local/share/lua/5.1/kong/init.lua:2130: in function 'admin_content'
        content_by_lua(nginx-kong.conf:447):2: in main chunk, client: 172.17.0.1, server: kong_admin, request: "POST / HTTP/1.1", host: "127.0.0.0"
172.17.0.1 - - [17/Jan/2025:13:12:41 +0000] "POST / HTTP/1.1" 500 42 "-" "-"

Source (260 instead of 267 due to 7 lines of copyright in the beginning of the file in the docker image): https://github.com/Kong/kong/blob/3.9.0/kong/api/api_helpers.lua#L260-L262

As I can see, sub function expects string in content_type variable (and receives it, if there is only one header with key Content-Type). However, if multiple lines Content-Type: ... provided, the content_type variable will contain a table.

Same issue happens if I change POST to PUT in the packet.txt.

Similar issue happens if I change POST to GET in the packet.txt, however, the error relates to /usr/local/share/lua/5.1/lapis/application.lua so I will create an issue in the Lapis repository. (leafo/lapis#794)

The issue will not happen if I change POST to DELETE in the packet.txt.

[khaled4vokalz]

It's not kind of valid to have multiple headers of the same type. Check https://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.7
The RFC doesn't force the implementations to handle both the headers if provided.

[Toliak]

@khaled4vokalz, Hello.

Yes, I agree, the HTTP packet I provided may be somewhat of invalid.
Nevertheless, (1) I think that Kong's Lua scripts should not crash even on the invalid HTTP packet and return an error 500 with the Lua stack trace in the nginx' error log.

Moreover, (2) other inconsistencies in HTTP packets (three examples below) are being handled correctly (anyway, there is always a way to check the type and return, for example, 400 instead of script crash).

kong/kong/plugins/jwt/handler.lua

Line 59 in 898882d

if type(token_header) == "table" then

kong/kong/tools/http.lua

Line 455 in 898882d

if type(accept_header) == "table" then

kong/kong/router/traditional.lua

Line 971 in 898882d

if type(req_header) == "table" then

Thus, based on points (1) and (2) I conclude that the behavior described in the issue is a bug in Kong.

[Water-Melon]

@Toliak I agree, this seems to be a bug. I'll create a ticket to track this issue. Thank you very much for your report!