Eval bug: llama-server stopped working after PR #11285 got merged

[tim-janik]

Name and Version

llama-server f30f099

Operating systems

Linux

GGML backends

CUDA

Hardware

RTX 4090, CUDA

Models

E.g. Code Qwen 2.5 7B-Chat (Q8)

Problem description & steps to reproduce

llama-server stopped generating any tokens for me, regardless of model, starting with commit f30f099 from #11285.
Simply reverting the above commit, e.g. on top of todays master (6171c9d) does fix the issue for me.

To reproduce, goto http://localhost:8080, enter a question hit return, nothing happens.

First Bad Commit

f30f099

Relevant log output

│ main: server is listening on http://0.0.0.0:8080 - starting the main loop                                                                                                                                                                                 │                
             │ srv  update_slots: all slots are idle                                                                                                                                                                                                                     │                
             │ request: GET / 127.0.0.1 200                                                                                                                                                                                                                              │                
             │ request: GET /favicon.ico  400                                                                                                                                                                                                                            │                
             │ request: POST /v1/chat/completions  400                                                                                                                                                                                                                   │

[ngxson]

Can you send the request via curl to see what is the response? (Or maybe see the response from browser's devtool)

From what I saw on your log, it responses with status code 400, meaning ERROR_TYPE_INVALID_REQUEST. This could potentially be due to upstream changes from httplib library, not a bug from llama-server.

[ngxson]

I'm not able to reproduce the bug on either my laptop (macbook M3) or on server (linux - nvidia T4)

[ngxson]

I did more tests but still can't reproduce the issue. Please provide more info about your setup:

• What is the browser you're using
• What happen when the request is sent via curl or wget or postman
• What show up if you access some other endpoints, for example /api/models

Also from your log, request: GET /favicon.ico 400, this means even non-API paths returns 400 error which is wrong. It's further strengthen my suspicion about bug in httplib.

[tim-janik]

I did more tests but still can't reproduce the issue. Please provide more info about your setup:

* What is the browser you're using

* What happen when the request is sent via `curl` or `wget` or postman

* What show up if you access some other endpoints, for example `/api/models`

Also from your log, request: GET /favicon.ico 400, this means even non-API paths returns 400 error which is wrong. It's further strengthen my suspicion about bug in httplib.

Thanks for the hint. I reverted f30f099 and only applied the httplib 0.18.5 change.
Turns out I can only trigger this issue if the URL contains a %0A, like this:

http://localhost:8080/?something=%0A

Which explains why only I saw it, I probably still had some q= or m= arg left over from playing with #11150 (btw, I'd appreciate answers to the two questions I posed in that PR if you could afford the time).

Which means this httplib change breaks multi-line use cases of #11150, do you have any idea why the newer httplib would break with URL encoded newlines?

[ngxson]

Ok sorry I forgot about #11150, I'll have a look later because it's not the priority.

I'm tagging author of httplib @yhirose too, maybe you have an idea why it returns 400 even for non-handler endpoints? (For example, /favicon.ico that should have returned 404 instead of 400)

[ngxson]

OK I think I pinpointed the problem. It only happens on OPTIONS request, which causes the memory to be corrupted somewhere. I repeatedly send OPTIONS request and got this in the log:

request:    400
request:    400
request: OPTIONS /tokenize 127.0.0.1 200
request: ndled   400
request: root   400
request: root   400

[ngxson]

Related to yhirose/cpp-httplib#2028