accept_tcp_from_lwip error: queue overrun; shouldn't happen with lwIP listen backlog

[rinor]

under what other conditions, beside high cpu load/usage/throttle can this happen

[eyberg]

do you have a fuller trace or sample code we can look at?

[rinor]

I'll try to reproduce it, but atm don't have more info other than that log from production systems. I'm just trying to narrow it down and understand so I can focus on that area.

[francescolavra]

Did you get that error message from an instance with multiple vCPUs? I can see in the code that this could happen when there are multiple concurrent connection requests that are processed in parallel by different vCPUs; it's a flaw in the code that we should be able to fix by reworking the logic that protects a listening socket from concurrent access to its struct fields.
Assuming you are using multi-vCPU instances, basically this could happen when your server has a TCP listen backlog close to the maximum supported value (255) and there is a high rate of client connection requests.

[rinor]

In this case there is no smp involved. The service is deployed on gcp f1-micro and single cpu is available/exposed from nanos perspective. What happened there was a high cpu usage that lasted for a long time and being a shared cpu with 0.2 cpu thread available (burstable to 0.4 for few seconds) things start to fall apart. I did expect troubles under such conditions, but nanos had no other major issues handling that.