Max thread limit reached with 1000+ connections

[zekronium]

Hi,

There seems to be some sort of an issue, with about 1000 connections it starts to panic

We are running on a 16 core server with 64gb ram on unix.

We are also opening many different ports via gost instead of one, which could explain the potential scaling issue

runtime: program exceeds 10000-thread limit
fatal error: thread exhaustion

runtime stack:
runtime.throw({0x14bb823?, 0x0?})
    /opt/hostedtoolcache/go/1.22.5/x64/src/runtime/panic.go:1023 +0x5c fp=0xc000067e10 sp=0xc000067de0 pc=0x43e0dc
runtime.checkmcount()
    /opt/hostedtoolcache/go/1.22.5/x64/src/runtime/proc.go:868 +0x8e fp=0xc000067e38 sp=0xc000067e10 pc=0x44226e
runtime.mReserveID()
    /opt/hostedtoolcache/go/1.22.5/x64/src/runtime/proc.go:884 +0x2f fp=0xc000067e60 sp=0xc000067e38 pc=0x4422af
runtime.startm(0xc00007e508?, 0x0, 0x0)
    /opt/hostedtoolcache/go/1.22.5/x64/src/runtime/proc.go:2862 +0x111 fp=0xc000067eb0 sp=0xc000067e60 pc=0x4457b1
runtime.handoffp(0xc0ffffffff?)
    /opt/hostedtoolcache/go/1.22.5/x64/src/runtime/proc.go:2911 +0x336 fp=0xc000067ed8 sp=0xc000067eb0 pc=0x445cb6
runtime.retake(0x1e8de6a9cb3a)
    /opt/hostedtoolcache/go/1.22.5/x64/src/runtime/proc.go:6165 +0x23d fp=0xc000067f38 sp=0xc000067ed8 pc=0x44dadd
runtime.sysmon()
    /opt/hostedtoolcache/go/1.22.5/x64/src/runtime/proc.go:6071 +0x345 fp=0xc000067fa0 sp=0xc000067f38 pc=0x44d785
runtime.mstart1()
    /opt/hostedtoolcache/go/1.22.5/x64/src/runtime/proc.go:1729 +0x93 fp=0xc000067fc8 sp=0xc000067fa0 pc=0x443d93
runtime.mstart0()
    /opt/hostedtoolcache/go/1.22.5/x64/src/runtime/proc.go:1686 +0x76 fp=0xc000067ff8 sp=0xc000067fc8 pc=0x443cd6
runtime.mstart()
    /opt/hostedtoolcache/go/1.22.5/x64/src/runtime/asm_amd64.s:394 +0x5 fp=0xc000068000 sp=0xc000067ff8 pc=0x476d45

[ginuerzh]

Please share your configuration and usage.

[zekronium]

gost -L http://:$port?interface=$interface -L http://:$port?interface=$interface -L http://:$port?interface=$interface... thats it, we bind different ports to different interfaces on the local machine. In some cases have tested with thousand plus ports/interfaces bound. In every case port and interface is unique.

[ginuerzh]

I'm not sure if this problem is caused by high concurrency or timeout, so I need more detailed runtime information.
Can you start GOST with profiling enabled?

GOST_PROFILING=:6060 gost -L http://:$port?interface=$interface ...

then go to http://localhost:6060/debug/pprof/ and monitor the goroutine and threadcreate.

[zekronium]

I managed to capture this last screenshot before it dying. It had about 12k goroutines. Used this version 3.0.0-nightly.20240806

[zekronium]

Maybe we can simply disable timeout, since in this case both the receiving server and the client both control their timeouts. Should a proxy be opinionated about when to time out?

Maybe possible to use a timer that does not make a platform thread?

[ginuerzh]

You can disable the timeout by setting a negative value:

gost -L http://:8080?dialTimeout=-1