Skip to content

Commit

Permalink
Fix high gateway CPU usage (#300)
Browse files Browse the repository at this point in the history
It turns out, when an http endpoint request is pending, and the request
buffer shuts down (if there are no containers running, or the serve is
cancelled, etc), we are not returning from this select, and it is
locking/unlocking indefinitely. Relevant pprof dump:


```
(pprof) top
Showing nodes accounting for 29.61s, 95.89% of 30.88s total
Dropped 301 nodes (cum <= 0.15s)
Showing top 10 nodes out of 47
      flat  flat%   sum%        cum   cum%
     9.03s 29.24% 29.24%      9.04s 29.27%  runtime.lock2
     7.94s 25.71% 54.95%      7.95s 25.74%  runtime.unlock2
     7.62s 24.68% 79.63%     28.65s 92.78%  runtime.selectgo
     1.48s  4.79% 84.42%     10.51s 34.03%  runtime.sellock
     1.03s  3.34% 87.76%      9.40s 30.44%  runtime.selunlock
     0.79s  2.56% 90.32%      0.79s  2.56%  runtime.fastrand (inline)
     0.73s  2.36% 92.68%      0.94s  3.04%  context.(*cancelCtx).Done
     0.39s  1.26% 93.94%     29.98s 97.09%  github.com/beam-cloud/beta9/pkg/abstractions/endpoint.(*RequestBuffer).ForwardRequest
     0.37s  1.20% 95.14%      0.37s  1.20%  runtime/internal/syscall.Syscall6
     0.23s  0.74% 95.89%      8.38s 27.14%  runtime.unlock (inline)
```
  • Loading branch information
luke-lombardi authored Jun 15, 2024
1 parent c2bfa06 commit 9d30d2f
Showing 1 changed file with 2 additions and 0 deletions.
2 changes: 2 additions & 0 deletions pkg/abstractions/endpoint/buffer.go
Original file line number Diff line number Diff line change
Expand Up @@ -106,8 +106,10 @@ func (rb *RequestBuffer) ForwardRequest(ctx echo.Context, payload *types.TaskPay
for {
select {
case <-rb.ctx.Done():
return nil
case <-done:
return nil
case <-time.After(100 * time.Millisecond):
}
}
}
Expand Down

0 comments on commit 9d30d2f

Please sign in to comment.