Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add strong warnings about XFF headers #181

Merged
merged 3 commits into from
Mar 17, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 68 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ store := memory.NewStore()
instance := limiter.New(store, rate)

// Alternatively, you can pass options to the limiter instance with several options.
instance := limiter.New(store, rate, limiter.WithTrustForwardHeader(true), limiter.WithIPv6Mask(mask))
instance := limiter.New(store, rate, limiter.WithClientIPHeader("True-Client-IP"), limiter.WithIPv6Mask(mask))

// Finally, give the limiter instance to your middleware initializer.
import "github.com/ulule/limiter/v3/drivers/middleware/stdlib"
Expand Down Expand Up @@ -125,6 +125,73 @@ You will find two stores:

When the limit is reached, a `429` HTTP status code is sent.

## Limiter behind a reverse proxy

### Introduction

If your limiter is behind a reverse proxy, it could be difficult to obtain the "real" client IP.

Some reverse proxies, like AWS ALB, lets all header values through that it doesn't set itself.
Like for example, `True-Client-IP` and `X-Real-IP`.
Similarly, `X-Forwarded-For` is a list of comma-separated IPs that gets appended to by each traversed proxy.
The idea is that the first IP _(added by the first proxy)_ is the true client IP. Each subsequent IP is another proxy along the path.

An attacker can spoof either of those headers, which could be reported as a client IP.

By default, limiter doesn't trust any of those headers: you have to explicitly enable them in order to use them.
If you enable them, **you must always be aware** that any header added by any _(reverse)_ proxy not controlled
by you **are completely unreliable.**

### X-Forwarded-For

For example, if you make this request to your load balancer:
```bash
curl -X POST https://example.com/login -H "X-Forwarded-For: 1.2.3.4, 11.22.33.44"
```

And your server behind the load balancer obtain this:
```
X-Forwarded-For: 1.2.3.4, 11.22.33.44, <actual client IP>
```

That's mean you can't use `X-Forwarded-For` header, because it's **unreliable** and **untrustworthy**.
So keep `TrustForwardHeader` disabled in your limiter option.

However, if you have configured your reverse proxy to always remove/overwrite `X-Forwarded-For` and/or `X-Real-IP` headers
so that if you execute this _(same)_ request:
```bash
curl -X POST https://example.com/login -H "X-Forwarded-For: 1.2.3.4, 11.22.33.44"
```

And your server behind the load balancer obtain this:
```
X-Forwarded-For: <actual client IP>
```

Then, you can enable `TrustForwardHeader` in your limiter option.

### Custom header

Many CDN and Cloud providers add a custom header to define the client IP. Like for example, this non exhaustive list:

* `Fastly-Client-IP` from Fastly
* `CF-Connecting-IP` from Cloudflare
* `X-Azure-ClientIP` from Azure

You can use these headers using `ClientIPHeader` in your limiter option.

### None of the above

If none of the above solution are working, please use a custom `KeyGetter` in your middleware.

You can use this excellent article to help you define the best strategy depending on your network topology and your security need:
https://adam-p.ca/blog/2022/03/x-forwarded-for/

If you have any idea/suggestions on how we could simplify this steps, don't hesitate to raise an issue.
We would like some feedback on how we could implement this steps in the Limiter API.

Thank you.

## Why Yet Another Package

You could ask us: why yet another rate limit package?
Expand Down
4 changes: 3 additions & 1 deletion drivers/middleware/stdlib/middleware.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ type Middleware struct {
Limiter *limiter.Limiter
OnError ErrorHandler
OnLimitReached LimitReachedHandler
KeyGetter KeyGetter
ExcludedKey func(string) bool
}

Expand All @@ -21,6 +22,7 @@ func NewMiddleware(limiter *limiter.Limiter, options ...Option) *Middleware {
Limiter: limiter,
OnError: DefaultErrorHandler,
OnLimitReached: DefaultLimitReachedHandler,
KeyGetter: DefaultKeyGetter(limiter),
ExcludedKey: nil,
}

Expand All @@ -34,7 +36,7 @@ func NewMiddleware(limiter *limiter.Limiter, options ...Option) *Middleware {
// Handler handles a HTTP request.
func (middleware *Middleware) Handler(h http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
key := middleware.Limiter.GetIPKey(r)
key := middleware.KeyGetter(r)
if middleware.ExcludedKey != nil && middleware.ExcludedKey(key) {
h.ServeHTTP(w, r)
return
Expand Down
20 changes: 20 additions & 0 deletions drivers/middleware/stdlib/options.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@ package stdlib

import (
"net/http"

"github.com/ulule/limiter/v3"
)

// Option is used to define Middleware configuration.
Expand Down Expand Up @@ -45,6 +47,24 @@ func DefaultLimitReachedHandler(w http.ResponseWriter, r *http.Request) {
http.Error(w, "Limit exceeded", http.StatusTooManyRequests)
}

// KeyGetter will define the rate limiter key given the gin Context.
type KeyGetter func(r *http.Request) string

// WithKeyGetter will configure the Middleware to use the given KeyGetter.
func WithKeyGetter(handler KeyGetter) Option {
return option(func(middleware *Middleware) {
middleware.KeyGetter = handler
})
}

// DefaultKeyGetter is the default KeyGetter used by a new Middleware.
// It returns the Client IP address.
func DefaultKeyGetter(limiter *limiter.Limiter) func(r *http.Request) string {
return func(r *http.Request) string {
return limiter.GetIPKey(r)
}
}

// WithExcludedKey will configure the Middleware to ignore key(s) using the given function.
func WithExcludedKey(handler func(string) bool) Option {
return option(func(middleware *Middleware) {
Expand Down
87 changes: 76 additions & 11 deletions network.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,35 +14,59 @@ var (
)

// GetIP returns IP address from request.
// If options is defined and either TrustForwardHeader is true or ClientIPHeader is defined,
// it will lookup IP in HTTP headers.
// Please be advised that using this option could be insecure (ie: spoofed) if your reverse
// proxy is not configured properly to forward a trustworthy client IP.
// Please read the section "Limiter behind a reverse proxy" in the README for further information.
func (limiter *Limiter) GetIP(r *http.Request) net.IP {
return GetIP(r, limiter.Options)
}

// GetIPWithMask returns IP address from request by applying a mask.
// If options is defined and either TrustForwardHeader is true or ClientIPHeader is defined,
// it will lookup IP in HTTP headers.
// Please be advised that using this option could be insecure (ie: spoofed) if your reverse
// proxy is not configured properly to forward a trustworthy client IP.
// Please read the section "Limiter behind a reverse proxy" in the README for further information.
func (limiter *Limiter) GetIPWithMask(r *http.Request) net.IP {
return GetIPWithMask(r, limiter.Options)
}

// GetIPKey extracts IP from request and returns hashed IP to use as store key.
// If options is defined and either TrustForwardHeader is true or ClientIPHeader is defined,
// it will lookup IP in HTTP headers.
// Please be advised that using this option could be insecure (ie: spoofed) if your reverse
// proxy is not configured properly to forward a trustworthy client IP.
// Please read the section "Limiter behind a reverse proxy" in the README for further information.
func (limiter *Limiter) GetIPKey(r *http.Request) string {
return limiter.GetIPWithMask(r).String()
}

// GetIP returns IP address from request.
// If options is defined and TrustForwardHeader is true, it will lookup IP in
// X-Forwarded-For and X-Real-IP headers.
// If options is defined and either TrustForwardHeader is true or ClientIPHeader is defined,
// it will lookup IP in HTTP headers.
// Please be advised that using this option could be insecure (ie: spoofed) if your reverse
// proxy is not configured properly to forward a trustworthy client IP.
// Please read the section "Limiter behind a reverse proxy" in the README for further information.
func GetIP(r *http.Request, options ...Options) net.IP {
if len(options) >= 1 && options[0].TrustForwardHeader {
ip := r.Header.Get("X-Forwarded-For")
if ip != "" {
parts := strings.SplitN(ip, ",", 2)
part := strings.TrimSpace(parts[0])
return net.ParseIP(part)
if len(options) >= 1 {
if options[0].ClientIPHeader != "" {
ip := getIPFromHeader(r, options[0].ClientIPHeader)
if ip != nil {
return ip
}
}
if options[0].TrustForwardHeader {
ip := getIPFromXFFHeader(r)
if ip != nil {
return ip
}

ip = strings.TrimSpace(r.Header.Get("X-Real-IP"))
if ip != "" {
return net.ParseIP(ip)
ip = getIPFromHeader(r, "X-Real-IP")
if ip != nil {
return ip
}
}
}

Expand All @@ -56,6 +80,11 @@ func GetIP(r *http.Request, options ...Options) net.IP {
}

// GetIPWithMask returns IP address from request by applying a mask.
// If options is defined and either TrustForwardHeader is true or ClientIPHeader is defined,
// it will lookup IP in HTTP headers.
// Please be advised that using this option could be insecure (ie: spoofed) if your reverse
// proxy is not configured properly to forward a trustworthy client IP.
// Please read the section "Limiter behind a reverse proxy" in the README for further information.
func GetIPWithMask(r *http.Request, options ...Options) net.IP {
if len(options) == 0 {
return GetIP(r)
Expand All @@ -70,3 +99,39 @@ func GetIPWithMask(r *http.Request, options ...Options) net.IP {
}
return ip
}

func getIPFromXFFHeader(r *http.Request) net.IP {
headers := r.Header.Values("X-Forwarded-For")
if len(headers) == 0 {
return nil
}

parts := []string{}
for _, header := range headers {
parts = append(parts, strings.Split(header, ",")...)
}

for i := range parts {
part := strings.TrimSpace(parts[i])
ip := net.ParseIP(part)
if ip != nil {
return ip
}
}

return nil
}

func getIPFromHeader(r *http.Request, name string) net.IP {
header := strings.TrimSpace(r.Header.Get(name))
if header == "" {
return nil
}

ip := net.ParseIP(header)
if ip != nil {
return ip
}

return nil
}
22 changes: 22 additions & 0 deletions options.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,16 @@ type Options struct {
// IPv6Mask defines the mask used to obtain a IPv6 address.
IPv6Mask net.IPMask
// TrustForwardHeader enable parsing of X-Real-IP and X-Forwarded-For headers to obtain user IP.
// Please be advised that using this option could be insecure (ie: spoofed) if your reverse
// proxy is not configured properly to forward a trustworthy client IP.
// Please read the section "Limiter behind a reverse proxy" in the README for further information.
TrustForwardHeader bool
// ClientIPHeader defines a custom header (likely defined by your CDN or Cloud provider) to obtain user IP.
// If configured, this option will override "TrustForwardHeader" option.
// Please be advised that using this option could be insecure (ie: spoofed) if your reverse
// proxy is not configured properly to forward a trustworthy client IP.
// Please read the section "Limiter behind a reverse proxy" in the README for further information.
ClientIPHeader string
}

// WithIPv4Mask will configure the limiter to use given mask for IPv4 address.
Expand All @@ -32,8 +41,21 @@ func WithIPv6Mask(mask net.IPMask) Option {
}

// WithTrustForwardHeader will configure the limiter to trust X-Real-IP and X-Forwarded-For headers.
// Please be advised that using this option could be insecure (ie: spoofed) if your reverse
// proxy is not configured properly to forward a trustworthy client IP.
// Please read the section "Limiter behind a reverse proxy" in the README for further information.
func WithTrustForwardHeader(enable bool) Option {
return func(o *Options) {
o.TrustForwardHeader = enable
}
}

// WithClientIPHeader will configure the limiter to use a custom header to obtain user IP.
// Please be advised that using this option could be insecure (ie: spoofed) if your reverse
// proxy is not configured properly to forward a trustworthy client IP.
// Please read the section "Limiter behind a reverse proxy" in the README for further information.
func WithClientIPHeader(header string) Option {
return func(o *Options) {
o.ClientIPHeader = header
}
}