L7 · security · reviewed
Reverse proxies
HTTP reverse proxying and request forwarding
An intermediary the client cannot see, which routes by hostname and terminates TLS — and in doing so replaces every fact the application had about who is calling it.
Presenter modeEmbed this figure
Why it exists
An application listening on a port can do one thing well. Putting a proxy in front of it separates the concerns that have nothing to do with the application from the application: which hostname maps where, which certificate to present, what to cache, how many connections to accept, and what to do when a backend stops answering. IntermediariesRFC 9110 · INTERNET STANDARD · June 2022
It also solves a counting problem. Public addresses and port 443 are scarce; applications are not. One proxy on one address can front dozens of backends, because HTTP carries the hostname in the request and TLS carries it in the handshake, so the intermediary can tell them apart. Host and :authorityRFC 9110 · INTERNET STANDARD · June 2022
A forward proxy and a reverse proxy are the same machinery pointed in opposite directions. A forward proxy is configured in the client and acts for the client; a reverse proxy is deployed by the operator and acts as the origin server. Which one you have determines who is being protected from whom.
The cost is that the application loses its connection. Every fact it used to read from the socket — who is calling, over what scheme, to what hostname — now has to be forwarded to it in a header, and every one of those headers is an assertion by a machine rather than an observation.
One address, several applications
Three hostnames resolve to one proxy. It reads the Host header, picks a backend, and hands the request on — replacing the client’s address with its own in the process.
Three names — app, api and docs — all resolve to the same address. There is one public endpoint and the client has no way to know there are three applications behind it. Client · 203.0.113.44. Reverse proxy · 198.51.100.7: Listening 198.51.100.7:443, Backends 3. app · 192.168.4.40:8080. api · 192.168.4.41:9000. docs · 192.168.4.42:80.
- Link
- Blocking
- Packet in flight
- Discarded
- Emphasis
Text equivalent of this diagram
| Element | Kind | State |
|---|---|---|
| Client · 203.0.113.44 | host | — |
| Reverse proxy · 198.51.100.7 | firewall | Listening: 198.51.100.7:443 · Backends: 3 |
| app · 192.168.4.40:8080 | host | — |
| api · 192.168.4.41:9000 | host | — |
| docs · 192.168.4.42:80 | host | — |
| Client · 203.0.113.44 — Reverse proxy · 198.51.100.7 | link | up · https · 443 |
| Reverse proxy · 198.51.100.7 — app · 192.168.4.40:8080 | link | up · http · 8080 |
| Reverse proxy · 198.51.100.7 — api · 192.168.4.41:9000 | link | standby · http · 9000 |
| Reverse proxy · 198.51.100.7 — docs · 192.168.4.42:80 | link | standby · http · 80 |
Three names — app, api and docs — all resolve to the same address. There is one public endpoint and the client has no way to know there are three applications behind it.
This is the distinction from a forward proxy in one sentence: a forward proxy is chosen by the client and configured in the client; a reverse proxy is chosen by the operator and the client is not told it exists.
What changed
- Reverse proxy · 198.51.100.7: Listening → 198.51.100.7:443
- Reverse proxy · 198.51.100.7: Backends → 3
How it works
A request arrives on the proxy’s listener. If TLS terminates there, the proxy has already used the SNI name from the handshake to choose a certificate; it then decrypts, reads the Host header, and matches it against its configured backends. Server Name IndicationRFC 6066 · PROPOSED STANDARD · January 2011
The proxy opens a separate connection to the chosen backend and forwards the request on it. There are now two connections, and the backend’s peer is the proxy — which is the correct description of what happened, not a bug to be worked around. Message ForwardingRFC 9110 · INTERNET STANDARD · June 2022
To give the backend what it lost, the proxy appends forwarding headers. X-Forwarded-For accumulates a comma-separated list in which each hop adds the address it received the connection from; X-Forwarded-Proto and X-Forwarded-Host carry the scheme and hostname the client used. Forwarded ForRFC 7239 · PROPOSED STANDARD · June 2014
RFC 7239 standardised all of that into one header, Forwarded, with for, by, host and proto parameters. It arrived after the X- versions were already universal, so most deployments emit both and read whichever is configured — and the standard header solves the syntax problem, not the trust problem. Forwarded HTTP Header FieldRFC 7239 · PROPOSED STANDARD · June 2014
Some headers do not survive a hop by design. Connection, and anything it names, applies to a single connection only and must be consumed rather than forwarded — which is why an upgrade to WebSocket has to be arranged explicitly at every proxy in the path instead of passing through on its own. ConnectionRFC 9110 · INTERNET STANDARD · June 2022
The one standard header describing the path is Via, which each intermediary appends to. It is genuinely specified, widely emitted, and almost never read — the de facto X- headers won because they answer the question people actually have. ViaRFC 9110 · INTERNET STANDARD · June 2022
Try it
Trace the hops
Add hops, move where TLS ends, and change how many you trust — then read which address the application ends up believing.
- Client
203.0.113.44 - Reverse proxy
192.168.4.210TLS ends here - Origin
192.168.4.40:8080
The application reads 203.0.113.44 — the real client.
| Header | Value |
|---|---|
| Host | netknife.lab.alexflux.comPreserved end to end unless a hop is configured to rewrite it. Rewriting it is what breaks redirects and cookie domains. |
| X-Forwarded-For | 203.0.113.44Leftmost is the original client only if no client supplied one first. Every entry to the right of that was appended by a hop, and each hop can only vouch for the one it added. |
| X-Forwarded-Proto | httpsWhat the client used, not what the backend hop used. Without it an application behind a terminating proxy generates http:// redirects. |
| X-Forwarded-Host | netknife.lab.alexflux.comThe hostname the client asked for, in case a hop rewrote Host. |
| Via | 1.1 reverseRFC 9110. The only one of these that is actually a standard request header for proxies, and the one nobody reads. |
- Reverse proxy
- Chosen by the operator, invisible to the client. It maps a hostname to a backend, terminates TLS, and is where certificates and virtual hosts live.
On the wire
Constructed examples, encoded from the field table below them — not captured traffic.
- TCP and TLS
- Almost always port 443 inside TLS, with the protocol agreed by ALPN during the handshake rather than by an upgrade. RFC 9113
- Frame header
- Length, type, flags, and the stream this frame belongs to. RFC 9113
- Frame payload
- Format determined by the type: headers, data, settings, a window update, a reset. RFC 9113
Configure it
server { listen 443 ssl default_server; server_name _; ssl_certificate /etc/nginx/tls/default.crt; ssl_certificate_key /etc/nginx/tls/default.key; return 421; }A default that refuses. Without it the first server block becomes the default, so any hostname pointed at this address routes into whichever application happens to be configured first.
Common mistake: Omitting this entirely. It is how a staging application ends up serving requests for a name nobody configured.
RFC 9110 §15.5.20
server { listen 443 ssl; http2 on; server_name app.example.com; ssl_certificate /etc/nginx/tls/app.crt; ssl_certificate_key /etc/nginx/tls/app.key;One block per hostname. The name here is matched against SNI to pick the certificate and against Host to pick the route — two independent lookups that normally agree.
RFC 6066 §3
set_real_ip_from 198.51.100.0/24; real_ip_header X-Forwarded-For; real_ip_recursive on;The trust boundary, stated as addresses rather than as a hop count. nginx walks X-Forwarded-For from the right and keeps discarding while the entry belongs to a trusted range, so the first untrusted address it reaches is the client.
Common mistake: Setting `real_ip_header` without `set_real_ip_from`, or listing 0.0.0.0/0 as trusted. Either one makes nginx believe the leftmost entry, which is the one a client can write.
RFC 7239 §8.1
location / { proxy_pass http://127.0.0.1:8080; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host;nginx forwards no headers you do not name, so all four are deliberate. `$proxy_add_x_forwarded_for` appends the peer address to whatever arrived — append, not replace, which is why the trust boundary above has to exist.
RFC 7239 §5.2
proxy_set_header X-Forwarded-Proto $scheme;Without this the application sees plain HTTP, infers the scheme from its own connection, and emits http:// redirects that the proxy sends straight back to https.
Common mistake: Setting it and stopping there. Most frameworks ignore the header until they are told which proxies may set it — the proxy change and the application change are two separate things.
RFC 9110 §15.4
proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $connection_upgrade; proxy_read_timeout 3600s; } }Connection is hop-by-hop: nginx consumes it rather than forwarding it, so a WebSocket upgrade has to be passed on explicitly. The timeout matters too, or an idle but healthy socket is closed as dead.
RFC 9110 §7.6.1
Verify
nginx -t- Configuration parses before reloading.
nginx -T | grep -E "set_real_ip_from|real_ip_header"- The trust boundary as it is actually loaded, not as the file reads.
curl -H "X-Forwarded-For: 198.18.0.1" -sv https://app.example.com/- The application should log the real client, not 198.18.0.1.
openssl s_client -connect app.example.com:443 -servername app.example.com- Which certificate the SNI name selects.
Caveats
- `proxy_set_header` inherited from an outer block is replaced wholesale as soon as one is declared in an inner block, not merged with it.
- `$connection_upgrade` is not built in; it comes from a `map $http_upgrade` block that has to be declared at http level.
- nginx does not emit Via by default, so a chain of nginx hops leaves no standard record of the path.
When it breaks
Symptom first, because that is what you have when it happens.
Rate limits are bypassed, allow lists admit the wrong clients, or logs show addresses that cannot be real.
Narrow it down
- Find where the application reads the client address, and whether it takes the first or last entry.
- Send a request with your own X-Forwarded-For and see what gets logged.
- Count the hops actually in the path and compare that to the application’s trusted-proxy configuration.
Cause
The application reads the leftmost entry of X-Forwarded-For. Proxies append rather than replace, so a header the client supplied ends up in exactly the position being trusted.
Fix
Count from the right by the number of hops you operate, or have the outermost hop replace the header. Configure the trusted-proxy list in the framework rather than parsing the header by hand.
Header Validity and IntegrityRFC 7239 · PROPOSED STANDARD · June 2014The browser reports too many redirects. Requests alternate between http and https, and the application works when reached directly.
Narrow it down
- Follow the redirects with `curl -IL` and read each Location header.
- Check whether the proxy sets X-Forwarded-Proto and whether the application is configured to trust it.
- Look for a proxy-side http-to-https redirect combined with an application-side https-to-http one.
Cause
The proxy terminates TLS and forwards plain HTTP. The application infers the scheme from its own connection, generates an http:// redirect, and the proxy sends it back to https.
Fix
Set X-Forwarded-Proto and enable the framework’s trusted-proxy handling for it. Setting the header alone is not enough — most frameworks ignore it until told which proxies may set it.
Redirection 3xxRFC 9110 · INTERNET STANDARD · June 2022WebSocket connections fail with 400 or 426, or connect and close immediately. Ordinary requests to the same application work.
Narrow it down
- Confirm the proxy forwards the Upgrade and Connection headers explicitly.
- Check the HTTP version between proxy and backend — an upgrade needs HTTP/1.1.
- Look at the proxy read timeout, which will close an idle but healthy socket.
Cause
Connection is hop-by-hop and is consumed rather than forwarded, so an upgrade does not pass through a proxy unless the proxy is configured to pass it.
Fix
Forward Upgrade and Connection for the affected routes, ensure HTTP/1.1 upstream, and raise the read timeout for long-lived connections.
ConnectionRFC 9110 · INTERNET STANDARD · June 2022A TLS name mismatch for one hostname while others on the same address are fine, or the default site’s certificate is served.
Narrow it down
- Request the certificate with an explicit SNI name and compare it to the one without.
- Check which server block is the default for the listener.
- Confirm the certificate covers the exact name, including whether the wildcard depth matches.
Cause
No server block matched the SNI name, so the listener’s default certificate was presented. One address serving many names depends entirely on that match.
Fix
Add the name to a server block, or issue a certificate covering it. A wildcard covers one label, so it does not cover a further subdomain.
Server Name IndicationRFC 6066 · PROPOSED STANDARD · January 2011A user sees another user’s content, or a logged-out visitor sees a personalised page.
Narrow it down
- Read the Cache-Control and Vary headers on the affected response.
- Determine the cache key and whether it includes what made the response different.
- Check whether the application marks authenticated responses private.
Cause
A shared cache stored a response that varied on something absent from the cache key — a cookie, an authorization header, or a header the application read but did not declare.
Fix
Mark user-specific responses private or no-store, and Vary on whatever the response genuinely depends on. Declaring it is what makes the cache key correct.
Calculating Cache Keys with the Vary Header FieldRFC 9111 · INTERNET STANDARD · June 2022Intermittent 502 or 504 on long requests, or connections dropped mid-response under load.
Narrow it down
- Compare the proxy’s read timeout with the backend’s own request timeout.
- Compare keep-alive idle timeouts at each tier — the shorter one closes first.
- Check whether the proxy retried a non-idempotent request after a timeout.
Cause
Timeouts configured independently at each tier. The shortest one governs, and a proxy that gives up first turns a slow success into a gateway error.
Fix
Set the proxy timeout longer than the backend’s so the backend’s own error surfaces, and make backend keep-alive idle timeouts longer than the proxy’s so the proxy closes idle connections rather than discovering closed ones.
502 Bad GatewayRFC 9110 · INTERNET STANDARD · June 2022
Design notes
Never read the leftmost entry of X-Forwarded-For as the client. Count from the right by the number of hops you operate: those entries were written by your own infrastructure, and everything further left was written by something you do not control — including, at the far end, the client. Header Validity and IntegrityRFC 7239 · PROPOSED STANDARD · June 2014
Better still, have the outermost trusted hop replace the header instead of appending to it. It is the only hop that observes the client address first-hand, so it is the only one that can discard whatever the client sent — and then there is no depth to count and no count to get wrong when a tier is added.
Set X-Forwarded-Proto and configure the application to trust it. An application that does not know the client used HTTPS will emit http:// redirects, set cookies without the Secure attribute, and build absolute URLs that downgrade — all from one missing header.
Preserve Host unless you have a specific reason to rewrite it. Rewriting it changes what the application thinks it is called, which changes redirect targets, cookie domains and any absolute URL it generates. Host and :authorityRFC 9110 · INTERNET STANDARD · June 2022
Give an unmatched Host a default backend that reveals nothing. A proxy that falls through to the first configured server turns any name pointed at its address into a route into that application.
Two hops that disagree about what a request means is a request-smuggling condition. Keep the proxy and the backend on the same HTTP implementation family where you can, reject ambiguous framing rather than normalising it, and do not put a permissive parser behind a strict one. Risks of IntermediariesRFC 9110 · INTERNET STANDARD · June 2022
A shared cache in front of an application caches for everyone. Vary on whatever actually distinguishes responses, and mark anything user-specific private — a cache key that omits the thing that made the response different serves one user’s page to another. Calculating Cache Keys with the Vary Header FieldRFC 9111 · INTERNET STANDARD · June 2022
Misconceptions
- “The first entry in X-Forwarded-For is the client address.”
- It is the first entry that was in the header, which is not the same thing. Proxies append, so if the client sent the header itself, its value sits leftmost — in exactly the position that gets trusted. Count from the right by the hops you operate. Header Validity and IntegrityRFC 7239 · PROPOSED STANDARD · June 2014
- “Using the standard Forwarded header instead of X-Forwarded-For fixes the spoofing problem.”
- It fixes the syntax problem. Forwarded is still a request header a client can set and still a list each hop appends to, so it needs the same trust boundary and the same counting. The RFC says so in its own security considerations. Header Validity and IntegrityRFC 7239 · PROPOSED STANDARD · June 2014
- “A reverse proxy is transparent to the application behind it.”
- It replaces the connection. The peer address, the scheme, the HTTP version and often the hostname are all now the proxy’s rather than the client’s, and each one has to be forwarded deliberately. Applications that were written without a proxy in mind break in exactly these four places.
- “Setting X-Forwarded-Proto on the proxy makes the application generate https URLs.”
- Most frameworks ignore the header until they are told which proxies may set it, because believing it unconditionally would let any client claim any scheme. The proxy setting and the application trusting it are two separate changes.
- “TLS pass-through is the same thing with better security.”
- It removes every reason the proxy was there. Without decrypting, the proxy cannot route on Host, cache, compress, rewrite, or add a forwarding header — so the application also loses the client address, and getting it back needs the PROXY protocol underneath rather than a header inside.
- “A load balancer and a reverse proxy are different devices.”
- A layer-7 load balancer is a reverse proxy that also distributes across a pool. A layer-4 one forwards without reading the request, so it appends no headers at all — which matters if an application counts hops, because adding an L4 tier changes nothing in the header and everything about the count.
More walkthroughs
A rate limit that anyone can walk pastfailure
The application reads the leftmost entry of X-Forwarded-For to identify the client. A client that sends its own X-Forwarded-For chooses what the application believes.
The application rate-limits by client address, and it reads that address from the first entry in X-Forwarded-For. That sounds right: the leftmost entry is the original client. Client · 203.0.113.44. CDN edge · 198.51.100.7. Reverse proxy · 192.168.4.5. Application: Client from XFF[0], Limit 10 requests/minute.
- Link
- Blocking
- Packet in flight
- Discarded
- Emphasis
Text equivalent of this diagram
| Element | Kind | State |
|---|---|---|
| Client · 203.0.113.44 | host | — |
| CDN edge · 198.51.100.7 | cloud | — |
| Reverse proxy · 192.168.4.5 | firewall | — |
| Application | host | Client from: XFF[0] · Limit: 10 requests/minute |
| Client · 203.0.113.44 — CDN edge · 198.51.100.7 | link | up |
| CDN edge · 198.51.100.7 — Reverse proxy · 192.168.4.5 | link | up |
| Reverse proxy · 192.168.4.5 — Application | link | up |
The application rate-limits by client address, and it reads that address from the first entry in X-Forwarded-For. That sounds right: the leftmost entry is the original client.
The reasoning is sound and the conclusion is wrong, which is what makes this durable. The leftmost entry is the original client only if the original client did not send the header itself.
What changed
- Application: Client from → XFF[0]
- Application: Limit → 10 requests/minute
Where TLS should enddesign-choice
Terminate at the edge, re-encrypt to the backend, or pass through untouched. Each choice decides what the proxy can do and what the application can know.
Terminating at the proxy is the usual choice. It decrypts, so it can route on Host, cache, compress, rewrite and inspect — every layer-7 feature depends on being able to read the request. Client. Reverse proxy: TLS terminated here. Application.
- Link
- Blocking
- Packet in flight
- Discarded
- Emphasis
Text equivalent of this diagram
| Element | Kind | State |
|---|---|---|
| Client | host | — |
| Reverse proxy | firewall | TLS: terminated here |
| Application | host | — |
| Client — Reverse proxy | link | up · https |
| Reverse proxy — Application | link | up · http |
Terminating at the proxy is the usual choice. It decrypts, so it can route on Host, cache, compress, rewrite and inspect — every layer-7 feature depends on being able to read the request.
One certificate, in one place, renewed by one process. That operational simplification is usually the real reason, and it is a good one.
What changed
- Reverse proxy: TLS → terminated here
- Emphasis: Encrypted
- Emphasis: Cleartext
Terms
- Reverse proxy
- An intermediary deployed by the operator that answers as the origin server. The client is not told it exists, which is what distinguishes it from a forward proxy.
- Forward proxy
- An intermediary chosen and configured by the client, which asks it to fetch on its behalf. Same machinery as a reverse proxy, pointed the other way.
- TLS termination
- Decrypting at the proxy rather than the backend. It is what makes routing, caching and header rewriting possible, and it is why the backend has to be told the client used HTTPS.
- X-Forwarded-For
- A comma-separated list in which each hop appends the address it received the connection from. Never standardised; read it from the right by the hops you operate, because the left end is whatever the client sent.
- SNI
- Server Name Indication: the hostname a client names in the TLS handshake, in the clear, so one address can present the right certificate for many names.