TCP — Transmission Control Protocol

Why it exists

IP delivers packets that may be lost, duplicated or reordered. Almost no application wants to handle that, so TCP presents a byte stream instead: what is written at one end appears at the other, in order, once, or the connection fails and says so.

The harder problem is not reliability but rate. A sender has no way to measure the path, so TCP infers capacity from what comes back — and the inference is a separate algorithm that the protocol deliberately does not specify.

Almost every TCP performance problem is one of two things: a window too small for the path, or an inference broken by something the sender cannot see. Both are visible in a capture of the first three packets.

Three packets, and everything you will ever negotiate

The handshake synchronises sequence numbers. It is also the only chance either end gets to agree on the options that decide how the connection performs.

Link
Blocking
Packet in flight
Discarded
Emphasis

Select a device to read its state. Arrow keys walk the topology.

Text equivalent of this diagram

Devices and links at this step
Element	Kind	State
Client	host	State: SYN-SENT
Firewall	firewall	—
Server	host	—
Client — Firewall	link	up
Firewall — Server	link	up

1 / 6

The client sends SYN with an initial sequence number, chosen unpredictably so an off-path attacker cannot guess where the stream starts.

What changed

Client: State → SYN-SENT
SYN: Client → Server

How it works

The three-way handshake synchronises sequence numbers and negotiates every option the connection will ever have — maximum segment size, window scale, SACK, timestamps. None can be added later, so a middlebox that strips one from the SYN sets the connection’s performance for its whole life.

Two windows govern how much may be in flight. The receive window is advertised in every ACK; the congestion window is the sender’s private estimate and appears nowhere. The sender uses the smaller.

Throughput is therefore window divided by round-trip time, not link rate. Sixty-four kilobytes on a 100 ms path is about five megabits per second regardless of how fast the link is, which is what window scaling exists to fix.

Segment size comes from path MTU discovery: send with Don’t Fragment set and learn from the ICMP that comes back. Filtering that ICMP does not make the network safer; it makes the mechanism fail silently.

On the wire

Constructed examples, encoded from the field table below them — not captured traffic.

TCP: Options, if any, follow this header and are covered by the data offset. RFC 9293

Configure it

```
ss -ti dst <peer>
```
Before tuning: this prints the congestion window, the round-trip time and the retransmission count for a live socket. If the congestion window is the limit, no buffer setting will help.
Common mistake: Raising buffers first. The receive window is the ceiling and the congestion window is usually the floor; changing the ceiling when the floor is the problem produces no measurable difference and a lot of confidence.
RFC 9293 §3.8.6
```
sysctl -w net.ipv4.tcp_window_scaling=1
```
On by default everywhere, and worth confirming rather than assuming — a middlebox stripping the option produces the same symptom as having it disabled, and only a capture tells them apart.
RFC 7323 §2.2
```
sysctl -w net.ipv4.tcp_rmem="4096 131072 16777216"
sysctl -w net.ipv4.tcp_wmem="4096 65536 16777216"
```
Minimum, default and maximum. The maximum should cover bandwidth times delay — 16 MB carries a gigabit at 100 ms — and the default should not, because every socket pays for the default.
```
sysctl -w net.ipv4.tcp_congestion_control=bbr
sysctl -w net.core.default_qdisc=fq
```
BBR models bandwidth and round-trip time instead of treating loss as congestion, which is what makes it behave differently on paths that lose packets for other reasons. It expects fair queueing beneath it, so the two settings go together.
RFC 9438 §1
```
sysctl -w net.ipv4.tcp_mtu_probing=1
```
The host-side answer to a black hole: when retransmissions of a full-size segment go unanswered, try a smaller one instead of the same one forever. It is a workaround for a path that eats ICMP, not a substitute for fixing it.
RFC 9293 §3.7.2
```
sysctl -w net.ipv4.tcp_syncookies=1
```
Defers state allocation until the third packet arrives, which removes the asymmetry a SYN flood exploits. It is on by default on most distributions and worth confirming on anything internet-facing.
RFC 9293 §3.5

Verify

ss -ti: Congestion window, round-trip time and retransmissions per socket.
ss -s: Socket counts by state — where TIME_WAIT exhaustion shows.
sysctl net.ipv4.tcp_congestion_control: Which algorithm is actually in use.
nstat -az TcpExtTCPTimeouts TcpRetransSegs: Whether loss or timeout is the limiting factor.

Caveats

The receive window is a ceiling. If loss is holding the congestion window down, raising it changes nothing.
Window scaling can only be negotiated in the handshake; a stripped option looks identical to a disabled setting.
MTU probing works around a filtered ICMP path; it does not fix one.

When it breaks

Symptom first, because that is what you have when it happens.

A connection establishes, transfers a little data, then stalls until it times out. Small requests work.
Narrow it down
1. Find the largest packet that gets through with the do-not-fragment bit set.
2. Look for retransmissions of full-size segments with nothing coming back.
3. Check every MTU in the path, including tunnels and tagged links.
Cause
Path MTU discovery is broken. Something is dropping the full-size segment and the ICMP that should report it is being filtered, so the sender has no evidence the size is the problem.
Fix
Stop filtering ICMP type 3 code 4, or clamp the MSS on the tunnel so the sender never builds a segment that cannot fit.
A server accumulates thousands of sockets in TIME_WAIT and eventually refuses new connections.
Narrow it down
1. Count sockets by state and confirm TIME_WAIT dominates.
2. Establish which side closes first — the side that sends the first FIN owns the TIME_WAIT.
3. Check whether the application opens a new connection per request instead of reusing one.
Cause
TIME_WAIT lasts twice the maximum segment lifetime so a delayed duplicate cannot be mistaken for data on a new connection with the same four-tuple. It is correct behaviour with a wrong caller.
Fix
Have the client close first, or keep connections alive and reuse them. Lowering the timeout trades a real correctness guarantee for headroom.
Throughput sits far below the link rate on a long path, and is the same number no matter how much bandwidth is added.
Narrow it down
1. Multiply the advertised receive window by one over the round-trip time and compare against what you see.
2. Check whether window scaling was negotiated on the SYN.
3. Look for a middlebox stripping unknown TCP options.
Cause
The window caps throughput at window ÷ RTT regardless of capacity. Without scaling the ceiling is 65 535 octets, which is nothing on a long path.
Fix
Enable window scaling at both ends, and find whatever is stripping the option on the SYN — it can only be negotiated during the handshake.
A server reports a flood of connections in SYN_RECV and legitimate clients cannot connect.
Narrow it down
1. Count sockets in SYN_RECV against the accept backlog.
2. Check whether the source addresses are plausible or scattered.
3. Confirm whether SYN cookies are enabled.
Cause
The handshake commits state on the server after the first packet. An attacker who never sends the third packet costs themselves nothing and the server a socket.
Fix
Enable SYN cookies so no state is allocated until the ACK arrives, and rate limit at the edge.
A request-response application pauses about forty milliseconds per transaction, on a network with a sub-millisecond round trip.
Narrow it down
1. Check whether the delay is a constant rather than proportional to size — a fixed 40 ms is the signature.
2. Look at whether the application writes a request in two calls, a header then a body.
3. Confirm whether the socket has Nagle disabled.
Cause
Nagle holds a small segment until the previous one is acknowledged; delayed acknowledgement holds the acknowledgement waiting for data to piggyback on. Each is waiting for the other, and the deadlock breaks only when the delayed-ACK timer fires.
Fix
Write the request in one call so there is no second small segment, or disable Nagle on the socket. Both work; only the first also reduces syscalls.

Design notes

Capture the SYN on both sides of any middlebox before tuning anything. Which options survive the path determines the ceiling, and no amount of kernel tuning raises a ceiling set by a stripped option.

Clamp the MSS on every tunnel interface. It is the fix that does not depend on ICMP surviving a path you do not control, and the alternative is a class of failure where the connection establishes and then transfers nothing.

Size buffers from bandwidth times delay, and check the congestion window before raising them. If loss is holding the congestion window down, a larger receive window changes nothing at all.

Have clients close connections rather than servers, or reuse connections. TIME_WAIT belongs to whichever side closes first, and it is correct behaviour that becomes a capacity problem when the busy side owns it.

Misconceptions

“TCP guarantees delivery.”: It guarantees that data arrives in order and without duplication, or that the connection fails. If the path stays broken the connection resets — nothing is delivered and the application is told.
“Congestion control is part of the TCP header.”: Nothing in the header names an algorithm. Reno, CUBIC and BBR all produce ordinary TCP on the wire; they differ only in when the sender chooses to transmit. Two ends can run different algorithms and never know.
“A larger receive window always means faster transfer.”: It raises the ceiling; it does not fill it. The sender is limited by the smaller of the receive window and the congestion window, and the congestion window is set by loss and delay, not by configuration.
“Slow start is the slow part of a transfer.”: It doubles the congestion window every round trip — the fastest growth TCP ever does. What is slow is congestion avoidance afterwards, adding one segment per round trip, which is why a single loss on a long path costs seconds to recover from.

More walkthroughs

Two windows, and only one of them is on the wiredesign-choice

The receiver advertises what it can hold. The sender guesses what the network can carry. The smaller of the two decides everything.

Link
Blocking
Packet in flight
Discarded
Emphasis

Select a device to read its state. Arrow keys walk the topology.

Text equivalent of this diagram

Devices and links at this step
Element	Kind	State
Sender	host	—
Path · 100 ms	cloud	—
Receiver	host	rwnd: 4 MB (scaled)
Sender — Path · 100 ms	link	up
Path · 100 ms — Receiver	link	up

1 / 6

The receive window is advertised in every ACK. It says how much the receiver can buffer, and it is the only one of the two windows that appears in the header.

What changed

Receiver: rwnd → 4 MB (scaled)
Emphasis on Path · 100 ms ↔ Receiver

The connection that opens and then goes quietfailure

The handshake succeeds because SYNs are small. The first full-size segment is discarded, and the message that would explain it is filtered.

Link
Blocking
Packet in flight
Discarded
Emphasis

Select a device to read its state. Arrow keys walk the topology.

Text equivalent of this diagram

Devices and links at this step
Element	Kind	State
Client · MTU 1500	host	State: ESTABLISHED
Tunnel · MTU 1400	router	—
Server	host	—
Client · MTU 1500 — Tunnel · MTU 1400	link	up
Tunnel · MTU 1400 — Server	link	up

1 / 6

The handshake completes normally. SYN, SYN-ACK and ACK are all small, so nothing on the path objects.

That is why the symptom is so misleading. The connection is genuinely established, the application has a socket, and the failure looks like the server not answering.

What changed

Client · MTU 1500: State → ESTABLISHED
Emphasis on Client · MTU 1500 ↔ Tunnel · MTU 1400

Terms

Congestion window: The sender’s private estimate of what the network will carry. It appears nowhere in the header, the receiver never learns it, and it is usually the smaller of the two limits on how much may be in flight.
Receive window: How much the receiver can buffer, advertised in every acknowledgement. It sets a ceiling on throughput of window divided by round-trip time, which is why it needs scaling on a long path.
MSS clamping: A router rewriting the maximum segment size option in a passing SYN so both ends agree on a size that fits. It is the path-MTU fix that does not depend on ICMP surviving a network you do not control.

Check yourself

A transfer stalls on full-size packets while small requests succeed, and ICMP is filtered at the firewall. What is happening?
Which side of a connection accumulates TIME_WAIT sockets?
Without window scaling, what caps throughput on a 100 ms path?
Window scaling was not negotiated. When can it be enabled?
Which of these can be turned on after a connection is established?
A sender has a 4 MB receive window advertised to it and a congestion window of 40 KB. How much may be in flight?
What does selective acknowledgement change?
A request-response application pauses exactly 40 ms per transaction on a sub-millisecond network. What is the likely cause?

Why it exists

Three packets, and everything you will ever negotiate

How it works

On the wire

Configure it

Verify

Caveats

When it breaks

A connection establishes, transfers a little data, then stalls until it times out. Small requests work.

Narrow it down

Cause

Fix

A server accumulates thousands of sockets in TIME_WAIT and eventually refuses new connections.

Narrow it down

Cause

Fix

Throughput sits far below the link rate on a long path, and is the same number no matter how much bandwidth is added.

Narrow it down

Cause

Fix

A server reports a flood of connections in SYN_RECV and legitimate clients cannot connect.

Narrow it down

Cause

Fix

A request-response application pauses about forty milliseconds per transaction, on a network with a sub-millisecond round trip.

Narrow it down

Cause

Fix

Design notes

Misconceptions

More walkthroughs

Two windows, and only one of them is on the wiredesign-choice

The connection that opens and then goes quietfailure

Terms

Check yourself

Related

QUIC

TLS

HTTP