What is the issue with re-marking and OoO packets?

This question, taken from the CCDE Written Outline, taught me something new. Since that’s the purpose of any certification, I welcome these interesting, if somewhat vague, questions.

The main issue with re-marking and Out-of-Order packets can be found in RFC3819: (aka BCP89)

“reordering does come at a cost with TCP as it is currently defined. Because TCP returns a cumulative acknowledgment (ACK) indicating the last in-order segment that has arrived, out-of-order segments cause a TCP receiver to transmit a duplicate acknowledgment. When the TCP sender notices three duplicate acknowledgments, it assumes that a segment was dropped by the network and uses the fast retransmit algorithm [Jac90] [RFC 2581] to resend the segment. In addition, the congestion window is reduced by half, effectively halving TCP’s sending rate. If a subnetwork reorders segments significantly such that three duplicate ACKs are generated, the TCP sender needlessly reduces the congestion window and performance suffers.”

In English, this means that if your Service Provider is doing any sort of re-marking (with a traffic policer, for example) your TCP flows may suffer due to re-ordering within the network. Let’s say your SP provides a subrate Ethernet attachment circuit to you at 10Mbps. You’re permitted to go over this rate but your traffic will be classified as less-than-best-effort (COS=1). You begin transmitting a large PDF via FTP. Once your flow goes over the committed rate, it was remarked to COS=1. Next, those packets are delayed within the core through a shaping mechanism (realizing that in the real world, less-than-best-effort would most likely be policed). Anyhow, the packets are delivered out-of-order due to this shaping policy. Meanwhile, the TCP session established by FTP notices segments are arriving out-of-order and performs the fast retransmit algorithm. This causes your FTP transfer to slow down during the transfer.

Now that we know this, how will it influence our design?

The first and most obvious result would be to configure an outbound shaping policy to match the SP’s subrate Ethernet AC. In general, reordering is not something that we can design for, as once we hand-off the packet to the provider we cannot control the path it takes. Within our own network, we can reduce the number of paths a packet takes by using traffic engineering or reducing the number of equal-cost links. Within Cisco devices, CEF attempts to ensure the same path is taken by using a hash of the socket information. This is one of the reasons equal-cost paths may not show equal utilization.