TCP Anycast Shift

What happens when routing changes for a stateful network protocol and unexpectedly arrives at a device without a session?

Terminology

Symptoms

tcpdump from the origin illustrates the different behaviors when accessing the destination via regular and anycast IP address. The regular returned the expected flags [S][.][P.]...[F.] while anycast returned [S][.][P.]...[R]. Where [S] (Start Connection), [P] (Push Data), [R] (Reset Connection), and [F] (Finish Connection). Connection reset was reproducible with:

Explanation

Take the following interaction diagram illustrates the above and TCP’s stateful nature:

sequenceDiagram participant Origin participant DestDeviceOne participant DestDeviceTwo Origin->>DestDeviceOne: SYN=i DestDeviceOne-->>Origin: SYN=j, ACK=i+1 Origin->>DestDeviceOne: ACK=j+1 Note over Origin,DestDeviceOne: 3-way handshake loop alt Origin->>DestDeviceOne: SEQ=a, ACK=b, DATA=c DestDeviceOne-->>Origin: ACK, SEQ=b, ACK=a+c Note over Origin,DestDeviceOne: data transfer else Origin->>DestDeviceTwo: SEQ=a, ACK=b, DATA=c DestDeviceTwo-->>Origin: RST Note over Origin,DestDeviceTwo: anycast shift end end %%alt %% Origin->>DestDeviceTwo: SEQ=a, ACK=b, DATA=c %% DestDeviceTwo-->>Origin: RST %% Note over Origin,DestDeviceTwo: anycast shift %%else %% Origin->>DestDeviceOne: FIN, ACK, SEQ=m, ACK=n %% DestDeviceOne-->>Origin: ACK, ACK=m+1 %% Origin->>DestDeviceOne: FIN, ACK, SEQ=n, ACK=m+1 %% DestDeviceOne-->>Origin: ACK, ACK=n+1 %% Note over Origin,DestDeviceTwo: 4-way termination %%end

Let’s focus on “anycast shift” where a packet unexpectedly arrives at a device without a session. This can result from origins with routes using equal-cost multipath. The splitting of packets across links means the destination anycast IP may resolve to a different device resulting in reset.

Example flow with multipath:

graph LR %%{init:{'flowchart':{'nodeSpacing': 80, 'rankSpacing': 50}}}%% Origin(Origin1) DestDeviceOne(DestDevice1) DestDeviceTwo(DestDevice2) HopOne((Hop)) HopTwo((Hop)) HopThreeA((Hop)) HopThreeB((Hop)) Origin -->|1.| HopOne -->|2.| HopTwo HopTwo -->|3. equal-cost multipath| HopThreeA --> |4. anycast| DestDeviceOne HopTwo -->|5. equal-cost multipath| HopThreeB --> |6. anycast| DestDeviceTwo