Consider a case where you have a router balancing over four links. It chooses which link based on a hash of some information from the packet.
Now imagine you have four more routers on each of those links, hashing out over four more links. So you have a tree with 16 outputs.
If the second-tier router uses the same hash algorithm as the first one, all the packets it receives will hash to the same link, because it's doing the same calculation as the router before it.
Thus the 2nd tier of four routers will only use 4 of their outputs, instead of all 16.
I think you are describing a CLOS(spine and leaf) network data center topology here.
>"Now imagine you have four more routers on each of those links, hashing out over four more links. So you have a tree with 16 outputs."
I'm not really understanding this as a link exists exactly between only two routers do you mean "path" instead?
>"If the second-tier router uses the same hash algorithm as the first one, all the packets it receives will hash to the same link, because it's doing the same calculation as the router before it.
And this is kind of the tax of flow based preservation which is fine compared to the price of TCP reordering no? Efficient hash-based ECMP utilization is going to be a function of the distribution of source IP and port in the 5 tuple used in hashing. You can see this outside of EMCP for example when running LVS with the hashing algo and you have customers that are all behind the same NAT box. But also there's nothing stopping you from using different hashing on your spine tier than you do on your leaf tier. You could assign a 4 tuple on one and a 5 tuple on the other.
At any rate a common CLOS ECMP design with BGP is to put each ToR switch in its own ASN and then load balance across ASNs. So using your example and a 3 tier CLOS network. If Tier 2 had the 16 outputs then the a router should ECMP over the 16 different ToR ASNs to the destination.
> And this is kind of the tax of flow based preservation which is fine compared to the price of TCP reordering no? Efficient hash-based ECMP utilization is going to be a function of the distribution of source IP and port in the 5 tuple used in hashing.
mcpherrinm's example is worse than what I think you're suggesting. Because a given second-tier router will only receive packets that hash to a specific set of values, if that router has the same number of downstream links that the first tier router had, it will send all packets it receives to the same link, ignoring other links. Which is a terrible trade-off for avoiding OOO packets.
The more reasonable trade-off is giving up on utilizing all routers for a single flow, to avoid OOO packets.
>"The more reasonable trade-off is giving up on utilizing all routers for a single flow, to avoid OOO packets."
Huh? No you would never want to use "all" routers for a single flow anyway. Each router just needs to make a deterministic selection for each packet. The alternative to a hash based scheme would be per packet load balancing which is practically never used b/c it gives you TCP packet reordering.
>"if that router has the same number of downstream links that the first tier router had, it will send all packets it receives to the same link, ignoring other links. Which is a terrible trade-off for avoiding OOO packets"
No it would not be a terrible trade off. Optimizing for maximum link utilization only matters if you have congestion in your network and even then ECMP is congestion agnostic. In reality your leaf network has less downstream link that it has upstream. A common topology is 4x4x2 where each leaf node has two downstream links to two ToR switches.
Pedantic: Clos isn’t an acronym and only the first letter should be capitalized. It’s a non-blocking switched network named after its inventor, Charles Clos.
Thank you for this explanation, I understand now. Originally I thought you would want all routers in the path to hash the same flow the same way, but didn't think about how it interacts with layers of routers.
On first guess, I would think a per-device salt is another way to address polarization. What are other ways?
Per device salt is the usual way. See https://docs.cumulusnetworks.com/plugins/servlet/mobile#cont... for some example documentation (ctrlf hash seed). Or you can change the inputs to the hash function (eg, one tier hashes on source is and the other doesn't) but that's more troublesome.
The property you want is that all packets for a given flow take the same path through the network (absent topology changes), to minimize out-of-order packets and ease troubleshooting.
A1 +- B1 +- C1
| `- C2
|
`- B2 +- C3
`- C4
In this example, if all routers compute the same hash value for a given packet, links B1 -> C2 and B2 -> C4 never get used, for any flows.
So all you really need is for each router to make a consistent decision about packets in the same flow. They don't have to use the exact same hashing function. In addition to the cumulus networks link provided in the GP, it looks like Cisco gear also has a per-device salt: https://www.cisco.com/c/en/us/support/docs/ip/express-forwar...
Note that this has nothing to do with endpoint selection, just the intermediate hops.
Edit: I'm also leaving out how the final modulo can produce different selections between layers. So in practice, if you have layers that are different sizes you'd get better utilization. That seems like a fragile thing to depend upon, though.
>"So all you really need is for each router to make a consistent decision about packets in the same flow."
Indeed this is what I was trying to articulate but maybe I didn't do a good job of that. I mentioned somewhere else you can also just change the hash at each tier in your network - add IP protocol in one, don't use IP protocol in another. This should achieve the same as adding a "seed." to a router's hash.
The problem is if all of the equipment hashes the same way, many of your links will be underutilized.
If A has a link to B1 and B2, and each of B1 and B2 have two links to C, and everything hashes the same, there will only be two possible paths to C, instead of the four you should have.
Ex: if packet hashes to 0, it goes to B1, and then over the 0 link to C. If packet hashes to 1, it goes to B2 and over the 1 link to C. If the Bs hash on different values than A (including if they have a different salt), you'll have better distribution on the B to C links, and actually be able to hit all 4 paths.
The term for unwanted oscillation between different paths is, IIRC, "flapping".
However, when working with ECMP (equal-cost multi path), you actually do want to use all your paths (hence "equal cost") simultaneously (for load-balancing purposes, say). "Polarisation" refers to the unwanted condition when the hash algorithm that decides which path a flow/packet takes is not properly distributing flows/packets across links, leading to underutilisation and/or overutilisation of links/routes.
Yeah I think you could seed each router's hashing with something unique like a mac address hardware ID but you could also just use different hashing at each tier in your network like using source IP + dest IP on one tier and source IP + dest IP + IP protocol on another tier. This should achieve the same result.
Yeah but then you need to keep coming up with new ways of hashing things as you add tiers. It's easier to generate a new salt than it is to figure out a new but still-valid way of hashing the packets.
Practically speaking this isn't an issue though. You don't see many spine and leaf networks that are deeper than 3 tiers(edge, distribution, ToR) which means one link upstream and one down. It's not like people build networks arbitrarily deep.
