Egress Peer Engineering extends regular BGP policies to provide more flexibility.
Why use Egress Peer Engineering
When a network operator has multiple external connections, such as IP transit, private peerings or Internet Exchange (IXP), there is often a need to flexibly steer traffic over all those connections. The considerations can include:
- Cost savings – usually private peering is cheaper than IXP and IXP is cheaper than IP transit
- Ensuring low latency/better performance path for real time traffic
- Load balancing across multiple exits to avoid congestion
Limitations of classical BGP
Usually, network operators rely on BGP policies to pick the preferred exit for egress traffic. The 2 most common approaches are:
- Cold potato routing – send traffic to the most optimal exit – usually achieved by tuning BGP Local Preference
- Hot potato routing – send traffic to the nearest exit – usually achieved by setting the same Local Preference and ignoring AS Path for BGP best path, so that the exit is picked by lowest IGP metric
In more complex scenarios, a combination of these 2 approaches can be used. For example, the operator might prefer site A to use local IP transit instead of sending traffic over an expensive intercity link to site B which has a private peering.
Anyway, BGP has some limitations:
- Lack of source routing. When the operator defines a preferred exit by setting higher Local Preference, the entire Autonomous System (AS) will prefer that exit
- Performance obliviousness. BGP will keep sending traffic to the preferred link even when it’s congested and there is more bandwidth available on other links
- Unpredictable convergence. When the traffic is evenly balanced, and some links fail so the traffic is rerouted via other links, all the manual load balancing falls apart and congestions can occur even when unused bandwidth is available
- Operational complexity. Attempting to solve the aforementioned problems by configuring more BGP policies leads to very complex configurations that are difficult to manage and are prone to errors, leading to network outages
The need for Egress Peer Engineering
Below is the typical example demonstrating the BGP limitations and the need for EPE. Consider the topology:
AS400 advertises prefix 192.168.0.0/24, and AS666 has 3 paths to it: via AS100, AS200 (both connected to ASBR1), and AS300 (connected to ASBR2).
If the operator wants to steer traffic to AS400 as shown on the diagram, there is no way to achieve that with classical BGP.
However, with EPE and source routing it is possible to steer traffic this way, or any other way in any topology.
In order for this to work, the ASBR must allocate an MPLS label per each egress peer (BGP nexthop) and the ingress routers attach this label at the bottom of the label stack. Upon receiving a packet with EPE label, the ASBR pop the label and send the packet to the relevant egress peer, ignoring the regular routing table.
SR or not SR?
EPE works best with Segment Routing. There are multiple reasons for that.
The way regular Traffic Engineering with SR works is that the headend pushes a set of MPLS labels on the packet to steer it via the desired path:
In order to add EPE to this logic, R1 just need to add an EPE label on the bottom of stack:
The only change is that the endpoint is now not the node but the egress peer. It is possible to reuse all other features of SR-TE such as automated steering, bandwidth reservations, TE constraints, flex algo etc.
If the network is not running Segment Routing, it is still possible to run EPE, but with some limitations. EPE label will not extend the existing LSP (signaled by LDP or RSVP), but recursively resolve over it.
To illustrate the difference in control plane logic:
It is also not possible to use automated steering, and path constraints (if used) need to be specified separately for RSVP LSP and for EPE.
Controller or not?
Most EPE designs require a controller. It’s not as bad as it sounds, because router<>controller communication uses standard protocols such as BGP and PCEP, and even if the controller fails, network will just fallback to the usual BGP best path rules.
Besides, advanced SR-TE with bandwidth reservations or multi-domain capability requires a controller anyway, so the same controller can also support EPE.
However, some simple EPE designs don’t need a controller and can still be useful.
EPE example without a controller
Consider the topology:
The goal is to steer traffic from R1 to AS400 via AS100 and from R2 to AS400 via AS200. Without regular BGP policies it’s not possible, but if ASBR1 supports EPE and advertises a BGP-LU route for each egress peer (172.16.0.100 and 172.16.0.200), it’s possible to achieve such routing logic.
ASBR1 and RR use add-path to advertise both copies of 192.168.0.0/24 to R1 and R2, and set the respective communities. R1 and R2 choose the relevant route, which is recursively resolved via a BGP-LU EPE route.
Conclusion
The simple example shown above doesn’t need a controller. However, a good EPE controller enables much more advanced EPE designs, especially when combined with Segment Routing. BGP-LS and peer SID, path constraints, bandwidth reservations, automated steering and more.
In the next several posts I will expand more on the subject and show more examples of EPE.
Further reading
There is a lot of information about EPE. Aside from some theory, I prepared labs with Arista cEOS and Traffic Dictator as an EPE controller.
RFC/Drafts
[RFC9086]
[draft-gredler-idr-bgplu-epe-15]
Books
[Segment Routing Part II] – chapters 14 and 17.4
[MPLS in the SDN Era]- chapters 13, 21