Tag: networking

1. Advanced BGP Configurations
2. BGP on Loopback Interfaces
3. Route Reflectors
4. OSPF + iBGP on Loopbacks Configuration on FortiGate

Advanced BGP Configurations

Recursive Resolution

When a router learns a route via BGP, the route says “to reach network X, send traffic to next-hop Y.” But what if next-hop Y isn’t directly connected to your router? The router needs to figure out how to reach Y first. Recursive resolution is when the router looks up the next-hop address in its own routing table to find the actual outgoing interface and gateway to use.
config router bgp
set recursive-next-hop enable
end

By default, BGP routes are not considered when a BGP next hop requires recursive resolution (only static, connected, OSPF, etc.). They are considered when recursive-next-hop is enabled. Recursive resolution will resolve to one level.

Conditional Advertisement

BGP Conditional Advertisement allows the router to advertise a route only when certain configured conditions are met. 

config router bgp
config neighbor
edit "2.2.2.2"
config conditional-advertise
edit "2224"
set condition-routemap "2814" "2824"
set condition-type non-exist
next
end
next
end
end

The above example configuration states that the route maps objects, 2814 and 2824, will only be advertised if they do not exist. In this scenario, the route map has match conditions for three subnets, 172.28.1.0/24, 172.28.2.0/24 and 172.22.2.0/24. This means that if the peer network advertises any of these three subnets to the FortiGate, the FortiGate will not advertise these three networks back out. 

• The route maps listed within set condition-routemap are the conditions that must be met in order to execute the advertise-routemap (“2224” in this example).
• Given that two condition route maps are set in this example, both the 2814 and 2824 route maps route map conditions must not be met (non-exist) in order for the 2224 route map to be advertised. 

Neighbor Range

A Neighbor Range allows for you to specify a subnet such that neighbors will be configured automatically if their source IP falls within that range.

Neighbor Group

Neighbor Group can be used to share the same neighbor attributes among group members, avoiding having to repeat configurations for multiple neighbors.
config router bgp
config neighbor-group
edit "sites"
set soft-reconfiguration enable
set remote-as 65510
set local-as 65520
set local-as-no-prepend enable
set local-as-replace-as enable
set route-map-in "filter1"
set keep-alive-timer 30
set holdtime-timer 90
set update-source "port1"
set weight 1000
set password ENC ******
next
end
end

This could be useful when you are peering from a hub to many spokes that all use the same password. You can assign all of your spokes to the same neighbor-group and simply set password ENC [password].

As seen in the section above, you can also apply a neighbor-group from within a neighbor-range, which will set the neighbor-group conditions to any peers within the specified range.

When using a Neighbor Group, the FortiGate will respond to peering requests within that range but not initiate requests itself. This is a default behavior that can be changed via set passive disable.

Address and Address Group objects as BGP Networks (FortiOS 7.6+)

Rather than setting and keeping up to date your list of BGP networks manually, you can include addresses and address groups.

• Within your Address or Address Group object,  set allow-routing enable.
• Only IP subnets and Interface subnet objects (address objects auto-created when you create an interface) can be referenced, not FQDNs, IP Ranges, Geographies, dynamic address objects, etc. 

Default Originate

Enabling the capability-default-originate command within a peer configuration will cause the FortiGate to advertise the 0.0.0.0/0 default route to the peer without including 0.0.0.0/0 in the list of BGP networks.

Secure BGP sessions with MD5 hashing

Within a peer configuration you can set password [password] to force authentication with your peer. The same password will need to be set on the peer.

Soft Reconfiguration

BGP is configured such that inbound filtering only occurs when routes are received from your peer. When soft reconfiguration is enabled, it stores a copy of the pre-filter routes locally in memory. This allows you to update filters and “re-run” the routes through them to result in an updated post-filter set of routes without necessitating a BGP hard-reset.

When soft reconfiguration is enabled, you gain the ability to see received routes pre-local-filtering. 

• get router info bgp neighbors [peer IP] received-routes will show your pre-filter received routes.
• get router info bgp neighbors [peer IP] routes will show your post-filter received routes.

Route Refresh

Just like Soft Reconfiguration, Route Refresh was created to solve the issue of necessitating a hard BGP reset every time you change your filters. Route Refresh, however, allows for this capability without the additional memory and CPU load created by storing a pre-filter set of routes. Route Refresh simply sends a message to the peer when the local filters are updating, asking for updated pre-filter routes. 

• The one downside of Route Refresh has a slightly slower update/refresh time, roughly 30 seconds, based on the default eBGP advertisement-interval.
• Route Refresh is enabled by default on modern FortiGates. 

Soft Reset and Hard Reset

A hard reset drops all of your peering’s BGP routes from the routing table and forces a temporary shutdown of the BGP session itself. 

execute router clear bgp [ip | all | ipv6] <neighbor IP>

• As this will affect active traffic flows, it is recommended to use a hard reset only after attempting a soft reset. 
• A hard reset will occur when enabling/disabling BGP capabilities and settings, such as Route Refresh.
  • Enabling/Disabling the Soft Reconfiguration capability will not trigger a Hard Reset.
• A hard reset is required to reflect changes to BGP timer values.

A soft reset applies new or changes to existing routing objects such as route maps, without the traffic disruption caused by a hard reset.

execute router clear bgp ip <neighbor IP> soft [in | out]

• Technically you should not “need” to soft reset after an inbound route filter change, as the enabled-by-default Route Refresh capability should send an update message after the change, however I often soft reset out of habit and extreme impatience.
• Outbound route filter changes will necessitate a soft reset as Soft Reconfiguration and Route Refresh capabilities do not apply to advertised routes. 

Summary: Soft Reconfiguration, Route Refresh, Soft Reset, Hard Reset

• Soft Reconfiguration: Store received routes before policy application in memory

• Route Refresh: BGP capability to request peer to re-advertise all prefixes

• Soft Reset: Reapply policies, keep the session

• Hard Reset: Drop TCP session, full BGP re-establishment

Next Hop Self

The next-hop-self configuration will need to be applied to a peering any time the next hop for a route it is advertising is not reachable by its peer. This is commonly seen with iBGP routers advertising eBGP routes and in hub-and-spoke topologies where spokes peer through the hub. 

• Next Hop Self is not required in eBGP peerings as eBGP changes the next hop for routes it advertises. 
• next-hop-self-rr is used to set the next hop to self for routes a RR reflects.
  • Commonly used to force hub transit.

Maximum Prefixes

You can configure a maximum number of prefixes that are allowed to be received by a peer. 

• When the maximum prefix amount is hit, the BGP peering will be torn down.
• The BGP session will continue to flap up and down each advertisement-interval, 30 seconds by default, until the number of prefixes received falls back below the limit. 
• You can set maximum-prefix-warning-only enable to only produce a log when the prefix limit is reached. This will show up in System Events > Router Events and could then be used via Automation Stitches or FortiAnalyzer Event Handlers to notify NOCs or attempt auto-remediations. 
• A use case here is stopping a peer network that you don’t control from injecting mass amounts of networks into your, in case of a misconfiguration. 

Local AS

You can config the Local AS setting to mask your actual AS, from the perspective of your peer. I recently ran into a real use case for this when standing up a BGP peering with one of our partners at work. We use a very common private AS and so do they. We set local-as [AS] and we appeared as the masked AS from their perspective, allowing us to peer without conflict or issue.

• Setting local-as-replace-as will stop your real AS from being included in the AS path from the perspective of your peer (their received AS path from you). 
• Setting local-as-no-prepend will stop your local AS from being included in the AS path on inbound routes. 

BGP Multipath

While covered more comprehensively in the dedicated ECMP document, it’s worth mentioning the BGP-specific ECMP configurations. ECMP, or Equal Cost Multipath, allows for multiple paths (interfaces) to the same destination (networks) to be used simultaneously. To allow multipath, the following BGP configuration should be applied:

config router bgp
set ebgp-multipath enable
end

And/Or:

config router bgp
set ibgp-multipath
end

For paths to be eligible for eBGP/iBGP multipath, they must have the same local preference, AS path, origin, MED, and next-hop reachability. The neighbor BGP router ID and neighbor IP address attributes are not required to match.

The maximum number of eBGP/iBGP additional paths to be considered for multipath can be configured with the following commands:

config router bgp
set additional-path enable
set additional-path-select [2 - 255]
end

BGP Aggregation/Summarization

Aggregation allows you to summarize multiple specific routes into a less specific route including the specific members. The only decent metaphor I can think of for this is when you visit a large hotel. You walk down the hallway and there are normally signs that point you towards a group of room numbers. Rather than the sign having an arrow for every specific room number, it has a range of room numbers.

Even if the room “205” didn’t exist and was actually just a utility closet without a room number, you’d still say rooms “201-216” were to the right. This is what we see in route summaries. For example we might have the following routes advertised to our neighbor:

Rather than the neighbor receiving 8 routes for these 8 specific destinations, we can advertise a summary route that includes all 8 destinations. In this case, we’ll advertise 10.100.1.0/28. Here’s how we configure it:

config router bgp
config aggregate-address
edit 1
set prefix 10.100.1.0/28
set summary-only enable
end
end
end

After a soft reset, when we check out routes received by our peer we see this:

You can imagine how valuable this becomes, especially if you’re advertising hundreds or thousands of specific routes. One thing to note is that 10.100.1.0/28 includes the IP host range 10.100.1.1 – 10.100.1.14, meaning our peer will route 10.100.1.9 – 10.100.1.14 to us, even though we don’t actually have a route to these hosts. 

It’s also important to note that in our configuration we enabled the summary-only setting. Let’s turn that off, soft reset, and check our peer received routes again:

You can see we actually received the summary route as well as all of the specific routes. 

Route Redistribution into BGP

Redistribution allows you to inject other routes on your FortiGate into BGP, allowing you to advertise the injected routes to your BGP neighbors. On FortiGate, you can redistribute the following route types:

In our lab we have these 8 static routes that our peer doesn’t know about, but we want them to.

In our BGP settings we simply turn on Static redistribute and the routes will be advertised:

However, in our lab, we actually have quite a few additional static routes that we don’t want our peer to know about, so we’ll apply a filter. We only want to advertise our routes that fall within the 10.100.1.0/28 subnet, so we’ll create an Access List to match that subnets within (exact match disabled). We’ll then create a Route Map with an Allow rule matching the Access List, and apply that Route Map to the Static redistribution configuration.

We’ll soft reset our BGP peering and check received routes on our peer router:

Something to consider regarding redistributing routes vs. simply adding them within your BGP networks configuration – when redistributed routes are received, the origin code will be a ?, while routes configured in BGP networks will show an iBGP origin.

• In the picture above, the 10.100.1.1 network is included in our BGP networks configuration, while the other 3 networks are redistributed.

BGP Network Import Check

By default, BGP will not advertise a route included in its networks configuration if the route doesn’t already exist in the FortiGate’s routing table. For example, you may include the 192.168.5.0/24 route in your BGP config because that subnet represents a connected interface on your FortiGate. BGP will be able to advertise the route because there is already a 192.168.5.0/24 directly connected route in the RIB. If you attempt to advertise a route via BGP that you don’t have a route for, it won’t be sent out. You can allow this non-default behavior with the set network-import-check disable setting.

Let’s try it in the lab! We’ll add a network to BGP for which we don’t have an existing route in our RIB (192.168.5.0/24):

We check our advertised routes and see that subnet isn’t being advertised:

Now let’s turn off network-import-check:

When we check our advertised routes again, our route is there!

BGP on Loopback Interfaces

The common situation that leads to the BGP on loopbacks design is seen when you have peerings between two sites with redundant connections. For example you may have a hub-and-spoke design, with each spoke site having 2 or more ISP connections for redundancy. You’ll likely have IPSec tunnels or private links back to your hub. This will require two BGP peerings and four peer IPs, resulting in at least 2X the configuration time and a possibility of asynchronous routing. If you expand to 3 redundant links, now we’re looking at 3X the configuration time and a serious headache.

BGP on loopbacks solves this by creating your hub-to-spoke peerings using the loopback interface IPs themselves.

BGP on Loopback (Over IPSec VPN) Requirements:

1. BGP Peering Settings: 
   • update-source set to the loopback interface
   • enforce-ebgp-multihop enabled (eBGP only)
2. Firewall Policy:
   • A firewall policy allowing TCP port 179 traffic to and from your loopback, to and from your link port (IPsec tunnel interface, physical port, VLAN, etc.)
3. Routing:
   • A static route from your loopback to the peer loopback via each link port OR
   • An SD-WAN health check with a destination of the peer loopback and the two link ports as members

Route Reflectors

Route Reflectors are used in iBGP to reduce the number of peerings required by a full-mesh topology. Their use is best depicted by the transition between these two diagrams:

Simply described, rather than peering all routers with all other routers, all routers just need to peer with a route reflector. 

• Routes advertised to the router reflector are then advertised to all of its other peers.
• The router reflector FortiGate will act as such when you enable set route-reflector-client within its configuration with each of its peers.
• A router reflector cannot modify BGP attributes for routes that it reflects, but it can block reflected routes via route maps (or other BGP filtering methods).
  • Starting in 7.6.1, with the application of the rr-attr-allow-change config, the RR can modify attributes.
• Routes received by the RR, from a non-RR client, will be advertised to RR clients only.
• Routes received by the RR, from an RR client, will be advertised to both RR clients and non-clients.
• Routes received by the RR, from an eBGP neighbor, will be advertised to both RR clients and non-clients.

OSPF + iBGP on Loopbacks Configuration on FortiGate

This routing configuration is commonly recommended in the networking community for its scalability and simplicity. OSPF is configured to share routes to loopback IPs. BGP is sourced from the same loopback IPs to allow for dual-WAN and multiple IPSec tunnels from each spoke router without having to stand up multiple BGP peerings. Here’s how to configure it in its simplest form between two FortiGates. 

For this example, my lab is running two Azure FortiGate VMs, connected via an IPSec tunnel. Our loopbacks and tunnel IPs are as follows:

AzFW01

Loopback: 10.100.1.1

Tunnel IP: 172.20.0.1

AzFW02

Loopback: 10.200.1.1

Tunnel IP: 172.20.0.2

CLI Configurations:

On our tunnel interface, we ensure we set our remote IP as our far side tunnel interface IP.

Now that our interfaces are configured, we can configure OSPF. 

Key considerations:

• For simplicity’s sake we’re using a single area for our entire hub-and-spoke topology (scaling out the lab to 20+ spokes). 
• The OSPF interface network type must be manually set to point-to-point. It will default to broadcast.

With the above configuration on both firewalls, our OSPF adjacency has come up:

And we have a route to our far-side loopback:

Finally, we’ll configure BGP on both routers and use the update-source setting to peer via our loopbacks:

Our BGP peering is up:

And we have received our neighbor’s 10.251.0.0/16 route:

1. Routing Objects
2. BGP States
3. BGP Timers
4. FortiGate BGP Troubleshooting

Routing Objects

Access Lists

• Access Lists are simply a list of subnets with an action of permit or deny applied. 

• Access lists are commonly referenced within Route Maps like variables in programming or address objects/address groups on FortiGates – or they are applied directly to a BGP peering.
• Access lists are not exclusive to BGP, they are also frequently used in OSPF route control.
• A common configuration within an Access List entry is the “Exact Match” attribute, which will only “match” a subnet if it is exactly the same IP and subnet mask. If the Exact match attribute is disabled, as it is by default, the list will match any subnet equal to or smaller than it.
  • For example, without exact match and 10.5.0.0/16 configured as an Access List entry, the networks 10.5.10.0/24, 10.5.59.0/29, and 10.5.150.10/32 will all match the entry. 
• An Access List can be directly attributed to a BGP peering by enabling the distribute-list-in and/or distribute-list-out and selecting the Access List object.
• Access Lists can also be applied to route maps.
• An access list, when applied directly to a BGP peering, will have an implicit deny functionality for subnets not matched by the list.

Prefix Lists

Prefix Lists are very similar to Access Lists in that they also are used to match subnets. Prefix tangents the capability of Access Lists by allowing you to set a greater than (ge) or lesser than (le) operator in relation to the subnet’s CIDR notation integer.

• For example, if you set a Prefix List with the value 10.0.0.0/8 le 16, it will match all subnets less than or equal to /16 within 10.0.0.0/8. 10.1.0.0/10 would be allowed while 10.250.10.0/24 would not. 
• In BGP, Prefix Lists can be applied directly to a BGP neighbor via the prefix-list-in/prefix-list-out configuration, or applied to route maps (which are then applied to BGP neighbors). 
• A prefix list, when applied directly to a BGP peering, will have an implicit deny functionality for prefixes not matched by the list. 

AS Path List

• An AS Path List uses Regex (Regular Expressions) to match an AS and be used either in filtering or matching criteria within a route map.
  • Regex is a way to describe a pattern of text you’re looking for—like telling a FortiGate “find me anything that looks like an IP address” instead of searching for one specific IP.
    config router aspath-list
edit "list1"
config rule
edit 1
set action permit
set regexp "^6550[1-3]$"
next
end
next
end
    

• Practically, an AS Path List can be used to match, for example:
  • ASs ending in 001
  • ASs starting with 20
  • ASs ending with even numbers
  • Anything above AS 65400
• An AS Path List can also be used as a matching criteria for a neighbor group, allowing you to dynamically group peers based on any pattern of AS. When using the remote-as-filter on a neighbor group, you cannot also use the remote-as attribute.
  config router bgp
set as <local AS>
config neighbor-group
edit <name>
set remote-as-filter <BGP filter>
next
end
end
  

Community Lists and Community Strings

Community lists are lists of BGP attributes called Community Strings, commonly used to filter and control routes.  
config router community-list
edit <name>
set type {standard | expanded}
config rule
edit <id>
set action {deny | permit}
set regexp <string>
set match <string>
next
end
next
end

• There are two types of community lists: Standard and Expanded. Standard lists match strings exactly and are used for simple filtering. Expanded lists are based on Regex patterns, to match patterns of communities. 

• Community strings are created in the [AS]:[#] format, for example 65001:1. 
• Community strings are transitive, meaning they are shared with your peers’ peers. However, it is the default behavior on FortiGate that community strings are not sent to peers unless you configure set sent-community enable on the peering.
• Each community list rule is associated with a permit or deny action, allowing or blocking routes matching the community list. 
• Community lists can be used within route maps to apply actions such as modifications of attributes of routes. 
• Creating match conditions for multiple community strings (one ID per rule) within a single community list will create an “OR” match condition. 
• Creating matching conditions for multiple community strings (multiple listed in the same rule, separated by a space) will create an “AND” match condition.

Well Known Communities

These are community strings known by FortiGate, as well as most routing network devices that support BGP. They have specific instructions for your peers and are often applied to route maps.

Some examples include:

• NO_ADVERTISE: Tells the peer not to advertise the routes it receives from our peering.
• NO_EXPORT: Tells the peer not to advertise the routes it receives from our peering to eBGP peers. Advertisement to iBGP peers is still allowed.
• INTERNET: Applied by default. Tells the peer to advertise routes to all peers. Indicates the internet as a destination. All BGP-speaking networking devices belong to this community.
• LOCAL_AS: Similar to no_export but allows advertisement within the sub-AS of a confederation. 

Route Maps

A route is a list of rules that are applied in ascending order of ID, beginning with rule 1 on FortiGate. They can be used to apply custom actions based on matched conditions and are often used in conjunction with the other routing objects explained in this document. The most common usage I see is access lists matching specific BGP routes, applied to route maps using the route-map-in and route-map-out to permit those routes and apply local preference and AS Path prepending. 

config router route-map
edit <name>
config rule
edit <id>
set action {permit | deny}
set match-as-path <string>
set match community <string>
set match-ip-address <string>
set match ip6-address <string>
set match-ip-nexthop <string>
set match-ip6-nexthop <string>
next
end
next
end

• Every route map has a permit or deny action, with the default set to permit if an action is not set.
• Route map entries appear in the order that they are configured. To “reorder” then, set a rule’s ID to a less or greater value than other rules in the list. They will still appear out of order in the GUI, but will be evaluated ascendingly by their ID. 
• Route map entries are evaluated in order, once a rule is matched, processing stops. 
• Route maps can also be used with OSPF and RIP.
• If no rule in a route map matches a route, the route is implicitly denied. 

Additional Notes on Routing Objects

• After making changes to Routing Objects already applied to BGP peerings, changes will take effect in roughly 2-3 minutes, so long as the default Capability: Route Refresh has not been disabled within the peering settings.
  • If you wish to reflect a change sooner, “Soft Reset” the peering. 

BGP States

When you configure a BGP peering from your FortiGate, then via the GUI or CLI check whether your peering is “up” with your neighbor, you’ll see one of the above states listed. The most common that you’ll see are Idle, Connect, and Established. Established is what you want, meaning the BGP peering is “up” and good to begin advertising and receiving routes.

• Idle
  • The initial state after you have configured a neighbor from your FortiGate. No BGP connections have been initiated and all incoming connections are refused.
• Connect
  • FortiGate has initiated a connection to the neighbor. FortiGate is listening for TCP connections from the neighbor. 
  • FortiGate will send 6 total connection attempts over time (after 1, 2, 4, 8, 16, and 32 seconds). Waiting a cumulative maximum period of 63 seconds. BGP will then move into the Active state.
  • If any of the above connection attempts result in a successful TCP handshake, an Open message is sent to the neighbor and the state moves to OpenSent.
• Active
  • FortiGate is listening for a TCP connection.
  • When this state begins, the ConnectRetryTimer begins counting down. Once it expires, the BGP peering moves back to the Connect state.
• OpenSent
  • BGP is waiting for an Open message from the neighbor.
  • When an Open message is received, FortiGate checks it for errors. If no errors are found, BGP moves to the OpenConfirm state.
• OpenConfirm
  • BGP is waiting for a Keepalive message acknowledging its Open message.
• Established
  • The connection is confirmed and update messages are sent to exchange routing information.

BGP Timers

The following timers are set on a global BGP basis, with the exception of set connect-timer.

1. set keepalive-timer [3-65535]
   • Frequency at which FortiGate sends keepalive messages to established peers.
   • Can be set per-peer via set keep-alive-timer [3-65535].
   • The default keep alive timer is 60 seconds. 
   • The keep alive timer should always be ⅓ or less than the holdtime timer. If a ratio above ⅓ is configured, the FortiGate will use a keep alive that is exactly ⅓ of the configured holdtime. 
   • BGP must be hard reset to apply changes to this value.
2. set holdtime-timer [3-65535]
   • Maximum amount of time FortiGate will wait before marking a peer as dead.
   • Can be set per-peer via set holdtime-timer [3-65535].
   • The default holdtime timer is 180 seconds.
   • Keep alive and holdtimes are negotiated between peers, with the lesser value between the two being chosen. 
   • BGP must be hard reset to apply changes to this value.
3. set advertisement-interval [0-600]
   • Minimum interval FortiGate will wait to send routing updates.
   • The default advertisement interval is 30 seconds. 
   • This value is not required to match between peers.
   • Lower advertisement interval = faster convergence but more BGP UPDATE traffic.
4. set connect-timer [1-65535]
   • Maximum amount of time for the ConnectRetryTimer.
   • When ConnectRetryTimer expires, FortiGate will initiate a new TCP connection attempt to the BGP peer.
   • The default connect timer is 120 seconds.
   • No global BGP configuration for this timer, it is exclusively set on a per-peer basis.
5. set scan-time [5-60]
   • Background interval in which BGP checks next-hop availability of routes.
     • If a route’s next hop is not reachable, the route will not be used.
   • The default scan time is 60 seconds.
   • Can be set to 0 second to disable scan time, however this is strongly discouraged as the FortiGate will skip proactive next hop checks entirely.

FortiGate BGP Troubleshooting

Troubleshooting Commands

show router bgp

get router info bgp summary

get router info bgp network

get router info routing-table bgp

get router info bgp neighbors

exec router clear bgp all 

What routes am I sending to my peer?

get router info bgp neighbors [peer IP] advertised-routes

What routes am I receiving from my peer? (Pre-inbound-filter, soft reconfiguration enabled)

get router info bgp neighbors [peer IP] received-routes

What routes am I receiving from my peer? (Post-inbound-filter)

get router info bgp neighbors [peer IP] routes

Check if the BGP port is listening

diagnose sys tcpsock | grep 179

Run a packet capture to verify BGP TCP session establishment

diagnose sniffer packet any 'tcp and port 179' 4

Soft Reset all peers

execute router clear bgp all soft out 

Hard Reset all peers

exec router clear bgp all

Turn on BGP debugging

diagnose ip router bgp level info
diagnose ip router bgp all enable
diagnose debug enable

Flapping Routes

Route Flap occurs when there are changes to a network that cause a peering to go offline and routing to reorient. This is often seen when a FortiGate reboots, fails over to an HA pair, or intermittent packet loss is experienced on a circuit.

While the obvious solution is to solve the root problem in the case of circuit or power issues, there are often factors outside of your control that will require you to give BGP a bit more slack.

Flapping Routes – Option #1 – Holdtime Timer

The Holdtime timer is the maximum amount of time a BGP speaker waits for a keepalive message from a peer before declaring the peer dead. Any time a keepalive is received, the timer is reset to, by default, 180 seconds.

• Lowering this value will cause BGP to failover faster by declaring peers dead sooner. Raising this value may be useful in a situation where BGP goes down for a period longer than 180 seconds before being restored.

Flapping Routes – Option #2 – Dampening

Dampening is a “smarter” solution in that it only applies once a route has flapped twice, and then will adjust based on whether the route continues to flap or not. Once dampening begins, a countdown timer begins, similar to the Holdtime timer, however with dampening the timer will extend itself each time the route continues to flap.

config router bgp
set dampening {enable | disable}
set dampening-max-suppress-time <minutes>
set dampening-reachability-half-life <minutes>
set dampening-reuse <integer>
set dampening-route-map <string>
set dampening-suppress <integer>
set dampening-unreachability-half-life <minutes>
end

The countdown timer can also be thought of as a penalty, like being put in the penalty box or a child given a timeout. If the route continues to “misbehave” while in timeout, their penalty will continue longer.

You can adjust the levels depicted on the graph above in FortiGate using the equivalent configurations:

• dampening-suppress = Suppress Limit (Default 2000)
  • This is the penalty level at which the route will be suppressed.
  • The first penalty is 1000 by default, hence suppression beginning after the second flap once the penalty has reached 2000. In actuality, due to the penalty decay over time, the route will not actually be suppressed until the third penalty. 
• dampening-reuse = Reuse Limit (Default 750)
  • When the penalty falls below this level, the route is no longer suppressed.
• dampening-reachability-halflife, in minutes (Default 15)
  • After 15 minutes, the penalty will reach 50% of the original value.
  • Increase this value for a longer penalty “recovery” time.
  • Only applicable to reachable routes.
• dampening-unreachability-halflife, in minutes (Default 15)
  • Same as above but applicable to routes that are unreachable or withdrawn.
  • Reduce this value to allow suppressed routes to become eligible for reuse more quickly after they’ve stopped flapping.
• dampening-max-suppress-time, in minutes (Default 60)
  • The maximum amount of time a route can be suppressed for.

Once the penalty falls below 50% of the Reuse Limit, the entire penalty is removed. 

Flapping Routes – Option #3 – Graceful Restart

Graceful Restart is recommended to be used in situations when routing isn’t interrupted, but the router is unresponsive to routing update advertisements. The most common configurations affected by these situations are FortiGates in HA pairs, particularly during failovers and FortiOS upgrades (a occurs during upgrades). 

config router bgp
set graceful-restart {disable | enable}
set graceful-restart-time <seconds>
set graceful-stalepath-time <seconds>
set graceful-update-delay <seconds>
config neighbor
edit 10.1.1.1
set capability-graceful-restart {enable | disable}
next
end
end

When Graceful Restart is enabled and a FortiGate plans to go offline (FortiOS upgrade and other semi-controlled failovers), it will send a message to BGP peers stating that it plans to go offline temporarily and for how long. The peer routers will keep the FortiGate in their routing tables until the expected downtime is exceeded, preventing routing flaps.

• In an HA pair, the BGP daemon only runs on the primary unit. When a failover occurs, Graceful Restart will tell the peer to keep the FortiGate its routing table, allowing the peering to stay up while the secondary is in the process of coming online. 

• The higher level config router bgp > set graceful-restart will enable Graceful Restart globally, while the config neighbor > set capability-graceful-restart only enables it with the single peer.
• Enabling Graceful Restart will cause BGP to flap (reset).
• For Graceful Restart to work, it needs to be configured on both sides of the peering.
• Graceful Restart will delay the time until actual outages are realized by peers. The standard keepalive timeout for BGP is 3 minutes. With GR enabled, an additional 2 minutes are added by default, for a total of 5 minutes.
• It is not recommended to use Graceful Restart concurrently with BFD.
  • BFD will detect the peering failure before GR can signal a Graceful Restart.

HA + BGP Graceful Restart for no traffic interruption during failover: https://community.fortinet.com/t5/FortiGate/Technical-Tip-Configuring-FortiGate-HA-and-BGP-graceful-restart/ta-p/196150

Bidirectional Forwarding Detection (BFD)

BFD allows for fast detection of failures between peerings by sending unique BFD packets between the devices.

• BFD works with static routes, OSPF, and BGP.
• BFD can be configured globally, by VDOM, or by interface.
  • Once BFD is enabled to one of the above scopes, it can be enabled on a route or routing protocol.
• For BFD to work with BGP, both sides of a peering must have it configured.
• It is not recommended to use BFD for peerings that occur over WAN connections, rather it is recommended for directly connected peers.
  • Using BFD with peers >1 hop away (not directly-connected) will require the config multihop-template configuration.

Peering Establishment Verification List:

1. Verify configured peer AS from both sides
   • Your BGP debug will print a “Bad Peer AS” if you have configured your peer AS incorrectly.
2. Verify configured peer IP from both sides
3. Verify BGP authentication settings from both sides
4. Run a packet capture to your peer IP
   • Check for full TCP handshakes
5. Run a Debug Flow to your peer IP
   • Check for any and all blocks, drops or errors
6. Verify your FortiGate is listening on port 179 for BGP traffic
   • diagnose sys tcpsock | grep 179
   • Check local-in policies (BGP is allowed by default)

1. BGP Basics
2. IPSec Connection Method with eBGP Peering Example
3. eBGP vs iBGP
4. Route Selection Process

BGP Basics

BGP is a routing protocol that shares routing information between peers, identified by their AS (Autonomous System) numbers. Peers are two or more networks that wish to share routing information.

BGP Basic Peering Requirements:

1. AS Number
2. Peer IP
3. (Advertised) Networks

AS Numbers

• Public AS Ranges: 1 – 64495

• These numbers must be unique when peering out into the public internet.
• You must register publicly with IANA to obtain an assigned number.
• Reserved AS Ranges: 64496 – 64511, 65535
• Private AS Ranges: 64512 – 65534
• These numbers may only be used within your private network (or eBGP peerings to other private networks) and may not be used when peering to the internet. 

• Public Extended (4-byte) AS Ranges: 65536 – 4199999999
• Private Extended (4-byte) AS Ranges: 4200000000 – 4294967294
• Reserved Extended (4-byte) AS Ranges: 4294967295

Peer IP

A peer IP is used to identify the network you are peering with, particularly how you will reach it. For every peering, each peer will choose a Peer IP, typically one of two IPs within a /30 subnet.

(Advertised) Networks

Your networks are subnets configured within your BGP settings that may be advertised to your peers. These are commonly your local subnets that you’d like to provide your peer access to.

Connection Method

Every BGP peering occurs on top of a connection method. This could be a direct connection (an Ethernet cable connecting two routers), an IPSec VPN tunnel, an MPLS connection, or any way two networks can physically (or logically) connect. The BGP peering is not a connection in itself, like a VPN tunnel is; rather it shares routing information between those networks once they are connected.

IPSec Connection Method with eBGP Peering Example

Here I’ve actually added two elements: the IPSec tunnel interfaces (ToFW2 and ToFW1) and the advertised and received networks, but let’s just focus on the former for now. The tunnel interfaces are virtual (logical) interfaces created when the IPSec VPN is stood up between the two networks. Within the interface configuration, from Firewall 1’s perspective we configure our IP (10.5.0.1) as well as a remote IP and subnet (10.5.0.2/30). From Firewall 2’s perspective we do the same, but with the values flipped (IP 10.5.0.2 and remote IP 10.5.0.1/30). 

eBGP vs iBGP

• eBGP (External BGP) is used when peering with a different AS. This is most commonly seen when peering with a partner organization, peering to the internet (an ISP, provider network), or peering to a cloud network (ex. peering to an on-ramp to Azure Cloud). 
• iBGP (Internal BGP) is used when peering within the same AS (internal to the same AS). This is most commonly used when sharing routing information internally – to other routers on your organization’s network. 

Differences between eBGP and iBGP

1. Major Difference – Route Advertisement Default Behavior

• Routes learned from iBGP peers are not automatically advertised to other iBGP peers, to prevent loops. By default, eBGP peers will advertise all of its configured networks and re-advertise all networks it receives from eBGP peers to its iBGP and other eBGP peers. 
• Default eBGP behavior:
• Default iBGP behavior:

2. Minor Difference – Next-Hop Behavior depends on Connection Method

• In iBGP, the remote peer IP is most commonly a direct connection. Therefore a route to the remote peer IP is created automatically.
  • For instance, if the local IP is 10.5.0.1/30 and the remote IP is 10.5.0.2/30, there will be a route via direct connection to 10.5.0.2/30 via the directly connected interface.
• In eBGP, the remote peer IP may often need a route to be configured manually.
  • This is not the case for a peering over an IPSec VPN tunnel with the remote IP configured on the tunnel interface. The FortiGate will automatically create a route to the remote IP when you configure the interface. 
  • If you are experiencing issues getting a peering to become established, check that you have a route in your routing table to the remote IP.

3. Minor Difference – Policy Control Purpose

• Policy Control refers to the application of route maps, access lists, filtering, and preferences.
• In iBGP, Policy Control is typically used for the purpose of preventing routing loops and ensuring proper route distribution. 
• In eBGP, Policy Control is typically used for the purpose of controlling which routes are shared between different organizations or WAN networks.

config router community-list
edit <name>
set type {standard | expanded}
config rule
edit <id>
set action {permit | deny}
set regexp <string>
set match <string>
next
end
next
end

Route Selection Process

Let’s say your FortiGate is peered with 2 different external networks and 3 internal routers. Your FortiGate receives the same 10.120.0.0/16 subnet from two different peers. Which peer (routing direction) will it prefer over the other? This is where the route selection steps come into play. If at any time one router “beats” another router, the route selection process stops and the winning peer is used for routing.

1. Higher Weight

• The default weight is 0 for routes learned from BGP peers and 32768 for locally originated routes.
• Weight is a local only attribute and is never propagated to peers.
• It is typically not advised to use this to manipulate routing unless you’re looking to route your default 0.0.0.0/0 traffic via a BGP peer rather than a static route.  

2. Higher Local Preference

• The default local preference is 100.
• This is the most common attribute used to manipulate routing decisions.
• Local preference only applies to the local router. Therefore it should only be used to manipulate routing decisions outbound (preference or depreference a peer for outbound traffic). 

3. Prefer Route Originated by Local Router

• Routes created directly (ex. your BGP networks) on the FortiGate will be preferred over routes sourced from peers.

4. Shortest AS Path

• Each router traversed to a final network destination is considered an AS hop.
• FortiGate will choose the path with the fewest AS hops, minimizing the number of networks traversed to reach the destination.
• Routes received by peers can be preferred or depreferred by “prepending” AS hops, or adding additional AS hops to a peer’s path. This is the most common method of preferring and depreferring received networks.

5. Lowest Origin Code

• Only relevant if multiple dynamic routing protocols are in use.
• Routes learned through IGP (Internal Gateway Protocol) are prioritized over EGP (External Gateway Protocol) and incomplete (unknown origin) routes.

6. Lowest MED

• The Multi-Exit Discriminator (MED) attribute helps control inbound traffic from external autonomous systems. FortiGate favors the route with the lowest MED if MEDs are exchanged between peers. MED can only pass between adjacent autonomous systems, meaning that it is not transferable through the Internet.

7. Prefer EBGP Over IBGP
8. Prefer Path Through Closest IGP Neighbor

• Only relevant if multiple dynamic routing protocols are in use.
• If multiple iBGP paths exist, FortiGate will prefer the one through the closest IGP neighbor (lowest IGP metric)

9. Oldest Route for eBGP Paths
10. Lowest Neighbor BGP Router ID

• Router ID is a manually assigned attribute within BGP settings on the FortiGate.

11. Lowest Neighbor IP Address

• Neighbor IP address = Peer IP address

Route Selection Process – Manipulating Routes in eBGP

The list above is a great technical breakdown of how your FortiGate will choose the best BGP path. In practice, the following attributes are what are typically used to manipulate eBGP routing.

• Inbound/Received Routes

• These are the routes received from your peers.
• Set a higher or lower Local Preference to preference or depreference a route.
• Outbound/Advertised Routes

• These are the routes advertised to your peers.
• AS Path prepend the routes you wish to depreference.

iBGP Real-World-Use Disclaimers

• iBGP networks should be fully meshed in design and will auto-determine their best paths via the Route Selection Process. 
• Route manipulation is not as common as it is in eBGP, but when it is necessary, Local Preference is typically used.
• iBGP is typically used in combination with another dynamic routing protocol such as OSPF.
• It is becoming more commonly recommended that smaller internal networks utilize a simpler routing protocol such as OSPF or EIGRP over iBGP. 
• eBGP is most commonly seen on large and/or service provider networks, but eBGP implementations are more and more being recommended over iBGP for even small-mid-sized enterprise networks, especially within datacenter environments. 

eBGP Route Manipulation via Local Preference and AS Path Prepending – Example

Let’s start with the above scenario. We have a dual-hub-and-spoke network design with a single spoke router. Functionally, a full mesh topology has been formed and routing will occur symmetrically. If Hub 1 needs to reach Hub 2 or Spoke 1, it will route directly over the direct connection.

Additionally, in this scenario, our stakeholders have asked that we steer all routing from Spoke 1 to Hub 1, even traffic destined for Hub 2. The reasoning behind this is that the link WAN between Spoke 1 and Hub 2 is extremely expensive and slow and should only be used as a backup (we will not address the backup routing in this scenario, for simplicity). 

As stated in one of the sections above, we should use Local Preference and AS Path Prepending. Given the simplicity of this scenario we can exclusively use Local Preference if we want, but outbound route-map-out depreferencing via AS Path Prepending is the correct way. For the sake of brevity, this example will only use LP.

Step 1: Customize Local Preference attributes via Route-Map-In

Route Selection Process, explained:

• Spoke 1’s perspective:
  • All routes received from Hub 1 have a Local Preference of 100
  • All routes received from Hub 2 have a Local Preference of 75
  • All traffic to the 10.20.0.0/16 and 10.40.0.0/16 networks will travel to the Hub 1 next hop as preferred by Local Preference.
• Hub 1’s perspective:
  • The 10.60.0.0/16 route received from Spoke 1 has a LP of 100
  • The remaining received routes from Spoke 1, 10.20.0.0/16 and 10.40.0.0/16 have a LP of 75.
  • The 10.40.0.0/16 route received from Hub 2 has a LP of 100
  • The remaining received routes from Spoke 2, 10.20.0.0/16 and 10.40.0.0/16 have a LP of 75.
  • Traffic to 10.20.0.0/16 will be preferred via Administrative Distance (considered before BGP attributes) as 10.20.0.0/16 is a local connected route with an AD of 0 and the route received via eBGP has an AD of 20.
  • Traffic to the 10.40.0.0/16 network will travel to the Hub 2 next hop as preferred by local preference.
• Hub 2’s perspective:
  • All routes received from Spoke 1 have a LP of 75.
  • All routes received from Hub 1 have a LP of 100.
  • Traffic to 10.40.0.0/16 will be preferred via Administrative Distance (considered before BGP attributes) as 10.40.0.0/16 is a local connected route with an AD of 0 and the route received via eBGP has an AD of 20.
  • All traffic to the 10.20.0.0/16 and 10.60.0.0/16 networks will travel to the Hub 1 next hop as preferred by Local Preference.

Administrative Distance (AD)

AD acts as a comparative factor for equivalent routes between different routing protocols. This is why a default (0.0.0.0/0) static route “beats” a default eBGP route. The mechanic is that static routes have a default AD of 0 and eBGP routes have a default AD of 20, and routers prefer the route with the lower AD. The following are the default ADs for common routes types on FortiGates:

• Connected: 0
• Static: 10
• eBGP: 20
• OSPF: 110
• IS-IS: 115
• RIP: 120
• iBGP: 200

The default BGP Administrative Distances can be overridden within the BGP global configuration.

To change the default AD for routes learned from eBGP peers:

config router bgp
set distance-external [1-255]
end

To change the default AD for routes learned from iBGP peers:

config router bgp
set distance-internal [1-255]
end

To change the default AD for routes locally learned from the BGP network or redistribute configurations:

config router bgp
set distance-local
end

MED (Multi-Exit Discriminator)

MED’s functionality is described by its name. It is commonly used when there are multiple exit paths to the same AS. You can “discriminate” traffic by setting a custom MED value. 

The lower the MED value, the more preferred the path is to the receiving peer. 

As an example, you could imagine a FortiGate with two VPN tunnels to the same destination AS, underlaid by different WAN circuits. If you’d like to steer traffic over the primary ISP (ISP1), you can set ISP2’s MED value to 100, higher than the default value of 0. 

More about MED

• MED is not a transitive property, meaning when it is received by a node, it is not passed on to other peered nodes.
• The MED attribute in BGP is only compared between routes received from the same autonomous system (AS).
  • You can however, force BGP to compare MED attributes on routes received by different AS’s by configuring the following setting:
    config router bgp
set always-compare-med enable
end