A BGP Unnumbered Underlay Across Two Vendors: The Route Was There, the Ping Was Not

Configuring BGP unnumbered#

I have been going back over BGP recently, and BGP unnumbered turns out to be widely used in the underlay of large data centers and AI data centers, because it substantially improves a network’s zero-touch provisioning story. With that as the background, I built a lab to check what I had been reading against something real, using one Arista and one Cisco:

The Cisco configuration. I will not walk through it line by line — the official documentation covers it.

interface Ethernet1/1
  no switchport
  ipv6 address use-link-local-only
  no shutdown

interface loopback0
  ip address 10.100.0.2/32

router bgp 65001
  router-id 10.100.0.2
  address-family ipv4 unicast
    network 10.100.0.2/32
  neighbor Ethernet1/1
    remote-as 65000
    address-family ipv4 unicast
    address-family ipv6 unicast

The Arista configuration. Arista’s own documentation has no detail on this, so the reference here is a community write-up.

interface Ethernet1
   no switchport
   ipv6 enable
!
interface loopback0
  ip address 10.100.0.1/32
!     
ip routing ipv6 interfaces 
!
ipv6 unicast-routing
!
router bgp 65000
   router-id 10.100.0.1
   neighbor CISCO peer group
   neighbor interface Et1 peer-group CISCO remote-as 65001
   !
   address-family ipv4
      neighbor CISCO activate
      neighbor CISCO next-hop address-family ipv6 originate
      network 10.100.0.1/32
   !
   address-family ipv6
      neighbor CISCO activate

Next, check the neighbour relationship and what routes were learned.

Checking neighbours and the routing table#

This is the Arista side. Both the neighbour and the learned routes look fine.

And the Cisco side. Routes and neighbour both fine here too.

Verifying IPv4 connectivity#

Time to test reachability. Pinging Arista from the Cisco side: the capture shows no ping packets leaving the Cisco interface at all. So let me try it the other way, from the Arista side, and see what the reverse direction looks like.

The capture shows Arista does put the packet on the wire, and Cisco never replies — which is what makes the ping fail. So now go and look at the Cisco configuration.

What is going on here? Start by comparing the two configurations. On the Arista side there is neighbor CISCO next-hop address-family ipv6 originate, which tells the device to encode its own IPv4 routes with an IPv6 next hop when sending them to the neighbour — the rule defined in RFC 5549. So the next step is to see whether Cisco’s documentation says anything about this. Following that thread turned up the key detail: if an interface has no IPv4 address configured but still needs to forward IPv4 traffic through it, ip forward is required.

It is worth being precise about what that command is. It does not enable RFC 5549 or RFC 8950. It grants an interface with no IPv4 address the ability to forward IPv4 traffic.

Before changing anything, confirm the current state: the neighbour is Established and 10.100.0.1/32 is installed in the routing table, so the BGP control plane is basically working and the problem is most likely in the data plane.

Cisco# show bgp sessions 
Total peers 1, established peers 1
ASN 65001
VRF default, local ASN 65001
peers 1, established peers 1, local router-id 10.100.0.2
State: I-Idle, A-Active, O-Open, E-Established, C-Closing, S-Shutdown

Neighbor        ASN    Flaps LastUpDn|LastRead|LastWrit St Port(L/R)  Notif(S/R)
fe80::5242:b3ff:fe5a:d518%Ethernet1/1
                65000 3     01:42:23|00:00:29|00:00:21 E   179/43601      0/3
Cisco# show ip route bgp
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

10.100.0.1/32, ubest/mbest: 1/0
    *via fe80::5242:b3ff:fe5a:d518%default, Eth1/1, [20/0], 01:42:25, bgp-65001,
 external, tag 65000

While we are here, note the shape of that route: the prefix is IPv4, 10.100.0.1/32, and the next hop is an IPv6 link-local address, fe80::5242:b3ff:fe5a:d518. That is the most direct view of RFC 8950 you will get in a routing table — we will come back to it when we get to next hop encoding.

Now configure ip forward under Ethernet1/1, giving this interface with no IPv4 address the forwarding qualification it lacks, and ping again:

Cisco(config)# interface Ethernet1/1
Cisco(config-if)# ip forward
Cisco(config-if)# ping 10.100.0.1
PING 10.100.0.1 (10.100.0.1): 56 data bytes
64 bytes from 10.100.0.1: icmp_seq=0 ttl=64 time=44.491 ms
64 bytes from 10.100.0.1: icmp_seq=1 ttl=64 time=13.764 ms
64 bytes from 10.100.0.1: icmp_seq=2 ttl=64 time=11.453 ms
64 bytes from 10.100.0.1: icmp_seq=3 ttl=64 time=8.607 ms
64 bytes from 10.100.0.1: icmp_seq=4 ttl=64 time=12.279 ms

--- 10.100.0.1 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 8.607/18.118/44.491 ms

The ping succeeds, so traffic is reaching its destination, and the cross-vendor BGP unnumbered underlay is complete. Throughout all of this, the BGP neighbour and the routing table never changed. The only thing that changed was the interface’s ability to forward IPv4 — a point we will return to.

RFC 8950: advertising IPv4 NLRI with an IPv6 next hop#

Having sorted out the fault, I followed the thread back to RFC 5549 and found that the IETF has since updated it — the current standard is RFC 8950. Since this was a cross-vendor BGP unnumbered setup, it is worth spending a bit more time on what this RFC actually does underneath, because knowing that something works is not the same as knowing why.

In short: conventionally, IPv4 NLRI uses an IPv4 address as its next hop. In the data center BGP unnumbered scenario, point-to-point links often no longer carry an IPv4 address at all and adjacency is built purely on IPv6 link-local addresses. That raises a problem — how should an IPv4 route express its next hop? RFC 8950 is exactly the answer to that question: it permits IPv4 NLRI to use an IPv6 address as the next hop, so a device can still advertise and learn IPv4 routes over BGP even with no IPv4 addressing on the link. Two parts to the mechanism:

• Capability negotiation (Capability Code 5). When forming the neighbour relationship, the two sides first exchange a signal — the Extended Next Hop Encoding capability — in the OPEN message. Note the direction of that signal: it means willing to receive IPv4 routes encoded with an IPv6 next hop. So whether Arista sends routes that way depends on whether Cisco declared it. If the far end has not declared it, the encoding is not used, which avoids sending something the peer cannot understand and blackholing the route.
• The next hop encoding format is determined by AFI/SAFI together with the Length of Next Hop field. For ordinary IPv4 unicast, 16 or 32 bytes indicates an IPv6 next hop; VPN-IPv4 and similar cases use the corresponding 24- or 48-byte encodings.

Note that BGP unnumbered is not the same thing as RFC 8950. BGP unnumbered describes a design in which point-to-point links are not assigned conventional IP addresses and BGP adjacency is built on the interface or on link-local addresses; RFC 8950 solves the separate problem of how IPv4 NLRI can use an IPv6 next hop. RFC 8950 is one of the important protocol foundations for this kind of design, but it is not BGP unnumbered itself.

RFC 8950 supersedes RFC 5549. One of the significant changes is that it re-specifies how VPN-IPv4 is encoded when using an IPv6 next hop, resolving problems in the corresponding definitions in RFC 5549, and it adds cases such as IPv4 multicast VPN.

The capability negotiation is also visible in the extended capability field of the BGP OPEN message:

As shown below, with neighbor CISCO next-hop address-family ipv6 originate enabled on Arista, the BGP OPEN message carries the capability (left); with it disabled, the OPEN message no longer carries it (right).

Back to the problem we tripped over:

• On the Arista side, neighbor CISCO next-hop address-family ipv6 originate acts on the control plane, and it does more than decide whether local routes are encoded with an IPv6 next hop — it directly determines whether that neighbour’s BGP OPEN message declares the Extended Next Hop Encoding capability (Capability Code 5). In the EOS version and configuration scenario used in this lab, removing the command makes the Extended Next Hop Encoding capability disappear from the capture. So the command affects not only whether IPv4 NLRI uses an IPv6 next hop, but also the Extended Next Hop capability associated with that neighbour.

• The ip forward typed under the interface on the Cisco side solves a data plane problem. This interface has no IPv4 address of its own — on what basis would the forwarding engine allow it to handle and forward IPv4 traffic? This has nothing to do with how the BGP control plane negotiates capabilities or encodes the next hop. It purely grants forwarding qualification to an interface that is missing an IPv4 address.

In other words, end-to-end IPv4 communication over an RFC 8950 BGP unnumbered link requires both that the control plane gets next hop encoding and capability negotiation right, and that the data plane interface is capable of forwarding IPv4. Comparing the EOS and NX-OS versions used in this lab side by side makes it clearer:

Note that the Arista example configuration also requires the global ip routing ipv6 interfaces. From the configuration itself and from community material, this command clearly relates to IPv4 routing behaviour on interfaces with no IPv4 address — but since Arista’s public documentation says relatively little about its semantics, this article will not go further and equate it with Cisco’s ip forward.

Conclusion#

In this lab, the Arista side had both its control plane and data plane configuration complete, so communication worked. The Cisco side had a working control plane with the route successfully installed, but IPv4 forwarding on an interface with no IPv4 address had not been enabled — which produced exactly the symptom of “the route is right there in the table and the ping still fails”.

In hyperscale data centers, BGP unnumbered really is well suited to zero-touch provisioning and removes a great deal of address planning work on point-to-point links. But when integrating across vendors, each vendor’s CLI and implementation can differ noticeably, and underneath it all you end up back at the protocol itself. For problems of this kind — routing is fine but traffic does not pass — separating the control plane from the data plane and then verifying each against the RFC and the vendor documentation usually finds the real cause faster than staring at BGP neighbour state.

References

• RFC 8950 — Advertising IPv4 NLRI with an IPv6 Next Hop
• Cisco Nexus 9000 — Unicast Routing Configuration Guide
• ipSpace — Arista interface eBGP

Behaviour described here comes from the EOS and NX-OS versions used in this lab; other platforms and versions may differ.