EVPN On Nokia 7450 Notes Part 1
Background
I discovered that Nokia’s road map for MC-LAG does not include active/active. After discussion with Nokia, they suggest EVPN was the best solution to introduce this capability. So after my research on EVPN it was time to POC it. This post assume basic knowledge of EVPN. If you would like to review EVPN, this blog post is a great starting point.
Scenario 1: All-active Multi-homing
In the above topology, PE1 and PE2 will be part of an EVPN-VPLS. LAG 60 will be a multi-homed active/active lag to CE1. LAG 81 and 82 will be traditional LAGs to GW1 and GW2.
The building blocks for EVPN in this solution are:
- OSPF
- RSVP-TE
- MPLS
- BGP
- LAG
- EVPN
- VPLS
Configuration
OSPF, RSVP-TE and MPLS will not be covered as it is very vanilla.
We will start from BGP configuration on PE1. PE2 is very similar. I will bold lines that are important.
*A:PE1>config>router>bgp# info
----------------------------------------------
family evpn
graceful-restart
stale-routes-time 300
exit
damping
group "EVPN"
local-as 65456
peer-as 65456
local-address 192.168.58.210
neighbor 192.168.58.211
exit
exit
no shutdown
----------------------------------------------
Port Configuration to CE1 from PE1.
A:PE1>config>port# info
----------------------------------------------
ethernet
mode access
encap-type dot1q
down-when-looped
keep-alive 30
no shutdown
exit
lldp
dest-mac nearest-bridge
admin-status tx-rx
notification
tx-tlvs sys-name
tx-mgmt-address system
exit
exit
no autonegotiate
exit
no shutdown
----------------------------------------------
LAG configuration to CE1:
- Both PE1 and PE2 have the same system ID and admin key.
*A:PE1>config>lag# info ---------------------------------------------- mode access encap-type dot1q port 2/2/13 lacp active administrative-key 60 system-id 00:00:00:00:05:60 no shutdown ----------------------------------------------
ESI constructs and flag options for multi-homing:
- RD uses :0 to inform the box to self derive per EVI.
- Can not use all zeros in bytes 2-7 for ESI.
- Service carving enforces the PEs to load balance DF role.
- Multi-homing can either be all-active or active-standby.
A:PE1>config>service>system>bgp-evpn# info ---------------------------------------------- route-distinguisher 192.168.58.210:0 ethernet-segment "ESI-60" create esi 00:11:22:33:44:55:66:77:88:60 es-activation-timer 0 service-carving mode auto exit multi-homing all-active lag 60 no shutdown exit ----------------------------------------------
VPLS service construct:
- Unique EVI per VPLS.
- Due to ingress-replication-bum-label. PE advertises another label for BUM traffic.
- MPLS will auto bind tunnels for data plane. We couldn’t get SDP contstruct to work.
A:PE1>config>service>vpls# info ---------------------------------------------- description "EVPN service 1 " bgp exit bgp-evpn evi 60 #needs to be unique per VPLS vxlan shutdown exit mpls ingress-replication-bum-label auto-bind-tunnel resolution any exit no shutdown exit exit stp shutdown exit service-name "EVPN service 1" sap lag-60:81 create description "access_sap" exit sap lag-81:81 create exit no shutdown ----------------------------------------------
Verification
PE1
Physical port from PE1 to CE1 is up and in the correct LAG.
A:PE1# show port 2/2/13 =============================================================================== Ethernet Interface =============================================================================== Description : Interface : 2/2/13 Oper Speed : 1 Gbps Link-level : Ethernet Config Speed : 1 Gbps Admin State : up Oper Duplex : full Oper State : up - Active in LAG 60 Config Duplex : full Physical Link : Yes MTU : 1518 <output omitted>
LAG from PE1 to CE1 is up.
A:PE1# show lag 60 detail =============================================================================== LAG Details =============================================================================== Description : N/A ------------------------------------------------------------------------------- Details ------------------------------------------------------------------------------- Lag-id : 60 Mode : access Adm : up Opr : up Thres. Exceeded Cnt : 16 Port Threshold : 0 Thres. Last Cleared : 09/20/2017 10:22:24 Threshold Action : down Dynamic Cost : false Encap Type : dot1q Configured Address : a0:f3:e4:5f:73:ac Lag-IfIndex : 1342177340 Hardware Address : a0:f3:e4:5f:73:ac Adapt Qos (access) : distribute Hold-time Down : 0.0 sec Port Type : standard Per-Link-Hash : disabled Include-Egr-Hash-Cfg: disabled Forced : - Per FP Ing Queuing : disabled Per FP Egr Queuing : disabled Per FP SAP Instance : disabled LACP : enabled Mode : active LACP Transmit Intvl : fast LACP xmit stdby : enabled Selection Criteria : highest-count Slave-to-partner : disabled MUX control : coupled Subgrp hold time : 0.0 sec Remaining time : 0.0 sec Subgrp selected : 1 Subgrp candidate : - Subgrp count : 1 System Id : 00:00:00:00:05:60 System Priority : 32768 Admin Key : 60 Oper Key : 60 Prtr System Id : 08:96:ad:4e:5b:e4 Prtr System Priority : 32768 Prtr Oper Key : 60 Standby Signaling : lacp Port weight speed : 0 gbps Number/Weight Up : 1 Weight Threshold : 0 Threshold Action : down ------------------------------------------------------------------------------- Port-id Adm Act/Stdby Opr Primary Sub-group Forced Prio ------------------------------------------------------------------------------- 2/2/13 up active up yes 1 - 32768 ------------------------------------------------------------------------------- Port-id Role Exp Def Dist Col Syn Aggr Timeout Activity ------------------------------------------------------------------------------- 2/2/13 actor No No Yes Yes Yes Yes Yes Yes 2/2/13 partner No No Yes Yes Yes Yes No Yes ===============================================================================
Transport labels exchanged between PEs:
- Label for BUM traffic.
- Label for Unicast traffic.
A:PE1# show service id 600 evpn-mpls
===============================================================================
BGP EVPN-MPLS Dest
===============================================================================
TEP Address Egr Label Num. MACs Mcast Last Change
Transport
-------------------------------------------------------------------------------
192.168.58.211 262131 0 Yes 10/04/2017 12:12:23
rsvp
192.168.58.211 262132 1 No 10/06/2017 11:12:06
rsvp
-------------------------------------------------------------------------------
Number of entries : 2
-------------------------------------------------------------------------------
===============================================================================
<output omitted>
VPLS is up.
A:PE1# show service id 600 all =============================================================================== Service Detailed Information =============================================================================== Service Id : 600 Vpn Id : 0 Service Type : VPLS Name : EVPN service 1 for MAS Description : EVPN service 1 for MAS Customer Id : 1 Creation Origin : manual Last Status Change: 09/11/2017 13:55:56 Last Mgmt Change : 09/12/2017 13:00:16 Etree Mode : Disabled Admin State : Up Oper State : Up MTU : 1514 Def. Mesh VC Id : 600 SAP Count : 2 SDP Bind Count : 0 Snd Flush on Fail : Disabled Host Conn Verify : Disabled SHCV pol IPv4 : None Propagate MacFlush: Disabled Per Svc Hashing : Disabled Allow IP Intf Bind: Disabled Fwd-IPv4-Mcast-To*: Disabled Def. Gateway IP : None Def. Gateway MAC : None Temp Flood Time : Disabled Temp Flood : Inactive Temp Flood Chg Cnt: 0 VSD Domain : SPI load-balance : Disabled TEID load-balance : Disabled -------------------------------------------------------------------------------
VPLS BGP RD and RT are auto derived from EVI.
A:PE1# show service id 600 bgp =============================================================================== BGP Information =============================================================================== Vsi-Import : None Vsi-Export : None Route Dist : None Oper Route Dist : 192.168.58.210:60 Oper RD Type : derivedEvi Rte-Target Import : None Rte-Target Export: None Oper RT Imp Origin : derivedEvi Oper RT Import : 65456:60 Oper RT Exp Origin : derivedEvi Oper RT Export : 65456:60 PW-Template Id : None ------------------------------------------------------------------------------- ===============================================================================
SAP is up in VPLS.
A:PE1# show service id 600 sap
===============================================================================
SAP(Summary), Service 600
===============================================================================
PortId SvcId Ing. Ing. Egr. Egr. Adm Opr
QoS Fltr QoS Fltr
-------------------------------------------------------------------------------
lag-60:81 600 1 none 1 none Up Up
lag-81:81 600 1 none 1 none Up Up
-------------------------------------------------------------------------------
Number of SAPs : 2
-------------------------------------------------------------------------------
===============================================================================
VPLS BGP labels allocated and tunnel is activated.
:PE1# show service id 600 bgp-evpn =============================================================================== BGP EVPN Table =============================================================================== MAC Advertisement : Enabled Unknown MAC Route : Disabled CFM MAC Advertise : Disabled VXLAN Admin Status : Disabled Creation Origin : manual MAC Dup Detn Moves : 5 MAC Dup Detn Window: 3 MAC Dup Detn Retry : 9 Number of Dup MACs : 0 IP Route Advertise*: Disabled EVI : 60 ------------------------------------------------------------------------------- Detected Duplicate MAC Addresses Time Detected ------------------------------------------------------------------------------- ------------------------------------------------------------------------------- =============================================================================== * indicates that the corresponding row element may have been truncated. =============================================================================== BGP EVPN MPLS Information =============================================================================== Admin Status : Enabled Force Vlan Fwding : Disabled Control Word : Disabled Split Horizon Group: (Not Specified) Ingress Rep BUM Lbl: Enabled Max Ecmp Routes : 0 Ingress Ucast Lbl : 261591 Ingress Mcast Lbl : 261569 =============================================================================== =============================================================================== BGP EVPN MPLS Auto Bind Tunnel Information =============================================================================== Resolution : any Filter Tunnel Types: (Not Specified) ===============================================================================
The DF for this Ethernet Segment has been selected and the multi-homing flags are correct.
A:PE1# show service system bgp-evpn ethernet-segment name "ESI-60" all =============================================================================== Service Ethernet Segment =============================================================================== Name : ESI-60 Admin State : Enabled Oper State : Up ESI : 00:11:22:33:44:55:66:77:88:60 Multi-homing : allActive Oper Multi-homing : allActive Source BMAC LSB : Lag Id : 60 ES Activation Timer : 0 secs Exp/Imp Route-Target : target:11:22:33:44:55:66 Svc Carving : auto ES SHG Label : 261578 =============================================================================== =============================================================================== EVI Information =============================================================================== EVI SvcId Actv Timer Rem DF ------------------------------------------------------------------------------- 60 600 0 yes ------------------------------------------------------------------------------- Number of entries: 1 =============================================================================== ------------------------------------------------------------------------------- DF Candidate list ------------------------------------------------------------------------------- EVI DF Address ------------------------------------------------------------------------------- 60 192.168.58.210 60 192.168.58.211 ------------------------------------------------------------------------------- Number of entries: 2 ------------------------------------------------------------------------------- ------------------------------------------------------------------------------- =============================================================================== <output omitted>
The 4 BGP route types for EVPN are received from PE2.
A:PE1# show router bgp neighbor 192.168.58.211 received-routes evpn
===============================================================================
BGP Router ID:192.168.58.210 AS:65456 Local AS:65456
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
l - leaked, x - stale, > - best, b - backup, p - purge
Origin codes : i - IGP, e - EGP, ? - incomplete
===============================================================================
BGP EVPN Auto-Disc Routes
===============================================================================
Flag Route Dist. ESI NextHop
Tag Label
-------------------------------------------------------------------------------
u*>i 192.168.58.211:60 00:11:22:33:44:55:66:77:88:60 192.168.58.211
0 LABEL 262132
u*>i 192.168.58.211:60 00:11:22:33:44:55:66:77:88:60 192.168.58.211
MAX-ET LABEL 0
u*>i 192.168.58.211:61 00:11:22:33:44:55:66:77:88:61 192.168.58.211
0 LABEL 262129
u*>i 192.168.58.211:61 00:11:22:33:44:55:66:77:88:61 192.168.58.211
MAX-ET LABEL 0
-------------------------------------------------------------------------------
Routes : 4
===============================================================================
===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag Route Dist. MacAddr ESI
Tag Mac Mobility Ip Address
NextHop
Label1
-------------------------------------------------------------------------------
u*>i 192.168.58.211:60 08:96:ad:4c:19:4a ESI-0
0 Seq:0 N/A
192.168.58.211
LABEL 262132
u*>i 192.168.58.211:60 08:96:ad:4e:5b:e1 00:11:22:33:44:55:66:77:88:60
0 Seq:0 N/A
192.168.58.211
LABEL 262132
-------------------------------------------------------------------------------
Routes : 2
===============================================================================
===============================================================================
BGP EVPN Inclusive-Mcast Routes
===============================================================================
Flag Route Dist. OrigAddr NextHop
Tag
-------------------------------------------------------------------------------
u*>i 192.168.58.211:1 192.168.58.211 192.168.58.211
0
u*>i 192.168.58.211:60 192.168.58.211 192.168.58.211
0
u*>i 192.168.58.211:61 192.168.58.211 192.168.58.211
0
u*>i 192.168.58.211:88 192.168.58.211 192.168.58.211
0
-------------------------------------------------------------------------------
Routes : 4
===============================================================================
===============================================================================
BGP EVPN Eth-Seg Routes
===============================================================================
Flag Route Dist. ESI NextHop
OrigAddr
-------------------------------------------------------------------------------
u*>i 192.168.58.211:0 00:11:22:33:44:55:66:77:88:60 192.168.58.211
192.168.58.211
u*>i 192.168.58.211:0 00:11:22:33:44:55:66:77:88:61 192.168.58.211
192.168.58.211
-------------------------------------------------------------------------------
Routes : 2
===============================================================================
<output omitted>
CE1
LAG on CE1 has both ports active/active.
RP/0/RSP0/CPU0:CE1#show bundle bundle-ether 61
Thu Oct 5 16:48:05.211 UTC
Bundle-Ether61
Status: Up
Local links <active/standby/configured>: 2 / 0 / 2
Local bandwidth <effective/available>: 2000000 (2000000) kbps
MAC address (source): 0222.2222.2222 (Configured)
Inter-chassis link: No
Minimum active links / bandwidth: 1 / 1 kbps
Maximum active links: 64
Wait while timer: 2000 ms
Load balancing: Default
LACP: Operational
Flap suppression timer: Off
Cisco extensions: Disabled
mLACP: Not configured
IPv4 BFD: Not configured
Port Device State Port ID B/W, kbps
-------------------- --------------- ----------- -------------- ----------
Gi0/0/0/6 Local Active 0x8000, 0x0004 1000000
Link is Active
Gi0/0/0/7 Local Active 0x8000, 0x0005 1000000
Link is Active
RP/0/RSP0/CPU0:RTCSR0000003NAT#
Ping GW1/2 physical and Virtual IP to verify connectivity.
RP/0/RSP0/CPU0:CE1#ping 8.8.8.1 source 8.8.8.60 Thu Oct 5 20:26:40.002 UTC Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 8.8.8.1, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/2 ms RP/0/RSP0/CPU0:CE1T#ping 8.8.8.2 source 8.8.8.60 Thu Oct 5 20:26:43.089 UTC Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 8.8.8.2, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/2 ms RP/0/RSP0/CPU0:CE1#ping 8.8.8.3 source 8.8.8.60 Thu Oct 5 20:26:45.713 UTC Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 8.8.8.3, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/2 ms
PE1 and PE2
VPLS forward database:
- Since GW1 is master for VRRP. Most traffic will transverse PE1. We can see 2 MAC addresses are learnt on PE1 then advertise to PE2 via EVPN.
- The MAC address for CE1’s LAG is learnt on both PE1 and PE2.
- PE1 learnt GW2’s MAC address from PE2 via EVPN.
A:PE1# show service id 600 fdb detail
===============================================================================
Forwarding Database, Service 600
===============================================================================
ServId MAC Source-Identifier Type Last Change
Age
-------------------------------------------------------------------------------
600 00:00:5e:00:01:51 sap:lag-81:81 L/0 10/04/17 12:12:23
600 08:96:ad:4c:19:4a eMpls: Evpn 10/07/17 21:35:14
192.168.58.211:262132
600 08:96:ad:4e:5b:e1 sap:lag-60:81 L/9 10/07/17 21:34:53
600 08:96:ad:4e:b1:69 sap:lag-81:81 L/0 10/07/17 21:34:51
-------------------------------------------------------------------------------
No. of MAC Entries: 4
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static
===============================================================================
A:PE2# show service id 600 fdb detail
===============================================================================
Forwarding Database, Service 600
===============================================================================
ServId MAC Source-Identifier Type Last Change
Age
-------------------------------------------------------------------------------
600 00:00:5e:00:01:51 eMpls: Evpn 10/04/17 12:12:30
192.168.58.210:261591
600 08:96:ad:4c:19:4a sap:lag-82:81 L/23 10/07/17 21:34:33
600 08:96:ad:4e:5b:e1 sap:lag-60:81 L/0 10/07/17 21:34:30
600 08:96:ad:4e:b1:69 eMpls: Evpn 10/07/17 21:34:32
192.168.58.210:261591
-------------------------------------------------------------------------------
No. of MAC Entries: 4
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static
===============================================================================