#CiscoLive

Lessons Learned From Multi-

Domain IBN Architectures in
SDA, SDWAN and ACI
Dhrumil Prajapati, Jeremy Bowman
Delivery Architects
BRKENS-3000

#CiscoLive
Agenda
• Introduction
• Design and Deployment Best Practices
• SDA & SDWAN Integration
• SDA and ACI Integration
• SDWAN and ACI Integration
• 100,000ft view on Multi-Domain Design
• Deployment and Migration Lessons
Learned from Large Scale Deployments

#CiscoLive BRKENS-3000 © 2021 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Who are we?
Dhrumil Prajapati
Delivery Architect
Technology and Transformation Group – CX
6+ Years @ Cisco
CCIE #28071 (R/S, SP)
Specialized in: SD-Access, SD-WAN, MPLS,
Campus LAN and WAN
@DhruPrajapati

#CiscoLive BRKENS-3000 © 2021 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Who are we?
Jeremy Bowman
Delivery Architect
Cisco CX
6+ Years @ Cisco
CCIE #51241 (R/S, Security)
CCDE #2018::16
Specialized in: Full Enterprise IBN with Security
@ciscojdb1
[email protected]

#CiscoLive BRKENS-3000 © 2021 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Design and
Deployment
Best Practices
Why Multi-Domain?
• Individual architectures introduce
• Segmentation
• Automation
• Within a single enterprise domain

• Multi-Domain Architectures
• Extend Segmentation
• Utilize orchestration
• Make the entire enterprise one IBN enclave

#CiscoLive BRKENS-3000 © 2021 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
What Is Involved In SDA & SDWAN Integration?
• Steps
• DNAC and vManage integration
• vManage owns each cEdge and assigns to DNAC
• Provision SDA specific changes through DNAC, SDWAN
specific changes via vManage

• Results
• SDA VNs and SDWAN Service VPNs tied together
• SDA SGT information propagated via SDWAN
• cEdge participates in both fabric domains
• Consistent application and security policy
• API based communication between DNAC and vManage

#CiscoLive BRKENS-3000 © 2021 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
SDA and SDWAN Integration

#CiscoLive BRKENS-3000 © 2021 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
What Is Involved In SDA & ACI Integration?
• Steps
• DNAC and APIC both integrate with ISE
• API based interconnection

• Results
• SDA VNs and ACI Contexts tied together
• SDA SGTs and ACI EPGs mapped
• Consistent policy throughout

#CiscoLive BRKENS-3000 © 2021 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
What Is Involved In ACI & SDWAN Integration?
• Steps
• APIC integrates with vManage
• Associate WAN SLA Policy with Contracts
• ACI Tenants matched to SDWAN VPNs

• Results
• Tenants control SDWAN AAR
• DC Segmentation is maintained to the
branch

#CiscoLive BRKENS-3000 © 2021 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
Really Really High-Level View

#CiscoLive BRKENS-3000 © 2021 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
 • SDA
• Endpoints dynamically assigned
SGTs and placed into VNs
• SDWAN
• Extends segmentation

100,000 ft view • Applies APIC/DNAC per-VPN

security and application policy.
• ACI
• End-to-end policy and
segmentation automatically
enforced

BRKENS-3000 © 2021 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Lessons
Learned From
Large Scale
Deployments
SDA and SDWAN Deployments
• Today available in fully automated “one-box” solution or partly
manual “two-box” solution
• One-box solution (integrated solution)
• Features SDA BN/CP and SDWAN WAN Edge in a single box.
• Must be an ASR 1000 or ISR 4000 series router
• Two-box solution (non-integrated solution)
• Clear demarcation between SDA and SDWAN architectures
• SDA BNs can be ISR4K, ASR1K or Cat9K switches, SDWAN edges can
be ISR4K or ASR1K series routers
• SDA and SDWAN designs can be implemented at a different pace

#CiscoLive BRKENS-3000 © 2021 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
SDA and SDWAN Deployments Contd.
• Majority of customers have employed two-box solution for
modularity of deployment and flexibility in operations
• Mapping of VNs and VPNs is crucial
• Inter-site traffic flow greatly depends on SDWAN tunnel design and
SDWAN underlay.
• For Multi-Regional (Global) networks, consistency across multiple
DNAC clusters is key.
• Special consideration for inter-VN routing within the site

#CiscoLive BRKENS-3000 © 2021 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
SDA to SDWAN Integration (One-Box)
MANAGEMENT Cisco DNA Center

Is
AP

ISE VPN IDs created in vManage are

exchanged with Cisco DNA Center.

MANAGEMENT vManage

SD-Access SD-WAN Fabric SD-Access

Fabric Site Fabric Site

WAN Edge WAN Edge

LISP OMP LISP CONTROL PLANE

VXLAN SGT (16 bits) IPSec CMD SGT MPLS VPN VXLAN SGT (16 bits)
DATA PLANE
Header VNID (24 bits) Header Header (16 bits) Labels (20 bits) Header VNID (24 bits)

#CiscoLive BRKENS-3000 © 2021 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
 SDA to SDWAN Integration (Two-Box)
MANAGEMENT Cisco DNA Center

Is
AP

ISE

vManage
MANAGEMENT

vBond vSmart

SD-Access SD-Access
SD-WAN Fabric
Fabric Site Fabric Site

WAN Edge WAN Edge

BGP BGP
LISP VRF-lite OMP VRF-lite LISP CONTROL PLANE

802.1Q 802.1Q
SGT (16 bits) SGT VPN SGT (16 bits)
VXLAN SGT (16 bits) IPSec CMD MPLS SGT (16 bits) VXLAN
DATA PLANE
Header Header Header (16 bits) Labels (20 bits) Header
VNID (24 bits) VLAN ID (12 bits) VLAN ID (12 bits) VNID (24 bits)
Inline Tagging Inline Tagging

#CiscoLive BRKENS-3000 © 2021 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
ACI and SDA Deployments (Phase 2 Integration)
• SGT to EPG mapping is critical, leverage ISE for consistency.
• Create contracts on both side of the fabric - SDA and ACI
• Integration strategies:
• Border/CP at Data Center by treating DC as a site
• VRF-Lite / Tunnels from HQ BN/CP to DC
• BGP/EVPN with VRF-Lite to extend macro and micro segmentation
• Leveraging CMD between SDA Border Nodes and ACI Border Leafs
• A good use case for Multi-Site Remote Border!

#CiscoLive BRKENS-3000 © 2021 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
SDA and ACI Integration (Phase 2)

App Groups, VRFs, App Groups, VRFs, BGP

BGP neighbors neighbors, IP-EPG
APIC

User & Device Groups, BGP

Management neighbor, VN

& Policy
SGT-EPG
Translation
table ACI

SD-Access
Border Border Leaf

Control Plane LISP BGP/EVPN COOP

1
Users VXLAN SGT (16 bits) SGT (16 bits)
iVXLAN EPG (16 bits)

Data Plane Header VNID (24 bits)

iVXLAN
Header
VNID (24 bits) Header VNID (24 bits)

CLASS ID (24 bits)

EPG-SGT
Translate SGT-EPG
Translate

#CiscoLive BRKENS-3000 © 2021 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
ACI and SDWAN Deployments
• ACI Border Leaf to SDWAN cEdge – Standardize Naming/VLANs
• Scale of BGP Peering Sessions
• Visualize traffic flow – Source and Destination
• Verify and document contracts and AAR policies to ensure efficient
routing through WAN.
• WAN MTU consideration crucial
• Very limited capability in current phase

#CiscoLive BRKENS-3000 © 2021 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
Lessons
Learned From
Large Scale
Migrations
SDA and SDWAN Migrations
• Order of operations is key!
• Underlay of SDA and Trusted VN needs to be bridged to overlay of
SDWAN
• DC first approach – get those cEdge headends built first
• At branch, install SDWAN first, test it and then proceed with SDA
• Infrastructure and UAT testing is very critical
• TrustSEC needs to be configured on SDWAN first and then SDA BN
• For sub-interfaces, TrustSEC must be enabled on physical and all
sub-interfaces

#CiscoLive BRKENS-3000 © 2021 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
ACI and SDA Migrations
• Border nodes and Border Leafs integration is key
• Data center as a site architecture with BGP/EVPN/VXLAN
• Currently SDWAN in the middle is not supported
• SXP configuration on BNs crucial for end-to-end segmentation
• Always verify and test this in a lab and use it as a certification test
bed

#CiscoLive BRKENS-3000 © 2021 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
ACI and SDWAN Migrations
• Order of operations is critical
• ACI to cEdge Aggregation Layer facilitates migration of
hosts/applications to ACI and non-migrated WAN to SDWAN
independently.
• Convert cEdge to CLI mode > fine-tune ACI to SDWAN connectivity
> update SDWAN template > reattach template for efficient turn up
of the solution
• More enhancements are in roadmap.

#CiscoLive BRKENS-3000 © 2021 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
Thank you

#CiscoLive
#CiscoLive