Cisco Catalyst 8000 Edge Platforms: An Intermediate Guide

A practical, intermediate-level guide to the Cisco Catalyst 8000 Edge platform family — hardware, IOS XE SD-WAN software, deployment, routing, security, and operations.

Table of Contents

• Chapter 1: Introduction to the Catalyst 8000 Edge Platform Family
• Chapter 2: Hardware Architecture and Models
• Chapter 3: IOS XE Software and Operating Modes
• Chapter 4: Licensing with Cisco DNA and Smart Licensing
• Chapter 5: SD-WAN Fundamentals on the Catalyst 8000
• Chapter 6: Routing and WAN Connectivity
• Chapter 7: Integrated Security and SASE
• Chapter 8: Application-Aware Routing and Quality of Experience
• Chapter 9: Deployment, Provisioning, and Day-0 Operations
• Chapter 10: Monitoring, Troubleshooting, and Lifecycle Management

Chapter 1: Introduction to the Catalyst 8000 Edge Platform Family

Learning Objectives

• Describe the role of the Catalyst 8000 Edge family within Cisco’s enterprise WAN and SASE strategy
• Distinguish the Catalyst 8200, 8300, 8500, and 8000V from each other and from legacy ISR/ASR platforms
• Explain the convergence of routing, SD-WAN, and security in a single edge platform

The Modern Network Edge

For most of networking history, the “edge” of an enterprise network was a quiet place. A branch office had a single router whose job was to push packets toward headquarters over a leased line or an MPLS circuit. The internet, when it was used at all, was reached by backhauling traffic all the way to a central data center, inspecting it there, and sending it back out. That model worked when applications lived in the corporate data center. It breaks down badly when the applications live in the cloud.

Branch, WAN aggregation, and cloud edge use cases

The modern enterprise edge has to serve three distinct roles, and the Catalyst 8000 family is built to cover all three:

• The branch edge. This is the router sitting in a retail store, a regional office, or a clinic. It connects local users and devices to the rest of the organization and, increasingly, directly to the internet and cloud services. A coffee-shop chain with 800 locations has 800 branch edges.
• The WAN aggregation edge. This is the large device at a central site or colocation facility that terminates the connections coming in from all those branches. If each branch is a spoke, the aggregation router is the hub. It must handle thousands of encrypted tunnels at once.
• The cloud edge. As workloads move into AWS, Azure, and Google Cloud, organizations need a router inside the cloud to extend their private network into those environments and to apply consistent policy. This is a software router running on a virtual machine rather than a physical box.

Think of these three edges like a postal system. The branch edge is your neighborhood post office, the aggregation edge is the regional sorting facility that consolidates mail from many neighborhoods, and the cloud edge is a satellite sorting office that opened up wherever a major new customer base appeared. Historically Cisco sold a different product line for each of these roles. The Catalyst 8000 family was designed to provide one consistent platform and one operating system across all of them [Source: https://www.cisco.com/c/en/us/products/collateral/routers/cloud-edge/at-a-glance-c45-744084.html].

Figure 1.1: The three edge roles and how branch traffic flows to aggregation and cloud

flowchart LR
    Branch1["Branch Edge<br/>(retail store)"]
    Branch2["Branch Edge<br/>(regional office)"]
    Branch3["Branch Edge<br/>(clinic)"]
    Agg["WAN Aggregation Edge<br/>(central hub / colo)"]
    Cloud["Cloud Edge<br/>(virtual router in AWS / Azure / GCP)"]
    Internet["Internet & SaaS<br/>(Microsoft 365, Salesforce)"]

    Branch1 -->|encrypted tunnel| Agg
    Branch2 -->|encrypted tunnel| Agg
    Branch3 -->|encrypted tunnel| Agg
    Branch1 -.->|local breakout| Internet
    Agg --> Cloud
    Cloud --> Internet

Drivers: SD-WAN, SASE, and cloud adoption

Three industry shifts pushed Cisco to rethink the edge.

The first is cloud adoption. When the applications a business depends on (Microsoft 365, Salesforce, Workday) live on the public internet, backhauling all traffic to a central data center for inspection adds latency and wastes bandwidth. Branches need to break out to the internet locally while still being protected.

The second is SD-WAN (Software-Defined Wide Area Network). Rather than statically pinning traffic to a single expensive MPLS circuit, SD-WAN builds an intelligent overlay across any available transport, including broadband, MPLS, and cellular, and steers each application down the best path in real time. Cisco’s SD-WAN fabric (originally the Viptela platform, now branded Cisco Catalyst SD-WAN) has three logical planes: a data plane of edge routers, a control plane of vSmart controllers, and a management plane called vManage, now also branded Catalyst SD-WAN Manager [Source: https://www.cisco.com/c/m/en_us/solutions/enterprise-networks/sd-wan/new-landscape-of-networking.html].

Figure 1.2: The three logical planes of the Cisco Catalyst SD-WAN fabric

flowchart TD
    subgraph Management["Management Plane"]
        vManage["vManage<br/>(Catalyst SD-WAN Manager)"]
    end
    subgraph Control["Control Plane"]
        vSmart["vSmart Controllers<br/>(distribute policy & routes)"]
    end
    subgraph Data["Data Plane"]
        Edge1["Edge Router<br/>(cEdge)"]
        Edge2["Edge Router<br/>(cEdge)"]
    end

    vManage -->|configure & monitor| vSmart
    vManage -->|provision| Edge1
    vManage -->|provision| Edge2
    vSmart -->|control policy| Edge1
    vSmart -->|control policy| Edge2
    Edge1 <-->|encrypted overlay tunnel| Edge2

The third is SASE (Secure Access Service Edge), pronounced “sassy.” SASE is the convergence of networking and security into a single, cloud-delivered service. Instead of shipping a firewall appliance to every branch, an organization steers traffic from the edge router up to cloud-based security services. Cisco explicitly calls Catalyst SD-WAN “the foundation for a SASE-enabled architecture,” with the edge router acting as the on-ramp into cloud security products such as Cisco Umbrella and Cisco Secure Access [Source: https://www.cisco.com/c/m/en_us/solutions/enterprise-networks/sd-wan/new-landscape-of-networking.html] [Source: https://www.tufin.com/blog/cisco-sase-architecture].

Where the Catalyst 8000 fits

The Catalyst 8000 Edge Platforms are Cisco’s current-generation enterprise WAN edge routers. They run Cisco IOS XE (the same modern, modular operating system used across Cisco’s enterprise routing and switching portfolio) and are positioned as the gateway to hybrid and multicloud applications, emphasizing application performance, visibility, and security at the WAN edge [Source: https://blogs.cisco.com/networking/catalyst-8000-edge-platforms] [Source: https://www.cisco.com/c/en/us/products/collateral/routers/cloud-edge/at-a-glance-c45-744084.html].

Crucially, a single Catalyst 8000 device can operate in either of two modes: traditional IOS XE routing (CLI-driven, sometimes called SD-Routing or “classic” mode) or as an SD-WAN cEdge router controlled centrally by vManage. The same physical box can therefore start life as a conventional router and later be converted into an SD-WAN node without a hardware swap [Source: https://conscia.com/dk/blog/catalyst-8000-platforms-new-kids-on-the-blog/] [Source: https://www.progent.com/Cisco_Routers_Consulting.htm]. In short, Catalyst 8000 is the modern WAN router family that combines what Cisco’s older ISR and ASR routers did, but with SD-WAN and cloud-edge capabilities built in from the start [Source: https://blogs.cisco.com/networking/catalyst-8000-edge-platforms].

Key Takeaway: The network edge has grown from a single backhaul router into three distinct roles (branch, aggregation, and cloud), driven by cloud adoption, SD-WAN, and SASE. The Catalyst 8000 family is Cisco’s IOS XE-based answer that covers all three roles and can run in either classic routing or SD-WAN mode on the same hardware.

Platform Family Overview

The Catalyst 8000 family is not a single product. It is three physical hardware lines plus one virtual router, each tuned for a different point on the size-and-throughput spectrum. They all share IOS XE and the same SD-WAN feature set, which is what makes the family coherent. The table below previews the lineup before we examine each member.

Figure 1.3: The Catalyst 8000 platform family tree

graph TD
    Family["Catalyst 8000 Edge Platform Family<br/>(shared IOS XE & SD-WAN feature set)"]
    Family --> C8200["Catalyst 8200<br/>Small / medium branch<br/>Fixed 1RU"]
    Family --> C8300["Catalyst 8300<br/>Large branch / regional headend<br/>Modular 1RU or 2RU"]
    Family --> C8500["Catalyst 8500 / 8500L<br/>Core / aggregation headend<br/>Fixed high-density"]
    Family --> C8000V["Catalyst 8000V<br/>Cloud / virtual edge<br/>Software only (VM)"]

    C8200 --> C8200a["C8200-1N-4T<br/>(8-core, app hosting)"]
    C8200 --> C8200b["C8200L-1N-4T<br/>(lean, 4-core)"]

[Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8200-series-edge-platforms/nb-06-cat8200-series-edge-plat-ds-cte-en.html] [Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8200-series-edge-platforms/catalyst-8300-8200-series-edge-platforms-architecture-wp.html]

Catalyst 8200 Series

The Catalyst 8200 Series targets small and medium branches that still need enterprise-grade routing and SD-WAN, but in a compact, cost-effective chassis. These are fixed-configuration platforms with integrated WAN interfaces rather than large modular bays, optimized for secure SD-WAN and branch connectivity with integrated services such as IPsec, firewalling, and cloud onramp [Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8200-series-edge-platforms/nb-06-cat8200-series-edge-plat-ds-cte-en.html] [Source: https://www.cisco.com/c/en/us/products/collateral/routers/cloud-edge/at-a-glance-c45-744084.html].

There are two main models, and the difference between them is instructive. Both are 1RU boxes with four 1-Gigabit WAN ports (two SFP plus two RJ-45) and one NIM slot plus one PIM slot:

• C8200-1N-4T has an 8-core CPU and 8 GB of DRAM (upgradeable), and it supports application hosting / edge compute, meaning you can run containerized applications directly on the router.
• C8200L-1N-4T is the “lean” variant with a 4-core CPU and 4 GB of DRAM, and it does not support on-box containers. It is aimed at branches that get their security from the cloud (a SASE model) and therefore do not need local compute [Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8200-series-edge-platforms/catalyst-8200-series-edge-platforms-faq.html] [Source: https://data.nag.wiki/Cisco/Datasheets/C8000/C8200%20Hardware%20Installation%20Guide.pdf].

Both deliver up to roughly 3.8 Gbps of unencrypted forwarding (Cisco Express Forwarding, or CEF) and on the order of 500 Mbps to 1 Gbps of IPsec throughput once security services are running [Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8200-series-edge-platforms/nb-06-cat8200-series-edge-plat-ds-cte-en.html].

Worked example. A small branch that previously ran an ISR 4321 or 4331 for a 100–200 Mbps internet/MPLS connection can move to a Catalyst 8200 to gain SD-WAN, much higher crypto throughput, and direct cloud access, all without stepping up to a larger modular chassis [Source: https://blog.router-switch.com/2022/02/why-upgrade-to-cisco-catalyst-8000-series-routers/].

Catalyst 8300 Series

The Catalyst 8300 Series is the workhorse of the family, built for medium-to-large branches and regional offices that need modular interfaces, higher throughput, and strong SD-WAN performance [Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8200-series-edge-platforms/catalyst-8300-8200-series-edge-platforms-architecture-wp.html] [Source: https://buyrouterswitch.com/blog/ins-and-outs-of-cisco-catalyst-8300-series-edge-platforms/].

Where the 8200 is fixed, the 8300 is modular. It comes in 1RU and 2RU variants with six built-in WAN ports and slots for one or two Network Interface Modules (NIMs), one or two Service Modules (SMs), and a dedicated PIM slot. More than 70 modules are supported, giving rich connectivity options including multiple WAN types, high-density Ethernet, and 5G. Models whose names end in “4T2X” include 10G SFP+ ports; “6T” models are all 1G [Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8200-series-edge-platforms/catalyst-8300-8200-series-edge-platforms-architecture-wp.html] [Source: https://www.youtube.com/watch?v=Xu6wvBcnrTM]. Representative SKUs include the C8300-2N2S-6T, C8300-2N2S-4T2X, and C8300-1N1S-4T2X (the “2N2S” prefix indicates two NIM and two SM slots).

Under the hood is an x86 system-on-chip with 8 or 12 cores, delivering up to 18.8 Gbps of CEF forwarding and up to 8.6 Gbps of hardware-accelerated IPsec, roughly five times the 8200’s ceiling. The 8300 is also explicitly 5G-ready [Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8200-series-edge-platforms/catalyst-8300-8200-series-edge-platforms-architecture-wp.html] [Source: https://buyrouterswitch.com/blog/ins-and-outs-of-cisco-catalyst-8300-series-edge-platforms/].

Worked example. A large regional office running an ISR 4451 with several T1/E1 and Ethernet modules can move to a Catalyst 8300, keep its modular connectivity, and scale to much higher encrypted throughput while deploying SD-WAN overlays [Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8200-series-edge-platforms/catalyst-8300-8200-series-edge-platforms-architecture-wp.html] [Source: https://blog.router-switch.com/2022/02/why-upgrade-to-cisco-catalyst-8000-series-routers/].

Catalyst 8500 Series

The Catalyst 8500 (and its smaller sibling, the 8500L) is not a branch device at all. It is a high-performance aggregation and data-center platform, the kind of box that sits at a colocation facility or central hub and terminates the encrypted tunnels arriving from hundreds or thousands of branches [Source: https://blog.router-switch.com/2022/02/why-upgrade-to-cisco-catalyst-8000-series-routers/] [Source: https://www.layer23-switch.com/blog/cisco-catalyst-8000-series-router-comparison.html].

These are fixed, high-density platforms with multiple 10/40/100-Gigabit ports, with no branch-style NIM or SM slots, aimed at large-scale WAN aggregation and SD-WAN hubs. Throughput runs from tens of gigabits per second into the 100+ Gbps range, depending on the SKU (examples include the C8500-12X and C8500-12X4QC, and the lighter C8500L-8S4X) [Source: https://www.layer23-switch.com/blog/cisco-catalyst-8000-series-router-comparison.html].

Worked example. A central site using an ASR 1001-HX or 1002-HX as its internet and SD-WAN hub can migrate to a Catalyst 8500 to gain tighter SD-WAN integration, more efficient hardware for encrypted traffic, and cloud-centric features such as Cloud OnRamp [Source: https://blog.router-switch.com/2022/02/why-upgrade-to-cisco-catalyst-8000-series-routers/] [Source: https://www.layer23-switch.com/blog/cisco-isr-vs-asr-difference.html].

Catalyst 8000V virtual router

The Catalyst 8000V is the family member with no front panel. It is a software-only virtual router that runs as a virtual machine or cloud instance across hypervisors and public clouds (VMware, KVM, AWS, Azure, GCP) and provides the same IOS XE feature set as its physical siblings [Source: https://www.cisco.com/site/us/en/products/networking/sdwan-routers/catalyst-8000-edge-platforms/index.html] [Source: https://www.cisco.com/c/en/us/products/collateral/routers/cloud-edge/at-a-glance-c45-744084.html].

Because it has no fixed silicon, the 8000V has no fixed maximum throughput. Performance scales with the licensed throughput tier and the number of vCPUs you allocate, ranging from a few hundred Mbps to multiple Gbps. Its typical jobs are virtual customer premises equipment (vCPE), a virtual SD-WAN edge inside a cloud region, or cloud transit and aggregation. Deployed in AWS or Azure and onboarded into vManage, it becomes the cloud edge described earlier, mapping branch VPNs to cloud VPCs/VNETs so that the same segmentation and security policy follows workloads into the cloud [Source: https://www.ciscolive.com/c/dam/r/ciscolive/global-event/docs/2024/pdf/BRKSEC-2092.pdf].

Key Takeaway: The family scales from the fixed, compact 8200 for small branches, through the modular 8300 workhorse for large branches and regional hubs, up to the fixed high-density 8500 for aggregation, plus the software-only 8000V for cloud. They differ in form factor and throughput but share IOS XE and one SD-WAN feature set, which is what makes managing them as a single fabric possible.

Migration from Legacy Platforms

The Catalyst 8000 family did not appear in a vacuum. It is the strategic successor to two earlier, separate Cisco product lines, and understanding that lineage is essential to understanding why the family is shaped the way it is.

ISR 4000 and ASR 1000 comparison

Two legacy families dominated the enterprise edge for the previous decade:

• The ISR 4000 (Integrated Services Router) series was the branch router. The ISR 4321 and 4331 served small sites; the 4451 and 4461 served larger ones. These were general-purpose multi-service routers handling routing, voice, and security at the branch.
• The ASR 1000 (Aggregation Services Router) series was the high-performance aggregation and WAN-edge platform. Models such as the ASR 1001-HX and 1002-HX commonly served as internet edges and SD-WAN or DMVPN hubs at central sites.

Cisco maps the new family onto the old one quite explicitly. The Catalyst 8300 is described as “an upgrade to the 4400 Integrated Services Router,” and the Catalyst 8500 is described as “an update to the 1001-HX and 1002-HX” ASR models. The 8200 and 8300 together take over the ISR 4000 branch roles, while the 8500 inherits the ASR 1000 aggregation role [Source: https://blog.router-switch.com/2022/02/why-upgrade-to-cisco-catalyst-8000-series-routers/]. The mapping is summarized below.

Figure 1.5: Migration mapping from legacy ISR/ASR platforms to the Catalyst 8000 family

flowchart LR
    ISR1["ISR 4321 / 4331<br/>(small/medium branch)"] --> C8200["Catalyst 8200"]
    ISR2["ISR 4451 / 4461<br/>(large branch)"] --> C8300["Catalyst 8300"]
    ASR["ASR 1001-HX / 1002-HX<br/>(aggregation / WAN hub)"] --> C8500["Catalyst 8500"]
    CSR["CSR1000v / virtual ISR<br/>(cloud / NFV router)"] --> C8000V["Catalyst 8000V"]

[Source: https://blog.router-switch.com/2022/02/why-upgrade-to-cisco-catalyst-8000-series-routers/] [Source: https://www.cisco.com/c/en/us/products/collateral/routers/cloud-edge/at-a-glance-c45-744084.html]

The single biggest technical distinction between old and new is where SD-WAN lives. On the ISR 4000 and ASR 1000, SD-WAN was effectively a bolt-on capability. On the Catalyst 8000, SD-WAN is a primary design target: the hardware is crypto-optimized to sustain high IPsec throughput, the family is 5G-ready, and the same operating system family spans branch through aggregation, simplifying lifecycle and configuration management [Source: https://blogs.cisco.com/networking/catalyst-8000-edge-platforms] [Source: https://www.progent.com/Cisco_Routers_Consulting.htm].

Why Cisco converged the portfolio

Maintaining two distinct hardware lines, each with its own modules, software images, and support processes, is expensive for Cisco and confusing for customers. Convergence onto one IOS XE-based family yields several concrete benefits. A single operating system spans the entire edge, so an engineer who knows one box knows them all. The same SD-WAN fabric and the same vManage console manage branch, hub, and cloud devices together. And the hardware is purpose-built for the encrypted, cloud-bound traffic that dominates modern networks, rather than for the legacy WAN technologies (such as TDM serial circuits) that older platforms were designed around [Source: https://blogs.cisco.com/networking/catalyst-8000-edge-platforms] [Source: https://www.layer23-switch.com/blog/cisco-catalyst-8000-series-router-comparison.html].

The analogy is a manufacturer consolidating three separate engine families onto one modular platform. The small car, the SUV, and the truck now share a common engine architecture and diagnostic system; only the displacement and tuning differ. A mechanic trained on one can service all three, and the factory builds one line instead of three.

That said, convergence is a direction, not an overnight event. The ISR 4000 and ASR 1000 remain supported and still appear in Cisco’s routing portfolio, especially for niche unified-communications and TDM voice roles that the Catalyst 8000 ecosystem may not yet fully match. A staged migration is the norm: new and refreshed sites deploy Catalyst 8000, while legacy sites with special feature dependencies stay on ISR/ASR until their hardware refresh windows align [Source: https://www.cisco.com/c/dam/m/ru_ua/training-events/2021/cisco-tech-talks/pdf/routing.pdf] [Source: https://www.router-switch.com/faq/cisco-catalyst-8000-wan-edge-modernization.html].

Investment protection and licensing transition

Because a Catalyst 8000 can run in either classic IOS XE mode or SD-WAN mode, organizations can protect their investment by separating two changes that older migrations forced together. The hardware swap (replacing aging ISR/ASR boxes with Catalyst 8000) and the mode switch (moving from classic routing to a vManage-controlled SD-WAN fabric) are distinct projects. A common approach is to deploy Catalyst 8000 in classic mode first, keeping the existing routing design and operational model intact, then convert the same boxes to SD-WAN later once the controllers and policy design are ready [Source: https://conscia.com/dk/blog/catalyst-8000-platforms-new-kids-on-the-blog/] [Source: https://www.progent.com/Cisco_Routers_Consulting.htm].

Figure 1.4: Dual-mode operation of a single Catalyst 8000 device

stateDiagram-v2
    [*] --> ClassicMode
    ClassicMode: Classic IOS XE Routing<br/>(CLI-driven / SD-Routing)
    SDWANMode: SD-WAN cEdge Mode<br/>(controlled by vManage)
    ClassicMode --> SDWANMode: convert when controllers<br/>& policy are ready
    SDWANMode --> ClassicMode: revert (no hardware swap)
    ClassicMode --> ClassicMode: keep existing routing design
    SDWANMode --> SDWANMode: enforce segmentation<br/>& cloud security

The most significant commercial change is licensing. Where the older ISR/ASR generation leaned on perpetual licenses, Catalyst 8000 platforms use Cisco DNA subscriptions and IOS XE software licensing, particularly for SD-WAN and advanced security services. This is a subscription-heavy model [Source: https://www.cisco.com/c/en/us/products/collateral/routers/cloud-edge/at-a-glance-c45-744084.html] [Source: https://www.progent.com/Cisco_Routers_Consulting.htm]. The practical implication is that budgeting must account for recurring per-router subscription cost, not just the one-time hardware purchase. Migration planners should also inventory existing features (MPLS, advanced QoS, TDM voice, DMVPN) and confirm each is supported on the planned IOS XE release and in the intended mode, since some legacy WAN protocols available in classic mode are not available in SD-WAN mode. Module reuse deserves the same scrutiny: the compatibility of existing ISR 4000 cards with the Catalyst 8300 varies by module type and must be checked against official support before a swap [Source: https://www.router-switch.com/faq/cisco-catalyst-8000-wan-edge-modernization.html] [Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8200-series-edge-platforms/catalyst-8300-8200-series-edge-platforms-architecture-wp.html].

Key Takeaway: Catalyst 8000 succeeds the ISR 4000 (branch) and ASR 1000 (aggregation) lines, converging two product families onto one IOS XE platform built for SD-WAN and encrypted cloud traffic. Because a single box runs both classic and SD-WAN modes, hardware refresh and SD-WAN adoption can be staged separately, but planners must budget for the shift to subscription-based DNA licensing and verify feature and module support before migrating.

Chapter Summary

The enterprise network edge has been transformed by the move of business-critical applications into the cloud. What was once a simple backhaul router now has to serve three roles at once: connecting branches directly and securely to the internet and cloud, aggregating the encrypted traffic from those branches at central hubs, and extending the corporate network into public-cloud regions. Cisco’s answer to all three is the Catalyst 8000 Edge Platform family, a single IOS XE-based portfolio that anchors the company’s SD-WAN and SASE strategy. The same device can act as a conventional router or, under the control of vManage, as an SD-WAN cEdge that enforces segmentation and steers traffic into cloud-delivered security services such as Cisco Umbrella and Cisco Secure Access.

The family scales across four members that share one operating system and one feature set. The fixed, compact Catalyst 8200 serves small and medium branches, with a “lean” 8200L variant aimed at cloud-secured (SASE) sites. The modular Catalyst 8300 is the workhorse for larger branches and regional headends, offering 5G readiness, more than 70 interface modules, and up to 18.8 Gbps of forwarding. The fixed, high-density Catalyst 8500 handles core and data-center aggregation at tens to hundreds of gigabits per second. And the software-only Catalyst 8000V brings the same capabilities into the public cloud, scaling by license and vCPU rather than by fixed silicon.

Finally, the Catalyst 8000 is best understood as the convergence of two legacy lines, the ISR 4000 branch routers and the ASR 1000 aggregation routers, onto one modern platform with SD-WAN designed in rather than bolted on. That convergence simplifies operations, but it is a multi-year journey rather than a single event. Because each box runs both classic and SD-WAN modes, organizations can refresh hardware and adopt SD-WAN as separate, staged projects, protecting their existing investment while planning carefully for the move to subscription-based DNA licensing and verifying feature and module parity along the way.

Key Terms

Chapter 2: Hardware Architecture and Models

Learning Objectives

• Compare the hardware specifications and target use cases of the 8200, 8300, and 8500 series
• Identify interface options, module slots, and performance characteristics for each model
• Select the appropriate platform model for a given branch or aggregation scenario

The Cisco Catalyst 8000 Edge family is not a single product but a graduated lineup, deliberately tiered so that a network architect can match silicon to the size of the site. At the low end sits the 8200, built for small and medium branches. In the middle is the 8300, aimed at large, feature-rich branches and campus edges. At the top are the 8500 and 8500L, which are aggregation and headend platforms rather than branch routers at all [Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8200-series-edge-platforms/nb-06-cat8200-series-edge-plat-ds-cte-en.html]. As you climb the family, CPU core count, memory ceiling, encrypted throughput, and interface speed all increase together.

Figure 2.1: Catalyst 8000 Edge family hierarchy by role

graph TD
    F["Catalyst 8000 Edge Family<br/>(shared IOS XE + SD-WAN)"]
    F --> B["Branch routers"]
    F --> A["Aggregation / headend"]
    B --> M8200["8200 series<br/>small/medium branch"]
    B --> M8300["8300 series<br/>large branch / campus edge"]
    A --> M8500["8500 series<br/>DC / colo headend"]
    M8200 --> M8200L["8200L<br/>cost-optimized, 4-core"]
    M8200 --> M8200F["8200<br/>full-power, 8-core"]
    M8300 --> M83001["8300-1N1S<br/>8-core"]
    M8300 --> M83002["8300-2N2S<br/>12-core"]
    M8500 --> M8500L["8500L<br/>x86 + HW crypto"]
    M8500 --> M8500F["8500<br/>QFP 3.0 ASIC"]

A useful analogy is the engine lineup of a single car model. The same chassis and dashboard (here, the Cisco IOS XE software and SD-WAN feature set) is offered with a four-cylinder, a six-cylinder, and a turbocharged V8. The driving experience feels familiar across all three, but the towing capacity, top speed, and price differ dramatically. The chapters that follow walk through each “engine,” then close with a master comparison table and a decision framework for picking the right one.

Catalyst 8200 Hardware

The Catalyst 8200 series is the entry point into the Catalyst 8000 Edge family, engineered for small and medium branch offices where the WAN is typically sub-gigabit and most heavy security processing is offloaded to the cloud [Source: https://network-switch.com/blogs/routers/cisco-catalyst-8200-edge-platforms-routers]. There are two members of the family: the C8200-1N-4T (the full-power model) and the C8200L-1N-4T, almost always referred to simply as the 8200L (the cost-optimized model).

8200 and 8200L models

The headline difference between the two is the processor and the memory it feeds. The C8200-1N-4T ships with an 8-core Intel x86 CPU, while the 8200L is built on a 4-core Intel x86 CPU [Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8200-series-edge-platforms/nb-06-cat8200-series-edge-plat-ds-cte-en.html] [Source: https://www.router-switch.com/pdf2html/pdf/cis:c8200l-1n-4t-datasheet.pdf]. Those cores are general-purpose: on the 8200 there is no separate forwarding ASIC, so the same x86 silicon handles packet processing, encryption, and any on-box services or containers. Halving the core count, as the 8200L does, directly halves the budget available for all of that work.

Memory follows the same pattern. The C8200-1N-4T ships with 8 GB of DRAM, upgradeable to 16 GB or 32 GB. The 8200L ships with just 4 GB of DRAM, but can also be upgraded up to 32 GB [Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8200-series-edge-platforms/nb-06-cat8200-series-edge-plat-ds-cte-en.html] [Source: https://data.nag.wiki/Cisco/Datasheets/C8000/C8200%20Hardware%20Installation%20Guide.pdf]. Both models carry 8 GB of onboard flash that is soldered down and non-upgradable. Where they diverge again is removable storage: the 8200 includes a 16 GB M.2 module by default and can scale up to a 600 GB M.2 NVMe drive (the Hardware Installation Guide notes configurations up to 2 TB NVMe), whereas the 8200L ships with no default M.2 but accepts the same optional storage modules [Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8200-series-edge-platforms/nb-06-cat8200-series-edge-plat-ds-cte-en.html] [Source: https://data.nag.wiki/Cisco/Datasheets/C8000/C8200%20Hardware%20Installation%20Guide.pdf]. That extra storage matters when you intend to run application-hosting containers locally, such as a security agent or an analytics collector.

The practical takeaway is that the 8200L is not a different architecture; it is the same router with a smaller engine and an empty trunk. You choose it when the branch is small, the WAN is slow, and security lives in the cloud rather than on the box.

Form factor and interfaces

Both 8200 models occupy a compact 1RU (one rack unit) form factor, making them easy to slot into a small branch wiring closet or a half-rack [Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8200-series-edge-platforms/nb-06-cat8200-series-edge-plat-ds-cte-en.html]. The fixed interface set is identical on both: four 1-Gigabit Ethernet ports, delivered as two SFP ports plus two RJ-45 copper ports running 10/100/1000 [Source: https://data.nag.wiki/Cisco/Datasheets/C8000/C8200%20Hardware%20Installation%20Guide.pdf]. The “4T” in the part number C8200-1N-4T refers to these four fixed ports.

Beyond the fixed ports, expansion comes from two module bays:

• One NIM slot. A NIM (Network Interface Module) is a hot-pluggable card that adds WAN or LAN connectivity the chassis does not have natively, such as a T1/E1 circuit, a serial link, or additional Ethernet ports [Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8200-series-edge-platforms/nb-06-cat8200-series-edge-plat-ds-cte-en.html]. Think of the NIM slot as an expansion card slot on a PC motherboard: the base system is complete, but the slot lets you bolt on the specific connectivity a given site demands.
• One PIM slot. A PIM (Pluggable Interface Module) on the 8200 is typically used for cellular WAN, that is, a 4G/5G/LTE modem module [Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8200-series-edge-platforms/nb-06-cat8200-series-edge-plat-ds-cte-en.html]. This is what lets a small branch use a wireless carrier link as a backup, or even a primary, transport.

This “4 fixed + 1 NIM + 1 PIM” layout is the signature of the 8200 family and a quick way to recognize it on a data sheet.

Figure 2.2: Catalyst 8200 chassis interface layout

flowchart LR
    subgraph Chassis["C8200-1N-4T / 8200L (1RU)"]
        direction TB
        subgraph Fixed["Fixed ports (4T)"]
            SFP["2 x SFP<br/>1 GbE"]
            RJ["2 x RJ-45<br/>10/100/1000"]
        end
        NIM["1 x NIM slot<br/>(T1/E1, serial, Ethernet)"]
        PIM["1 x PIM slot<br/>(4G/5G/LTE cellular)"]
    end

Performance and scale targets

Because the 8200 forwards and encrypts packets entirely on its general-purpose CPU, its throughput tracks core count. The C8200-1N-4T delivers up to roughly 1 Gbps of IPsec SD-WAN throughput, while the 8200L, with half the cores, delivers up to roughly 500 Mbps of IPsec SD-WAN throughput [Source: https://network-switch.com/blogs/routers/cisco-catalyst-8200-edge-platforms-routers]. For context, the closely related uCPE (universal customer premises equipment) variant, the C8200-UCPE-1N8, is rated for up to 1.85 Gbps routed and about 500 Mbps SD-WAN IPsec, illustrating how much encryption alone subtracts from raw routing capacity [Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8200-series-edge-ucpe/nb-06-cat8200-series-edge-ucpe-ds-cte-en.html].

A worked example clarifies model selection. Suppose a retail chain has 300 stores, each with a 300 Mbps broadband circuit and an LTE backup, and a corporate mandate that all internet traffic be inspected by a cloud security service (a SASE model). Here the branch router only needs to build SD-WAN tunnels, apply basic policy, and steer traffic to the cloud; it does not run a heavy on-box firewall. The 8200L is the right fit: its ~500 Mbps IPsec ceiling comfortably exceeds the 300 Mbps circuit, the single PIM slot accepts the LTE backup module, and there is no need for the extra cores or storage of the full 8200. If, instead, those same stores needed to run an on-box security container and serve 700–900 Mbps of encrypted traffic, the C8200-1N-4T with its 8 cores and expandable storage becomes the correct choice.

Key Takeaway: The 8200 family is a compact 1RU branch router with four fixed 1 GbE ports, one NIM slot, and one PIM slot for cellular. The 8200L is a cost-optimized version with half the cores (4 vs 8) and about half the encrypted throughput (~500 Mbps vs ~1 Gbps), best suited to small, cloud-secured branches.

Catalyst 8300 Hardware

The Catalyst 8300 series steps up to larger, high-performance branches and campus edges. It keeps the same IOS XE software experience as the 8200 but adds more CPU cores, higher-speed 10 GbE uplinks, and far richer modularity [Source: https://www.youtube.com/watch?v=PTHsSilh4PY]. If the 8200 is the four-cylinder branch router, the 8300 is the six-cylinder that can also tow service modules.

8300 1N1S and 2N2S variants

The two branch chassis you will encounter most often are the C8300-1N1S-4T2X (usually called the 8300-1N1S) and the C8300-2N2S-4T2X (usually called the 8300-2N2S). The naming convention is worth memorizing because it directly encodes the modular capacity: “N” stands for NIM slots and “S” stands for service module (SM-X) slots [Source: https://www.scribd.com/document/705644263/cat-8300-series-edge-qa]. Therefore:

• 8300-1N1S has 1 NIM slot + 1 SM-X slot.
• 8300-2N2S has 2 NIM slots + 2 SM-X slots [Source: https://www.scribd.com/document/705644263/cat-8300-series-edge-qa].

The “4T2X” suffix describes the fixed ports, identical on both variants: four 1-Gigabit Ethernet copper ports (4T) plus two 10-Gigabit Ethernet SFP+ ports (2X) [Source: https://www.youtube.com/watch?v=PTHsSilh4PY]. Those two 10G ports are the defining hardware upgrade over the 8200, enabling the 8300 to terminate metro Ethernet or multiple high-speed circuits at a regional branch.

The two variants also differ in compute. The smaller 1N1S carries an 8-core x86 CPU, while the larger 2N2S steps up to a 12-core x86 CPU [Source: https://www.youtube.com/watch?v=PTHsSilh4PY] [Source: https://www.ciscolive.com/c/dam/r/ciscolive/global-event/docs/2022/pdf/BRKARC-2882.pdf]. DRAM is model-dependent, typically starting at 8–16 GB and scaling much higher than the 8200, with on-box service capacity to match.

NIM and SM-X module support

The 8300’s flexibility comes from its broad module ecosystem. It accepts both the new Catalyst 8000-class NIMs and most legacy ISR 4000 NIMs, which protects existing module investments for customers migrating from older ISR routers.

The new C-NIM family adds MACsec (Media Access Control Security, line-rate Layer 2 encryption) across the board [Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8300-series-edge-platforms/datasheet-c78-744088.html]:

In addition, Cisco states that the 8300 supports “all the NIM modules supported on the 4000 ISR models” with a small number of exceptions such as the NIM-1GE-CU-SFP [Source: https://www.scribd.com/document/705644263/cat-8300-series-edge-qa]. That legacy list includes serial and async modules (NIM-1T/2T/4T, NIM-16A/24A), ISDN BRI modules, T1/E1 multiflex trunk modules, voice and PVDM (Packet Voice Digital Signal Processor Module) modules, and the NIM-ES2 series of small Ethernet switch modules [Source: https://www.ciscolive.com/c/dam/r/ciscolive/global-event/docs/2022/pdf/BRKARC-2882.pdf].

The SM-X slots accept service modules in the larger SM-X form factor. An SM-X module is a service module that can host compute, switching, or specialized functions. Supported options include the SM-X-64A high-density async module, Cisco UCS-E M3 server blades (UCS-E160S-M3, UCS-E1120D-M3, UCS-E180D-M3) that effectively put a small x86 server inside the router for branch compute, and the C-SM-NIM-ADPT carrier card, which is an SM-X-format adapter that holds up to two NIMs in a single SM-X slot [Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8300-series-edge-platforms/datasheet-c78-744088.html] [Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8300-series-edge-platforms/catalyst-8000-gigabit-ethernet-wan-modules-ds.html]. One important architectural note: the 8300 does not use ASR 1000 SPA modules and has no SPA slots, so port adapters from that older platform cannot be reused [Source: https://www.scribd.com/document/705644263/cat-8300-series-edge-qa].

Onboard crypto and CPU offload

The 8300 still uses general-purpose x86 cores rather than a dedicated forwarding ASIC, but it allocates those cores intelligently and uses hardware crypto acceleration to keep encryption from monopolizing the CPU. Cisco’s published core allocation for the 12-core 2N2S model is illustrative: 1 core for the control plane, 2 cores for I/O, 4 cores for packet processing, and 5 cores for services (SD-WAN security and third-party applications) [Source: https://www.youtube.com/watch?v=PTHsSilh4PY] [Source: https://www.ciscolive.com/c/dam/r/ciscolive/global-event/docs/2022/pdf/BRKARC-2882.pdf].

Figure 2.3: 8300-2N2S 12-core CPU allocation

flowchart LR
    CPU["12-core x86 CPU<br/>(8300-2N2S)"]
    CPU --> Ctrl["1 core<br/>Control plane"]
    CPU --> IO["2 cores<br/>I/O"]
    CPU --> PP["4 cores<br/>Packet processing"]
    CPU --> Svc["5 cores<br/>Services<br/>(SD-WAN security, apps)"]

This split is the key to understanding why the 8300 can run a rich feature set at multi-gigabit speed. By reserving a dedicated pool of cores for services and another for packet forwarding, the platform avoids the contention that limits the 8200, where every function shares the same small pool. The result is multi-gigabit SD-WAN/IPsec throughput, well beyond what the 8200 can sustain, while still hosting security and application workloads on-box [Source: https://www.youtube.com/watch?v=PTHsSilh4PY] [Source: https://www.ciscolive.com/c/dam/r/ciscolive/global-event/docs/2022/pdf/BRKARC-2882.pdf]. The analogy is a kitchen with dedicated stations: one cook plates orders (control plane), two run the pass (I/O), four work the line (packet processing), and five handle specials (services), so a rush at one station does not stall the others.

Key Takeaway: The 8300 is a modular branch and campus-edge router whose name encodes its slots: 1N1S means 1 NIM + 1 SM-X, and 2N2S means 2 NIM + 2 SM-X. It adds two 10 GbE SFP+ uplinks, 8 to 12 x86 cores, a broad NIM/SM-X ecosystem (including legacy ISR 4000 modules), and a deliberate core-allocation scheme that delivers multi-gigabit encrypted throughput while still running on-box services.

Catalyst 8500 Hardware

The Catalyst 8500 and 8500L are a different class of device. Despite sharing the Catalyst 8000 branding and IOS XE software, they are aggregation and headend platforms, designed to terminate the SD-WAN tunnels coming from hundreds or thousands of branch routers, not to sit in a branch themselves [Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8500-series-edge-platforms/white-paper-c11-2395855.html]. Where the 8200 and 8300 are the cars in the parking lot, the 8500 is the toll plaza that all of those cars converge on.

8500 and 8500L aggregation models

There are two tiers. The Catalyst 8500 (non-L) is the high-performance, modular tier built for very high throughput and large scale. The Catalyst 8500L is the compact, fixed, lower-cost tier aimed at entry-level 1G/10G aggregation [Source: https://www.networkworld.com/article/969558/cisco-bolsters-edge-networking-family-with-expanded-sd-wan-security-options.html]. Both are aggregation-class; the 8500L is explicitly described as “a lower-end version of the 8500,” not a branch device [Source: https://www.sdxcentral.com/news/cisco-adds-5g-option-to-catalyst-router-portfolio/].

A representative 8500L model, the 8500L-8S4X, is a fixed chassis with 8 × 1G SFP ports and 4 × 10G SFP+ ports and no modular interface slots; the port count and type are baked into the SKU [Source: https://www.cisco.com/c/en/us/td/docs/routers/cloud_edge/c8500l/b-catalyst-8500L-8S4X-edge-platform-hig/m-overview_85L.html] [Source: https://www.networkgenetics.net/new-cisco-c8500l-8s4x-catalyst-8500-series-4x-sfp-and-8x-sfp-4x10ge-8x1ge/]. Higher-end 8500 models, such as the 8500-20X6C, add far more high-speed density, including multiple 10/25G ports and 40/100G interfaces, for data-center and colocation WAN edges [Source: https://www.nexusitx.com/products/cisco-catalyst-8500-series-edge-platforms] [Source: https://www.youtube.com/watch?v=9qSRDhRXhpk].

QuantumFlow and x86 architectures

The single most important hardware distinction in the 8500 family is the data plane. The two tiers solve the aggregation problem with fundamentally different silicon.

The Catalyst 8500 uses Cisco’s third-generation QuantumFlow Processor (QFP 3.0), a custom ASIC (application-specific integrated circuit) dedicated to the data plane [Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8500-series-edge-platforms/white-paper-c11-2395855.html]. The QFP offloads packet forwarding, QoS, NAT, and VPN/crypto from the general-purpose CPU, leaving the route processor free to focus almost entirely on control-plane work: routing protocols, SD-WAN orchestration, and management [Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8500-series-edge-platforms/white-paper-c11-2395855.html] [Source: https://blogs.cisco.com/networking/safeguard-your-wan-from-quantum-computing-threats]. This is the same architectural lineage as the venerable ASR 1000, long prized for its IPsec and services throughput [Source: https://www.reddit.com/r/Cisco/comments/k6kru9/catalyst_8500/].

The Catalyst 8500L, by contrast, uses an x86-based architecture with hardware crypto accelerators rather than a discrete QFP [Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8500-series-edge-platforms/intro-c8500l-edge-platform-wp.html] [Source: https://www.ciscolive.com/c/dam/r/ciscolive/global-event/docs/2022/pdf/BRKARC-2882.pdf]. Its goal is to deliver the same software experience as the ASR 1000 and Catalyst 8500 but with a general-purpose CPU driving the data plane, trading peak performance for lower cost [Source: https://www.ciscolive.com/c/dam/r/ciscolive/global-event/docs/2022/pdf/BRKARC-2882.pdf].

Figure 2.4: 8500 (QFP ASIC) vs 8500L (x86) data-plane architecture

flowchart LR
    subgraph C8500["Catalyst 8500"]
        RP1["Route processor<br/>(control plane only)"]
        QFP["QFP 3.0 ASIC<br/>forwarding, QoS, NAT, crypto"]
        RP1 -->|"offloads data plane"| QFP
        QFP --> Out1["Hundreds of Gbps<br/>thousands of tunnels"]
    end
    subgraph C8500L["Catalyst 8500L"]
        CPU2["x86 CPU<br/>control plane + forwarding"]
        HW["HW crypto accelerators"]
        CPU2 -->|"shares CPU cycles"| HW
        HW --> Out2["Tens of Gbps<br/>hundreds of tunnels"]
    end

The dedicated ASIC versus shared x86 distinction is exactly analogous to a graphics card versus integrated graphics in a PC. A discrete GPU (the QFP) does its specialized job at high speed without taxing the main CPU, while integrated graphics (the 8500L’s x86 crypto) shares the main processor and competes for its cycles. This has a direct operational consequence: on an 8500, crypto runs on dedicated silicon, so control-plane CPU stays relatively idle even at high tunnel counts and high IPsec throughput. On an 8500L, crypto competes for x86 CPU cycles even with hardware assist, so you reach throughput and scaling limits earlier as you add tunnels, QoS, NAT, and firewall features [Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8500-series-edge-platforms/intro-c8500l-edge-platform-wp.html] [Source: https://www.ciscolive.com/c/dam/r/ciscolive/global-event/docs/2022/pdf/BRKARC-2882.pdf]. For the same licensed bandwidth tier, the 8500 sustains higher encrypted throughput and more tunnels at lower CPU utilization than an 8500L at comparable load.

A note on monitoring follows from this. On an 8500L you watch route-processor CPU utilization closely, because it reflects both forwarding and control-plane health. On an 8500 the primary saturation indicator is QFP utilization, not route-processor CPU, because the ASIC is where the forwarding work actually happens [Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8500-series-edge-platforms/white-paper-c11-2395855.html].

High-throughput WAN aggregation

The 8500 family is positioned squarely for WAN aggregation, and the two tiers map cleanly onto two deployment scales. The 8500 targets high-performance SD-WAN aggregation for enterprise data centers, colocation facilities, and large campuses, with aggregate throughput in the hundreds-of-Gbps class and the ability to terminate many thousands of SD-WAN tunnels [Source: https://www.nexusitx.com/products/cisco-catalyst-8500-series-edge-platforms] [Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8500-series-edge-platforms/white-paper-c11-2395855.html]. The 8500L is positioned as an entry-level 1G/10G aggregation router for regional hubs and large-branch aggregation, with aggregate throughput in the tens of Gbps [Source: https://www.networkworld.com/article/969558/cisco-bolsters-edge-networking-family-with-expanded-sd-wan-security-options.html] [Source: https://www.cisco.com/c/en/us/td/docs/routers/cloud_edge/c8500l/b-catalyst-8500L-8S4X-edge-platform-hig/m-overview_85L.html].

Two worked examples make the boundary concrete:

• Global data-center hub. A requirement of 4 × 100G uplinks to MPLS, DIA, and cloud on-ramps, plus 10 or more 10/25G secondary links, terminating thousands of IPsec tunnels with multi-tenant segmentation and a full feature set. The QFP 3.0 of the Catalyst 8500 handles the encrypted throughput and feature-rich forwarding without overloading the route processor, and high-density models like the 8500-20X6C supply the required 10/25/40/100G ports. The 8500L would be undersized here [Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8500-series-edge-platforms/white-paper-c11-2395855.html] [Source: https://www.youtube.com/watch?v=9qSRDhRXhpk].
• Regional aggregation hub. A mix of 1G and 10G circuits from 100–200 sites, aggregate traffic well under high-end 8500 capacity, hundreds (not thousands) of tunnels, and tight budget and power constraints. The 8500L-8S4X is the right fit: its 8 × 1G + 4 × 10G fixed ports and x86-with-hardware-crypto design provide adequate performance at lower cost and footprint [Source: https://www.networkworld.com/article/969558/cisco-bolsters-edge-networking-family-with-expanded-sd-wan-security-options.html] [Source: https://www.cisco.com/c/en/us/td/docs/routers/cloud_edge/c8500l/b-catalyst-8500L-8S4X-edge-platform-hig/m-overview_85L.html].

Key Takeaway: The 8500 and 8500L are aggregation platforms, not branch routers. The 8500 uses a dedicated QuantumFlow Processor (QFP 3.0) ASIC that offloads forwarding and crypto for hundreds-of-Gbps, thousands-of-tunnel headends; the 8500L uses a cost-optimized x86 data plane with hardware crypto for entry-level 1G/10G regional aggregation in the tens of Gbps. For equal licensed bandwidth, the QFP-based 8500 always sustains more encrypted throughput at lower CPU than the 8500L.

Chapter Summary

The Catalyst 8000 Edge family is tiered by role, and hardware capability rises predictably across the tiers. The 8200 is a 1RU small/medium-branch router with four fixed 1 GbE ports, one NIM slot, and one PIM slot for cellular; its 8200L sibling halves the cores (4 vs 8) and throughput (~500 Mbps vs ~1 Gbps IPsec) for cost-sensitive, cloud-secured sites. The 8300 moves up to large branches and campus edges, adding two 10 GbE SFP+ uplinks, 8–12 x86 cores, and a rich modular ecosystem; its 1N1S and 2N2S names directly encode NIM and SM-X slot counts, and a deliberate core-allocation scheme lets it run on-box services at multi-gigabit speeds. The 8500 family leaves the branch entirely: the 8500 uses a dedicated QuantumFlow Processor (QFP 3.0) ASIC for hundreds-of-Gbps, thousands-of-tunnel aggregation, while the 8500L uses a cheaper x86 data plane with hardware crypto for tens-of-Gbps regional hubs.

The selection logic, end to end, is: choose the 8200L for small sub-Gbps cloud-secured branches; the 8200 when you need up to ~1 Gbps or on-box services; the 8300 when you need 10G uplinks, service modules, or multi-gigabit on-box features; the 8500L for regional/mid-size hubs in the tens of Gbps; and the 8500 for large data-center and colocation hubs needing hundreds of Gbps, dense high-speed ports, and thousands of tunnels.

The master comparison table below consolidates the specifications discussed in this chapter.

(Sources for table: 8200 data sheet and HIG; 8300 data sheet, Q&A, and Cisco Live BRKARC-2882; 8500/8500L white papers and 8500L-8S4X HIG, cited in full within the sections above.)

Key Terms

Chapter 3: IOS XE Software and Operating Modes

Learning Objectives

By the end of this chapter, you will be able to:

• Explain the IOS XE software architecture on Catalyst 8000 platforms, including the relationship between the Linux platform OS and the IOSd control-plane process.
• Differentiate autonomous mode from controller (SD-WAN) mode, and understand where SD-Routing fits between them.
• Describe how to switch between operating modes, what happens to your configuration when you do, and when each mode is the right choice.

IOS XE Architecture

A Catalyst 8000 router is not a single monolithic program running on bare metal. Modern Cisco IOS XE is a layered operating system: a Linux kernel and platform services at the bottom, the familiar IOS control plane running as a process on top, and a hardware-accelerated forwarding plane underneath it all. Understanding this layering is the key to understanding everything else in this chapter—because autonomous mode, controller mode, and SD-Routing are all just different personalities of the same software stack.

Think of it like a modern smartphone. The phone’s operating system (the equivalent of the Linux platform OS) is always the same. What changes is which apps are running and how the device is being managed—locally by you, or centrally by a mobile-device-management server at your company. The hardware and the OS don’t change; the management model does.

Linux Kernel and IOSd

At the foundation of IOS XE is a Linux-based platform operating system, often referred to internally as binos (the Linux-based platform OS that provides hardware abstraction, process isolation, and base services on IOS XE). The Linux layer handles low-level concerns: scheduling, memory protection, device drivers, and keeping individual processes isolated from one another so that a fault in one does not crash the entire router.

Running on top of that Linux layer is IOSd (IOS daemon)—the process that runs the classic Cisco IOS control plane, including the CLI, routing protocols, and feature logic you already know. When you type configure terminal or show ip route, you are talking to IOSd. The important conceptual shift from older monolithic IOS is this: IOSd is now just a process, scheduled and protected by Linux like any other process [Source: https://www.scribd.com/document/790338878/TECARC-2407-Architecture-Deployments-and-Troubleshooting-deep-dive-for-Catalyst-8000-Series-Edge-Platforms].

This modular design is what makes the dual-personality model possible. When the router runs as a Cisco SD-WAN node, additional processes and agents—the Overlay Management Protocol (OMP) engine, secure control-connection daemons, and management agents—run alongside IOSd on the same Linux platform [Source: https://www.cisco.com/c/en/us/td/docs/routers/cloud_edge/c8500/software-configuration-guide/c8500-software-config-guide/software-package-and-architecture/deploy-iosxe-sdwan.html]. The underlying OS, the hardware abstraction, and the forwarding plane never change.

Below the control plane sits the data plane: packet forwarding handled by dedicated hardware acceleration (the NPUs/ASICs on physical Catalyst 8000 platforms). IOSd programs the forwarding tables into this hardware regardless of which mode the router is running in. A critical implication follows: choosing autonomous versus controller mode does not change the raw forwarding performance of the box. It only changes how the control and management planes are orchestrated and how configuration is represented [Source: https://www.cisco.com/c/en/us/td/docs/routers/cloud_edge/c8500/software-configuration-guide/c8500-software-config-guide/software-package-and-architecture/deploy-iosxe-sdwan.html].

Figure 3.1: IOS XE layered architecture on Catalyst 8000

flowchart TD
    subgraph CP["Control Plane (process layer)"]
        IOSd["IOSd<br/>Classic IOS control plane<br/>(CLI, routing protocols, features)"]
        SDWAN["SD-WAN agents<br/>(OMP engine, control connections,<br/>management agents)<br/><i>controller mode only</i>"]
    end
    Linux["Linux platform OS (binos)<br/>Hardware abstraction, process isolation, base services"]
    Data["Data plane<br/>Hardware-accelerated forwarding (NPU / ASIC)"]

    IOSd --> Linux
    SDWAN --> Linux
    Linux --> Data
    IOSd -. "programs forwarding tables" .-> Data

Software Packaging and SMUs

Cisco consolidated IOS XE for Catalyst 8000/cEdge routers into a single unified image that contains both the autonomous router personality and the SD-WAN cEdge personality [Source: https://www.cisco.com/c/en/us/td/docs/routers/cloud_edge/c8500/software-configuration-guide/c8500-software-config-guide/software-package-and-architecture/deploy-iosxe-sdwan.html]. In the older world, autonomous IOS XE and SD-WAN (then called “IOS XE SD-WAN”) were separate images and you had to physically replace one with the other to switch. With the unified image, a single setting—the controller-mode flag—tells IOS XE which personality to boot. This is why a mode switch is a reboot, not a software reinstall.

Patching also benefits from the modular architecture. A SMU (Software Maintenance Upgrade) is a targeted, often hot-installable patch that fixes a specific defect or security issue without requiring a full image upgrade and the associated downtime. Because IOS XE components are packaged and isolated, a SMU can deliver a fix to a specific component, letting you address a critical bug quickly while deferring a full version upgrade to your normal maintenance schedule.

Day-0 Boot Process

Day-0 configuration is the initial, one-time setup that brings a brand-new (or freshly erased) device from “out of the box” to “reachable and ready for further configuration” [Source: https://www.cisco.com/c/en/us/td/docs/routers/C8000V/Configuration/c8000v-installation-configuration-guide/day-0-configuration/day0-configuration-overview.html]. It is the networking equivalent of the first-boot setup wizard on a new laptop: you provide just enough information—a name, an address, credentials—to take over from there.

A Catalyst 8000 ships in autonomous mode by default, so the out-of-box Day-0 experience is the classic one. You can deliver Day-0 configuration through several mechanisms:

• The interactive setup dialog at the console.
• A USB file or bootflash config file applied automatically at boot.
• Plug-and-Play (PnP/ZTP), where the device phones home to a provisioning server.
• For the virtual Catalyst 8000V, cloud-init / custom-data supplied by the hypervisor or cloud platform, which can even install Python packages and run scripts in Guestshell for richer automation [Source: https://www.cisco.com/c/en/us/td/docs/routers/C8000V/Configuration/c8000v-installation-configuration-guide/day-0-configuration/performing-day-0-configuration-using-custom-data.html].

Figure 3.2: Day-0 boot and provisioning flow

flowchart TD
    Start["Device powered on<br/>(out of box / freshly erased)"]
    Default["Boots in autonomous mode by default"]
    Check{"Day-0 config<br/>source?"}
    Setup["Interactive setup dialog<br/>(console)"]
    USB["USB / bootflash<br/>config file"]
    PnP["Plug-and-Play / ZTP<br/>(phone home to server)"]
    Cloud["cloud-init / custom-data<br/>(8000V hypervisor / cloud)"]
    Reachable["Device reachable and manageable"]
    Day1["Day-1: core service configuration"]
    Day2["Day-2: optimization, telemetry, ongoing changes"]

    Start --> Default --> Check
    Check --> Setup --> Reachable
    Check --> USB --> Reachable
    Check --> PnP --> Reachable
    Check --> Cloud --> Reachable
    Reachable --> Day1 --> Day2

The content of Day-0 differs dramatically by personality, and we will return to this when we contrast the modes. In autonomous mode, Day-0 is about basic connectivity (a management IP, a default route, login credentials). In controller mode, Day-0 is about onboarding into the SD-WAN fabric (system IP, site ID, organization name, and controller addresses). After Day-0, you move on to Day-1 (core service configuration) and Day-2 (optimization, telemetry, ongoing changes) [Source: https://www.cisco.com/c/en/us/td/docs/routers/C8000V/Configuration/c8000v-installation-configuration-guide/day-0-configuration/day0-configuration-overview.html].

Key Takeaway: IOS XE is a layered OS—a Linux platform (binos) hosting the IOSd control-plane process over a hardware-accelerated data plane—shipped as a single unified image whose controller-mode flag selects which personality boots. SMUs allow targeted patching, and Day-0 configuration gets a fresh device to a manageable state.

Autonomous vs Controller Mode

The same Catalyst 8000 hardware and the same unified IOS XE image can present two completely different operational personalities. They are mutually exclusive—the router boots into one or the other—and they differ in how the device is configured, managed, and integrated into the network [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/sdwan-xe-gs-book/install-upgrade-17-2-later.html].

Autonomous (Traditional) Mode

Autonomous mode is the default, out-of-box personality: the Catalyst 8000 behaves like a traditional standalone IOS XE router with no SD-WAN overlay running [Source: https://www.cisco.com/c/en/us/td/docs/routers/secure-routers/sec-8200-series/software-config-guide/secure-8200-series-scg/overview.html]. If you have configured any Cisco router in the last two decades, this will feel familiar.

Its key characteristics:

1. Local CLI is primary. You configure the device with configure terminal, interface and routing stanzas, and so on. You can also use NETCONF/RESTCONF, Ansible, or a controller like Cisco DNA Center for automation.
2. Classic routing and services. You run OSPF, BGP, EIGRP, RIP, and static routes, plus features such as IPsec VPNs (DMVPN, GETVPN, FlexVPN), NAT, QoS, and VRFs—all configured and stored locally. There is no OMP control plane.
3. Config lives on the box. Configuration is held in running-config and startup-config. You make changes locally and save them with copy running-config startup-config (or write memory).

Here is a representative autonomous-mode configuration—exactly the kind you would type into any classic Cisco router [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/sdwan-xe-gs-book/install-upgrade-17-2-later.html]:

! Autonomous mode – classic router
hostname BRANCH-RTR1
interface GigabitEthernet0/0
 ip address 10.10.10.1 255.255.255.0
 no shut
!
router ospf 10
 network 10.10.10.0 0.0.0.255 area 0
!
router bgp 65010
 neighbor 192.0.2.1 remote-as 65000

Everything here is configured and stored on the device itself. No SD-WAN controller is involved.

Controller (SD-WAN) Mode

Controller mode turns the same device into a cEdge (a Cisco SD-WAN edge router) that participates in an SD-WAN overlay fabric and is managed centrally by Cisco Catalyst SD-WAN Manager (the product formerly known as vManage) [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/onboarding-cedge-c8000v].

Its key characteristics:

1. Centralized, template-driven management. SD-WAN Manager is a single dashboard from which you define device templates, feature templates, segmentation, QoS, application-aware routing, and security policies—then push them to many cEdges at once [Source: https://www.cisco.com/c/en/us/solutions/collateral/enterprise-networks/sd-wan/nb-06-sd-wan-sol-overview-cte-en.html]. The CLI is still IOS XE-style, but it is transactional (config-transaction), and most configuration is generated from templates rather than typed by hand.
2. Secure control connections. The cEdge forms encrypted (DTLS/TLS) control connections to the SD-WAN controllers—the SD-WAN Validator and SD-WAN Controller (formerly vBond and vSmart) and SD-WAN Manager—validated by certificates [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/sdwan-xe-gs-book/cisco-sd-wan-overlay-network-bringup.html].
3. OMP-based overlay. The Overlay Management Protocol (OMP)—the SD-WAN control protocol that distributes routes, TLOCs (transport locators), and policies across the overlay—runs alongside the traditional protocols. BGP/OSPF are still supported, but they typically handle local LAN/WAN adjacency while OMP carries reachability across the fabric [Source: https://www.cisco.com/c/en/us/solutions/collateral/enterprise-networks/sd-wan/nb-06-sd-wan-ds-cte-en.html].
4. Manager is the source of truth. The device’s configuration is effectively shadowed by SD-WAN Manager. Local CLI changes that conflict with templates are overwritten or flagged as out-of-sync.

A minimal SD-WAN Day-0 bootstrap looks very different from the autonomous snippet above—notice it is about joining the fabric, not about routing [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/sdwan-xe-gs-book/cisco-sd-wan-overlay-network-bringup.html]:

! Minimal SD-WAN bootstrap
system
 host-name BRANCH-EDGE1
 system-ip 10.255.255.11
 site-id 101
 organization-name "MyOrg"
 vbond 203.0.113.10
!
interface GigabitEthernet0/0
 ip dhcp client
 tunnel-interface
  allow-service all
  color biz-internet
  nat

This bootstrap only gets the device into the SD-WAN fabric; the full configuration—VPNs, policies, routing—is pushed afterward from SD-WAN Manager.

There is also a middle ground worth knowing about. SD-Routing is an operational capability where the router stays in autonomous mode and forwards traffic with traditional routing, but is fully managed and monitored by Catalyst SD-WAN Manager—without joining a full SD-WAN overlay or building SD-WAN data-plane tunnels [Source: https://www.cisco.com/c/en/us/products/collateral/networking/sdwan-routers/catalyst-8000v-edge-software/nb-06-sd-routing-faq-cte-en.html]. It is not a third boot mode; it is an add-on to autonomous mode. The router establishes DTLS/TLS control connections to the SD-WAN Validator and Manager (just like an SD-WAN device does for management) so that one Manager can act as a single pane of glass across both your overlay devices and your traditional routers. SD-Routing was introduced in Cisco IOS XE 17.12.1 on routers paired with controllers at release 20.12.1, and it is supported on Catalyst 8000 platforms [Source: https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2024/pdf/BRKENT-1039.pdf]. We will look at how it is onboarded in the next section.

Feature Differences Between Modes

The clearest way to internalize the contrast is side by side. The triangle of autonomous, SD-Routing, and SD-WAN sorts out a question students often confuse: all three can be managed by SD-WAN Manager, but only one builds an overlay.

Figure 3.3: Autonomous, SD-Routing, and SD-WAN compared

flowchart LR
    subgraph Auto["Autonomous (classic)"]
        A1["Execution mode: Autonomous"]
        A2["No SD-WAN tunnels"]
        A3["Normally not managed by<br/>SD-WAN Manager"]
    end
    subgraph SDR["SD-Routing"]
        S1["Execution mode: Autonomous"]
        S2["No SD-WAN tunnels<br/>for user traffic"]
        S3["Managed via control channel<br/>to SD-WAN Manager"]
    end
    subgraph SDW["SD-WAN"]
        W1["Execution mode: Controller"]
        W2["Full SD-WAN overlays<br/>(IPsec, etc.)"]
        W3["Full SD-WAN policy and<br/>routing control"]
    end
    Mgr["Catalyst SD-WAN Manager"]
    SDR -. "control channel" .-> Mgr
    SDW == "policy + routing + overlay" ==> Mgr

[Source: https://www.cisco.com/c/en/us/products/collateral/networking/sdwan-routers/catalyst-8000v-edge-software/nb-06-sd-routing-faq-cte-en.html]

And comparing the two true boot modes head to head [Source: https://www.cisco.com/c/en/us/td/docs/routers/cloud_edge/c8500/software-configuration-guide/c8500-software-config-guide/software-package-and-architecture/deploy-iosxe-sdwan.html]:

Key Takeaway: Autonomous mode is the default, CLI-driven standalone router; controller mode makes the device an SD-WAN cEdge managed centrally through SD-WAN Manager with OMP-based overlay; and SD-Routing is a middle option that keeps the router autonomous while still managing it from SD-WAN Manager—without building an overlay.

Switching Modes

Because autonomous and controller mode are two personalities of one image, switching between them is a software operation rather than a reimage. But it is far from a trivial toggle: every mode switch erases the configuration and reboots the router. Treat it as a migration with a maintenance window, not a feature flag [Source: https://www.scribd.com/document/765359733/cat8300swcfg-xe-17-book].

Figure 3.4: Mode switching as a destructive state machine

stateDiagram-v2
    [*] --> Autonomous: ships autonomous by default
    Autonomous --> Controller: controller-mode enable<br/>(erase startup-config + reload)
    Controller --> Autonomous: controller-mode autonomous<br/>(erase startup-config + reload)
    Controller --> Autonomous: controller-mode reset<br/>(clear to clean Day-0)

    state Autonomous {
        [*] --> ClassicRouter
        ClassicRouter --> SDRoutingOnboarded: sd-routing bootstrap load<br/>(stays autonomous, no overlay)
    }

    note right of Autonomous
        System mode: Autonomous
        Verify: show platform software device-mode
    end note
    note right of Controller
        System mode: Controller-Managed
        cEdge in SD-WAN fabric
    end note

request platform / controller-mode Commands

All mode commands run from privileged EXEC (the Router# prompt)—not from configuration mode. Start by confirming which personality you are in [Source: https://www.scribd.com/document/765359733/cat8300swcfg-xe-17-book]:

Router# show platform software device-mode
System mode: Autonomous

The output reports either System mode: Autonomous or System mode: Controller-Managed (exact wording can vary slightly by release).

Worked example A — Catalyst 8300: autonomous → SD-WAN. To convert an autonomous router into a cEdge, use controller-mode enable. The router warns you that it will erase the startup-config, asks you to confirm the reload, then wipes its configuration and reboots into the SD-WAN personality [Source: https://0x2142.com/how-to-cisco-sd-wan-onboarding-a-catalyst-8000v/]:

C8300# show platform software device-mode
System mode: Autonomous

C8300# controller-mode enable
Enabling controller mode will erase the startup-config. Continue? [y/n]: y
This action will reload the system. Do you want to continue? [y/n]: y

! after reload
C8300# show platform software device-mode
System mode: Controller-Managed

Worked example B — Catalyst 8000V: SD-WAN → autonomous. The reverse operation turns a cEdge back into a classic IOS XE router with controller-mode autonomous. Note that on the virtual 8000V, the hypervisor/VM settings (vNICs, vCPU, memory) are untouched—only the IOS XE configuration is reset [Source: https://www.scribd.com/document/765359733/cat8300swcfg-xe-17-book]:

C8000V# show platform software device-mode
System mode: Controller-Managed

C8000V# controller-mode autonomous
Converting to autonomous mode will erase the startup-config. Continue? [y/n]: y
This action will reload the system. Do you want to continue? [y/n]: y

! after reload
C8000V# show platform software device-mode
System mode: Autonomous

Once in controller mode and onboarded, a new family of SD-WAN operational commands becomes available for verification and troubleshooting [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/onboarding-cedge-c8000v]:

Router# show sdwan system status
Router# show sdwan control connections
Router# show sdwan omp peers
Router# show sdwan bfd sessions

For SD-Routing, the workflow is different because the device stays autonomous. You generate a bootstrap file (ciscosdwan.cfg) in SD-WAN Manager, copy it to bootflash, ensure the device is in a clean Day-0 state (using controller-mode reset if it was in controller mode, or write erase plus a reload to clear an existing autonomous config), and then bring it up with [Source: https://www.cisco.com/c/en/us/td/docs/routers/sd-routing/1715x/sd-routing-onboard-routing-devices-to-sd-wan-manager-1715x.html]:

Router# sd-routing bootstrap load bootflash:ciscosdwan.cfg

This command establishes the control connections to the Validator and Manager and registers the device as an SD-Routing (autonomous) device. An alternative is activation by chassis number and Token ID (UUID) obtained from SD-WAN Manager [Source: https://www.cisco.com/c/en/us/td/docs/routers/sd-routing/1715x/sd-routing-onboard-routing-devices-to-sd-wan-manager-1715x.html].

Troubleshooting tip: If controller-mode ? shows no options, the image or platform may not support SD-WAN mode switching. Verify you are running a Catalyst 8000 SD-WAN-capable IOS XE image (typically 17.3.x or later for SD-WAN; 17.12.1+ for SD-Routing) and that your release/SKU supports the feature [Source: https://www.cisco.com/c/en/us/support/docs/routers/catalyst-8300-edge-platform/225727-upgrade-software-on-catalyst-8000-edge.html].

Configuration Persistence and Erase

This is the single most important operational fact in the chapter: mode switching is destructive. Both controller-mode enable and controller-mode autonomous are one-way, destructive operations each time they run—they wipe the current configuration and reload [Source: https://www.scribd.com/document/765359733/cat8300swcfg-xe-17-book].

Specifically, when you switch modes:

• The startup-config is deleted from NVRAM.
• The running configuration is lost on reload; the device comes up with only a minimal, factory-like base config appropriate to the new personality.
• Legacy features (VRFs, DMVPN, classic crypto) must be re-created afterward—either through SD-WAN templates (controller mode) or by reconfiguring from CLI (autonomous mode) [Source: https://0x2142.com/how-to-cisco-sd-wan-onboarding-a-catalyst-8000v/].

The reason is architectural: SD-WAN’s configuration model and data store do not map one-to-one onto classic IOS XE startup-config, so there is no clean way to translate one into the other. The system simply resets to a known-good baseline for the target personality.

The practical discipline that follows:

1. Back up first. Capture show running-config and copy it to TFTP/FTP/flash before any switch.
2. Model the target state ahead of time. If migrating autonomous → SD-WAN, build the equivalent routing, QoS, and VPN behavior as SD-WAN templates and policies before the cutover, so the cEdge can be onboarded immediately after it reboots.
3. Schedule downtime. The reload interrupts traffic. Capturing console output during the first conversion is valuable for change records and rollback planning [Source: https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2025/pdf/BRKTRS-2572.pdf].

A useful real-world analogy: switching modes is like converting a building from offices to apartments. The structure (hardware) and the plot of land (the Linux OS) stay the same, but you gut the interior and rebuild it to a new floor plan. You would never do it without blueprints (templates/backups) ready and the tenants temporarily moved out (a maintenance window).

When to Use Each Mode

The choice comes down to your operational model, feature needs, and team skill set [Source: https://www.reddit.com/r/networking/comments/1ai46hw/cisco_8200_controller_mode_or_autonomous_mode/].

• Choose autonomous mode when you need “just a router”—standard protocols and localized services—in smaller environments or where the team is comfortable with per-device CLI and generic automation (Ansible, RESTCONF). It carries the least architectural overhead.
• Choose controller (SD-WAN) mode when you want centralized policy, application-aware routing, integrated segmentation, overlay security, and template-based rollout across many sites. The Catalyst 8000 becomes a cEdge with SD-WAN Manager as the source of truth.
• Choose SD-Routing when you want unified management—a single SD-WAN Manager pane of glass over both overlay and traditional routers—without forcing every site into an overlay. It is especially valuable for phased migration.

A few illustrative scenarios make the decision concrete [Source: https://www.cisco.com/c/en/us/td/docs/routers/cloud_edge/c8500/software-configuration-guide/c8500-software-config-guide/software-package-and-architecture/deploy-iosxe-sdwan.html]:

• Scenario A — Simple dual-homed branch. A site that just needs MPLS + Internet routing with a few VPNs, operated mostly via CLI, is simplest in autonomous mode.
• Scenario B — Large enterprise WAN. Many sites needing centralized policy, per-application dynamic path selection, and template-driven rollout call for controller mode.
• Scenario C — Gradual transition. An estate of 200 classic BGP/MPLS routers can be upgraded to IOS XE 17.12.x+, brought up in SD-Routing to onboard them all into SD-WAN Manager for inventory and monitoring, and then migrated site-by-site to full SD-WAN with controller-mode enable during individual change windows [Source: https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2024/pdf/BRKENT-1039.pdf].

Because flipping modes is destructive, many teams in production avoid repeated switching altogether: they dedicate specific hardware or 8000V instances to SD-WAN and others to autonomous routing, maintain golden bootstrap configs for each personality, and lean on PnP/ZTP automation to minimize manual steps after any conversion [Source: https://www.scribd.com/document/765359733/cat8300swcfg-xe-17-book].

Key Takeaway: Switch modes from privileged EXEC with controller-mode enable (to SD-WAN) or controller-mode autonomous (back to classic), verifying with show platform software device-mode; onboard SD-Routing with sd-routing bootstrap load. Every switch erases the config and reboots, so back up first, model the target state ahead of time, and choose the mode that matches your scale, feature needs, and operating model.

Chapter Summary

A Catalyst 8000 runs a layered IOS XE operating system: a Linux-based platform OS (binos) hosts the IOSd control-plane process over a hardware-accelerated data plane. Cisco ships this as a single unified image containing two mutually exclusive personalities, selected by the controller-mode flag, with SMUs available for targeted patching and Day-0 mechanisms (setup dialog, USB, PnP, cloud-init) to bring fresh devices to a manageable state.

Autonomous mode is the default: a traditional CLI-driven router running classic routing protocols and services, with configuration stored locally. Controller mode turns the box into an SD-WAN cEdge—centrally managed by Catalyst SD-WAN Manager through templates, forming secure control connections to the SD-WAN controllers, and distributing routes via OMP across an overlay fabric. SD-Routing sits between them: the router stays autonomous and builds no overlay, yet is fully managed by SD-WAN Manager, making it ideal for unified management and phased migration.

Switching personalities is done from privileged EXEC with controller-mode enable and controller-mode autonomous, verified with show platform software device-mode, and SD-Routing is onboarded with sd-routing bootstrap load. The non-negotiable operational rule is that every mode switch erases the configuration and reboots the device—so always back up, pre-build the target configuration, and plan a maintenance window. Because the data plane is identical across modes, the decision is never about performance; it is about how you want to configure, manage, and integrate the device into your network.

Key Terms

Chapter 4: Licensing with Cisco DNA and Smart Licensing

Buying a Cisco Catalyst 8000 router is not like buying a laptop, where the box you purchase is the product you own forever. A Catalyst 8000 is closer to a smartphone bought on a carrier plan: you own the hardware, but most of what makes it useful — SD-WAN, advanced security, automation, analytics — arrives through a subscription you must keep current. Worse, even the speed of the box and its ability to encrypt traffic at full rate are gated by licensing, not just by the silicon inside.

This chapter untangles that model. We will start with the two-stack licensing structure (Network and DNA) and the tiers within each. We will then explain Smart Licensing Using Policy (SLP), the modern reporting mechanism that replaced the old “register-then-use” workflow. Finally, we will cover HSEC and throughput licenses — the controls that decide how fast your router can move encrypted packets — and walk through how to choose the right combination for a given site.

Learning Objectives

By the end of this chapter, you should be able to:

• Explain the Cisco DNA and Cisco Networking subscription licensing tiers available for the Catalyst 8000 family.
• Distinguish the Network-stack from the DNA-stack and describe how the two are tier-matched.
• Describe Smart Licensing Using Policy (SLP) and how usage is reported to Cisco Smart Software Manager (CSSM).
• Map throughput tiers and feature sets to the correct license selection for a given deployment.
• Explain why an HSEC license is required to exceed the export-control crypto throughput cap, and how Boost/throughput licenses interact with it.

Licensing Tiers

The Two-Stack Model: Network Stack vs. DNA Stack

The single most important idea in Catalyst 8000 licensing is that every router carries two licenses, not one. Cisco organizes its IOS XE licensing into two parallel “stacks” [Source: https://www.layer23-switch.com/blog/cisco-network-vs-dna-licensing.html]:

• The Network stack maps to the IOS XE routing feature set. It comes in two tiers — Network Essentials and Network Advantage — and it grants you the right to run the routing software you bought. Crucially, the Network stack behaves as a perpetual license: even after everything else expires, the box keeps routing with the feature tier you own [Source: https://www.layer23-switch.com/blog/cisco-network-vs-dna-licensing.html].

• The DNA stack is the subscription add-on that “rides on top of” the Network stack. It also comes in tiers — DNA Essentials, DNA Advantage, and a higher DNA Premier option — and it unlocks SD-WAN, automation, analytics, and selected security functions. Unlike the Network stack, the DNA stack is term-based: you rent it for 3, 5, or 7 years [Source: https://www.layer23-switch.com/blog/cisco-c8000-dna-ordering-guide.html].

A useful analogy: think of the Network stack as owning a car and the DNA stack as a subscription to a navigation-and-driver-assist service. When the subscription lapses, the car still drives — you simply lose the smart features layered on top.

The two stacks are tier-matched, a rule that trips up many first-time buyers: Network Essentials pairs with DNA Essentials, and Network Advantage pairs with DNA Advantage. You do not mix Network Essentials with DNA Advantage or vice versa [Source: https://www.layer23-switch.com/blog/cisco-network-vs-dna-licensing.html]. In ordering, this usually appears as a single bundled SKU — for example, “Network Advantage + DNA Advantage, 5-year term.”

Figure 4.1: The two-stack model and tier-matching

graph TD
    R["Catalyst 8000 Router"] --> NS["Network Stack (perpetual): routing rights"]
    R --> DS["DNA Stack (term-based): SD-WAN, automation, analytics"]
    NS --> NE["Network Essentials"]
    NS --> NA["Network Advantage"]
    DS --> DE["DNA Essentials"]
    DS --> DA["DNA Advantage"]
    DS --> DP["DNA Premier"]
    NE -. tier-matched .- DE
    NA -. tier-matched .- DA
    NA -. tier-matched .- DP

One more rule worth memorizing: on the Catalyst 8200 and 8300, the DNA subscription is mandatory. You cannot order these platforms “license-free,” and the older perpetual add-on licenses (SEC, UC, APP) that ISR buyers may remember have been retired in favor of the term-based DNA model [Source: https://www.layer23-switch.com/blog/cisco-c8000-dna-ordering-guide.html].

DNA Essentials, Advantage, and Premier

The DNA-stack tier you choose determines which subscription features light up. While exact feature lists evolve with software releases, the conceptual progression is consistent [Source: https://edgeium.com/blog/dna-essentials-vs-dna-advantage]:

• DNA Essentials delivers core SD-WAN (the overlay fabric and basic application-aware routing), basic device management and policy templates, and basic telemetry/visibility. For a typical branch router doing SD-WAN with direct internet access (DIA) break-out and standard NAT/firewalling, Essentials is often enough [Source: https://www.layer23-switch.com/blog/cisco-c8000-dna-ordering-guide.html].

• DNA Advantage adds the full SD-WAN feature set with advanced policies and application optimization, richer security and segmentation (more granular zone-based policy), deeper analytics, and integration with the broader Cisco ecosystem — Cisco ISE, Secure Network Analytics, and ThousandEyes — when paired with those subscriptions [Source: https://www.reddit.com/r/networking/comments/1mccpqz/what_is_the_use_of_cisco_dna_advantage_license/]. Independent pricing analyses note that the DNA portion of Advantage typically runs roughly 50–100% more than Essentials [Source: https://edgeium.com/blog/dna-essentials-vs-dna-advantage].

• DNA Premier sits above Advantage as a bundle that rolls in additional Cisco software entitlements (for example, advanced security and threat services) on top of the Advantage feature set, aimed at organizations that want a single, broad subscription rather than buying add-ons separately.

A critical practical point: the DNA subscription is the SD-WAN license. You do not buy a separate SD-WAN product. The same subscription covers both SD-WAN mode and traditional (“autonomous”) IOS XE routing mode on the 8200/8300/8500 [Source: https://www.cisco.com/c/en/us/products/collateral/routers/catalyst-8300-series-edge-platforms/cat-8300-8200-series-edge-plat-og.html].

The table below summarizes the tiers. Treat the feature columns as directional rather than exhaustive — Cisco shifts specific features between tiers across releases.

What Happens When the Term Expires

Because the DNA stack is a rental, expiry matters. When a DNA subscription term ends and is not renewed [Source: https://www.layer23-switch.com/blog/cisco-network-vs-dna-licensing.html]:

• The device keeps routing using the Network-stack tier you own (Essentials or Advantage routing features persist).
• You lose DNA-only capabilities: advanced SD-WAN control, certain automation and analytics, and cloud services tied to DNA.
• You lose entitlement to major new feature updates within the DNA scope and to the Cisco support associated with the subscription.

The danger is that this can happen quietly — routing continues, so nobody notices until an SD-WAN feature or support case is needed. Track end dates proactively and budget renewals against the hardware lifecycle (a 5-year term aligned to a 5-year refresh is a common pattern) [Source: https://ormsystems.com/blogs/cisco-license-cost-and-renewal-considerations-in-2026].

Figure 4.2: What persists and what is lost when the DNA term expires

flowchart TD
    E["DNA subscription term ends, not renewed"]
    E --> K["KEPT: routing via owned Network tier (Essentials/Advantage)"]
    E --> L["LOST: advanced SD-WAN control, automation, analytics, DNA cloud services"]
    E --> U["LOST: entitlement to new DNA-scope feature updates and subscription support"]
    K --> Q["Quiet failure: routing continues, so lapse goes unnoticed"]
    L --> Q
    U --> Q
    Q --> M["Mitigation: track end dates and budget renewals to lifecycle"]

Throughput-Based Tiers (T0/T1/T2/T3)

Beyond the feature stacks, Catalyst 8000 ordering includes a throughput tier — a software-enforced cap on how much traffic the router will forward [Source: https://www.layer23-switch.com/blog/cisco-c8000-dna-ordering-guide.html]. The platform supports multiple levels (commonly grouped as T0, T1, T2, T3, mapping to ascending bandwidth ranges such as 50 Mbps, 100 Mbps, 250 Mbps, 500 Mbps, 1 Gbps, and higher, depending on model) [Source: https://www.cisco.com/c/en/us/td/docs/routers/cloud_edge/configure-licenses-throughput-catalyst-8000-platforms.html].

IOS XE polices the data plane to the configured tier, so a router with capable hardware will still cap forwarding at the tier you licensed. On the virtual Catalyst 8000V you configure the throughput level explicitly; on hardware platforms it is generally tied to your DNA license selection [Source: https://www.cisco.com/c/en/us/td/docs/routers/C8000V/configure-licenses-throughput-c8000v.html]. As we will see in the HSEC section, the throughput tier and the crypto cap are independent controls — a fact that causes a great deal of confusion.

Cisco Networking Subscription: Why Routers Are Excluded

Cisco has introduced Cisco Networking Subscription (CNS) as a unified subscription model spanning many product families. It is tempting to assume the Catalyst 8000 falls under it — but routing is explicitly excluded from CNS today [Source: https://www.cisco.com/c/en/us/products/collateral/networking/software/networking-subscription-ds.html]. Catalyst 8000 routers remain on DNA subscriptions; you cannot migrate them into CNS. In a mixed environment, your switches and wireless may move to CNS (which carries a 36-month minimum term) while your routers stay on the Network + DNA stack model [Source: https://www.cisco.com/c/en/us/products/collateral/networking/software/networking-subscription-ds.html]. Plan budgets accordingly.

Key Takeaway: Every Catalyst 8000 carries two stacks — a perpetual Network stack (Essentials/Advantage) for routing rights and a term-based DNA stack (Essentials/Advantage/Premier) for SD-WAN, automation, and analytics. The two are tier-matched, the DNA subscription is the SD-WAN license, a separate throughput tier (T0–T3) caps forwarding speed, and routing is excluded from Cisco Networking Subscription.

Smart Licensing Using Policy

From “Register First” to “Report Usage”

Older Cisco licensing (legacy Smart Licensing, on IOS XE 16.x and early 17.x) followed an “obtain authorization before use” model: you generated a token, ran license smart register idtoken ..., and the device sat in evaluation mode until it received explicit authorization from Cisco. If it could not reach Cisco, it risked falling out of compliance with noisy syslog warnings [Source: https://www.cisco.com/c/en/us/td/docs/routers/sl_using_policy/b-sl-using-policy/info_about.html].

Smart Licensing Using Policy (SLP), introduced in IOS XE 17.3.2 and used by the Catalyst 8200/8300/8400/8500 family, flips that model on its head [Source: https://www.cisco.com/c/en/us/td/docs/routers/sl_using_policy/b-sl-using-policy/info_about.html]. SLP is best described as “trust, then verify by reporting.” Its defining characteristics are:

• No registration token required for unenforced licenses. You configure features immediately; no license smart register idtoken ... is needed [Source: https://www.cisco.com/c/en/us/td/docs/routers/sl_using_policy/b-sl-using-policy/info_about.html].
• No evaluation mode. The compliance-anxiety state of legacy licensing is gone for unenforced licenses.
• Unenforced licenses are authorized by default and always usable. Enforcement applies only to specific export-controlled or enforced license types when policy requires it [Source: https://www.cisco.com/c/en/us/td/docs/routers/sl_using_policy/b-sl-using-policy/info_about.html].
• Usage reporting replaces continuous authorization. The device records what it uses and periodically reports, on a schedule defined by a policy [Source: https://www.cisco.com/c/en/us/td/docs/routers/sl_using_policy/b-sl-using-policy/m-sle-how-smart-licensing-using-policy-works.html].

The contrast is summarized below [Source: https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2024/pdf/LTROPS-1007.pdf]:

CSSM, RUM Reports, and Trust

Cisco Smart Software Manager (CSSM) is the central cloud portal that manages your Smart Account, virtual accounts, and entitlements. Under SLP it is the system of record: for each license type it shows how many are Purchased, how many are In Use, and the resulting Balance (Purchased − In Use). For example, with 50 CUBE sessions purchased and one router using 20, CSSM shows Purchased 50, In Use 20, Balance 30. If usage exceeds entitlement (balance goes negative), CSSM flags Insufficient Licenses [Source: https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/voice/cube/ios-xe/config/ios-xe-book/m_smart_licensing.html].

The router feeds CSSM through RUM reports. A RUM report (Reporting Usage Metrics report) is structured data showing which licenses are in use, the quantity (throughput level, session count, and so on), device identifiers (serial number, product ID), timestamps, and policy metadata [Source: https://www.cisco.com/c/en/us/td/docs/routers/sl_using_policy/b-sl-using-policy/m-sle-how-smart-licensing-using-policy-works.html]. The product instance continuously records usage locally, generates RUM reports, stores them until they are sent, and logs alerts if a report becomes overdue.

The policy governs how often you must report. It can specify intervals such as 30, 90, or 365 days. As general guidance, both perpetual and subscription licenses require a report within 90 days of a usage change; if usage does not change, no new report is required [Source: https://edgeium.com/blog/disabling-smart-agents-for-enhanced-control]. A global default policy applies unless you install a custom policy downloaded from CSSM.

This is where trust enters. The legacy “registration + authorization code” handshake is replaced by a policy-driven trust state: the router sends a RUM report aligned to its current policy, CSSM associates it with the Smart/Virtual Account and returns confirmation that the report is accepted, and the device tracks whether it holds a valid, current policy and whether its last report landed within the allowed window [Source: https://www.cisco.com/c/en/us/td/docs/routers/sl_using_policy/b-sl-using-policy/m-sle-how-smart-licensing-using-policy-works.html]. If that trust state goes stale — an overdue report or missing policy — the router emits syslog warnings, but unenforced features keep working [Source: https://www.cisco.com/c/en/us/td/docs/routers/sl_using_policy/b-sl-using-policy/m-sle-how-smart-licensing-using-policy-works.html].

Reporting Methods: Online, On-Prem, and Air-Gapped

SLP supports three transports, letting you match licensing to your security posture [Source: https://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/ios-xe-17/217046-configure-smart-licensing-using-policy-o.html]:

1. Direct online (Smart Transport). The router talks HTTPS straight to Cisco’s cloud CSSM and sends RUM reports automatically at policy intervals. Configured with license smart transport smart; requires DNS, routing, and HTTPS reachability to Cisco.

2. On-prem via CSLU or CSSM Satellite. Common in enterprises that keep devices off the public internet. You deploy a Cisco Smart License Utility (CSLU) or on-prem CSSM registered to your Smart Account, then point the router at it:

   configure terminal
     license smart transport cslu
     license smart url cslu https://your-cslu.example.com:port/cslu/v1/pi
   end
   license smart sync all

   The on-prem server then syncs with cloud CSSM as your security policy allows [Source: https://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/ios-xe-17/217046-configure-smart-licensing-using-policy-o.html].

3. Offline / air-gapped (file exchange). For fully isolated networks, the router accumulates usage locally, you export a RUM report file (CLI or USB), upload it to CSSM from a connected machine, then download any updated policy or trust information and import it back to the router [Source: https://www.cisco.com/c/en/us/td/docs/routers/sl_using_policy/b-sl-using-policy/m-sle-how-smart-licensing-using-policy-works.html]. This satisfies reporting with zero direct connectivity.

Figure 4.3: RUM report flow from router to CSSM across the three transports

flowchart LR
    PI["Router (product instance): records usage, generates RUM reports per policy"]
    PI --> T1["Smart Transport: HTTPS direct"]
    PI --> T2["CSLU / on-prem CSSM"]
    PI --> T3["Offline file export (CLI/USB)"]
    T1 --> CSSM["Cloud CSSM: system of record (Purchased / In Use / Balance)"]
    T2 --> CSSM
    T3 --> FILE["Upload file from connected machine"]
    FILE --> CSSM
    CSSM --> ACK["Returns acceptance + updated policy/trust"]
    ACK --> PI

You can verify state at any time with show license all and show license usage, and manually trigger reporting with license smart sync all [Source: https://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/ios-xe-17/217046-configure-smart-licensing-using-policy-o.html]. If you encounter “evaluation mode” or “not authorized” messages, the router is usually not properly associated with the correct Smart/Virtual Account or DNA quantity — register it and confirm the Network and DNA entitlements exist for that product ID and term [Source: https://www.layer23-switch.com/blog/cisco-c8000-dna-ordering-guide.html]. As a last resort for corrupted licensing data, Cisco documents license factory reset (which requires a reload) [Source: https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2024/pdf/LTROPS-1007.pdf].

Key Takeaway: SLP (IOS XE 17.3.2+) replaces token registration and evaluation mode with a “use now, report later” model. Unenforced licenses work immediately; the router periodically sends RUM reports to CSSM — directly, via CSLU/on-prem, or by offline file exchange — within policy windows (typically 90 days after a usage change). Missing a deadline produces overdue warnings, not feature loss.

HSEC and Throughput

The HSEC License and the 250 Mbps Crypto Cap

Here is a fact that surprises many engineers: a Catalyst 8000 licensed for 1 Gbps of throughput may still refuse to push more than about 250 Mbps of encrypted traffic. The reason is export control, not hardware.

U.S. export regulations historically required vendors to limit high-speed strong encryption in certain territories. Cisco implements this as a soft cap of roughly 250 Mbps on encrypted (IPsec/crypto) throughput and on tunnel/security-association scale, lifted only when an export-control license is present [Source: https://www.cisco.com/c/en/us/td/docs/routers/cloud_edge/configure-licenses-throughput-catalyst-8000-platforms.html]. Two licenses matter here [Source: https://blog.router-switch.com/2014/09/cisco-sec-k9-license-vs-hsec-k9-license/]:

• SEC-K9 enables standard crypto (IPsec, SSL, and so on) but stays export-limited (~250 Mbps).
• HSEC-K9 — the High-Security Export Control license — removes that limitation so the router can use its full crypto capability, subject to whatever throughput tier you also hold.

The cap applies only to encrypted traffic. Plain Layer 3 forwarding can use the full licensed bandwidth. So on a 1 Gbps-tier router without HSEC, unencrypted traffic flows near 1 Gbps while IPsec stubbornly tops out around 250 Mbps [Source: https://www.cisco.com/c/en/us/td/docs/routers/cloud_edge/configure-licenses-throughput-catalyst-8000-platforms.html]. This is the classic symptom: tunnels that never exceed ~200–250 Mbps no matter how much WAN bandwidth you provision.

Throughput Tiers and HSEC Are Independent

The key mental model: the throughput tier and HSEC are two separate gates, and your encrypted throughput is limited by whichever is lower [Source: https://www.cisco.com/c/en/us/td/docs/routers/cloud_edge/configure-licenses-throughput-catalyst-8000-platforms.html].

• Throughput tier = 100 Mbps with HSEC → encrypted traffic capped at ~100 Mbps (the tier is the bottleneck).
• Throughput tier = 1 Gbps without HSEC → encrypted traffic capped at ~250 Mbps (export control is the bottleneck), while plaintext approaches 1 Gbps.
• Throughput tier = 1 Gbps with HSEC → encrypted traffic can approach 1 Gbps.

To push more than 250 Mbps of IPsec, you need both a throughput tier high enough for the target bandwidth and an active HSEC license [Source: https://www.cisco.com/c/en/us/td/docs/routers/cloud_edge/configure-licenses-throughput-catalyst-8000-platforms.html].

Figure 4.4: The two independent gates limiting encrypted throughput

flowchart TD
    TRAF["Encrypted (IPsec) traffic"] --> G1{"Gate 1: Throughput tier"}
    G1 --> G2{"Gate 2: HSEC license present?"}
    G2 -->|"No HSEC"| CAP["Export cap ~250 Mbps applies"]
    G2 -->|"HSEC active"| FULL["Limited only by throughput tier"]
    CAP --> RESULT["Effective encrypted rate = lower of the two gates"]
    FULL --> RESULT

Boost / Performance Licenses

On the older ISR 4000 series, Cisco used a Performance (Boost) license to lift an artificial cap on overall forwarding — letting the box run “unthrottled” up to its hardware limits rather than a lower base tier [Source: https://www.cisco.com/c/en/us/support/docs/routers/4000-series-integrated-services-routers/217135-performance-license-on-cisco-isr4000.html]. It is essential to understand that Boost is not HSEC: Boost removes the general forwarding cap, while HSEC removes the crypto cap. On an ISR 4K you can install Boost and still see IPsec stuck at ~250 Mbps because HSEC is missing — you need both [Source: https://www.reddit.com/r/networking/comments/1boy1n5/cisco_isr_4331_boost_license_unclear/]. On the Catalyst 8000, the Boost role is filled by the throughput tiers described earlier, but the same logic holds: high crypto throughput requires a high tier and HSEC [Source: https://www.cisco.com/c/en/us/td/docs/routers/cloud_edge/configure-licenses-throughput-catalyst-8000-platforms.html].

Installing and Verifying HSEC

Catalyst 8000 manages HSEC through Smart Licensing / SLP. Verify with [Source: https://www.cisco.com/c/en/us/support/docs/routers/catalyst-8300-series-edge-platforms/225417-install-hsec-license-in-catalyst-8300.html]:

show license all
show license usage
show platform hardware throughput level

Look for an HSEC / HSEC-K9 entry with status IN USE or AUTHORIZED, and confirm the throughput level is high enough. If HSEC is missing:

• In SD-WAN deployments, use the “Sync and Install HSEC Licenses” workflow in Cisco SD-WAN Manager (vManage) [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/licensing/managing-hsec-licenses-in-cisco-catalyst-sd-wan.html].
• In autonomous mode, follow the Catalyst 8200/8300/8500 online or offline HSEC install guide — register the device with Smart Licensing, then install/activate HSEC [Source: https://www.cisco.com/c/en/us/support/docs/routers/catalyst-8300-series-edge-platforms/225417-install-hsec-license-in-catalyst-8300.html].

On the C8000V you set the throughput level explicitly and reload; at or above the 250 Mbps/T1 level an HSEC license is required to avoid the export cap [Source: https://www.cisco.com/c/en/us/td/docs/routers/C8000V/configure-licenses-throughput-c8000v.html]:

conf t
 platform hardware throughput level 500000   ! example: 500 Mbps
end
write memory
reload

Finally, remember that feature overhead (zone-based firewall, IPS, NetFlow, heavy QoS) consumes data-plane cycles and can hold real throughput below the licensed ceiling even when HSEC and the tier are correct. Check show platform hardware qfp active datapath utilization summary if numbers fall short [Source: https://www.ciscolive.com/c/dam/r/ciscolive/global-event/docs/2023/pdf/BRKENT-2139.pdf].

Worked Example: Choosing the Right Tier

A regional architect must license two sites. Walk the decision the way you would in a real design review.

Site A — Small branch, simple SD-WAN. A Catalyst 8300 provides SD-WAN with DIA break-out, basic application-aware routing, standard NAT and firewalling, and a 200 Mbps internet circuit, all encrypted back to the hub.

• Feature need: basic SD-WAN and standard security → DNA Essentials, matched with Network Essentials.
• Throughput need: ~200 Mbps, all encrypted. That sits below the 250 Mbps export cap, so a 250 Mbps tier (T1-class) covers it and no HSEC is required — the encrypted traffic never crosses the cap.
• Term: align to lifecycle → 5-year subscription.
• Result: Network Essentials + DNA Essentials, 250 Mbps tier, 5 years, no HSEC [Source: https://www.layer23-switch.com/blog/cisco-c8000-dna-ordering-guide.html].

Site B — Regional hub, complex and high-speed. A Catalyst 8500 aggregates many branches with large-scale SD-WAN, advanced traffic engineering, segmentation, integrated security, and ecosystem integration (ISE, Secure Network Analytics, ThousandEyes), carrying ~800 Mbps of encrypted hub traffic.

• Feature need: advanced SD-WAN, rich segmentation, ecosystem integration → DNA Advantage, matched with Network Advantage.
• Throughput need: ~800 Mbps encrypted. This exceeds 250 Mbps, so you need a 1 Gbps tier (T2/T3-class) and an HSEC license — both gates must be cleared [Source: https://www.cisco.com/c/en/us/td/docs/routers/cloud_edge/configure-licenses-throughput-catalyst-8000-platforms.html].
• Term: a long-lived hub → 5- to 7-year subscription.
• Result: Network Advantage + DNA Advantage, ~1 Gbps tier, HSEC-K9, 5–7 years [Source: https://www.layer23-switch.com/blog/cisco-c8000-dna-ordering-guide.html].

The reusable rule of thumb: default to Essentials, escalate to Advantage by exception, and add HSEC the moment your encrypted throughput target crosses ~250 Mbps. Validating that each site genuinely needs Advantage — many branches do not — is one of the most effective ways to control licensing spend [Source: https://www.layer23-switch.com/blog/cisco-c8000-dna-ordering-guide.html].

Figure 4.5: License-selection decision tree

flowchart TD
    START["New site to license"] --> Q1{"Need advanced SD-WAN, rich segmentation, or ecosystem integration?"}
    Q1 -->|"No"| ESS["Network Essentials + DNA Essentials"]
    Q1 -->|"Yes"| ADV["Network Advantage + DNA Advantage (or Premier)"]
    ESS --> Q2{"Encrypted throughput > ~250 Mbps?"}
    ADV --> Q2
    Q2 -->|"No"| TIER1["Size throughput tier to circuit; no HSEC needed"]
    Q2 -->|"Yes"| TIER2["High throughput tier + HSEC-K9 license"]
    TIER1 --> TERM["Choose term (3/5/7 yr) aligned to lifecycle"]
    TIER2 --> TERM

Key Takeaway: HSEC-K9 lifts the U.S. export-control cap of ~250 Mbps on encrypted throughput; it is independent of the throughput tier, and your real crypto rate is limited by whichever gate is lower. Boost/Performance licenses (ISR) and high throughput tiers (Catalyst 8000) remove general forwarding caps but never substitute for HSEC. To exceed 250 Mbps of IPsec you need a high tier and active HSEC.

Chapter Summary

Catalyst 8000 licensing rests on a two-stack model: a perpetual Network stack (Essentials or Advantage) that grants routing rights, and a term-based DNA stack (Essentials, Advantage, or Premier) that rents SD-WAN, automation, analytics, and security. The two are tier-matched, the DNA subscription is the SD-WAN license, and on the 8200/8300 the subscription is mandatory. When DNA expires the router keeps routing but loses its subscription features — a quiet failure mode worth monitoring. Routing remains outside Cisco Networking Subscription, so these routers stay on DNA.

A separate throughput tier (T0–T3) caps forwarding speed in software. Compliance is handled by Smart Licensing Using Policy (SLP) from IOS XE 17.3.2 onward, which abolished token registration and evaluation mode: unenforced licenses work immediately, and the router simply sends periodic RUM reports to CSSM — online, via CSLU/on-prem, or by offline file exchange — within policy windows (typically 90 days after a usage change). Overdue reports generate warnings, not outages.

Finally, encryption speed is gated independently. The HSEC-K9 license lifts the export-control cap of ~250 Mbps on encrypted throughput; without it, IPsec stays capped even on a multi-gigabit tier while plaintext flows freely. Reaching high IPsec rates requires both a sufficient throughput tier and HSEC, and Boost/Performance licenses address general forwarding — never crypto. Choosing well means defaulting to Essentials, escalating to Advantage by exception, sizing the throughput tier to the circuit, and adding HSEC whenever encrypted throughput will exceed 250 Mbps.

Key Terms

Chapter 5: SD-WAN Fundamentals on the Catalyst 8000

The Catalyst 8000 family is more than a fast router — it is a purpose-built WAN Edge device designed to live inside a software-defined WAN (SD-WAN) fabric. To understand what a Catalyst 8000 actually does in production, you first have to understand the control architecture that surrounds it: the controllers that manage it, the protocol that tells it where everything is, and the encrypted tunnels it builds to carry real user traffic. This chapter walks through that architecture from the top down, then follows a brand-new router from the moment it is unboxed until it is forwarding application traffic across an intelligent overlay.

Learning Objectives

By the end of this chapter, you will be able to:

• Describe the Cisco Catalyst SD-WAN (Viptela) control plane components and the plane each one belongs to.
• Explain how a Catalyst 8000 WAN Edge onboards into an SD-WAN fabric using ZTP or PnP.
• Trace the relationship between the OMP control plane and the IPsec data plane, including TLOCs and colors.

SD-WAN Control Components

Traditional routing collapses everything — management, control decisions, and packet forwarding — into each individual router. Cisco Catalyst SD-WAN (the technology formerly known as Viptela) takes the opposite approach: it cleanly separates the network into four planes, each handled by a dedicated role [Source: https://www.cisco.com/c/en/us/td/docs/solutions/CVD/SDWAN/cisco-sdwan-design-guide.html].

A useful analogy is an airport. The WAN Edge routers are the aircraft that actually carry passengers (data). vBond is the security checkpoint and information desk at the entrance — it verifies your identity and tells you which gate to go to. vSmart is air traffic control, deciding which planes fly which routes. And Catalyst SD-WAN Manager is the airport operations center, where staff plan schedules, watch every screen, and set the rules everyone follows. No single component does everything, and that separation is exactly what makes the system scalable and easy to operate.

Figure 5.1: The four planes of the Catalyst SD-WAN architecture

flowchart TD
    vManage["Management Plane<br/>Catalyst SD-WAN Manager (vManage)<br/>Templates, policy, monitoring, device whitelist"]
    vSmart["Control Plane<br/>vSmart controllers<br/>OMP route reflector, policy enforcement"]
    vBond["Orchestration Plane<br/>vBond / Validator<br/>Authentication, controller discovery, NAT traversal"]
    Edge["Data Plane<br/>WAN Edge routers (Catalyst 8000)<br/>Forward user traffic across the overlay"]

    vManage -->|"Pushes config & policy"| vSmart
    vManage -->|"Holds whitelist consulted by"| vBond
    vSmart -->|"Distributes routes & policy via OMP"| Edge
    vBond -->|"Admits & hands controller list to"| Edge
    Edge -->|"Telemetry / config session"| vManage

vManage / Catalyst SD-WAN Manager

Catalyst SD-WAN Manager (its current name; you will still hear it called vManage) is the management plane — the network management system (NMS) and single pane of glass for the entire fabric [Source: https://www.ciscopress.com/articles/article.asp?p=3203553]. It performs four broad jobs.

First, it owns central configuration and templates. It stores the device inventory and per-site parameters such as system IP, site ID, and organization-name, and it uses device templates and feature templates to generate complete configurations that it pushes down to controllers and WAN Edges over secure connections [Source: https://www.cisco.com/c/en/us/td/docs/solutions/CVD/SDWAN/cisco-sdwan-design-guide.html].

Second, it is the place where policy is authored. Administrators define control policies (route filtering, route manipulation, topology) and data policies (traffic steering, QoS, service chaining, security) in the GUI. vManage translates those clicks into OMP policy constructs and hands them to the vSmart controllers, which enforce them on the edges [Source: https://www.ciscopress.com/articles/article.asp?p=3203553&seqNum=3].

Third, it handles monitoring, analytics, and lifecycle management — collecting telemetry on interfaces, tunnel SLAs, and application performance; presenting dashboards and alarms; and managing software images and upgrades [Source: https://www.ciscopress.com/articles/article.asp?p=3203553].

Fourth, and critically for onboarding, it is the device authority. vManage holds the whitelist of authorized serial numbers and UUIDs that vBond consults when a new device tries to join, and in IOS-XE deployments it integrates with Cisco Plug and Play [Source: https://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/SDWAN/sdwan-wan-edge-onboarding-deploy-guide-2020nov.pdf]. The practical takeaway: when a Catalyst 8000 reaches vBond but is rejected, the very first thing to check is whether its chassis/serial appears (and is authorized) in vManage’s device list [Source: https://www.grandmetric.com/cisco-viptela-sd-wan-components-connectivity-viptela-part-1/].

Key Takeaway: Catalyst SD-WAN Manager (vManage) is the management plane — it stores templates and inventory, authors all policy, monitors the fabric, and acts as the authoritative device whitelist used during onboarding.

vSmart Controllers

If vManage is where decisions are authored, vSmart is where they are distributed. vSmart is the centralized control-plane engine of Catalyst SD-WAN [Source: https://www.cisco.com/c/en/us/td/docs/solutions/CVD/SDWAN/cisco-sdwan-design-guide.html].

vSmart maintains a secure DTLS/TLS control connection to every WAN Edge in the fabric. Over those connections, edges advertise their reachable prefixes and their TLOCs (transport locators — covered in detail later) using the Overlay Management Protocol (OMP). vSmart behaves like a route reflector for the overlay: it collects everything, runs best-path selection, and redistributes the appropriate routes back out to each edge based on policy [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/omp-overview]. Edges never peer directly with one another at the control plane — they peer with vSmart, which gives the controller a complete, global view of the network.

Because vSmart sees the whole topology, it is also where topology and segmentation are realized. It builds full-mesh, hub-and-spoke, or regional-mesh designs purely by choosing which OMP routes to advertise where, and it enforces multi-tenancy by tagging and distributing routes per VPN (for example, keeping VPN 10 separate from VPN 20) [Source: https://www.cisco.com/c/en/us/td/docs/solutions/CVD/SDWAN/cisco-sdwan-design-guide.html]. Finally, vSmart is the policy enforcement point — it applies the control and data policies that vManage authored and pushes the resulting decisions to the edges, which implement them in their data plane.

A common field symptom illustrates vSmart’s role: if a Catalyst 8000 has a healthy management connection to vManage but builds no overlay tunnels to other sites, the problem is usually that its control connection to vSmart is down — or that a control policy on vSmart is deliberately blocking those routes [Source: https://www.ciscopress.com/articles/article.asp?p=3203553&seqNum=3].

Key Takeaway: vSmart is the centralized control plane — it runs OMP as a route reflector, holds a global view of all sites and TLOCs, builds VPN topologies, and enforces the control and data policies authored in vManage.

vBond / Validator Orchestrator

The vBond orchestrator — now also called the Validator — is the orchestration plane and, almost always, the first SD-WAN node a brand-new WAN Edge talks to [Source: https://study-ccnp.com/cisco-sd-wan-architecture-overview/]. Think of it as the bouncer at the door who also happens to know where every room in the building is.

vBond does three things. It performs authentication and admission control: it establishes a secure DTLS/TLS session with each WAN Edge (and with vSmart and vManage) using certificates, validates the device’s identity, and checks with vManage that the serial/UUID is on the authorized whitelist. If the device checks out, it is admitted; if not, it is rejected [Source: https://study-ccnp.com/cisco-sd-wan-architecture-overview/].

It performs controller discovery: once a device is admitted, vBond hands it the IP addresses or FQDNs of the available vSmart and vManage instances. The edge then builds direct control connections to those controllers — vBond steps out of the path after the introduction is made [Source: https://www.cisco.com/c/en/us/td/docs/solutions/CVD/SDWAN/cisco-sdwan-design-guide.html].

And it performs NAT traversal: because vBond sits on a public-facing IP and learns both the inside and outside addresses of every device, it can coordinate how a router behind NAT should connect, so that direct edge-to-controller sessions can form even across address translation [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/high-availability]. When onboarding stalls at “Attempting to contact vBond,” the usual culprits are a wrong or unreachable vBond FQDN in DNS, blocked ports, or a clock/certificate mismatch breaking TLS [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/high-availability]. For resilience, production fabrics typically deploy multiple vBond instances behind DNS round-robin.

Key Takeaway: vBond (the Validator) is the orchestration plane and first point of contact — it authenticates devices against the vManage whitelist, hands them the controller addresses, and handles NAT traversal, then drops out of the path once direct control connections form.

WAN Edge Onboarding

Getting a Catalyst 8000 from its shipping box into a live fabric is called onboarding, and the design goal is zero-touch: a non-technical installer should be able to rack the device, plug in the WAN cable, and power it on, with everything else happening automatically. Two workflows accomplish this — ZTP and PnP — and although they start differently, they converge on exactly the same end state.

ZTP and PnP Onboarding

Zero Touch Provisioning (ZTP) is the classic Viptela-style workflow, originally built for vEdge hardware and cloud-based edges, though it also works with IOS-XE SD-WAN [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/wan-edge-deployment]. The flow proceeds in stages:

1. Boot. A factory-default device powers on with its image and built-in identity certificate. Its WAN-facing interface uses DHCP to obtain an IP address, gateway, and DNS server.
2. Discover vBond. The device queries a well-known ZTP FQDN (or uses a statically configured ZTP/vBond address). The ZTP service validates the device’s serial/chassis ID and returns the organization name and the vBond FQDN/IP.
3. Connect to vBond. The edge opens a DTLS/TLS session to vBond using its certificate and organization name; vBond authenticates it and confirms with vManage that the serial is authorized.
4. Receive the controller list. If authorized, vBond sends the addresses (and often priorities) of the vSmart and vManage servers, frequently with multiple entries for redundancy.
5. Establish control connections. The edge forms a direct OMP control connection to each vSmart and a management connection to vManage.
6. Download configuration. vManage recognizes the device by serial and either auto-attaches a predefined template or waits for an operator to attach one, then pushes the full configuration — system parameters, VPNs, routing protocols, transport and IPsec settings.
7. Form the overlay. The edge advertises its TLOCs and prefixes via OMP; vSmart distributes them; IPsec tunnels come up and user traffic begins to flow [Source: https://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/SDWAN/sdwan-wan-edge-onboarding-deploy-guide-2020nov.pdf].

Figure 5.2: WAN Edge zero-touch onboarding handshake

sequenceDiagram
    participant Edge as WAN Edge (Catalyst 8000)
    participant vBond
    participant vManage
    participant vSmart

    Edge->>vBond: Boot, DHCP, then DTLS/TLS with certificate + org-name
    vBond->>vManage: Verify serial / UUID against whitelist
    vManage-->>vBond: Serial authorized
    vBond-->>Edge: Admitted; here are the vSmart & vManage addresses
    Edge->>vSmart: Establish direct OMP control connection
    Edge->>vManage: Establish management connection
    vManage-->>Edge: Push full configuration (VPNs, routing, IPsec)
    Edge->>vSmart: Advertise TLOCs & prefixes via OMP
    vSmart-->>Edge: Distribute remote routes; IPsec tunnels form and traffic flows

Plug and Play (PnP) is the workflow used for IOS-XE platforms — which includes the Catalyst 8000 family — and integrates with Cisco’s cloud PnP service or an on-prem PnP server [Source: https://www.lookingpoint.com/blog/cisco-sd-wan-pnp-onboarding]. A factory-default Catalyst 8000 boots, gets an IP via DHCP, and then reaches out over HTTPS to a default Cisco PnP host (a devicehelper FQDN) or to an on-prem server, identifying itself with its burned-in SUDI (Secure Unique Device Identifier) certificate. PnP looks up the device’s claim record and returns controller information — vBond/vManage/vSmart addresses and the organization name — plus an optional small bootstrap configuration (system IP, site ID, VPN 0 parameters). From that point on, the device follows steps 3–7 above, exactly like ZTP [Source: https://www.lookingpoint.com/blog/cisco-sd-wan-pnp-onboarding].

The simplest way to remember the distinction:

For air-gapped or offline networks, neither cloud service is reachable, so operators use an on-prem PnP/ZTP server or perform a manual Day-0 bootstrap via USB or console — keying in vbond, organization-name, system-ip, site-id, and VPN 0 addressing by hand — after which the device still uses vBond/vSmart/vManage for the overlay [Source: https://www.ciscolive.com/c/dam/r/ciscolive/global-event/docs/2025/pdf/BRKIOT-2911.pdf].

Key Takeaway: ZTP (classic Viptela/vEdge) and PnP (IOS-XE, including Catalyst 8000) are two ways to discover vBond automatically; PnP uses the SUDI certificate and a cloud/on-prem server, but both converge on authenticating to vBond, learning the controllers, and downloading configuration from vManage.

Certificates and Serial Whitelisting

Zero-touch onboarding only works because the fabric can trust a device it has never seen before, and that trust rests on two pillars: certificates and the serial whitelist.

Every WAN Edge ships with a built-in identity certificate. On a Catalyst 8000, that is the SUDI certificate, programmed into a tamper-resistant chip at the factory and signed by Cisco’s certificate authority [Source: https://www.lookingpoint.com/blog/cisco-sd-wan-pnp-onboarding]. When the device contacts vBond, the DTLS/TLS handshake proves cryptographically that the router is a genuine Cisco device with the identity it claims. Certificates also secure every other control connection in the fabric — edge-to-vSmart, edge-to-vManage, and controller-to-controller [Source: https://study-ccnp.com/cisco-sd-wan-architecture-overview/].

Certificates prove what a device is; the whitelist decides whether it is allowed in. An attacker could, in principle, present a valid Cisco certificate from some other router, so authentication alone is not enough. vManage maintains an authoritative list of the serial numbers and UUIDs that belong in this organization’s fabric, and vBond checks every joining device against it [Source: https://www.grandmetric.com/cisco-viptela-sd-wan-components-connectivity-viptela-part-1/]. A device must pass both tests — valid certificate and an authorized serial — before it is admitted.

Two failure modes dominate real deployments. The first is the dreaded “chassis not authorized” rejection, which means the serial is missing from (or not authorized in) vManage’s inventory [Source: https://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/SDWAN/sdwan-wan-edge-onboarding-deploy-guide-2020nov.pdf]. The second is subtler: time. TLS handshakes fail if the device clock is wrong or a certificate has expired, so an edge with no NTP and a battery-dead clock may present a perfectly valid certificate and still be unable to complete the handshake [Source: https://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/SDWAN/sdwan-wan-edge-onboarding-deploy-guide-2020nov.pdf]. A mismatched organization-name between the device and the controllers produces the same kind of admission failure.

Key Takeaway: Onboarding trust requires both a valid device certificate (SUDI on the Catalyst 8000) proving identity and a matching serial/UUID on the vManage whitelist proving authorization; mismatched serials, wrong organization-name, expired certs, or a bad clock are the classic failure causes.

Control Connections

Once a device is admitted, the fabric runs on control connections — the secure DTLS/TLS sessions that carry everything except user data. It is worth being precise about which sessions exist and what each one does, because troubleshooting almost always starts here.

After vBond makes the introduction, each WAN Edge establishes separate, direct DTLS/TLS sessions to vManage and to each vSmart controller [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/control-connections]. The session to vManage is used mainly to push and pull configuration and telemetry. The session to vSmart is the one that carries OMP — routes, TLOCs, services, policies, and even the encryption keys the edges will later use [Source: https://www.linkedin.com/pulse/sd-wan-control-connections-explained-how-dtlstls-sessions-abusaa-zb2ef]. An edge forms one OMP adjacency with each vSmart, which is what provides control-plane redundancy and lets the design scale out [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/control-connections].

Figure 5.3: Control connections versus the IPsec data plane

flowchart LR
    subgraph Controllers
        vManage["vManage<br/>(config & telemetry)"]
        vSmart["vSmart<br/>(OMP routes, TLOCs, keys)"]
    end
    EdgeA["WAN Edge A"]
    EdgeB["WAN Edge B"]

    EdgeA -.->|"DTLS/TLS control"| vManage
    EdgeA -.->|"DTLS/TLS + OMP"| vSmart
    EdgeB -.->|"DTLS/TLS control"| vManage
    EdgeB -.->|"DTLS/TLS + OMP"| vSmart
    EdgeA ==>|"IPsec data-plane tunnel (user traffic)"| EdgeB

The important mental separation is this: control connections (edge↔controller, DTLS/TLS) are completely independent of the IPsec data-plane tunnels that later form between edges. They run over different paths, secure different traffic, and fail in different ways. When diagnosing a fabric, the disciplined order is to verify the control plane first — are the DTLS/TLS sessions up to each controller, and is the OMP session exchanging routes? — before ever looking at data-plane tunnels [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/control-connections]. If control connections are up but no OMP routes are arriving, the cause is usually a control policy on vSmart rather than an underlay or IPsec problem.

Key Takeaway: After vBond’s introduction, each WAN Edge builds direct DTLS/TLS control connections to vManage (config/telemetry) and to every vSmart (OMP routes, TLOCs, and keys); these control sessions are independent of the data-plane tunnels, so troubleshooting always checks the control plane first.

Overlay Planes

With the device onboarded and its control connections up, we can finally look at how the fabric actually moves packets. This is where two distinct “planes” meet: the OMP control plane, which decides what exists and where, and the IPsec data plane, which physically carries the traffic. Keeping these two ideas separate is the single most useful concept in all of SD-WAN.

OMP Control Plane

The Overlay Management Protocol (OMP) is the brain of Catalyst SD-WAN. It is a TCP-based, path-vector protocol that is conceptually very similar to BGP — it advertises reachability with rich attributes and makes policy-driven best-path decisions [Source: https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2025/pdf/BRKENT-3115.pdf]. The crucial structural fact is that OMP runs between each WAN Edge and the vSmart controllers, never directly between edges, and it runs inside the secure DTLS/TLS control connection [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/omp-overview]. Locally learned routes on an edge — connected, static, OSPF, EIGRP, or BGP — can be redistributed into OMP and advertised across the overlay, much as you would redistribute into BGP at a traditional edge.

OMP carries three types of routes, and understanding the division of labor between them is what makes the overlay click:

A vRoute is a VPN prefix bound to a TLOC — for example, “192.168.1.0/24 in VPN 10, reachable via TLOC (1.1.1.1, mpls, ipsec).” A TLOC route describes a WAN attachment point itself, advertised in the transport VPN (VPN 0). A service route advertises something like a firewall so that policy can steer traffic through it [Source: https://www.dclessons.com/unicast-routing-overview].

The end-to-end flow ties these together. An edge learns LAN prefixes locally, then advertises its TLOC routes and vRoutes (and any service routes) up to vSmart over the OMP session. vSmart maintains a global OMP RIB, runs best-path selection across everything it has heard, applies centralized control policy, and advertises the selected, policy-allowed routes back down to each edge — tailored to that site’s topology. The receiving edge installs the vRoutes into its per-VPN routing tables and resolves each route’s next hop using TLOC information, which tells it which transport to use to reach the destination [Source: https://www.grandmetric.com/knowledge-base/design_and_configure/sd-wan-overlay-management-protocol-omp/]. In one line: local route → OMP vRoute/TLOC → vSmart global view and policy → remote edge → TLOC resolves the transport → tunnel.

Figure 5.4: End-to-end OMP route flow through vSmart

flowchart LR
    LAN["Local LAN prefixes<br/>(connected, static, OSPF, EIGRP, BGP)"]
    EdgeSrc["Source WAN Edge<br/>redistributes into OMP"]
    vSmart["vSmart<br/>Global OMP RIB<br/>best-path + control policy"]
    EdgeDst["Remote WAN Edge<br/>installs vRoutes per VPN"]
    Resolve["TLOC resolves next hop<br/>to a transport"]
    Tunnel["IPsec tunnel carries traffic"]

    LAN --> EdgeSrc
    EdgeSrc -->|"Advertise vRoutes, TLOC routes, service routes"| vSmart
    vSmart -->|"Selected, policy-allowed routes"| EdgeDst
    EdgeDst --> Resolve
    Resolve --> Tunnel

Key Takeaway: OMP is a BGP-like, path-vector control protocol running between edges and vSmart inside DTLS/TLS; it advertises vRoutes (what prefixes exist), TLOC routes (how sites attach to transports), and service routes (chainable services), and vSmart uses its global view plus policy to program each edge’s overlay.

IPsec Data Plane and TLOCs

OMP decides everything, but it carries no user data. The actual traffic rides IPsec tunnels built directly between WAN Edges — and the object that makes those tunnels possible is the TLOC.

A TLOC (Transport Locator) is the attachment point where a WAN Edge connects to a particular transport (Internet, MPLS, LTE, Metro-E). It is uniquely identified by a 3-tuple [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/what-is-a-tloc]:

TLOC = (system IP, color, encapsulation)

The system IP is a logical router ID — not the interface address — and it stays the same across all of a router’s TLOCs; color and encapsulation are what distinguish one of its WAN links from another [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/17-x/systems-interfaces/systems-interfaces-guide-17-x/tloc.html]. The encapsulation is the tunnel protocol — IPsec (the default, with encryption) or GRE — and the two ends must match for a tunnel to form. Default tunnel MTU depends on encapsulation: 1442 bytes for IPsec, 1468 for GRE, later adjusted by BFD-based path MTU discovery [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/17-x/systems-interfaces/systems-interfaces-guide-17-x/tloc.html].

Here is the relationship that ties the two planes together, and it is worth stating carefully. The DTLS/TLS control connections and the IPsec data tunnels are decoupled but interdependent. vSmart, via OMP, distributes not only routes but also the encryption keys and security parameters edges need to build IPsec sessions [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/sdwan-xe-gs-book/system-overview.html]. TLOC routes tell an edge which remote system-IP and transport to connect to; once two edges hold matching TLOC routes and the necessary crypto material, they negotiate IPsec directly and the data-plane tunnel comes up [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/sdwan-xe-gs-book/system-overview.html]. In other words, OMP builds the data plane: change OMP, and the IPsec topology follows.

This decoupling is also the key to troubleshooting. If an IPsec tunnel flaps while OMP stays up, suspect the underlay — transport reachability, NAT, or IPsec parameters — not control-plane routing. But if OMP/DTLS goes down, the edge stops receiving fresh routes and keys; existing tunnels may linger briefly, but once timers expire, routes and tunnels are withdrawn and connectivity is lost until OMP recovers [Source: https://www.cisco.com/c/en/us/support/docs/routers/sd-wan/222045-troubleshoot-performance-and-design-appl.pdf].

For each active tunnel, the two edges also run a BFD (Bidirectional Forwarding Detection) session inside the tunnel. BFD detects failures quickly and continuously measures latency, jitter, and loss; those measurements feed back to drive application-aware, SLA-based routing — letting the fabric prefer MPLS for a latency-sensitive app and fall back to Internet when MPLS degrades [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/17-x/systems-interfaces/systems-interfaces-guide-17-x/tloc.html].

Key Takeaway: A TLOC = (system IP, color, encapsulation) identifies each WAN attachment; OMP distributes TLOC routes and crypto keys so edges build IPsec tunnels per TLOC pair, each running its own BFD session. If IPsec flaps but OMP is up, look at the underlay; if OMP drops, the data plane eventually collapses with it.

Color and Transport Definitions

The middle element of the TLOC tuple — color — does more work than its name suggests. A color is a globally significant label that identifies a specific transport and tells the fabric whether that transport should be treated as public or private. That public/private classification, in turn, governs which IP address (and whether NAT traversal) is used when building a tunnel [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/tloc-color-and-carrier].

Cisco provides a fixed set of colors. A single WAN Edge cannot reuse the same color twice, so a router with two Internet links must use two different public colors (for example, biz-internet and public-internet).

The address-selection rule follows directly from the classification: if both colors are public — or if either is public — the edges try to form the tunnel over their public IP addresses; only when both colors are private do they use private addresses [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/tloc-color-and-carrier]. This is why choosing the right color is a design decision, not just a label: tag an MPLS link biz-internet and the fabric will try to reach it via a public address that may not exist, breaking tunnel formation.

Two controls refine the default behavior. The restrict keyword on a color forces a TLOC to build tunnels only to TLOCs of the same color. Tunnel groups add an orthogonal dimension: only TLOCs sharing a tunnel group will connect, regardless of color — useful in multi-carrier or multi-tenant designs [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/tloc-color-and-carrier].

Worked example — a Catalyst 8000 branch with MPLS and Internet. Consider a branch (system IP 1.1.1.1) and a data center (system IP 2.2.2.2), each with an MPLS link and a biz-internet link [Source: https://www.cisco.com/c/en/us/td/docs/solutions/CVD/SDWAN/cisco-sdwan-design-guide.html]:

• Branch TLOCs: A = (1.1.1.1, mpls, ipsec) and B = (1.1.1.1, biz-internet, ipsec)
• Data Center TLOCs: C = (2.2.2.2, mpls, ipsec) and D = (2.2.2.2, biz-internet, ipsec)

All four TLOCs are advertised to vSmart via OMP. By default, vSmart’s policy allows the matching pairs: A ↔ C (mpls ↔ mpls) forms an IPsec tunnel over the private MPLS addresses with no NAT, and B ↔ D (biz-internet ↔ biz-internet) forms an IPsec tunnel over the public Internet addresses, typically with NAT on each side. Cross-transport pairs (A ↔ D, B ↔ C) form only if the design allows them.

Figure 5.5: TLOC pairing and IPsec tunnels between a branch and data center

flowchart LR
    subgraph Branch["Branch (system IP 1.1.1.1)"]
        A["TLOC A<br/>(1.1.1.1, mpls, ipsec)"]
        B["TLOC B<br/>(1.1.1.1, biz-internet, ipsec)"]
    end
    subgraph DC["Data Center (system IP 2.2.2.2)"]
        C["TLOC C<br/>(2.2.2.2, mpls, ipsec)"]
        D["TLOC D<br/>(2.2.2.2, biz-internet, ipsec)"]
    end

    A ==>|"Private MPLS, no NAT"| C
    B ==>|"Public Internet, with NAT"| D
    A -.->|"Cross-transport, only if allowed"| D
    B -.->|"Cross-transport, only if allowed"| C
``` Each tunnel runs its own BFD session, so vSmart can steer latency-sensitive applications onto the MPLS path (A–C) and shift to Internet (B–D) when MPLS performance drops. Applying `restrict` to the colors would forbid the cross-transport tunnels entirely, guaranteeing mpls-to-mpls and biz-internet-to-biz-internet only [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/tloc-color-and-carrier].

To verify all of this on a live Catalyst 8000, operators use `show omp tlocs` to inspect learned TLOCs and `show sdwan bfd sessions` to confirm tunnels and their BFD state per TLOC pair [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/17-x/systems-interfaces/systems-interfaces-guide-17-x/tloc.html].

> **Key Takeaway:** Color labels a transport and classifies it as public or private, which decides whether tunnels use public IPs (with NAT) or private IPs; a router can't reuse a color, encapsulation must match, and `restrict`/tunnel-groups give finer control over which TLOC pairs are allowed to build IPsec tunnels.

### Chapter Summary

Cisco Catalyst SD-WAN cleanly separates the network into four planes, each with a dedicated role. **Catalyst SD-WAN Manager (vManage)** is the management plane — templates, policy authoring, monitoring, and the authoritative device whitelist. **vSmart** is the centralized control plane, running OMP as a route reflector and enforcing policy. **vBond (the Validator)** is the orchestration plane — the first-contact authenticator that admits devices and hands out controller addresses before stepping aside. The **WAN Edge** routers, including the Catalyst 8000, are the data plane that forwards real traffic.

A new Catalyst 8000 joins the fabric through zero-touch **onboarding**, using **ZTP** (classic Viptela) or, for IOS-XE platforms, **PnP**. Both converge on the same path: discover vBond, authenticate with a device certificate (the SUDI on a Catalyst 8000), pass the serial **whitelist** check in vManage, learn the controllers, build direct **control connections** to vManage and vSmart, and download a full configuration. The classic onboarding failures are an unauthorized serial, a mismatched organization-name, an expired certificate, or a bad clock breaking TLS.

Finally, the running fabric is best understood as two cooperating planes. **OMP** — a BGP-like path-vector protocol between edges and vSmart — distributes vRoutes, TLOC routes, and service routes, plus the encryption keys edges need. Edges then build **IPsec data-plane tunnels** directly between matching **TLOCs**, where each TLOC is the tuple (system IP, **color**, encapsulation). Color classifies a transport as public or private and thereby decides address selection and NAT behavior, while BFD inside each tunnel measures performance to drive application-aware routing. OMP builds the data plane: when control-plane routing changes, the IPsec topology follows.

### Key Terms

| Term | Definition |
|------|------------|
| Catalyst SD-WAN Manager | The management-plane NMS (formerly vManage) that stores templates and inventory, authors control and data policies, monitors the fabric, and holds the authoritative device whitelist used during onboarding. |
| vSmart | The centralized control-plane controller that runs OMP as a route reflector for the overlay, distributes routes/TLOCs/services per policy, builds VPN topologies, and enforces the policies authored in vManage. |
| vBond | The orchestration-plane component (also called the Validator) that is the first node a WAN Edge contacts; it authenticates devices via certificates, validates them against the vManage whitelist, hands out vSmart/vManage addresses, and handles NAT traversal. |
| OMP | Overlay Management Protocol — a TCP-based, BGP-like path-vector protocol running between WAN Edges and vSmart inside DTLS/TLS, distributing vRoutes, TLOC routes, service routes, and encryption keys that drive IPsec tunnel formation. |
| TLOC | Transport Locator — the attachment point of a WAN Edge to a specific transport, uniquely identified by the 3-tuple (system IP, color, encapsulation), used by the fabric to build IPsec/GRE data-plane tunnels. |
| Color | A globally significant label that identifies a WAN transport and classifies it as public (e.g., biz-internet, lte) or private (e.g., mpls, metro-ethernet), determining which IP address and NAT behavior are used to form a tunnel; a single WAN Edge cannot reuse a color. |
| ZTP | Zero Touch Provisioning — the classic Viptela-style automatic onboarding workflow (vEdge/cloud edges) in which a factory-default device discovers vBond via a ZTP DNS FQDN, authenticates, and downloads its configuration. |
| Plug and Play | PnP — the onboarding workflow for IOS-XE platforms (including the Catalyst 8000) in which a factory-default device uses its SUDI certificate to reach a Cisco cloud or on-prem PnP server, retrieve controller info and a bootstrap config, then join the fabric like ZTP. |

---

## Chapter 6: Routing and WAN Connectivity

### Learning Objectives

By the end of this chapter, you will be able to:

- Configure core routing protocols (OSPF, BGP, and EIGRP) on Catalyst 8000 platforms operating in SD-WAN mode.
- Explain the distinction between service-side and transport-side routing in SD-WAN deployments, and how each maps to the underlay and overlay.
- Describe WAN transport options including MPLS, public internet, and cellular/5G connectivity, and how each is represented as a TLOC.
- Configure VRF-based segmentation using service VPNs, and reason about inter-VRF routing and route leaking.

A Catalyst 8000 edge router rarely lives alone. It sits at the boundary between a branch LAN and one or more wide-area transports, stitching local networks into a global fabric. To do that job well, it has to speak two routing "languages" at once: the traditional protocols that talk to LAN switches and provider routers, and the Overlay Management Protocol (OMP) that talks to the rest of the SD-WAN fabric. This chapter walks through both, and shows how segmentation (VRFs) keeps traffic in its lane across the whole system.

A useful mental model up front: think of the Catalyst 8000 as a multilingual translator at an international airport. The **underlay** protocols (BGP, OSPF, static routes toward providers) are the languages spoken in the airport's home country — the local infrastructure. **OMP** is the universal language spoken between airports worldwide. The translator's job is to decide exactly which local announcements get translated into the universal language and broadcast to other airports, and which global announcements get translated back into the local tongue. Translate too freely and you get chaos (routing loops); translate too little and travelers get stranded (unreachable subnets).

---

### Routing Protocols

In SD-WAN mode, the Catalyst 8000 organizes everything around **VPNs**, which are the SD-WAN term for separate routing instances. There are three roles: VPN 0 is the transport underlay, VPN 512 is management, and VPN 1 through 511 (and beyond) are **service VPNs** — the user and data VRFs where customer traffic lives [Source: https://networklessons.com/cisco/cisco-sd-wan/cisco-sd-wan-ospf-configuration]. The critical rule for this section is that **BGP, OSPF, and EIGRP are configured *inside* service VPNs, not in the global table** [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/lab1-manipulating-site-local-bgp-routing]. Each protocol process is scoped to a VPN/VRF and uses that VRF's routing table.

Throughout this section we will use a single worked scenario: a branch Catalyst 8000 WAN Edge where **service VPN 10** is the user LAN VRF. Interface `GigabitEthernet0/0/1` lives in VPN 10 and faces a LAN switch running OSPF area 0, and the same VPN runs BGP toward an MPLS PE or data-center router (local AS 65010, neighbor 192.0.2.2 in AS 65000) [Source: https://networklessons.com/cisco/cisco-sd-wan/cisco-sd-wan-ospf-configuration].

#### BGP on the edge

**BGP (Border Gateway Protocol)** is the path-vector protocol that exchanges routing information between autonomous systems. On a Catalyst 8000 in SD-WAN mode, BGP is most commonly used in a service VPN to peer with an MPLS provider edge (PE) router or a data-center core, while OMP carries routes between SD-WAN sites.

The configuration uses Viptela-style `vpn N` blocks but remains IOS-like underneath [Source: https://networklessons.com/cisco/cisco-sd-wan/cisco-sd-wan-ospf-configuration]. Here is BGP in VPN 10 toward the PE:

```none
! BGP in VPN 10
vpn 10
 router bgp 65010
  bgp router-id 10.10.10.1
  neighbor 192.0.2.2 remote-as 65000
  neighbor 192.0.2.2 timers 10 30
  !
  address-family ipv4 unicast
   neighbor 192.0.2.2 activate
   ! Advertise local LAN routes into BGP
   redistribute connected
   redistribute ospf 10
  exit-address-family
 exit
!

Two details matter here. First, BGP is configured per VPN (vpn 10) so it draws from that VRF’s table [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/lab1-manipulating-site-local-bgp-routing]. Second, inside the IPv4 address family you must explicitly activate the neighbor and configure redistribution if you want LAN prefixes advertised toward the peer — without redistribute connected or redistribute ospf, the session can be established yet carry no prefixes [Source: https://theworldsgonemad.net/2022/sdwan-vpn0-bgp/].

That last point is the single most common BGP gotcha on these platforms. A show bgp vpn 10 summary that reports Estab looks healthy, but show bgp vpn 10 ipv4 unicast may show an empty table. The usual causes are: missing redistribution (LAN routes never enter BGP), the neighbor not activated under the IPv4 AF, or an outbound route-policy filtering everything [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/lab1-manipulating-site-local-bgp-routing]. The lesson: an “up” BGP session is necessary but not sufficient — always verify the prefix table.

BGP can also run on the transport side in VPN 0, peering with an ISP or MPLS provider to bring up the underlay (covered in the next section). When it does, it typically uses separate templates and far more restrictive advertisement, because you do not want raw transport routes leaking into the overlay [Source: https://theworldsgonemad.net/2022/sdwan-vpn0-bgp/].

OSPF and EIGRP service-side

OSPF (Open Shortest Path First) is a link-state IGP commonly run on the service side toward the branch LAN. Like BGP, the OSPF process is scoped to the service VPN rather than the global table:

! Service VPN 10 – define interface toward LAN
vpn 10
 interface GigabitEthernet0/0/1
  ip address 10.10.10.1 255.255.255.0
  no shutdown
 !
!
! OSPF in VPN 10
vpn 10
 router ospf 10
  router-id 10.10.10.1
  area 0
   interface GigabitEthernet0/0/1
    passive-interface disable
   exit
  exit
 exit
!

Notice two SD-WAN-specific habits. The OSPF process belongs to vpn 10, and interfaces are attached to an area directly (the area 0 / interface block) rather than with the classic network statement seen in traditional IOS [Source: https://networklessons.com/cisco/cisco-sd-wan/cisco-sd-wan-ospf-configuration]. Because SD-WAN templates often default LAN interfaces to passive, you must explicitly disable passive (passive-interface disable) on any interface where you expect to form an adjacency [Source: https://networklessons.com/cisco/cisco-sd-wan/cisco-sd-wan-ospf-configuration].

When an OSPF adjacency will not form, the failure modes mirror traditional OSPF: the interface was never added to the area, the interface is passive where it should be active, an MTU or network-type mismatch stalls the neighbors in EXSTART/2-WAY, or authentication type/keys disagree with the LAN peer [Source: https://networklessons.com/cisco/cisco-sd-wan/cisco-sd-wan-ospf-configuration].

EIGRP (Enhanced Interior Gateway Routing Protocol) is Cisco’s advanced distance-vector IGP. Like OSPF and BGP, when EIGRP is used on the service side of a Catalyst 8000 in SD-WAN mode it is configured within a service VPN/VRF, follows the same per-VPN scoping principle, and its routes are advertised into the overlay through OMP only when OMP is told to advertise them. The research material for this chapter is light on EIGRP specifics, so we will not invent CLI here — the key takeaway is structural: any service-side IGP, EIGRP included, lives inside its service VPN and reaches other sites only via OMP redistribution, exactly as OSPF does. In many branch designs OSPF or BGP is preferred precisely because the redistribution behavior with OMP is well documented; treat EIGRP as a valid but less commonly templated alternative and verify the exact syntax against your specific IOS-XE SD-WAN release before deploying.

Route redistribution with OMP

OMP (Overlay Management Protocol) is the SD-WAN overlay routing protocol that runs between WAN Edges and the vSmart controller. It is the “universal airport language” from our analogy. OMP advertises connected, static, OSPF, BGP (and EIGRP) routes from service VPNs into the overlay — but only for the route types you explicitly enable [Source: https://routing-guru.com/cisco-sd-wan-omp-protocol/].

! Global OMP parameters (simplified)
omp
 graceful-restart
 advertise bgp
 advertise ospf
 advertise connected
 advertise static
!
! Optionally per-VPN controls
vpn 10
 omp
  advertise connected
  advertise static
  advertise ospf
  advertise bgp
 exit
!

These advertise knobs are deliberately selective. If a LAN subnet is OSPF-learned but only advertise connected is enabled, OMP will never push that prefix into the overlay, and remote sites will not see the branch LAN — even though it appears in show ip route vpn 10. The fix is to enable advertise ospf specifically [Source: https://routing-guru.com/cisco-sd-wan-omp-protocol/].

Figure 6.1: Route redistribution between service-side protocols and OMP

flowchart LR
    LAN["Branch LAN switch<br/>OSPF area 0"] -->|OSPF routes| VRF["Service VPN 10 VRF<br/>(RIB/FIB)"]
    PE["MPLS PE / DC core<br/>AS 65000"] -->|BGP routes| VRF
    VRF -->|"advertise ospf / bgp"| OMP["OMP into overlay<br/>(to vSmart)"]
    OMP -->|"redistribute omp"| VRF
    OMP -->|vRoutes| REMOTE["Remote SD-WAN sites"]

The mirror-image problem is getting overlay routes back into the LAN. OMP routes are not automatically redistributed into local OSPF/BGP — you must explicitly configure redistribute omp per VPN [Source: https://journey2theccie.wordpress.com/2020/04/23/cisco-sd-wan-omp/]:

! Leak overlay (OMP) routes into OSPF in VPN 10
vpn 10
 router ospf 10
  redistribute omp metric 20 metric-type 1
 exit
!
! Leak overlay (OMP) routes into BGP in VPN 10 (if needed)
vpn 10
 router bgp 65010
  address-family ipv4 unicast
   redistribute omp
  exit-address-family
 exit
!

Here is where the airport translator must be disciplined. Do not blindly redistribute OMP into both OSPF and BGP at the same site without route tags and filters. Doing so can re-inject an overlay route back into the overlay through the other protocol, creating a routing loop [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/omp-redistribution-loop-prevention]. The recommended discipline is to pick a single “direction of trust” per site — for example OSPF ↔ OMP, but never OSPF ↔ BGP ↔ OMP all at once — and to tag redistributed routes so they cannot loop back in [Source: https://www.ciscolive.com/c/dam/r/ciscolive/apjc/docs/2024/pdf/BRKENS-2720.pdf]. A common, low-risk pattern is to redistribute OMP into only one protocol toward the LAN (often OSPF) and keep BGP restricted to the PE/DC side [Source: https://www.reddit.com/r/Cisco/comments/9aqs62/need_some_pointers_on_route_redistribution_and/].

Figure 6.2: Redistribution loop hazard versus a single direction of trust

flowchart TD
    subgraph LOOP["Hazard: OMP redistributed into both protocols"]
        OMP1["OMP overlay route"] --> OSPF1["OSPF (VPN 10)"]
        OMP1 --> BGP1["BGP (VPN 10)"]
        OSPF1 -->|"advertise ospf"| REINJECT["Re-injected into OMP"]
        BGP1 -->|"advertise bgp"| REINJECT
        REINJECT -->|routing loop| OMP1
    end
    subgraph SAFE["Discipline: one direction of trust"]
        OMP2["OMP overlay route"] -->|"redistribute omp (tagged)"| OSPF2["OSPF toward LAN"]
        BGP2["BGP toward PE/DC only"] -.->|"not redistributed back"| OMP2
    end

In production you generally do not type these commands directly. Instead you build vManage feature templates — a VPN template, an interface template, an OSPF template, a BGP template, and an OMP template — and attach them to a device template for the Catalyst 8000 [Source: https://networklessons.com/cisco/cisco-sd-wan/cisco-sd-wan-device-and-feature-templates]. The OSPF and BGP templates expose the same redistribution choices (including “Redistribute OMP” with metric/metric-type), and the OMP template exposes the per-VPN Advertise Connected/Static/OSPF/BGP checkboxes [Source: https://routing-guru.com/cisco-sd-wan-introduction-to-feature-templates/]. The resulting behavior is identical to the CLI; it is simply stored centrally and pushed consistently across many devices [Source: https://www.linkedin.com/pulse/sd-wan-templates-explained-device-feature-why-your-push-abusaa-p55lf].

Key Takeaway: On a Catalyst 8000 in SD-WAN mode, BGP, OSPF, and EIGRP all run inside service VPNs, never the global table. OMP carries routes between sites, but only the route types you explicitly advertise, and OMP routes reach the LAN only when you explicitly redistribute omp. Redistributing OMP into two protocols at one site invites loops — pick one direction of trust and use route tags.

Transport Options

Routing protocols decide what to advertise; transports decide how the bits physically move. This section is about the underlay — the world of VPN 0.

MPLS and internet underlays

The Catalyst 8000 separates two routing problems that beginners often conflate. Transport-side routing lives in VPN 0, the transport VPN, and answers the question: “How does this WAN Edge reach the internet/MPLS and the SD-WAN controllers so that tunnels can form?” [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/cisco-sd-wan-main-principles]. Service-side routing lives in the service VPNs and answers: “Once the tunnels exist, how do user VPNs at different sites reach each other?” [Source: https://www.certprepare.com/vpns-in-sd-wan-tutorial].

This maps cleanly onto two more terms. The underlay is the physical IP network that carries the SD-WAN tunnels — internet, MPLS, LTE. It uses ordinary IP routing (static, DHCP, IGP, BGP toward the ISP/PE) and has no awareness of SD-WAN VPNs or OMP; it sees only encrypted IP packets traveling between transport IP addresses [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/underlay-vs-overlay-routing]. The overlay is the logical fabric built on top, carrying OMP control traffic and segmented user data, and it is the only layer that understands “VPN 10 at Site A reaches VPN 10 at Site B via Internet or MPLS” [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/underlay-vs-overlay-routing]. In short: underlay ≈ transport-side (VPN 0); overlay ≈ service-side plus OMP control.

Figure 6.3: Underlay (transport-side, VPN 0) versus overlay (service-side plus OMP)

flowchart LR
    subgraph OVERLAY["Overlay (service-side + OMP)"]
        SVPN_A["Site A: VPN 10 users"] -. "OMP / segmented data" .- SVPN_B["Site B: VPN 10 users"]
    end
    subgraph UNDERLAY["Underlay (transport-side, VPN 0)"]
        EDGE_A["WAN Edge A<br/>VPN 0 transport IP"] -->|encrypted IPsec/GRE| NET["Internet / MPLS / LTE<br/>(plain IP routing)"]
        NET --> EDGE_B["WAN Edge B<br/>VPN 0 transport IP"]
    end
    SVPN_A --- EDGE_A
    SVPN_B --- EDGE_B

A common Catalyst 8000 branch pattern places each WAN circuit in VPN 0 [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/17-x/systems-interfaces/systems-interfaces-guide-17-x/vpn.html]:

• Gig0/0/0 in VPN 0 to the internet (color: public-internet), often getting its address via DHCP and a default route from the ISP.
• Gig0/0/1 in VPN 0 to the MPLS PE (color: mpls), often running BGP with the provider.

The router installs the internet default route and the BGP-learned MPLS prefixes in VPN 0 only. With that underlay reachability, it can now reach the controllers (vBond, vSmart, vManage) and the remote sites’ transport IPs, bring up DTLS/TLS control connections, and build IPsec/GRE data tunnels [Source: https://theworldsgonemad.net/2022/sdwan-vpn0-bgp/]. If VPN 0 routing is wrong — no default route, bad BGP, wrong next-hop — the tunnels and controllers never come up, and the service VPNs see no OMP routes even though the LAN itself looks fine [Source: https://theworldsgonemad.net/2022/sdwan-vpn0-bgp/].

The classic transport-side mistakes all stem from putting things in the wrong VPN: placing the ISP default route in a service VPN instead of VPN 0, forgetting NAT on an internet-facing VPN 0 interface, or using the same physical interface in both VPN 0 and a service VPN [Source: https://www.certprepare.com/vpns-in-sd-wan-tutorial]. Each of these can break or destabilize the control and data tunnels while LAN routing appears perfectly healthy — which is exactly why the underlay/overlay split is such a powerful troubleshooting lens.

Cellular and 5G connectivity

The underlay does not have to be wired. WAN transport options on the Catalyst 8000 include MPLS, public internet, and cellular/LTE/5G, each surfaced into the overlay as a TLOC with its own color [Source: https://www.cisco.com/site/us/en/products/networking/sdwan-routers/catalyst-8000-edge-platforms/index.html]. A cellular interface in VPN 0 behaves like any other underlay link: it obtains IP connectivity from the carrier and is assigned a color such as lte, so the overlay can build tunnels across it and prefer or avoid it via policy [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/cisco-sd-wan-main-principles].

This makes cellular/5G attractive both as a primary transport for sites where wired circuits are impractical, and as a resilient backup that comes online when MPLS or broadband fails. From the overlay’s perspective the cellular path is just another TLOC color competing for traffic — the segmentation and policy model is unchanged.

Loopback TLOCs and TLOC extension

A TLOC (Transport Location) is the unique identity of a WAN attachment point: the tuple of system-IP, color, and encapsulation (ipsec or gre) [Source: https://www.lookingpoint.com/blog/cisco-sd-wan-omp]. Every WAN-facing interface in VPN 0 that participates in SD-WAN becomes a TLOC, and only the TLOCs — not the raw VPN 0 routes — are advertised into the overlay [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/cisco-sd-wan-main-principles]. OMP’s overlay routes (vRoutes) point at TLOCs as their next-hops rather than at raw IP next-hops [Source: https://www.lookingpoint.com/blog/cisco-sd-wan-omp]. This is the airport analogy made concrete: other airports learn the gates (TLOCs) you can be reached through, not the internal taxiway map of your home country (the underlay routes).

In dual-router or dual-transport designs, TLOC extension lets one router “borrow” another router’s transport circuits over an intra-site link [Source: https://ciscolearningservices.my.site.com/cln/s/article/catalyst-sd-wan-dual-router-high-availability-with-ngfw]. The canonical scenario: Router A owns two underlay circuits (MPLS and Internet); Router B has only a LAN link to Router A. TLOC extension lets Router B’s overlay traffic exit through Router A’s TLOCs, so both routers share the transports without each needing its own physical circuit [Source: https://www.linkedin.com/pulse/cisco-sd-wan-lab-part-2-tloc-extensions-vrrp-nelson-paiva].

Figure 6.4: TLOC extension sharing transport circuits between two routers

flowchart LR
    INET["Internet<br/>(color: public-internet)"]
    MPLS["MPLS PE<br/>(color: mpls)"]
    RA["Router A<br/>owns both TLOCs (VPN 0)"]
    RB["Router B<br/>no direct circuits"]
    INET --- RA
    MPLS --- RA
    RB -->|"TLOC-extension link (VPN 0 only)"| RA
    RA -->|"borrowed TLOCs"| INET
    RA -->|"borrowed TLOCs"| MPLS

Conceptually:

vpn 0
 interface GigabitEthernet0/0/0
  ip address 198.51.100.2/30   ! underlay Internet
  tunnel-interface
   color public-internet
   encapsulation ipsec
  !
 !
 interface GigabitEthernet0/0/1
  ip address 192.0.2.2/30      ! link to Router B
  tloc-extension GigabitEthernet0/0/0
 !

The non-negotiable rule: the TLOC extension link must live in VPN 0 (the same VRF as the original TLOC) and must not carry service VPN user traffic [Source: https://www.cisco.com/c/en/us/support/docs/routers/sd-wan/221734-configure-and-troubleshoot-sd-wan-networ.html]. Placing a TLOC-extension interface into a user service VRF mixes transport and user data and creates hard-to-debug routing overlaps. Keep these links on dedicated VPN 0 subinterfaces or VLANs [Source: https://www.linkedin.com/pulse/cisco-sd-wan-lab-part-2-tloc-extensions-vrrp-nelson-paiva].

This split gives a sharp troubleshooting heuristic: if you see TLOC loss alarms, suspect a transport-side issue in VPN 0 (underlay reachability, or a color/tunnel misconfiguration). If TLOCs are up but applications are unreachable, the problem is on the service side — OMP, VPN IDs, or LAN routing [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/cisco-sd-wan-main-principles].

Key Takeaway: Transport-side routing (VPN 0, the underlay) brings up tunnels to controllers and remote TLOCs; service-side routing (service VPNs, the overlay) connects users across sites. Only TLOCs — system-IP + color + encapsulation — are advertised into the overlay; the underlay sees only encrypted packets. MPLS, internet, and cellular/5G are each just a TLOC color, and TLOC extension shares circuits between routers but must stay strictly inside VPN 0.

VRFs and Segmentation

The final piece is keeping different kinds of traffic separated end to end — corporate users, voice, IoT, and shared services should not bleed into one another by accident.

Service VPNs / VRFs

On IOS-XE SD-WAN, including the Catalyst 8000 and the virtual C8000V, each service VPN is literally implemented as a separate VRF in the data plane [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/17-x/systems-interfaces/systems-interfaces-guide-17-x/vpn.html]. vManage shows you “VPN 10,” but the device enforces it as a VRF:

vrf definition SDWAN-10
 rd 1:10
 !
 address-family ipv4
 exit-address-family

interface GigabitEthernet0/0/1
 vrf forwarding SDWAN-10
 ip address 10.10.10.1 255.255.255.0

So the practical equation is VPN = VRF. Each VRF has its own RIB and FIB; an interface belongs to exactly one VRF; and packets are switched only within the same VRF unless you explicitly configure inter-VRF routing [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/vpn-segmentation]. The VRF name is an IOS-XE artifact, but the VPN ID is what OMP and vSmart use to enforce segmentation across the fabric [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/17-x/systems-interfaces/systems-interfaces-guide-17-x/vpn.html]. A pleasant consequence: because the VRFs are genuinely separate tables, you can reuse overlapping IP ranges in different VPNs, since OMP tags every route with its VPN ID [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/vpn-segmentation]. When troubleshooting, lean on this equivalence — correlate show vrf, show ip route vrf <name>, and the SD-WAN VPN views together [Source: https://www.cisco.com/c/en/us/td/docs/routers/C8000V/Configuration/c8000v-installation-configuration-guide/m_configure_vrf_sharing.html].

Global VRF and transport VPN

Segmentation is enforced at two levels simultaneously. On each Catalyst 8000, the VRF level isolates the data plane: separate RIB/FIB per VRF, and no switching between VRFs without explicit configuration [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/vpn-segmentation]. Across the fabric, the VPN level enforces it through OMP: OMP advertises vRoutes tagged with a VPN ID, vSmart maintains a separate topology per VPN, and a route in VPN 10 never automatically appears in VPN 20 anywhere in the fabric [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/omp-overview].

It helps to lay the special VPNs alongside the service VPNs:

VPN 512 is reserved by default as the management VPN for out-of-band management — SSH, NETCONF, SNMP, vManage CLI access — and is segmented like any other VRF so management traffic is isolated from both transport and user traffic [Source: https://www.reddit.com/r/ccie/comments/g9sacp/sdwan_management_vpn_512/]. The discipline that keeps designs sane is keeping roles clean by VPN ID: VPN 0 carries transport only (no user LAN interfaces), VPN 512 carries management only (ideally truly out-of-band), and service VPNs carry user and tenant traffic with their LAN routing [Source: https://www.certprepare.com/vpns-in-sd-wan-tutorial]. Critically, segmentation exists only in the overlay — the underlay sees encrypted traffic with no segmentation context at all [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/underlay-vs-overlay-routing].

Inter-VRF routing

Because the default is total isolation, any communication between segments must be deliberately enabled through route leaking. There are three mechanisms, and choosing the right one depends on scope.

A worked example anchors the choice. Suppose you run VPN 10 (corporate users), VPN 20 (voice or OT/IoT), and VPN 30 (shared services such as DNS and Active Directory in the data center). By default, users in VPN 10 reach only other VPN 10 routes, voice reaches only VPN 20, and nobody reaches the DC services in VPN 30 [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/vpn-segmentation]. To let users and voice reach shared services, you leak selected prefixes from VPN 30 into VPNs 10 and 20.

Figure 6.5: VRF segmentation with selective route leaking to shared services

graph TD
    VPN10["VPN 10 VRF<br/>Corporate users"]
    VPN20["VPN 20 VRF<br/>Voice / OT-IoT"]
    VPN30["VPN 30 VRF<br/>Shared services (DNS, AD)"]
    VPN10 -. "isolated by default" .- VPN20
    VPN30 -->|"leak selected prefixes"| VPN10
    VPN30 -->|"leak selected prefixes"| VPN20
    VPN10 -.->|"no leak: stays isolated"| VPN20

1. Centralized route leaking via vSmart policy (recommended for fabric-wide consistency). You define source and destination VPNs, match routes from the source (using prefix lists, communities, or tags), and have vSmart clone those vRoutes into the destination VPN(s) by changing the VPN ID before advertising them [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/routing/ios-xe-17/routing-configuration-guide-17-x/route-leaking-between-vpns-ref/configure-route-leaking.html]. In our example, a host in VPN 10 then sees the DC services as routes in its own VRF, while the DC stays in VPN 30; packets traverse the fabric using the leaked OMP routes [Source: https://kbits.live/blog/sd-wan-route-leaking-between-vp-ns]. Use this when the inter-segment relationship must be consistent across many sites.

2. Route leaking between the global/transport VRF and service VPNs. Cisco documents controlled leaking between the global/VPN 0 context and service VPNs as a separate feature — useful when a service VPN needs reachability to something in the global/transport VRF, such as a shared NAT device or OOB management [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/routing/ios-xe-17/routing-configuration-guide-17-x/route-leaking-between-vpns-ref/configure-route-leaking.html]. You import selected prefixes into the service VPN and optionally export specific service-VPN routes back, configured centrally in SD-WAN Manager.

3. Local VRF Route Sharing on the Catalyst 8000 (device-local). The C8000V and Catalyst 8000 support sharing routes between VRFs through local import/export — for instance, between a “User” VRF and a local “Firewall” VRF for service chaining [Source: https://www.cisco.com/c/en/us/td/docs/routers/C8000V/Configuration/c8000v-installation-configuration-guide/m_configure_vrf_sharing.html]. This is device-local: it changes reachability on that one router but does not alter how vSmart/OMP sees the VPNs. Use it for site-local relationships such as steering a segment through a local security stack.

The guiding rule: for any segmentation that must hold across many sites, prefer vSmart route-leaking policies; reserve local VRF Route Sharing for relationships that are local to a single site [Source: https://www.cisco.com/c/en/us/td/docs/routers/C8000V/Configuration/c8000v-installation-configuration-guide/m_configure_vrf_sharing.html]. And in all cases, be conservative — use prefix lists and tags so you never accidentally leak a default route or an overly broad network between segments [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/vpn-segmentation].

Key Takeaway: Every service VPN on a Catalyst 8000 is a real IOS-XE VRF (VPN = VRF), giving each segment its own table, allowing overlapping IP ranges, and keeping VPN 10 routes out of VPN 20 by default. Inter-segment traffic requires deliberate route leaking — centralized vSmart policy for fabric-wide consistency, or local VRF Route Sharing for site-local service chaining — always scoped tightly with prefix lists and tags.

Chapter Summary

The Catalyst 8000 routes at two layers at once. On the service side, traditional protocols — BGP toward MPLS/DC peers, OSPF and EIGRP toward the LAN — run inside service VPNs, never the global table, each scoped to its own VRF. On the overlay, OMP carries prefixes between sites, but only the route types you explicitly advertise, and it pushes overlay routes back into the LAN only where you explicitly redistribute omp. The recurring hazard is redistribution looping: redistributing OMP into two protocols at the same site can re-inject overlay routes into the overlay, so you pick a single direction of trust and tag routes to prevent loops.

The transport side is the world of VPN 0 — the underlay — where static, DHCP, or BGP routing toward the ISP and MPLS PE brings up control connections to the controllers and IPsec/GRE data tunnels to other sites. Underlay maps to transport-side; overlay maps to service-side plus OMP. The fabric advertises only TLOCs (system-IP + color + encapsulation), never raw underlay routes, and MPLS, internet, and cellular/5G are simply different TLOC colors. TLOC extension lets routers share circuits but must stay strictly inside VPN 0. This split yields a sharp troubleshooting rule: TLOCs down means fix the underlay; TLOCs up but apps unreachable means fix the service side.

Finally, segmentation rests on the identity VPN = VRF: each service VPN is a genuine IOS-XE VRF with its own table, so segments are isolated by default and may even reuse IP ranges. Crossing between segments is always deliberate — centralized vSmart route-leaking for fabric-wide consistency, or local VRF Route Sharing for site-local service chaining — and should be done conservatively with prefix lists and tags. VPN 0 stays transport-only and VPN 512 stays management-only, keeping each role cleanly in its lane.

Key Terms

Chapter 7: Integrated Security and SASE

Learning Objectives

By the end of this chapter, you will be able to:

• Describe the integrated security features available on the Catalyst 8000 edge.
• Explain how the platform participates in a Cisco SASE architecture.
• Configure enterprise firewall, IPS, and secure internet access policies.

The Catalyst 8000 edge platforms are not just routers that happen to forward packets quickly. They are designed so that a single box at the WAN edge can also be the security checkpoint, the encryptor, and the on-ramp to a cloud-delivered security stack. Think of an airport: years ago you might have had separate buildings for ticketing, customs, baggage screening, and the departure gate. The modern WAN edge collapses all of those functions into one terminal. The Catalyst 8000 is that terminal — traffic that enters or leaves the branch passes through firewall inspection, intrusion prevention, malware analysis, encryption, and policy-based steering toward cloud security, all without leaving the device.

This chapter walks through three layers of that terminal: the security functions that run directly on the box, the connectivity features that protect traffic in transit, and the cloud integration that extends enforcement beyond the branch into Cisco’s SASE architecture.

On-Box Security

The first layer of protection happens inside the router itself. The Catalyst 8200 (small branch), 8300 (branch), and 8500 (aggregation and data center) all run Cisco Catalyst SD-WAN and ship with a full on-box security stack: zone-based enterprise firewall, Snort IPS, AMP/Cisco Malware Defense, TLS/SSL proxy, and URL filtering — all on a single WAN edge device. [Source: https://www.cisco.com/c/en/us/products/collateral/routers/cloud-edge/at-a-glance-c45-744087.pdf]

A crucial point about on-box security in SD-WAN mode: you rarely hand-write the firewall and IPS configuration on the device. Instead, you author policy centrally in Cisco SD-WAN Manager (vManage), and vManage translates that policy into the native IOS XE constructs — zones, zone-pairs, class-maps, and policy-maps — and pushes them to the router. The device CLI becomes your tool for verification and troubleshooting rather than authoring. [Source: https://www.cisco.com/c/en/us/td/docs/solutions/CVD/SDWAN/cisco-sdwan-security-policy-design-guide.html]

Zone-Based Firewall

A zone-based firewall (ZBFW) is a stateful firewall model where interfaces (or, in SD-WAN, VPN segments) are grouped into named zones, and policy is written between pairs of zones rather than per-interface. On the Catalyst 8000 this is delivered as the application-aware enterprise firewall, meaning it can match not only on Layer 3/4 criteria (source/destination IP, port, protocol) but also on Layer 7 application signatures, with built-in deep packet inspection (DPI). [Source: https://www.cisco.com/c/en/us/products/collateral/routers/cloud-edge/at-a-glance-c45-744087.pdf]

The mental model is a building with security doors between rooms. Each room is a zone (LAN, WAN, Data Center, the router’s own control plane). You never write a rule that says “this door”; you write a rule that says “traffic going from the lobby room to the vault room is allowed only if it is a verified employee.” That from-zone-to-zone rule is called a zone-pair, and its action is one of inspect (allow and statefully track), pass, or drop. [Source: https://www.cisco.com/c/en/us/support/docs/security/ios-firewall/98628-zone-design-guide.html]

Figure 7.1: Zone-based firewall zone-pair model

graph TD
    LAN["LAN Zone<br/>VPN 10 - Trusted inside"]
    WAN["WAN Zone<br/>VPN 0 - Untrusted outside"]
    DC["DC Zone<br/>VPN 20 - Semi-trusted services"]
    SELF["SELF Zone<br/>Router control plane / mgmt"]

    LAN -->|"Zone-pair: inspect HTTP/HTTPS/DNS"| WAN
    LAN -->|"Zone-pair: inspect"| DC
    WAN -.->|"No zone-pair = DROP by default"| LAN
    LAN -->|"Zone-pair: permit mgmt"| SELF

    classDef trusted fill:#1f6feb,stroke:#58a6ff,color:#fff
    classDef untrusted fill:#762020,stroke:#f85149,color:#fff
    classDef control fill:#3d2a6b,stroke:#a371f7,color:#fff
    class LAN,DC trusted
    class WAN untrusted
    class SELF control

A typical branch zone design looks like this:

The single most important rule of ZBFW behavior: any zone-pair you do not explicitly configure is denied by default once zones are attached. [Source: https://www.cisco.com/c/en/us/support/docs/routers/sd-wan/217758-cisco-sd-wan-zone-based-firewall-zbfw.html] This is the source of most “everything broke after I turned on the firewall” incidents. A second classic trap is the self-zone: if you do not write a policy permitting traffic to the router’s own control plane, you can lock yourself out of management. [Source: https://www.cisco.com/c/en/us/td/docs/solutions/CVD/SDWAN/cisco-sdwan-security-policy-design-guide.html]

Worked example — LAN-to-WAN with default-drop. Suppose a branch needs LAN users to reach the web and DNS, but nothing inbound from the Internet should reach the LAN. In vManage you create the LAN and WAN zones (mapping VPN 10 and VPN 0 respectively), then build the inter-zone rules:

• LAN to WAN: match applications HTTP, HTTPS, DNS (or ports 80, 443, 53) → action allow/inspect, with logging enabled.
• LAN to WAN default: match any → action drop.
• WAN to LAN default: match any → action drop. [Source: https://www.cisco.com/c/en/us/td/docs/solutions/CVD/SDWAN/cisco-sdwan-security-policy-design-guide.html]

After attaching the Enterprise Application-Aware Firewall policy to the branch site list and pushing it, vManage generates native ZBFW configuration on the router. You verify on the device with:

! See the security policy that was pushed
show sdwan running-config security

! See the generated zone-based firewall constructs
show running-config | section zone
show zone security

! Inspect active firewall sessions and drops in the data plane
show policy-map type inspect zone-pair sessions
show platform hardware qfp active feature firewall

If traffic between two VPNs suddenly fails after you attach zones, the cause is almost always a missing zone-pair policy (the default-drop behavior). If management access to the router itself breaks, check the self-zone policy. [Source: https://www.cisco.com/c/en/us/support/docs/routers/sd-wan/217758-cisco-sd-wan-zone-based-firewall-zbfw.html]

Snort IPS / IDS

Where the firewall decides whether a flow is allowed, an intrusion prevention system (IPS) decides whether an allowed flow is actually carrying an attack. The Catalyst 8000 integrates Snort IPS — the industry-standard Snort engine (Snort3) — as part of its multilayer security stack, available with the appropriate SD-WAN security licenses (for example, DNA Advantage or Premier). [Source: https://www.cisco.com/c/en/us/products/collateral/routers/cloud-edge/at-a-glance-c45-744087.pdf] [Source: https://www.networkworld.com/article/969558/cisco-bolsters-edge-networking-family-with-expanded-sd-wan-security-options.html]

If the firewall is the security guard who checks your badge, the IPS is the screening machine that opens your bag and looks for contraband even though your badge is valid. Snort inspects packet contents against a library of signatures — known exploit patterns, malware behaviors, web attacks — and either alerts on or blocks matches.

Figure 7.2: Packet inspection flow through ZBFW, Snort IPS, and AMP

flowchart TD
    PKT["Inbound flow<br/>(LAN to WAN zone-pair)"] --> ZBFW{"ZBFW: zone-pair<br/>match L3/L4 + L7 app?"}
    ZBFW -->|"No matching rule"| DROP1["Drop<br/>(default-deny)"]
    ZBFW -->|"Allowed / inspect"| IPS{"Snort IPS<br/>signature match?"}
    IPS -->|"Match - Prevent mode"| DROP2["Block + alert"]
    IPS -->|"Match - Detect-only"| ALERT["Alert, allow"]
    IPS -->|"Clean"| AMP{"AMP / Malware Defense<br/>file reputation?"}
    AMP -->|"Malicious file"| DROP3["Block download"]
    AMP -->|"Clean"| URL{"URL filtering<br/>category allowed?"}
    URL -->|"Blocked category"| DROP4["Block / warn"]
    URL -->|"Permitted"| FWD["Forward to destination"]

    classDef deny fill:#762020,stroke:#f85149,color:#fff
    classDef allow fill:#1f6feb,stroke:#58a6ff,color:#fff
    class DROP1,DROP2,DROP3,DROP4 deny
    class FWD,ALERT allow

On IOS XE, Snort runs inside the UTD (Unified Threat Defense) engine, a container/virtual-service installed on the router. [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/security/ios-xe-17/security-book-xe/intrusion-prevention.html] The IPS supports two operating modes and a choice of base rulesets:

Worked example — adding Snort IPS to the branch firewall. Building on the previous firewall example, the workflow in vManage is:

1. Prepare the engine and signatures. Upload the UTD Snort IPS image and signature packages to the vManage software repository so they can be delivered to the device. [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/security/ios-xe-17/security-book-xe/intrusion-prevention.html]
2. Create an IPS profile. Under Configuration → Security → Intrusion Prevention, create a profile named, say, Branch-IPS-Balanced — mode Prevent for high-risk signatures, base policy Balanced, with categories such as Malware and Exploits enabled. [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/security/ios-xe-17/security-book-xe/intrusion-prevention.html]
3. Bind IPS to traffic. Reference that profile on the LAN-to-WAN zone-pair so that flows the firewall allows are then handed to Snort for inspection. [Source: https://www.cisco.com/c/en/us/td/docs/solutions/CVD/SDWAN/cisco-sdwan-security-policy-design-guide.html]
4. Attach and push the combined firewall+IPS policy to the same branch site list. vManage ensures the UTD container is installed and running, then pushes the rule actions. [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/intrusion-prevention]

Verify the engine on the device:

! Confirm the UTD/Snort engine is installed and running
show platform software utd status
show utd engine standard status
show utd engine standard all

! Inspection counters and active sessions
show utd statistics
show utd active-sessions

Three diagnostic patterns are worth memorizing. If utd status shows not installed or not running, the container image failed to download — check the repository and device disk/memory. If you see license errors when UTD starts, the security/IPS feature license is missing. And if the IPS counters stay at zero despite traffic, the IPS is not bound to the right zone-pair, or traffic is not matching the firewall rules that invoke it. [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/security/ios-xe-17/security-book-xe/intrusion-prevention.html]

URL Filtering and AMP

Two more on-box capabilities round out content security. URL filtering lets the router enforce policy on web destinations by category — blocking, allowing, or warning — as part of the same application-aware firewall and IPS stack. [Source: https://www.cisco.com/c/en/us/products/collateral/routers/cloud-edge/at-a-glance-c45-744084.html] AMP (Advanced Malware Protection), branded Cisco Malware Defense on these platforms, performs file and malware analysis at the WAN edge, so a malicious file download can be identified and blocked before it ever reaches a branch endpoint. [Source: https://www.cisco.com/c/en/us/products/collateral/routers/cloud-edge/at-a-glance-c45-744087.pdf]

If IPS is the bag scanner looking for known attack patterns in network traffic, AMP is the lab that takes a suspicious file and analyzes it — including checking its reputation and detonating unknowns — to determine whether it is malware. Both URL filtering and AMP are advanced functions that require higher-tier DNA/SD-WAN security licenses. [Source: https://www.networkworld.com/article/969558/cisco-bolsters-edge-networking-family-with-expanded-sd-wan-security-options.html]

Key Takeaway: The Catalyst 8000 delivers a full on-box security stack — zone-based application-aware firewall, Snort IPS via the UTD engine, URL filtering, and AMP/Malware Defense. In SD-WAN mode you author policy centrally in vManage and use the device CLI to verify; remember that any unconfigured zone-pair drops by default and the self-zone must be explicitly permitted.

Secure Connectivity

On-box inspection protects traffic at the branch. The next layer protects traffic as it moves — encryption in flight, the ability to inspect encrypted traffic, and a lightweight security control at the DNS layer.

IPsec and Crypto Offload

Every SD-WAN tunnel in the Catalyst SD-WAN fabric is an encrypted IPsec tunnel, and the Catalyst 8000 platforms are built to encrypt and decrypt at line rate. The heavy mathematical work of encryption is handled by dedicated crypto offload hardware rather than the general-purpose CPU. The analogy is a dedicated mail-room that seals and unseals every envelope, so the executives (the CPU doing routing and policy) are not slowed down stuffing envelopes themselves. This matters enormously when the same box is also building IPsec tunnels to cloud security gateways — a topic covered in the next section — because those SIG tunnels are themselves IPsec, and the router may maintain several of them per WAN transport. [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/security/ios-xe-17/security-book-xe/m-secure-internet-gateway.html]

A practical design note: IPsec tunnels depend on a healthy underlay. Public-IP reachability, correct MTU, and reasonable NAT timeouts all matter. When those degrade, the symptom is intermittent tunnel flaps and degraded application performance rather than an obvious hard failure. [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/security/ios-xe-17/security-book-xe/m-secure-internet-gateway.html]

TLS/SSL Proxy

A growing problem for any inspection engine is that the overwhelming majority of web traffic is now encrypted with TLS. A firewall or IPS that can only read the outside of the envelope cannot see the threat inside. The TLS/SSL proxy solves this by terminating the encrypted session on the router, decrypting the traffic for inspection by the firewall, IPS, AMP, and URL filtering engines, and then re-encrypting it before forwarding — a controlled man-in-the-middle performed by the trusted edge device. [Source: https://www.cisco.com/c/en/us/products/collateral/routers/cloud-edge/at-a-glance-c45-744084.html]

Figure 7.3: TLS/SSL proxy decrypt-inspect-re-encrypt flow

flowchart LR
    CLIENT["Branch client<br/>(encrypted TLS)"] --> TERM["TLS proxy<br/>terminate + decrypt"]
    TERM --> INSPECT["On-box engines inspect<br/>ZBFW / IPS / AMP / URL filter"]
    INSPECT --> REENC["Re-encrypt<br/>(new TLS session)"]
    REENC --> SERVER["Internet / SaaS server"]

    classDef proxy fill:#3d2a6b,stroke:#a371f7,color:#fff
    classDef edge fill:#1f6feb,stroke:#58a6ff,color:#fff
    class TERM,REENC proxy
    class INSPECT edge

This is the customs officer who is authorized to open sealed diplomatic pouches, inspect the contents, and reseal them with an official seal before they continue. Because it is computationally expensive and security-sensitive, TLS/SSL proxy is an advanced function gated behind higher-tier DNA licenses on the Catalyst 8000 SD-WAN routers. [Source: https://www.cisco.com/c/en/us/products/collateral/routers/cloud-edge/at-a-glance-c45-744084.html]

DNS-Layer Security (Umbrella)

The cheapest, earliest place to stop a threat is at the moment a device asks “what is the IP address for this domain?” Cisco Umbrella provides DNS-layer security: DNS requests from the branch are resolved through Umbrella, which can block resolution of domains associated with malware, phishing, or command-and-control before any connection is ever attempted. [Source: https://www.cisco.com/c/en/us/solutions/collateral/enterprise-networks/sd-wan/sd-wan-sse-integrations-aag.html]

DNS-layer security is attractive because it requires no IPsec tunnels — it is a natural first phase of a SASE rollout. Many organizations begin with Umbrella DNS integrated into SD-WAN and later migrate to full SIG tunnels. [Source: https://umbrella.cisco.com/blog/extending-sd-wan-segmentation-to-umbrella-cloud-security] One caution worth flagging now: DNS policies and full Secure Web Gateway (SWG) policies can be configured independently, and inconsistent settings cause confusing results — for example, DNS allows a domain but the SWG later blocks the HTTP request to it. [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/security/ios-xe-17/security-book-xe/m-secure-internet-gateway.html]

Key Takeaway: Secure connectivity on the Catalyst 8000 combines hardware crypto offload for line-rate IPsec, a TLS/SSL proxy that lets on-box engines inspect encrypted traffic, and Umbrella DNS-layer security as a tunnel-free first line of defense and natural entry point into a SASE rollout.

SASE Integration

The third and outermost layer extends enforcement beyond the branch into the cloud. SASE (Secure Access Service Edge) is an architecture that converges WAN-edge functions — SD-WAN and routing — with cloud-delivered security services such as Secure Web Gateway (SWG), Cloud Access Security Broker (CASB), Zero Trust Network Access (ZTNA), and Firewall-as-a-Service (FWaaS) into a single, identity-aware service model. [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/what-is-sase]

Cisco’s SASE architecture has three pillars: Catalyst SD-WAN at the WAN edge (the Catalyst 8000 platforms), Cisco cloud security in the middle (Umbrella SIG / Cisco Secure Edge / Cisco Secure Access SSE), and identity and policy components such as Cisco ISE. [Source: https://www.prnewswire.com/news-releases/cisco-unveils-new-expanded-sase-architecture-delivering-complete-protection-from-endpoint-to-the-cloud-301258745.html] The result is an “identity-first” model: regardless of whether a user connects through the branch SD-WAN edge or via remote access, their traffic is consistently inspected and controlled in the cloud security layer before reaching the Internet or SaaS. The end-to-end path is:

User → Branch LAN → Catalyst SD-WAN edge → SIG IPsec tunnel → Umbrella POP → Internet/SaaS — all under SASE policies. [Source: https://www.wwt.com/video/umbrella-sig-integration-with-cisco-sd-wan]

Figure 7.4: Cisco SASE three-pillar architecture and end-to-end path

flowchart LR
    subgraph EDGE["Pillar 1: WAN Edge"]
        USER["User / Branch LAN"] --> C8K["Catalyst 8000<br/>SD-WAN edge"]
    end
    subgraph CLOUD["Pillar 2: Cloud Security"]
        POP["Umbrella SIG /<br/>Cisco Secure Access SSE<br/>POP"]
    end
    subgraph ID["Pillar 3: Identity"]
        ISE["Cisco ISE<br/>identity + policy"]
    end

    C8K -->|"SIG IPsec tunnel"| POP
    ISE -.->|"identity-first policy"| POP
    POP --> NET["Internet / SaaS"]

    classDef edge fill:#1f6feb,stroke:#58a6ff,color:#fff
    classDef cloud fill:#3d2a6b,stroke:#a371f7,color:#fff
    classDef ident fill:#1a5c3a,stroke:#3fb950,color:#fff
    class USER,C8K edge
    class POP cloud
    class ISE ident

Secure Internet Gateway (SIG) Tunnels

A Secure Internet Gateway (SIG) is a cloud-delivered security stack — Cisco Umbrella SIG, also called Cisco Secure Edge — deployed as a globally distributed set of security points of presence (POPs). It typically includes DNS-layer security, an SWG for URL/category filtering and web threat protection, a cloud-delivered firewall (FWaaS) for L3–L7 policy, and CASB-type controls, depending on license. [Source: https://www.cisco.com/c/en/us/solutions/collateral/enterprise-networks/sd-wan/sd-wan-sse-integrations-aag.html]

The connection between the Catalyst 8000 and the SIG is an automatically built IPsec tunnel. The branch edge sends Internet-bound traffic into that tunnel so every flow is inspected in the cloud before exiting to the Internet. The genuinely valuable part of the Catalyst SD-WAN approach is that you do not build these tunnels by hand. SD-WAN Manager (vManage) provides built-in SIG templates that automate the entire process. [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/security/ios-xe-17/security-book-xe/m-secure-internet-gateway.html]

Worked example — full Internet breakout via Umbrella SIG. Consider a branch with dual Internet circuits that should send all Internet traffic through Umbrella for inspection. The high-level workflow is:

1. Define the SIG provider profile in vManage — provider type (Umbrella/Secure Edge, or a third party), region/POP list, and API credentials so vManage can register the devices with the cloud service automatically. [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/security/ios-xe-17/security-book-xe/m-secure-internet-gateway.html]
2. Configure a SIG policy / configuration group that defines which edges use SIG, which WAN transports may carry SIG tunnels, and how SD-WAN VPNs map to SIG POPs. [Source: https://www.cisco.com/c/en/us/solutions/collateral/enterprise-networks/sd-wan/sd-wan-sse-integrations-aag.html]
3. Attach the SIG configuration to the device template. When pushed, each edge automatically initiates IKE/IPsec to its nearest POPs — commonly one tunnel per circuit (broadband and LTE each get a tunnel) for redundancy and load distribution. [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/security/ios-xe-17/security-book-xe/m-secure-internet-gateway.html]
4. Define data/centralized policies to steer traffic — for example, a default route (0.0.0.0/0) pointing into the SIG, with app-aware routing choosing which circuit-and-POP tunnel to use based on performance. [Source: https://www.cisco.com/c/en/us/solutions/collateral/enterprise-networks/sd-wan/nb-06-cloud-onramp-saas-aag-cte-en.html]

This automation generates the IKE/IPsec parameters and crypto maps and provisions redundancy automatically, avoiding manual per-router IPsec work. [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/security/ios-xe-17/security-book-xe/m-secure-internet-gateway.html]

A powerful refinement is segmentation preservation. SD-WAN VPNs (Corp, Guest, IoT) can map to separate Umbrella identities and policies, so isolation is maintained in the cloud — guest Wi-Fi can be forced through SIG-only with stricter category filtering while corporate traffic gets its own policy. [Source: https://umbrella.cisco.com/blog/extending-sd-wan-segmentation-to-umbrella-cloud-security]

The same SIG template framework also supports third-party SSE providers — Zscaler, Palo Alto Prisma Access, and Netskope — so the automation model is not locked to Umbrella. [Source: https://docs.paloaltonetworks.com/prisma-access/integration/integrate-third-party-sd-wans-with-prisma-access/cisco-catalyst-sd-wan-solution-guide/integrate-prisma-access-with-cisco-catalyst-sd-wan]

Cloud OnRamp for SaaS and IaaS

Sending everything through a cloud gateway is simple but not always optimal — backhauling a low-latency SaaS app through a distant POP can hurt user experience. Cloud OnRamp is the Catalyst SD-WAN framework that optimizes and secures access to cloud services. Cloud OnRamp for SaaS continuously measures performance to applications such as Microsoft 365 and Salesforce across multiple paths and selects the best one. Cloud OnRamp for IaaS extends the SD-WAN fabric into public cloud (AWS, Azure) so branch-to-cloud traffic rides an optimized, secured path. [Source: https://www.cisco.com/c/en/us/solutions/collateral/enterprise-networks/sd-wan/nb-06-cloud-onramp-saas-aag-cte-en.html]

The decision logic is what makes this elegant. For each SaaS application the edge probes three candidate paths — direct Internet breakout, data-center backhaul, and SIG tunnels — and the controller ranks them by latency, loss, jitter, and HTTP response time. The best-performing path wins. [Source: https://www.dclessons.com/cloud-onramp-for-saas-over-sig-tunnels]

Figure 7.5: Cloud OnRamp for SaaS path probing and selection

flowchart TD
    APP["SaaS app traffic<br/>(e.g. Microsoft 365)"] --> PROBE["Edge probes candidate paths<br/>latency / loss / jitter / HTTP response"]
    PROBE --> P1["Direct Internet breakout<br/>local on-box inspection"]
    PROBE --> P2["Data-center backhaul<br/>DC security stack"]
    PROBE --> P3["SIG tunnel<br/>full Umbrella SIG"]
    P1 --> RANK{"Controller ranks paths"}
    P2 --> RANK
    P3 --> RANK
    RANK -->|"Best-performing path wins"| BEST["Selected path to SaaS"]

    classDef path fill:#1f6feb,stroke:#58a6ff,color:#fff
    classDef best fill:#1a5c3a,stroke:#3fb950,color:#fff
    class P1,P2,P3 path
    class BEST best

When Cloud OnRamp for SaaS over SIG tunnels is enabled, the probes run across the SIG tunnels too, so the “best path” might be a specific Umbrella POP reached over a specific circuit. This delivers optimization and security simultaneously — traffic is inspected by the SIG while still following the lowest-latency route. [Source: https://www.dclessons.com/cloud-onramp-for-saas-over-sig-tunnels]

A common production pattern: trusted, well-behaved apps such as Microsoft 365 take the direct path when it tests best and policy allows, while unknown apps and risky categories are sent to the SIG for full inspection. [Source: https://www.dclessons.com/cloud-onramp-for-saas-over-sig-tunnels]

Cisco Secure Access Integration

The newest pillar of Cisco’s SASE architecture is Cisco Secure Access, Cisco’s Security Service Edge (SSE) offering that unifies the cloud security functions — SWG, CASB, FWaaS, and ZTNA — behind a single identity-aware policy plane, alongside identity sources such as Cisco ISE. [Source: https://www.prnewswire.com/news-releases/cisco-unveils-new-expanded-sase-architecture-delivering-complete-protection-from-endpoint-to-the-cloud-301258745.html] From the Catalyst 8000’s perspective the integration mechanics are familiar: the SD-WAN edge builds SIG/SSE tunnels and steers traffic through them exactly as it does for Umbrella, using the same vManage SIG template framework. [Source: https://www.cisco.com/c/en/us/solutions/collateral/enterprise-networks/sd-wan/sd-wan-sse-integrations-aag.html]

The difference is where enforcement decisions are made. Once traffic reaches the SSE, user identity, device posture, and application context drive the SWG, firewall, and CASB policies — completing the identity-first SASE story in which the same user gets the same policy whether they are at the branch behind a Catalyst 8000 or working remotely. [Source: https://www.prnewswire.com/news-releases/cisco-unveils-new-expanded-sase-architecture-delivering-complete-protection-from-endpoint-to-the-cloud-301258745.html]

This raises the key design question of any SASE deployment: which enforcement lives where? L3/L4 filtering can happen at the SIG or at the local on-box firewall; application-aware routing lives in SD-WAN while web app controls live in the SWG. Deciding this deliberately avoids duplicate or conflicting rules between the on-box and cloud layers. [Source: https://www.cisco.com/c/en/us/solutions/collateral/enterprise-networks/sd-wan/sd-wan-sse-integrations-aag.html]

Key Takeaway: In a Cisco SASE architecture the Catalyst 8000 is the WAN-edge pillar that builds automated IPsec SIG tunnels (Umbrella, Secure Access, or third-party SSE) via vManage templates, preserves VPN segmentation as cloud identities, and uses Cloud OnRamp to send each application over the best-performing secure path — while identity-aware policy is enforced in the cloud security layer.

Chapter Summary

The Catalyst 8000 edge platforms turn a single WAN-edge device into a complete security checkpoint across three layers. On-box, they run a zone-based application-aware enterprise firewall, Snort IPS via the UTD engine, URL filtering, and AMP/Cisco Malware Defense — authored centrally in SD-WAN Manager (vManage), which translates policy into native IOS XE zones and zone-pairs and pushes it to the device, where the CLI is used for verification. The two behaviors most likely to cause outages are ZBFW’s default-drop on unconfigured zone-pairs and the need to explicitly permit the self-zone for management.

For secure connectivity, hardware crypto offload allows line-rate IPsec, the TLS/SSL proxy decrypts traffic so the on-box engines can inspect it, and Umbrella DNS-layer security offers tunnel-free protection that is a natural first phase of a SASE rollout. Advanced functions — TLS/SSL proxy, AMP, URL filtering — require higher-tier DNA/SD-WAN security licenses.

For SASE integration, the Catalyst 8000 acts as the WAN-edge pillar of Cisco’s identity-first architecture. vManage SIG templates automate IPsec tunnel creation to Umbrella SIG, Cisco Secure Access SSE, or third-party providers; SD-WAN VPN segmentation maps to cloud identities to preserve isolation; and Cloud OnRamp for SaaS and IaaS probes direct, data-center, and SIG paths to steer each application over the best-performing secure route. The defining design decision in any such deployment is choosing which enforcement lives on the box versus in the cloud to avoid conflicting policy.

Key Terms

Chapter 8: Application-Aware Routing and Quality of Experience

In the early days of the wide-area network (WAN), a router’s job was simple: get the packet to its destination. Any path that reached the far side was, by definition, “good enough.” But the modern branch office sends a very different mix of traffic across its links — a voice-over-IP call that breaks up after 150 milliseconds of latency, a Microsoft Teams video stream that stutters with even mild jitter, and a nightly backup that happily tolerates packet loss as long as it eventually completes. Treating all of these flows identically, and choosing transport paths based only on reachability, produces a poor quality of experience for the users who matter most.

This chapter covers the three mechanisms Cisco Catalyst 8000 edge platforms use to deliver an application-tuned WAN. Application-Aware Routing (AAR) continuously measures each transport path and steers each application onto the tunnel that best meets its performance requirements. Quality of Service (QoS) manages congestion on those paths so that critical traffic is served first. And application visibility features — built on NBAR2 deep packet inspection and extended by Cloud OnRamp for SaaS — let the router identify exactly what is flowing across the link and optimize cloud-bound traffic accordingly.

Learning Objectives

By the end of this chapter, you will be able to:

• Explain application-aware routing (AAR) and SLA-based path selection, including how BFD probes measure path quality.
• Configure QoS policies for WAN traffic on Catalyst 8000 platforms, including per-tunnel, per-VPN, and adaptive QoS.
• Describe Cloud OnRamp and application visibility features built on NBAR2 deep packet inspection.

Application-Aware Routing

Routing protocols answer the question “Can I reach the destination?” Application-Aware Routing answers a harder one: “Of all the ways I can reach the destination, which one will my users actually enjoy using right now?” In Cisco Catalyst SD-WAN, AAR uses real-time path performance — loss, latency, and jitter measured by BFD — combined with SLA classes defined in an app-route policy to dynamically choose the best data-plane tunnel (IPsec or GRE) for each application flow [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/policies/ios-xe-17/policies-book-xe/application-aware-routing.html].

A useful analogy is a navigation app like Waze. The underlying road map (your routing protocol) tells you which roads physically connect your house to the office. But Waze layers live traffic data on top, and reroutes you onto the freeway, the side streets, or the toll road depending on which is moving fastest at this exact moment. AAR is the WAN equivalent: OMP, BGP, and OSPF still decide reachability, but AAR overrides the default tunnel choice based on the live “traffic conditions” of each path [Source: https://blogs.cisco.com/learning/understanding-application-aware-routing-aar-in-cisco-sd-wan].

The AAR solution rests on three pillars:

One subtlety worth remembering: AAR is only evaluated after the overlay route has been chosen and installed in the routing table. If OMP tie-breaking never installs the route, the app-route policy is never invoked for that destination — a common troubleshooting trap [Source: https://www.reddit.com/r/networking/comments/uwuhkd/cisco_sdwan_application_aware_routing_order_of/].

SLA Classes and Probes (BFD)

An SLA class (Service Level Agreement class) is a reusable object that defines the maximum tolerable loss, latency, and jitter for a category of traffic [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/configuring-application-aware-routing-policies]. It has three fields:

• loss-percentage — maximum acceptable packet loss, as a percentage.
• latency — maximum one-way delay, in milliseconds.
• jitter — maximum variation in delay, in milliseconds.

Think of an SLA class as the bouncer’s checklist at an exclusive club. To get in, a tunnel must satisfy every item on the list. Consider two classes:

• voice-gold: loss ≤ 1%, latency ≤ 100 ms, jitter ≤ 30 ms
• email-bronze: loss ≤ 5%, latency ≤ 300 ms, jitter ≤ 100 ms

A tunnel whose measured loss, latency, and jitter all fall at or below the configured thresholds is considered SLA-compliant for that class. Exceed any single threshold and the tunnel is marked SLA-violating and dropped from the candidate set for that class (unless policy specifies a fallback) [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/policies/ios-xe-17/policies-book-xe/application-aware-routing.html]. Because SLA classes are reusable objects, a single “voice-gold” definition can be referenced by many application rules, simplifying design.

A practical word of caution: set thresholds based on real WAN behavior. Overly strict SLAs cause every tunnel to be flagged “bad” most of the time, defeating the purpose; overly loose SLAs never trigger a reroute even when users suffer.

So where do the loss, latency, and jitter numbers come from? BFD (Bidirectional Forwarding Detection) sessions run over each data-plane tunnel between edge routers. BFD performs two jobs: it detects tunnel liveliness (fast failure detection) and it measures performance metrics — packet loss, latency, and jitter — recording them as PfR (Performance Routing) data per tunnel [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/policies/ios-xe-17/policies-book-xe/m-enhanced-application-aware-routing.html].

The timing details matter for understanding why standard AAR can feel sluggish:

• BFD collects statistics over a poll interval, which defaults to 10 minutes for app-route/SLA purposes.
• During each interval it computes averages for loss, latency, and jitter.
• A multiplier (default 6) means SLA decisions look across several poll intervals’ worth of weighted averages, smoothing out transient spikes [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/policies/ios-xe-17/policies-book-xe/m-enhanced-application-aware-routing.html].

This averaging makes standard AAR deliberately conservative. A 30-second burst of packet loss may be diluted into a 10-minute average that never breaches the SLA, so the router never reroutes. That conservatism prevents needless path-flapping, but it can let real-time traffic suffer through short-lived problems.

Figure 8.1: BFD vs. EAAR measurement loops feeding SLA evaluation

flowchart LR
    subgraph Standard["Standard AAR (BFD probe loop)"]
        direction TB
        A1["BFD probes sent over tunnel"] --> A2["Collect loss / latency / jitter<br/>over 10-min poll interval"]
        A2 --> A3["Compute weighted average<br/>(multiplier x6)"]
        A3 --> A4["Update per-tunnel PfR metrics"]
        A4 --> A1
    end
    subgraph Enhanced["Enhanced AAR (inline data-plane)"]
        direction TB
        B1["Inspect live application packets"] --> B2["Derive loss / latency / jitter inline"]
        B2 --> B3["Apply dampening<br/>(prevent flapping)"]
        B3 --> B4["Report metrics back via BFD"]
        B4 --> B1
    end
    A4 --> SLA{"Tunnel meets<br/>SLA class?"}
    B4 --> SLA
    SLA -->|Faster reaction via EAAR| OUT["SLA-compliant / SLA-violating state"]

Enhanced Application-Aware Routing (EAAR), introduced around release 20.12, addresses exactly this. Instead of relying solely on BFD probe averages gathered over the control plane, EAAR derives performance metrics from inline data-plane packets — the live application traffic itself — and reports the results back via BFD [Source: https://lostintransit.se/2024/02/19/catalyst-sd-wan-enhanced-application-aware-routing/]. The payoff is faster detection of a degrading tunnel and quicker reroute. EAAR also includes a dampening mechanism to prevent rapid flapping when metrics hover near a threshold. Two operational notes: EAAR is disabled by default, and it must be enabled on both the local and remote edges of a tunnel to function correctly [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/policies/ios-xe-17/policies-book-xe/m-enhanced-application-aware-routing.html].

Key Takeaway: SLA classes define the loss/latency/jitter thresholds an application demands; BFD continuously measures those metrics on every tunnel over a 10-minute averaging window, while EAAR uses live data-plane packets for much faster reaction.

App-Route Policies

The rules that tie applications to SLA classes live in a centralized app-route policy, built in vManage (Cisco SD-WAN Manager) and pushed to edge routers by vSmart [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/policies/ios-xe-17/policies-book-xe/application-aware-routing.html]. This is a dedicated policy type, distinct from security or basic data policies. Its building blocks are:

The policy is organized as match → action sequences. The match section selects traffic (by VPN, prefix, or application); the action section sets the SLA class to evaluate, plus optional preferred-color/TLOC preferences and fallback behavior [Source: https://www.youtube.com/watch?v=atuoyIVmbNM]. A simple two-sequence policy:

• Sequence 1: match the voice subnet → action: use SLA-class voice-gold.
• Sequence 2: match the email subnet → action: use SLA-class email-bronze.
• Default action: leave all other traffic unchanged.

The app-probe-class field deserves emphasis. WAN paths can behave very differently per QoS queue — the priority queue carrying voice may show 20 ms latency while the best-effort queue shows 200 ms. By sending BFD probes with a specific DSCP (say, EF for voice), AAR measures the path quality of the actual queue your voice traffic will use, rather than generic best-effort, making the resulting decisions far more accurate [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/configuring-application-aware-routing-policies].

Key Takeaway: An app-route policy is a centralized match/action policy that maps applications (by VPN, prefix, or NBAR2 app-list) to SLA classes and color preferences; the app-probe-class lets you probe a specific DSCP so measurements reflect the real QoS queue the traffic will traverse.

Path Preference and Failover

With SLA classes defined and the app-route policy attached, here is how the edge selects a tunnel for a given flow, assuming the overlay route is installed, BFD is running, and multiple TLOCs (e.g., MPLS and Internet colors) exist to the destination:

1. Identify traffic and SLA — match the flow against the app-route policy and determine the applicable SLA class.
2. Gather metrics — retrieve current loss/latency/jitter from BFD (and EAAR if enabled) for each candidate tunnel.
3. Filter by SLA compliance — discard any tunnel where loss, latency, or jitter exceeds the threshold; what remains is the SLA-compliant set.
4. Apply preferences — narrow to tunnels matching the configured preferred-color or TLOC list; if several qualify, choose the best by performance.
5. Forward — install a forwarding entry steering the application over the chosen tunnel, overriding the default selection.
6. React to change — if a tunnel later violates the SLA, mark it bad, remove it from the candidate set, and reroute to a still-compliant tunnel. EAAR makes this reroute much faster.
7. If no tunnel meets the SLA — depending on configuration, use the best-available tunnel (lowest loss/latency even if non-compliant, with a logged violation), fall back to default routing, or stay on the current path to avoid flapping.

Figure 8.2: AAR path-selection decision flow against the SLA

flowchart TD
    START["New application flow arrives"] --> MATCH["Match flow against<br/>app-route policy"]
    MATCH --> SLA["Determine applicable SLA class"]
    SLA --> METRICS["Gather loss / latency / jitter<br/>from BFD + EAAR per tunnel"]
    METRICS --> FILTER["Filter: discard tunnels that<br/>exceed any SLA threshold"]
    FILTER --> ANY{"Any SLA-compliant<br/>tunnel remaining?"}
    ANY -->|Yes| PREF["Apply preferred color / TLOC list"]
    PREF --> MULTI{"Multiple candidates<br/>qualify?"}
    MULTI -->|Yes| BEST["Choose best by performance"]
    MULTI -->|No| BEST
    BEST --> FWD["Install forwarding entry<br/>over chosen tunnel"]
    ANY -->|No| FALLBACK["Fallback action:<br/>best-available, default routing,<br/>or stay to avoid flapping"]
    FALLBACK --> FWD
    FWD --> WATCH{"Chosen tunnel<br/>later violates SLA?"}
    WATCH -->|Yes - EAAR reroutes fast| METRICS
    WATCH -->|No| WATCH

Worked example. A branch connects to the data center over two transports: MPLS (color mpls) and Internet (color biz-internet). The design defines:

• voice-gold: loss ≤ 1%, latency ≤ 100 ms, jitter ≤ 30 ms
• data-silver: loss ≤ 3%, latency ≤ 200 ms, jitter ≤ 50 ms

The app-route policy says: voice subnets (VPN 10) use voice-gold with preferred color mpls; general data uses data-silver with preferred color biz-internet.

At runtime, BFD reports:

The result: voice traffic sees that only MPLS meets voice-gold (and MPLS is preferred anyway), so all voice flows take MPLS. Data traffic finds both tunnels meet data-silver, but policy prefers biz-internet, so data uses the Internet path — conveniently keeping MPLS bandwidth free for voice [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/policies/ios-xe-17/policies-book-xe/application-aware-routing.html].

Now suppose MPLS suddenly degrades to 5% loss. BFD detects the higher loss and AAR marks MPLS as no longer meeting voice-gold. If Internet still satisfies the voice thresholds and policy permits that color for voice, the voice flows shift to the Internet tunnel — preserving the user experience at the cost of changing transport. With EAAR enabled, that transition happens far faster because the degradation is seen through inline data rather than waiting on the 10-minute average [Source: https://lostintransit.se/2024/02/19/catalyst-sd-wan-enhanced-application-aware-routing/].

Key Takeaway: Path selection filters tunnels by SLA compliance, then applies color/TLOC preferences; when the chosen tunnel degrades, AAR reroutes to a still-compliant path — and EAAR with dampening makes that failover both faster and more stable.

Quality of Service

AAR decides which path a flow takes. Quality of Service (QoS) decides what happens to flows once they share a path and that path becomes congested. When a 100 Mbps interface is asked to send 120 Mbps, something has to wait or be dropped — QoS ensures it is the backup job that waits, not the CEO’s video call.

On Catalyst 8000 platforms, QoS is configured per WAN edge and pushed at scale via vManage device templates and localized policies. The standard per-interface workflow is [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/qos]:

1. Define forwarding classes and map them to hardware queues.
2. Configure QoS schedulers (bandwidth or priority) per forwarding class.
3. Group the schedulers into a QoS map.
4. Apply the QoS map and a shaping rate on the egress interface.
5. Classify traffic into forwarding classes (via data policy, ACL, or DSCP).
6. Optionally configure egress DSCP/CoS rewrite.

A crucial design principle runs through all of this: classification is separated from queuing/shaping. The centralized data policy decides “what goes into which forwarding class,” while the localized QoS policy decides “how each class is queued and shaped.” Keeping these separate makes policy modular and far easier to troubleshoot [Source: https://www.cisco.com/c/en/us/support/docs/routers/sd-wan/222695-understand-qos-fundamentals-and-class-de.html].

Per-Tunnel and Per-VPN QoS

A forwarding class is a logical QoS class (voice, real-time, critical data, bulk) mapped to an underlying hardware queue on the Catalyst 8000, with scheduling properties (priority versus bandwidth) and optional policers [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/qos]. In vManage, you define forwarding classes under Configuration → Policies → Localized Policy → QoS and map each to a queue number, for example:

A class list groups traffic that will share the same forwarding class. A QoS map is the SD-WAN equivalent of an IOS policy-map: it ties each forwarding class to a scheduler, specifying minimum bandwidth percentage, whether the queue is strict priority (LLQ), and optional WRED drop profiles [Source: https://www.youtube.com/watch?v=m4QEZdFndWc]. The QoS map and a shaping rate are then applied to the WAN interface through the Cisco VPN Interface Ethernet feature template, with the localized policy attached to the device template.

Standard QoS operates at the physical interface level — fine for a branch with one uplink, but limiting for a hub serving hundreds of spokes. Two advanced models address larger topologies:

Per-Tunnel QoS shapes and prioritizes traffic per IPsec tunnel to each spoke, not just per physical interface. This is essential at a hub: without it, a single busy spoke could consume the whole interface and starve the others. Key constraints [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/qos/ios-xe-17/qos-book-xe/per-tunnel-qos.html]:

• Supports hub-to-spoke topology only — spoke-to-spoke direct tunnels are not supported.
• Hubs must be IOS XE Catalyst SD-WAN devices (cEdges / Catalyst 8000).
• The number of sessions is configurable from 100 to 6000, default 2000.

Architecturally, Per-Tunnel QoS uses a per-tunnel QoS aggregator on the hub that enforces bandwidth per spoke tunnel, a QoS map (referenced as underlay dqos) defining per-tunnel bandwidth reservations, and a physical shaping rate on the hub WAN interface representing the total bandwidth to divide among tunnels. Configuration spans the branch interface template (set upstream/downstream bandwidth, enable Per-Tunnel QoS), the hub interface template (enable Per-Tunnel QoS and the aggregator, set the physical shaping rate and QoS map name), and a localized policy defining the underlay dqos map [Source: https://www.youtube.com/watch?v=GG47Iuif0n0].

Figure 8.3: Per-Tunnel QoS hierarchy on a hub WAN interface

graph TD
    PHY["Hub WAN interface<br/>(physical shaping rate = total bandwidth)"] --> AGG["Per-Tunnel QoS aggregator"]
    AGG --> T1["Per-spoke tunnel A<br/>(underlay dqos share)"]
    AGG --> T2["Per-spoke tunnel B<br/>(underlay dqos share)"]
    AGG --> T3["Per-spoke tunnel C<br/>(underlay dqos share)"]
    T1 --> Q1A["fc-voice (Q0): LLQ / priority"]
    T1 --> Q1B["fc-critical (Q1): guaranteed BW"]
    T1 --> Q1C["fc-default (Q2): best effort"]
    T1 --> Q1D["fc-bulk (Q3): scavenger"]
    T2 --> Q2["Same 4-queue QoS map per tunnel"]
    T3 --> Q3["Same 4-queue QoS map per tunnel"]

Per-VPN QoS instead partitions interface bandwidth per VPN — useful when different VPNs represent different tenants or business units that each need a guaranteed slice. It requires controller and vManage version 20.6.1 or later, applies a VPN-QoS policy that affects outbound traffic only, and mandates a shaping rate on the interface [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/qos/ios-xe-17/qos-book-xe/per-vpn-qos.html]. Because IPsec packets may be forwarded out of order during congestion, Per-VPN QoS requires enabling the IPsec extended anti-replay window on both ends to avoid dropping valid packets.

Two interaction rules are non-negotiable:

• Per-Tunnel QoS and Per-VPN QoS are mutually exclusive on the same interface [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/qos/ios-xe-17/qos-book-xe/per-vpn-qos.html].
• In every model, the sum of minimum bandwidths must be less than the interface shaping rate [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/qos/ios-xe-17/qos-book-xe/per-tunnel-qos.html].

Key Takeaway: Forwarding classes map traffic to hardware queues governed by a QoS map and shaping rate; Per-Tunnel QoS gives each spoke its own bandwidth share on a hub (hub-to-spoke only, 100–6000 sessions), while the mutually exclusive Per-VPN QoS partitions bandwidth per VPN.

Shaping, Policing, and Queuing

These three are the workhorses of congestion management, and it helps to keep them straight:

• Shaping limits the outbound rate by buffering excess traffic and releasing it smoothly, smoothing bursts to match the rate the service provider can accept. On Catalyst 8000, you set the interface shaping rate in the VPN Interface Ethernet template’s ACL/QoS section; it acts as the parent shaper for any per-VPN or per-tunnel allocations beneath it [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/qos].
• Policing also enforces a rate, but instead of buffering it drops (or re-marks) traffic that exceeds the limit — useful where buffering delay is undesirable.
• Queuing decides, when packets are waiting, which to send first. Each forwarding class has its own queue, and the QoS map scheduler determines whether a queue gets strict priority (LLQ, for voice) or a weighted share of remaining bandwidth.

A water analogy makes the shaping/policing distinction concrete: shaping is a funnel that holds back excess water and lets it through at a steady trickle, while policing is a valve that simply spills anything beyond the allowed flow onto the floor. Always set the interface shaper at or below the real ISP bandwidth — shaping to a rate higher than the provider delivers means congestion (and drops) happen invisibly inside the provider’s network, where your QoS policy cannot help.

Key Takeaway: Shaping buffers excess traffic to a smooth rate (the parent for per-VPN/per-tunnel allocations), policing drops or re-marks traffic over the limit, and queuing — driven by the QoS map scheduler — decides which class’s packets are served first during congestion.

Adaptive QoS

Static shaping has a blind spot: it assumes a fixed link rate. But broadband and LTE links deliver variable bandwidth, and a shaper hard-coded to 100 Mbps will needlessly drop traffic when the link briefly delivers only 60 Mbps — or waste capacity when it delivers 120 Mbps. Adaptive QoS solves this by dynamically adjusting the shaping rate based on real-time link conditions in hub-to-spoke topologies on Catalyst cEdge devices [Source: https://www.thenetworkdna.com/2024/02/introduction-to-adaptive-qos-in-cisco.html].

Its requirements and limitations are specific [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/qos/ios-xe-17/qos-book-xe/m-adaptive-qos.html]:

• Only IOS XE Catalyst SD-WAN devices are supported (both hub and spokes must be cEdges).
• Hub-to-spoke topology only.
• A WAN interface already configured as a hub cannot also run Adaptive QoS.
• Support on DIA/DCA interfaces depends on overlay session throughput; on loopback interfaces it works only if a single loopback is bound to a single physical interface.

Configuration is refreshingly simple: in the Cisco VPN Interface Ethernet feature template, open the ACL/QoS section and change Adaptive QoS from its default Deactivated to Global / On [Source: https://www.thenetworkdna.com/2024/02/introduction-to-adaptive-qos-in-cisco.html]. The router then uses telemetry such as measured throughput and link conditions to continuously tune how traffic is shaped and queued, within the constraints of the configured forwarding classes.

Key Takeaway: Adaptive QoS dynamically tunes the shaping rate to actual link conditions on hub-to-spoke cEdge deployments — enabled by a single ACL/QoS template setting — avoiding the waste and drops of a fixed shaper on variable-bandwidth links.

Application Visibility

Every mechanism so far depends on one prerequisite: the router must know what the traffic is. Steering “Webex” onto MPLS or allowing direct internet access only for “Office 365” is impossible if the router sees nothing but anonymous TCP/443 connections. Application visibility is the foundation that makes intent-based WAN policy possible.

NBAR2 and DPI

NBAR2 (Network Based Application Recognition, version 2) is the deep packet inspection (DPI) engine that identifies applications on Catalyst 8000 edge routers. DPI means inspecting packet contents — not just IP/port headers — to recognize the actual application. From SD-WAN release 20.6.1, NBAR2 is the DPI engine for all Cisco SD-WAN edge routers, integrated into the SD-WAN Application Intelligence Engine (SAIE); what older documentation called the “DPI flow” is now the SAIE flow [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/policies/ios-xe-17/policies-book-xe/m-custom-applications.html].

NBAR2 classifies traffic using protocol signatures, heuristics, stateful inspection, and the TLS Server Name Indication (SNI), mapping each flow to a specific application ID such as office365, webex, or salesforce. A clever trick handles the modern reality that almost everything is encrypted and many flows are short-lived: NBAR2 correlates DNS lookups with the connections that follow, allowing it to classify a flow on its first packet using the learned DNS information [Source: https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2025/pdf/BRKENT-2336.pdf].

For performance, classification and forwarding are split. A new flow first traverses the DPI/SAIE path, where NBAR2 inspects enough packets (plus DNS context) to identify the application. Once identified, subsequent packets are handled in a fast path using the cached application ID, so the router does not re-inspect every packet — preserving the high throughput Catalyst 8000 platforms are built for [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/policies/vedge-20-x/policies-book/m-config-deep-packet-inspection.html].

Figure 8.4: NBAR2 first-packet classification and fast-path forwarding

sequenceDiagram
    participant C as Client
    participant R as Catalyst 8000 (NBAR2 / SAIE)
    participant D as DNS resolver
    participant App as SaaS application
    C->>R: DNS query for SaaS FQDN
    R->>D: Forward / observe DNS lookup
    D-->>R: DNS response (learned FQDN to IP)
    Note over R: Correlate DNS with upcoming flow
    C->>R: First packet of new flow (TCP/443)
    Note over R: DPI/SAIE path - classify on first packet<br/>using signature, SNI, DNS context (e.g. office365)
    R->>App: Forward with cached application ID
    C->>R: Subsequent packets
    Note over R: Fast path - reuse cached app ID<br/>(no re-inspection)
    R->>App: Forward at line rate

NBAR2 classification combined with flow statistics (via Flexible NetFlow) forms Cisco’s Application Visibility and Control (AVC) framework, which reveals which applications are running, who uses them, and how they perform per WAN link [Source: https://www.liveaction.com/resources/blog-post/what-is-cisco-application-visibility-and-control/]. The most powerful consequence is trusted DIA (Direct Internet Access): rather than writing a brittle rule like “allow DIA for TCP/443 to these IP ranges,” you write “allow DIA only for office365, webex, and dropbox-business” by application name. Because NBAR2 identifies the real application regardless of the underlying IP, the policy stays robust even as SaaS providers constantly change their address ranges [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/allowing-dia-to-trusted-application-only].

This DNS dependency is also a key design constraint: if clients use encrypted DNS (DoH/DoT) directly to the internet, the router never sees the DNS query, degrading first-packet classification. Branches should be designed so the Catalyst 8000 (or a resolver behind it) sees the SaaS FQDN queries [Source: https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2025/pdf/BRKENT-2336.pdf].

Key Takeaway: NBAR2 is the SD-WAN DPI engine (from release 20.6.1, part of the SAIE) that identifies applications by signature, SNI, and DNS correlation — enabling first-packet classification and robust application-name policies like trusted DIA, provided the router can see DNS queries.

Cloud OnRamp for SaaS

Application-Aware Routing optimizes paths across the SD-WAN overlay, but much of today’s traffic heads to SaaS clouds reachable several different ways: local DIA over each transport, backhaul to the data center, or through a Secure Internet Gateway (SIG). Cloud OnRamp for SaaS continuously measures, scores, and selects the best of these paths per SaaS application and per site [Source: https://www.cisco.com/c/m/en_us/solutions/enterprise-networks/sd-wan/optimize_application.html].

It works in three stages:

1. Continuous path monitoring. The edge enumerates all viable paths to a SaaS app and sends periodic HTTP/HTTPS probes over each one — toward the application or its regional endpoints — measuring latency, loss, and sometimes server response time. When a SIG is in use, probes run inside the SIG tunnels too, so that path is evaluated alongside DIA and backhaul [Source: https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2025/pdf/BRKENT-2630.pdf].
2. Real-time scoring. The solution computes a quality score per path and per application, for a supported set of popular SaaS apps (Microsoft 365, Salesforce, Webex, Box, Dropbox, and others) [Source: https://www.cisco.com/c/en/us/solutions/collateral/enterprise-networks/sd-wan/nb-06-cloud-onramp-saas-aag-cte-en.html].
3. Automatic steering. Based on the scores, Cloud OnRamp steers new flows for each SaaS app onto the best path, and shifts flows away when the current path’s quality drops — with hysteresis to prevent flapping [Source: https://www.cisco.com/c/m/en_us/solutions/enterprise-networks/sd-wan/optimize_application.html].

Figure 8.5: Cloud OnRamp for SaaS path measurement and steering

flowchart TD
    SAAS["SaaS app per site<br/>(e.g. Microsoft 365, Webex)"] --> PROBE["Send HTTP/HTTPS probes<br/>over every viable path"]
    PROBE --> P1["DIA over biz-internet (ISP1)"]
    PROBE --> P2["DIA over public-internet (ISP2)"]
    PROBE --> P3["Backhaul to data center"]
    PROBE --> P4["Secure Internet Gateway (SIG) tunnel"]
    P1 --> SCORE["Compute quality score<br/>per path, per app"]
    P2 --> SCORE
    P3 --> SCORE
    P4 --> SCORE
    FEED["App Feedback for Path Selection<br/>(M365 true-experience telemetry)"] -.-> SCORE
    SCORE --> STEER["Steer new flows to best path<br/>(hysteresis prevents flapping)"]
    STEER --> MONITOR{"Current path<br/>quality drops?"}
    MONITOR -->|Yes| PROBE
    MONITOR -->|No| STEER

A distinctive capability is Application Feedback for Path Selection. For certain providers — notably Microsoft 365 — Cloud OnRamp can incorporate telemetry from the SaaS service itself about real user experience, not just network metrics. In the Catalyst SD-WAN interface, an “Enable Application Feedback for Path Selection” option lets the router exchange telemetry with the application and fold it into path scoring. The router may then prefer a path that yields a better true experience even when its raw latency is slightly higher [Source: https://www.youtube.com/watch?v=AbVa8-EDjmc].

Worked example. A branch with a Catalyst 8300 runs Microsoft 365 and Webex. NBAR2 recognizes the flows as office365, teams, and webex rather than generic HTTPS. Policy permits DIA only for those trusted apps; everything else goes via SIG. Cloud OnRamp is enabled for the two SaaS apps, sending HTTP/HTTPS probes over DIA on biz-internet (ISP1), DIA on public-internet (ISP2), and the SIG tunnel, measuring each path’s quality. If ISP1 develops high loss while ISP2 stays clean, new Teams and Webex flows steer to the ISP2 DIA path automatically. With Application Feedback enabled, Microsoft 365 telemetry can further refine the choice toward the path with the best actual user experience. The operational result: users notice fewer Teams and Webex freezes during ISP trouble, with no manual intervention at the branch [Source: https://www.cisco.com/c/en/us/solutions/collateral/enterprise-networks/sd-wan/nb-06-cloud-onramp-saas-aag-cte-en.html].

Key Takeaway: Cloud OnRamp for SaaS probes every viable path to a SaaS application (DIA, backhaul, SIG), scores them per app and per site, and steers flows to the best path with hysteresis — optionally using SaaS Application Feedback to optimize for true user experience rather than raw latency alone.

Application Performance Monitoring

Application visibility is not only an input to policy — it is also how operators see whether the network is meeting its goals. On Catalyst 8000 SD-WAN, NBAR2-based visibility provides, per application, traffic volumes and top talkers, per-application performance KPIs (loss, latency, jitter, and sometimes a voice MOS score) when combined with AAR SLAs, and the actual per-app routing decision being made — for example, office365 going DIA while unknown-https backhauls to the data center [Source: https://www.liveaction.com/resources/blog-post/what-is-cisco-application-visibility-and-control/].

For monitoring the routing and QoS mechanisms themselves, key tools include:

A few recurring troubleshooting themes tie the whole chapter together. AAR does nothing if the policy is not attached, or if the overlay route was never installed [Source: https://www.reddit.com/r/networking/comments/uwuhkd/cisco_sdwan_application_aware_routing_order_of/]. SLA thresholds that are too strict leave tunnels perpetually “bad”; thresholds too loose never trigger a reroute. EAAR silently fails to function unless enabled on both ends of a tunnel. And note that vManage QoS monitoring does not directly monitor Per-VPN QoS policy, so plan alternate verification for that feature [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/qos/ios-xe-17/qos-book-xe/per-vpn-qos.html]. Resource pressure is also worth watching: DPI and per-flow analytics consume CPU and QFP capacity, and very high-scale sites should validate flow, app, and tunnel counts against platform limits, narrowing full DPI to the SaaS apps that matter most if pressure appears [Source: https://miercom.com/wp-content/uploads/2024/10/Miercom-Report-Cisco-Catalyst-8000v-Edge-Router-Performance-DR210226F-FINAL-2.pdf].

Key Takeaway: NBAR2 visibility plus AAR SLAs surface per-application volume, performance, and routing decisions, while app-route stats, BFD/QoS CLI commands, and data-policy counters let operators verify that policy is working — keeping in mind common pitfalls like uninstalled routes, mis-tuned SLAs, one-sided EAAR, and DPI resource consumption.

Chapter Summary

The Catalyst 8000 edge platform delivers quality of experience through three cooperating layers. Application-Aware Routing measures every data-plane tunnel with BFD — loss, latency, and jitter over a 10-minute averaging window — and compares those metrics against reusable SLA classes defined in a centralized app-route policy. Matching traffic is steered onto an SLA-compliant tunnel honoring color/TLOC preferences, and rerouted when a path degrades; Enhanced AAR uses inline data-plane packets and a dampening mechanism for much faster, more stable failover, but must be enabled on both tunnel ends.

Quality of Service then manages congestion on the chosen paths. Forwarding classes map traffic to hardware queues controlled by a QoS map and a shaping rate, with classification (centralized data policy) deliberately separated from queuing and shaping (localized policy). Per-Tunnel QoS gives each spoke a guaranteed share on a hub (hub-to-spoke only), the mutually exclusive Per-VPN QoS partitions bandwidth per VPN, and Adaptive QoS tunes the shaper to real-time link conditions. Shaping buffers excess to a smooth rate, policing drops or re-marks it, and queuing decides serving order.

Underpinning everything, NBAR2 deep packet inspection identifies applications by signature, SNI, and DNS correlation — enabling first-packet classification, robust trusted-DIA policy by application name, and rich per-app visibility. Cloud OnRamp for SaaS extends this to the cloud, probing and scoring every path (DIA, backhaul, SIG) per SaaS application and steering flows to the best one, optionally guided by SaaS Application Feedback. Together these features let the Catalyst 8000 continuously adapt the WAN to deliver the experience each application demands.

Key Terms

Chapter 9: Deployment, Provisioning, and Day-0 Operations

When an enterprise rolls out fifty new branch offices, the last thing it wants is to fly a network engineer to each location to type configuration commands into a router console. The promise of Cisco Catalyst 8000 edge platforms in a Catalyst SD-WAN fabric is that a non-technical person at the branch can rack the router, plug in power and a network cable, and walk away — while the device provisions itself from the cloud. This chapter explains how that “Day-0” automation actually works, how administrators define the configurations that get pushed to devices, and how to design a branch that keeps running even when a router, link, or power supply fails.

We will move from the moment a router powers on out of the box, through the central configuration models in Cisco SD-WAN Manager, and finally to the high-availability designs that protect a branch from single points of failure.

Learning Objectives

By the end of this chapter, you will be able to:

• Describe zero-touch provisioning (ZTP) and Plug and Play (PnP) onboarding workflows for Catalyst 8000 edge platforms.
• Configure templates and configuration groups in Cisco Catalyst SD-WAN Manager.
• Plan a branch deployment that includes high availability and redundancy.

Zero-Touch Provisioning

Think of zero-touch provisioning like activating a new smartphone. You take it out of the box, connect it to Wi-Fi, and it phones home to the manufacturer’s cloud, identifies itself by serial number, downloads its profile, and configures itself — no manual setup required. Catalyst 8000 routers use the same idea: the device contacts Cisco’s cloud, proves who it is, learns which controllers it belongs to, and pulls down its full configuration. For Catalyst 8000 SD-WAN edge routers, this is achieved either by phoning home to Cisco’s Plug and Play (PnP) cloud and automatically discovering the SD-WAN controllers, or by bootstrapping the router with a pre-generated configuration file from Cisco SD-WAN Manager [Source: https://www.grandmetric.com/zero-touch-provisioning-ztp-cisco-sd-wan-work/].

Zero-Touch Provisioning (ZTP) is the overall outcome: ship a router from the factory, plug it in, power it on, and it self-provisions into the SD-WAN fabric with no command-line work at the branch. Plug and Play (PnP) is the underlying mechanism that delivers that outcome — a software agent built into IOS XE plus a cloud registry — that makes ZTP possible [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/wan-edge-deployment]. On Catalyst 8000, Cisco effectively uses the PnP infrastructure as the ZTP mechanism for IOS XE SD-WAN, while bootstrap files provide a more controlled, “almost-zero-touch” alternative.

Before going further, it helps to know the cast of characters involved in Day-0 onboarding:

The whole onboarding flow is really about getting the router from “factory default” to “has an IP address, has an identity, knows where its controllers are, and can pull a template from SD-WAN Manager” [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/wan-edge-deployment].

PnP Connect portal

PnP Connect is a cloud service that lives under your Cisco Smart Account / Virtual Account and holds a registry of devices along with their onboarding instructions [Source: https://www.lookingpoint.com/blog/cisco-sd-wan-pnp-onboarding]. It performs three key jobs for Catalyst 8000 SD-WAN onboarding.

First, device registration and claiming: every device is associated with your Smart/Virtual Account by serial number, product ID (PID), or Secure Unique Device Identifier (SUDI). This can happen automatically at purchase, or you can import devices manually by CSV or API [Source: https://www.lookingpoint.com/blog/cisco-sd-wan-pnp-onboarding].

Second, association to an SD-WAN profile: you link each device to a controller profile that contains the SD-WAN Manager, vBond, and vSmart FQDNs or IPs along with the organization name and tenant details. Alternatively, PnP can send the router just enough information to complete onboarding on its own [Source: https://www.cisco.com/c/en/us/td/docs/routers/sd-routing/1713x/sd-routing-onboard-routing-devices-to-sd-wan-manager-1713x.html].

Third, mapping to SD-WAN Manager: SD-WAN Manager can pull device information from the PnP portal and create a matching WAN Edge entry. This enables “cloud ZTP” — the router phones home to PnP, PnP says “you belong to this SD-WAN Manager and vBond,” and the router continues onboarding with those controllers [Source: https://www.lookingpoint.com/blog/cisco-sd-wan-pnp-onboarding].

In practice, before shipping a Catalyst 8000 to a branch you make sure the device is in the correct Virtual Account, associated with the correct SD-WAN onboarding profile, and that SD-WAN Manager is ready to accept it with certificates in place.

Bootstrap files and day-0 config

Not every site has open internet access to reach Cisco’s cloud. For those environments, SD-WAN Manager can generate a bootstrap configuration — a Day-0 file that contains everything the router needs to find its controllers on its own [Source: https://0x2142.com/how-to-cisco-sd-wan-onboarding-a-catalyst-8000v/].

A bootstrap file for an IOS XE SD-WAN WAN Edge typically includes:

• Basic system parameters: hostname, system-IP, and site-ID.
• The SD-WAN organization name and vBond/orchestrator address (FQDN or IP plus port).
• WAN transport interfaces configured as tunnel-interface (for example, on GigabitEthernet0/0/0) with basic IP or DHCP settings.
• Certificate, CSR, or token details so the device can authenticate to the controllers.
• Optional initial device-template variables, such as VPN 0 and the VPN 512 management VPN [Source: https://0x2142.com/how-to-cisco-sd-wan-onboarding-a-catalyst-8000v/].

To generate one, you go to Configuration → Devices in SD-WAN Manager, select the unused Catalyst 8000, and click Generate Bootstrap Configuration. You then choose the transport type (static IP or DHCP), indicate whether the router is behind NAT, and download the resulting file [Source: https://lostintransit.se/2023/07/08/catalyst-sd-wan-bootstrapping-a-catalyst8000v-using-a-file-on-bootflash/].

How the file gets to the router depends on the form factor. For a physical Catalyst 8000, you copy the file to a USB stick or bootflash, ensuring the filename matches what the Day-0 image expects (for example, a ciscosdwan.cfg style name). For a virtual Catalyst 8000V, you inject the file as day-0 userdata in the hypervisor or copy it to the VM’s bootflash [Source: https://0x2142.com/how-to-cisco-sd-wan-onboarding-a-catalyst-8000v/].

On first boot with no startup-config, the router looks for this Day-0 file. If it finds one, it loads it as the initial configuration, immediately knowing its org name, vBond address, system-IP, site-ID, and which interfaces to use for transport. It then uses that information to reach vBond, then SD-WAN Manager and vSmart [Source: https://lostintransit.se/2023/07/08/catalyst-sd-wan-bootstrapping-a-catalyst8000v-using-a-file-on-bootflash/]. This approach is not strictly zero-touch — someone has to attach the USB stick or inject the file — but it avoids all CLI work at the branch and is the standard method when cloud connectivity or PnP access is constrained.

ZTP over cellular and internet

When a Catalyst 8000 boots at factory default with internet reachability, the closest thing to pure ZTP unfolds automatically. The sequence is worth walking through step by step [Source: https://www.grandmetric.com/zero-touch-provisioning-ztp-cisco-sd-wan-work/]:

1. Power on and DHCP — The WAN/internet port (for example, Gi0/0/0) is connected to a network providing DHCP and outbound HTTPS. The router obtains an IP address, default gateway, and DNS server.
2. PnP discovery — The PnP agent in IOS XE runs automatically and attempts discovery in a typical order: check for DHCP options (43, 60, 143) pointing to a local PnP/ZTP server, then resolve known PnP FQDNs such as pnpserver.<domain>, then fall back to Cisco’s global cloud PnP Connect over HTTPS [Source: https://www.youtube.com/watch?v=t_k0ijpO3fI].
3. Phone-home to PnP Connect — The router opens an HTTPS session (TCP 443) to PnP Connect and presents its SUDI/serial and product ID. The portal looks up this identity in your Virtual Account and finds the associated SD-WAN onboarding profile [Source: https://www.lookingpoint.com/blog/cisco-sd-wan-pnp-onboarding].
4. PnP returns SD-WAN information — PnP sends back either the SD-WAN Manager/vBond FQDN and organization name, or a lightweight bootstrap that includes those details.
5. Connect to vBond — Now knowing where vBond is, the router initiates a secure DTLS/TLS connection (often port 12346 or 443). vBond authenticates the router by certificate, confirms its serial is in the trust database, and returns reachable addresses for SD-WAN Manager and vSmart along with NAT-traversal information [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/wan-edge-deployment].
6. Enroll with SD-WAN Manager — The router builds a tunnel to SD-WAN Manager, appears in the device list as “reachable,” and is matched to its entry by serial/chassis ID. SD-WAN Manager then pushes the full Day-1 device configuration [Source: https://0x2142.com/how-to-cisco-sd-wan-onboarding-a-catalyst-8000v/].
7. Join the fabric — The router establishes OMP sessions with vSmart and starts exchanging routes and policy [Source: https://www.ciscolive.com/c/dam/r/ciscolive/apjc/docs/2025/pdf/BRKENT-2108.pdf].

Figure 9.1: ZTP phone-home onboarding sequence

sequenceDiagram
    participant R as Catalyst 8000 (WAN Edge)
    participant D as DHCP / DNS
    participant P as PnP Connect (cloud)
    participant B as vBond (Validator)
    participant M as SD-WAN Manager (vManage)
    participant S as vSmart (Controller)

    R->>D: Power on, request IP / gateway / DNS
    D-->>R: IP address, default gateway, DNS
    R->>P: HTTPS (443) phone-home: SUDI / serial / PID
    P-->>R: SD-WAN Manager + vBond FQDN, org name
    R->>B: DTLS/TLS connect (12346 or 443), present certificate
    B-->>R: Authenticate; return Manager + vSmart addresses, NAT info
    R->>M: Build tunnel, appear as "reachable", match by chassis ID
    M-->>R: Push full Day-1 device configuration
    R->>S: Establish OMP session
    S-->>R: Exchange routes and policy (joined to fabric)

Crucially, this works over a cellular transport just as it does over a wired internet circuit. A Catalyst 8000 with an LTE/5G module can obtain its IP from the cellular network and phone home over that same HTTPS path, which is why ZTP is so attractive for branches where the only Day-0 connectivity is a cellular modem. The only onsite steps remain: rack the device, connect power, and provide a DHCP-enabled path with outbound HTTPS and DNS.

If onboarding fails, the troubleshooting commands to reach for are show platform software pnp status and debug pnp discovery to inspect the DHCP/DNS/PnP attempts, and show sdwan control connections to check the vBond/vSmart/Manager status once a bootstrap has loaded [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/wan-edge-deployment].

Key Takeaway: Catalyst 8000 Day-0 onboarding uses Cisco’s PnP infrastructure as its ZTP mechanism. Cloud ZTP (rack-and-cable only) requires DHCP, DNS, and outbound HTTPS to reach PnP Connect over the internet or cellular; bootstrap files generated by SD-WAN Manager provide a near-zero-touch alternative for closed or MPLS-only environments. Either way, the router gets an IP, learns its controllers, authenticates to vBond, and pulls its full configuration.

Templates and Configuration Groups

Once a router is reachable in SD-WAN Manager, something has to define what configuration gets pushed to it. SD-WAN Manager offers two configuration models for IOS XE edge devices like the Catalyst 8000: the traditional template model and the newer Configuration Group model. Both ultimately generate IOS XE SD-WAN configuration, but they differ in how you design, reuse, and deploy it [Source: https://lostintransit.se/2023/07/11/catalyst-sd-wan-introduction-to-configuration-groups/].

Feature and device templates

In the traditional model, feature templates are the basic building blocks of a device’s configuration [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/17-x/systems-interfaces/systems-interfaces-guide-17-x/config-devices.html]. Each feature template represents one feature on the router. Common examples include:

• System — hostname, system-IP, site-ID, organization name, timezone.
• VPN — VPN 0 for transport, VPN 512 for management, and service VPNs.
• VPN interface — WAN interfaces, LAN sub-interfaces, tunnel interfaces.
• Routing — OMP, BGP, OSPF, static routes.
• Security, AAA, logging, SNMP, NTP, and so on.

The power of feature templates comes from parameterization. Many fields can be set as device-specific variables rather than hard-coded values. The same VPN-interface template can be reused across hundreds of routers, with each router supplying its own IP address through a variable like vpn0_if1_ip. A useful analogy is a form letter: the body of the letter is identical for everyone, but the name and address are filled in per recipient.

A device template is the higher-level object that ties multiple feature templates together and binds them to a specific hardware platform [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/17-x/systems-interfaces/systems-interfaces-guide-17-x/config-devices.html]. You select a device type — for example, a Catalyst 8300 — and then assemble the System template, the VPN 0 and VPN 512 templates, one or more VPN-interface templates, OMP, BFD, and routing templates into a complete, model-aware definition.

A typical worked flow for provisioning a Catalyst 8000 with device templates looks like this:

1. Onboard the router (via PnP/ZTP or bootstrap) and confirm it is validated and present in SD-WAN Manager.
2. Create reusable feature templates: System, VPN 0, VPN 512, LAN VPNs, OMP, BFD, logging.
3. Create a device template for the specific model, for example a C8300-1N1S-4T2X.
4. Associate the feature templates with the device template.
5. Attach the device template to one or more routers.
6. Fill in per-device variables (manually or by CSV).
7. Review and deploy; SD-WAN Manager renders all the feature templates and pushes the combined configuration via NETCONF.

Figure 9.2: Device-template provisioning workflow

flowchart TD
    A["Onboard router (PnP/ZTP or bootstrap)"] --> B["Confirm validated and present in SD-WAN Manager"]
    B --> C["Create reusable feature templates:<br/>System, VPN 0, VPN 512, LAN VPNs, OMP, BFD, logging"]
    C --> D["Create device template for specific model<br/>(e.g. C8300-1N1S-4T2X)"]
    D --> E["Associate feature templates with device template"]
    E --> F["Attach device template to one or more routers"]
    F --> G["Fill in per-device variables (manually or by CSV)"]
    G --> H["Review and deploy"]
    H --> I["SD-WAN Manager renders templates and pushes config via NETCONF"]

The main drawback of this model is that device templates are model-specific. Even when the logical design is identical, you often need a separate device template for a C8200, a C8300, and a C8500. Across a fleet with many SKUs, this leads to a proliferation of nearly identical templates — the very pain point that motivated Configuration Groups.

Configuration Groups and parcels

Starting with SD-WAN Manager 20.8 and IOS XE SD-WAN releases around 17.9 and later, Cisco introduced Configuration Groups, a more modular, intent-based construct that is now the strategic direction for IOS XE devices [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/config-groups/configuration-group-guide/using-config-groups.html]. Two scope rules are essential to remember: Configuration Groups are supported only on IOS XE-based devices (which includes the Catalyst 8000 family), and a device can be managed by either a device template or a Configuration Group — never both at once [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/config-groups/configuration-group-guide/using-config-groups.html].

A Configuration Group is a logical grouping of devices that share a common role or business intent, configured using modular building blocks called feature profiles [Source: https://lostintransit.se/2023/07/11/catalyst-sd-wan-introduction-to-configuration-groups/]. Unlike a device template tied to one hardware SKU, a single Configuration Group can span multiple Catalyst 8000 models that share the same design — for instance, a “Small branch with dual internet” group covering both C8200 and C8300 variants [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/config-groups/configuration-group-guide/using-config-groups.html].

The reusable buckets inside a Configuration Group are feature profiles [Source: https://www.ciscolive.com/c/dam/r/ciscolive/global-event/docs/2025/pdf/BRKENT-1313.pdf]. The main types are:

• System Profile — system basics: hostname, system-IP, site-ID, banner, AAA, logging, DNS, NTP.
• Transport Profile — underlay WAN connectivity: interfaces in VPN 0, tunnel settings, TLOC colors.
• Service Profile — service VPNs and the LAN side: VPNs, LAN interfaces, BGP/OSPF, DHCP pools.

Within each profile, configuration is broken into smaller atomic elements called features or parcels. A parcel corresponds to a well-defined section of config — a specific service VPN, a transport interface with its addressing, a BGP process, or an NTP stanza [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/config-groups/configuration-group-guide/using-config-groups.html]. Because configuration is this granular, a small change like adding a new LAN VPN can be localized to a single parcel rather than requiring edits to a large monolithic template.

Figure 9.3: Configuration Group structure (profiles and parcels)

flowchart TD
    CG["Configuration Group<br/>(e.g. BRANCH_DUAL_INTERNET)"] --> SP["System Profile"]
    CG --> TP["Transport Profile"]
    CG --> SVP["Service Profile"]
    SP --> SP1["Parcel: hostname / system-IP / site-ID"]
    SP --> SP2["Parcel: AAA, logging, DNS, NTP"]
    TP --> TP1["Parcel: VPN 0 interfaces"]
    TP --> TP2["Parcel: tunnel settings, TLOC colors"]
    SVP --> SVP1["Parcel: service VPN + LAN interfaces"]
    SVP --> SVP2["Parcel: BGP/OSPF, DHCP pools"]
    CG --> DEV["Spans multiple IOS XE models<br/>(C8200, C8300, ...)"]

The table below contrasts the two models:

To create a Configuration Group, you use the Workflow Library → Create Configuration Group wizard, which walks you through naming the group (for example, BRANCH_DUAL_INTERNET), building the System, Transport, and Service profiles with smart defaults, associating the group with target devices, and previewing the compiled configuration before committing it [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/config-groups/configuration-group-guide/config-group-workflows.html]. The model also supports fine-grained role-based access control (RBAC): you grant read/write permissions per profile type, so a WAN team can own Transport profiles while a LAN team owns Service profiles [Source: https://www.cisco.com/c/en/us/td/docs/routers/sd-routing/1713x/sd-routing-onboard-routing-devices-to-sd-wan-manager-1713x.html].

When a feature is not yet modeled natively, CLI templates (specifically CLI add-on templates) remain available in either model. These let you inject a free-form chunk of IOS XE configuration that SD-WAN Manager merges with the model-driven config — handy for advanced QoS or proprietary monitoring during migrations, at the cost of less validation and harder reuse.

Variables and CSV onboarding

For large deployments, filling in per-device variables one screen at a time would be unbearable. Both models support bulk provisioning through a CSV file, where each row is a device and each column maps to a template variable [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/17-x/systems-interfaces/systems-interfaces-guide-17-x/config-devices.html].

Three columns are mandatory and must come first:

After those three, you add a column for each variable you want to populate — WAN IP, MPLS IP, site-ID, and so on. As a worked example, imagine onboarding twenty branch routers that all share one device template. You would build a spreadsheet with twenty rows, the three mandatory columns identifying each router, plus columns such as vpn0_if1_ip and vpn10_lan_ip. At attachment time, SD-WAN Manager ingests the CSV, matches each row to its device by serial number, and renders all twenty configurations in one operation. If a required variable is left blank, template attachment can fail or the router may receive an incomplete configuration — so validating the CSV before deployment is a critical habit.

Key Takeaway: SD-WAN Manager offers two configuration models. The legacy template model (feature templates → device templates → CLI add-ons) is model-specific and well understood; the newer Configuration Group model (Configuration Groups → System/Transport/Service profiles → parcels) is IOS XE-only, reuses building blocks across Catalyst 8000 models, and adds guided workflows, smart defaults, and RBAC. A device uses one model or the other, never both. CSV onboarding — with the mandatory csv-deviceId, csv-deviceIP, and csv-host-name columns — scales variable population across large fleets.

High Availability

Getting a single router online is only part of the job. A branch that runs a hospital, a trading floor, or a 24-hour distribution center cannot tolerate an outage just because one router reloads or one circuit goes dark. High-availability (HA) design ensures that a user’s default gateway, the SD-WAN control connections, and at least one data-plane path always remain up, with rapid automatic failover [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/26x-later/high-availability/high-availability-book-xe/ha.html]. Redundancy is the underlying principle: deploy duplicate components so that the failure of any one does not interrupt service.

Dual-router branch design

In Cisco SD-WAN, a dual-router branch is modeled as one site with two devices. Both Catalyst 8000 routers share the same site-ID (giving them a common branch identity and consistent policy), but each has a unique system IP — analogous to a unique loopback in classic WAN design — and both use the same organization/domain name [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/26x-later/high-availability/high-availability-book-xe/ha.html]. Both routers independently build control connections to redundant vBond, vSmart, and vManage controllers over every available transport.

A common physical topology pairs:

• Two Catalyst 8000 routers, each with WAN interfaces toward one or more ISPs or MPLS providers.
• A switch stack or pair of access/distribution switches, dual-homing each router (usually via LACP port-channels for link and node redundancy).
• VRRP between the two routers on each user VLAN to present a single virtual gateway.
• Optionally, a TLOC extension link so each router can use the other’s WAN circuits.

        ┌─────────────┐      ┌─────────────┐
  DIA ──│  Router A   │      │  Router B   │── DIA2 / MPLS
        │ site 100    │◄────►│ site 100    │
        │ sys 10.255. │ TLOC │ sys 10.255. │
        │ 100.1       │ ext. │ 100.2       │
        └──────┬──────┘      └──────┬──────┘
               │ LACP              │ LACP
            ┌──┴───────────────────┴──┐
            │   Access switch stack   │
            └────────────┬────────────┘
                         │  VRRP VIP 10.10.10.1
                    LAN users (VLAN 10)

Figure 9.4: Dual-router HA branch topology

flowchart TD
    DIA1["DIA / ISP 1"] --> RA["Router A<br/>site 100<br/>sys 10.255.100.1<br/>VRRP master (prio 120)"]
    DIA2["DIA 2 / MPLS"] --> RB["Router B<br/>site 100<br/>sys 10.255.100.2<br/>VRRP backup (prio 100)"]
    RA <-->|"TLOC extension"| RB
    RA -->|LACP| SW["Access switch stack"]
    RB -->|LACP| SW
    SW --> VIP["VRRP VIP 10.10.10.1"]
    VIP --> LAN["LAN users (VLAN 10)"]

Treating both routers as a single SD-WAN site keeps policy and routing symmetric and simplifies application-aware routing and SLA tracking at the branch [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/26x-later/high-availability/high-availability-book-xe/ha.html].

VRRP and TLOC redundancy

On the LAN side, VRRP (Virtual Router Redundancy Protocol) is a first-hop redundancy protocol that presents a single virtual IP and virtual MAC as the host default gateway [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/service-side-redundancy-using-vrrp]. One router becomes the VRRP master and forwards traffic for the virtual IP; the other is the backup and takes over if the master fails. A typical VLAN 10 design might use:

• Router A SVI: 10.10.10.2 (VRRP priority 120, master)
• Router B SVI: 10.10.10.3 (VRRP priority 100, backup)
• VRRP virtual IP: 10.10.10.1 (the gateway every client points to)

The clever enhancement on IOS XE Catalyst SD-WAN is VRRP Interface Tracking. Instead of failing over only when the router chassis dies, the master can lower its priority — and hand over the master role — when tracked WAN interfaces or SD-WAN SLA trackers detect loss or degradation [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/17-x/systems-interfaces/systems-interfaces-guide-17-x/vrrp-interface-tracking.html]. A good design tracks the DIA interface state, the MPLS/backup link state, and an SD-WAN SLA tracker (loss, latency, jitter) toward the controllers or a cloud destination.

Consider the failure scenario: Router A is master and holds the primary internet circuit. If Router A loses all of its usable WAN circuits, or its SD-WAN tracker detects loss to the controllers, its VRRP priority drops below Router B’s, so Router B becomes master. The LAN default gateway effectively moves to the router that still has working WAN connectivity, preventing the classic problem of hosts continuing to send traffic to a “dead” router. Always enable VRRP preempt on the preferred router so it reclaims the master role once its WAN health is restored [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/service-side-redundancy-using-vrrp].

Figure 9.5: VRRP interface-tracking failover sequence

flowchart TD
    A["Router A is VRRP master (prio 120)<br/>holds primary internet circuit"] --> B{"Router A WAN healthy?<br/>(tracked interface + SLA tracker)"}
    B -->|Yes| A
    B -->|"No: all WAN circuits lost or<br/>SLA tracker detects loss"| C["Router A priority drops below Router B"]
    C --> D["Router B becomes VRRP master"]
    D --> E["LAN gateway (VIP 10.10.10.1)<br/>now served by Router B"]
    E --> F{"Router A WAN health restored?"}
    F -->|"Yes + preempt enabled"| A
    F -->|No| E

On the WAN side, redundancy centers on the TLOC (Transport Location), which identifies an edge transport attachment point by color, encapsulation, system-IP, and interface IP, and is used for both control (OMP) and data tunnels [Source: https://www.networkacademy.io/ccie-enterprise/sdwan/wan-edge-deployment]. Each Catalyst 8000 forms control connections from each active TLOC to all controllers, with BFD monitoring each TLOC for fast failover.

TLOC extension takes redundancy further by letting one router “borrow” the other’s WAN circuits over an inter-router link (commonly a redundant LACP port-channel) [Source: https://www.linkedin.com/pulse/cisco-sd-wan-lab-part-2-tloc-extensions-vrrp-nelson-paiva]. If Router A owns DIA1 and Router B owns DIA2, TLOC extension lets Router A use both DIA1 (local) and DIA2 (via Router B) as valid TLOCs, and vice versa. Each router can then survive the loss of its own WAN ports by exiting through the peer’s transports. Because the inter-router link becomes critical, it should be redundant and carry QoS so TLOC-extension traffic is not starved, and SD-WAN policy must avoid sending traffic back and forth needlessly.

Chassis and power redundancy

Redundancy is not only a network-protocol concern; it extends down to the hardware itself. Within a single Catalyst 8000 chassis, many models support dual power supplies so that the failure of one PSU — or one feed from a power distribution unit — does not drop the router. Pairing each power supply with a separate circuit or UPS protects against an upstream electrical fault as well.

At the branch level, the dual-router design is itself the chassis-redundancy strategy: two physically separate routers mean that a hardware failure, a software crash, or even a maintenance reload of one chassis leaves the second fully operational. When Router A fails completely, VRRP on every VLAN converges to Router B, which continues using its own (and any TLOC-extended) circuits, so the site stays up [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/26x-later/high-availability/high-availability-book-xe/ha.html].

A worked end-to-end example ties the layers together. Two Catalyst 8300 routers share site-ID 100, with system IPs 10.255.100.1 and 10.255.100.2, joined to the same SD-WAN domain and controller cluster. Each forms an LACP port-channel toward the switch stack, with SVIs living on the port-channel. For VLAN 10, Router A is VRRP master at priority 120 and Router B is backup at 100, with the VIP at 10.10.10.1. VRRP tracks each router’s DIA and MPLS interfaces plus an SD-WAN SLA tracker. TLOC extension runs in both directions across the inter-router port-channel, so each router advertises both its own and the peer’s TLOCs. Application-aware routing prefers MPLS for voice and critical apps (failing over to DIA if the SLA degrades) and prefers DIA for general internet and SaaS. The result: if Router A’s DIA fails but MPLS survives, internet traffic shifts to Router B’s DIA (possibly via TLOC extension) while MPLS traffic stays on A; if a whole router or power supply fails, the site rides through on the survivor [Source: https://www.linkedin.com/pulse/cisco-sd-wan-lab-part-2-tloc-extensions-vrrp-nelson-paiva].

The accepted HA best practices pull all of this together: deploy at least two controllers per role in separate locations; provide at least two diverse transport circuits per site (such as DIA plus MPLS, optionally with LTE/5G backup); use BFD and SLA trackers tuned to avoid flapping; apply QoS so control, BFD, and VRRP traffic are prioritized; match MTU/MSS across links and TLOC extensions; align VRRP roles with any inline firewall design to keep traffic symmetric; and always test failover scenarios (router reload, WAN down, partial loss, controller failure) before production cutover [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/ha-scaling/ios-xe-17/high-availability-book-xe/m-high-availability-and-scaling.html].

Key Takeaway: A high-availability dual-router branch is one SD-WAN site with two routers sharing a site-ID but having unique system IPs. VRRP with Interface Tracking keeps the LAN gateway on the router with healthy WAN connectivity; TLOC and controller redundancy plus TLOC extension keep data and control paths diverse; and dual power supplies and the second chassis itself protect against hardware failure. Combine diverse transports, BFD/SLA trackers, QoS, and thorough failover testing for a resilient branch.

Chapter Summary

Day-0 operations turn the chore of branch deployment into an automated, repeatable process. Zero-touch provisioning lets a Catalyst 8000 phone home through Cisco’s Plug and Play infrastructure — discovering controllers via DHCP options, DNS, or PnP Connect over the internet or cellular — so that onsite staff only need to rack and cable the device. Where cloud access is constrained, a bootstrap file generated by SD-WAN Manager delivers near-zero-touch onboarding by carrying the org name, vBond address, and transport details on a USB stick or as VM userdata. In every case, the router gets an IP, authenticates to vBond, learns its controller addresses, connects to SD-WAN Manager and vSmart, and pulls its full configuration.

That configuration is defined through one of two models in SD-WAN Manager. The legacy feature template and device template model assembles parameterized, per-feature building blocks into model-specific device definitions. The newer Configuration Group model organizes configuration into System, Transport, and Service feature profiles built from granular parcels, spans multiple Catalyst 8000 models, and adds guided workflows and RBAC — though a device is always managed by one model or the other. CSV onboarding, anchored by the mandatory device-ID, device-IP, and hostname columns, scales variable population across large fleets.

Finally, a resilient branch is engineered with redundancy at every layer: two routers modeled as one site, VRRP with interface tracking for LAN gateway failover, redundant TLOCs and controllers plus TLOC extension for path diversity, and dual power supplies and chassis for hardware resilience. Tested failover, diverse transports, and prioritized control traffic complete a Cisco-aligned high-availability design.

Key Terms

Chapter 10: Monitoring, Troubleshooting, and Lifecycle Management

Building, deploying, and securing a Catalyst 8000 edge fabric is only the beginning. The real test of any network is what happens on an ordinary Tuesday afternoon when a branch goes quiet, an application crawls, or a software release reaches the end of its supported life. This final chapter brings together the operational disciplines that keep a Catalyst 8000 SD-WAN fabric healthy over its lifetime: watching it (monitoring and telemetry), fixing it (structured troubleshooting), and renewing it (lifecycle management). Think of it as the difference between building a car and running a fleet — the engineering matters, but the maintenance logbook, the diagnostic scanner, and the recall schedule are what keep the wheels turning year after year.

Learning Objectives

By the end of this chapter, you will be able to:

• Use Catalyst SD-WAN Manager dashboards and tools to monitor edge health, and extend that visibility with model-driven telemetry and ThousandEyes.
• Apply a structured troubleshooting methodology for control-plane and data-plane issues, using real IOS XE SD-WAN show commands and admin-tech bundles.
• Plan software upgrades, configure rollback safety nets, and manage the end-of-life roadmap for Catalyst 8000 platforms.

Monitoring and Telemetry

You cannot troubleshoot what you cannot see. Effective operations begin with layered visibility: a control plane that tells you the fabric’s overall state, streaming telemetry that captures the fine detail, and synthetic tests that measure the experience your users actually receive.

SD-WAN Manager monitoring dashboards

Cisco Catalyst SD-WAN Manager (formerly vManage) is the central point for monitoring and maintaining a Catalyst SD-WAN fabric, including all of its Catalyst 8000 edge routers [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/Monitor-And-Maintain/monitor-maintain-book/m-dashboard-screen.html]. It presents network-wide dashboards for overlay health, transport performance, and device status, with drill-downs to individual routers and circuits. For full feature and integration compatibility, run SD-WAN Manager 20.6.x or later; some advanced monitoring integrations explicitly require 20.6.3 or later [Source: https://www.logicmonitor.com/support/cisco-catalyst-sd-wan-monitoring].

The Overview / Dashboard pages summarize the whole fabric at a glance: the count of healthy versus unhealthy sites and devices, control-plane status (OMP and control connections), transport health (loss, latency, and jitter by circuit), and alarm counts by severity [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/Monitor-And-Maintain/monitor-maintain-book/vmanage-monitor-overview.html]. The analogy is an aircraft cockpit: a wall of green tells you to relax, while a single amber light tells you exactly where to look next.

A typical day-to-day monitoring flow runs like this:

1. Start on the main dashboard and scan for red or orange indicators under Network Health, Transport Health, or Alarms [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/Monitor-And-Maintain/monitor-maintain-book/vmanage-monitor-overview.html].
2. If a site looks degraded, drill into Monitor > Network > Devices and filter by model (for example, Catalyst 8300 or 8500) to see affected routers and their CPU, memory, interface, and tunnel status [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/Monitor-And-Maintain/monitor-maintain-book/m-dashboard-screen.html].
3. Use the time-range selector (last 1 hour / 24 hours) to distinguish a sustained trend from a momentary spike.
4. Use the Real Time view on a device page to confirm whether a problem is still happening right now — packet drops, BFD flaps, or high CPU.

Each Catalyst 8000 router also has its own device-level dashboard that aggregates the metrics that matter most:

Source for the device panels above: [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/Monitor-And-Maintain/monitor-maintain-book/m-dashboard-screen.html].

Alongside the dashboards, SD-WAN Manager generates alarms and logs events for device and network conditions [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/Monitor-And-Maintain/monitor-maintain-book/m-alarms-events-logs.html]. Examples include OMP Peer Down, BFD Session Down, high CPU or memory, and interface flaps. View active and historical alarms with severity filters under Monitor > Alarms, and detailed timestamped history under Monitor > Events / Logs. This is what lets you turn a vague user report (“the branch was down around 10:05”) into a precise correlation with the exact control-plane, tunnel, or interface event that caused it.

Many organizations supplement SD-WAN Manager by polling routers via SNMP and pulling SD-WAN-specific data through SD-WAN Manager APIs, feeding both into external observability platforms for consolidated dashboards and alerting [Source: https://www.logicmonitor.com/support/cisco-catalyst-sd-wan-monitoring] [Source: https://github.com/tamersaid2022/sdwan-health-monitor].

Key Takeaway: SD-WAN Manager is your single pane of glass — start at the fabric-wide dashboard, drill into the per-device panels (system, control, BFD, interface, VPN), and use alarms and events to correlate incidents with precise timestamps.

Model-driven telemetry

Dashboards are excellent for human eyes, but their charts are coarse and SNMP polling is periodic. To catch short-lived events — a five-second burst of loss, a brief CPU spike — you need higher-resolution data. That is the role of Model-Driven Telemetry (MDT): a streaming framework that continuously pushes device data to external collectors using YANG data models, rather than relying on periodic polling [Source: https://www.cisco.com/c/en/us/products/collateral/switches/catalyst-9300-series-switches/model-driven-telemetry-wp.html].

The key concepts are:

• Subscriptions define which YANG data paths (sensors) to monitor — interface counters, QoS queues, routing, or platform health [Source: https://www.cisco.com/c/en/us/td/docs/iosxr/cisco8000/telemetry/24xx/configuration/guide/b-telemetry-cg-8000-24xx/dial-out-telemetry-session-from-router-to-destination.html].
• Encodings and transports: data is typically sent over gRPC/gNMI or TCP using encodings such as GPB or JSON [Source: https://www.cisco.com/c/en/us/td/docs/iosxr/cisco8000/telemetry/711x/configuration/guide/b-telemetry-cg-8000-711x/dial-in-telemetry-session-from-destination-to-router.html].
• Dial-out sessions: the router pushes updates at configured intervals to one or more collectors (the router initiates). In dial-in, the collector initiates the subscription toward the router [Source: https://www.cisco.com/c/en/us/td/docs/iosxr/cisco8000/telemetry/24xx/configuration/guide/b-telemetry-cg-8000-24xx/dial-out-telemetry-session-from-router-to-destination.html].

If SNMP is taking a photograph every five minutes, MDT is shooting continuous video — you see the moment of the spike, not just a smoothed-over average.

A practical telemetry pattern for a Catalyst 8000 SD-WAN router:

1. Identify critical sensors — interface counters (bytes, packets, drops, errors), CPU and memory, BFD/tunnel SLA statistics, and app-aware routing KPIs (loss, latency, jitter per SLA class).
2. Create subscriptions — for example, a 30-second sample interval for general system health and a tighter 10-second interval for critical tunnel/BFD sensors.
3. Configure a dial-out collector destination — stream to a metrics/TSDB stack such as InfluxDB or Prometheus (via an agent like Telegraf), or a commercial NPM/APM platform [Source: https://www.cisco.com/c/en/us/td/docs/iosxr/cisco8000/telemetry/24xx/configuration/guide/b-telemetry-cg-8000-24xx/dial-out-telemetry-session-from-router-to-destination.html].
4. Verify the subscription — confirm the router is actively streaming and the collector is receiving and parsing the data.

Two cautions: over-sampling (for example, 1-second intervals across many sensors) can load both the router and the collector, so balance fidelity against overhead; and because telemetry exposes detailed operational data, protect the transport with TLS and access controls [Source: https://www.cisco.com/c/en/us/products/collateral/switches/catalyst-9300-series-switches/model-driven-telemetry-wp.html]. Treat telemetry as a complement to SD-WAN Manager, not a replacement — Manager still provides the overlay control context that raw telemetry cannot.

Key Takeaway: Model-driven telemetry streams YANG-modeled sensor data over gRPC/gNMI at sub-minute intervals, giving you the high-resolution history needed to catch microbursts and brief spikes that coarse dashboards and SNMP polling miss.

ThousandEyes integration

Dashboards and telemetry tell you how the fabric is behaving, but not necessarily how it feels to a user reaching Microsoft 365 across the public internet. ThousandEyes closes that gap with synthetic, end-to-end testing. Its Enterprise Agents can run directly on supported Cisco devices — including Catalyst 8000 edge routers — and be orchestrated through SD-WAN Manager [Source: https://thousandeyes.github.io/enna-study-guide/platforms-and-architecture/cisco-integration/].

The core ideas:

• Enterprise Agents on the router: SD-WAN Manager can install a ThousandEyes Enterprise Agent on selected Catalyst 8000 routers, letting the device itself initiate synthetic tests toward SaaS apps and other targets [Source: https://thousandeyes.github.io/enna-study-guide/platforms-and-architecture/cisco-integration/].
• Joint dashboards: ThousandEyes provides path visualization plus DNS and HTTP tests, while SD-WAN Manager and vAnalytics show overlay routing, SLA decisions, and tunnel health [Source: https://www.youtube.com/watch?v=sVg7yqm_7A4].
• WAN Insights: ThousandEyes WAN Insights uses SD-WAN telemetry to produce proactive path recommendations, integrating with vAnalytics and SD-WAN Manager [Source: https://docs.thousandeyes.com/product-documentation/wan-insights/introducing/wan-insights-vanalytics-vmanage].

Worked example — slow Office 365 from a Catalyst 8300 branch. Users complain that Office 365 feels sluggish, but ping looks fine and the branch has both MPLS and direct internet access (DIA).

1. From the embedded ThousandEyes agent, run HTTP and network tests to Office 365 targets. The branch LAN and overlay look normal, but the tests show high latency and retransmits in the upstream ISP segment near the SaaS provider [Source: https://www.youtube.com/watch?v=sVg7yqm_7A4].
2. In SD-WAN Manager, the transport dashboard shows internet-tunnel loss spiking for that branch at the same time [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/Monitor-And-Maintain/monitor-maintain-book/vmanage-monitor-overview.html].
3. WAN Insights suggests an alternate path (backup ISP or MPLS) to meet the SLA [Source: https://docs.thousandeyes.com/product-documentation/wan-insights/introducing/wan-insights-vanalytics-vmanage].
4. You adjust the SLA class or preferred path for Office 365 in SD-WAN policy, then re-check both ThousandEyes metrics and SD-WAN transport health to confirm the improvement.

The power here is correlation: ThousandEyes tells you where on the end-to-end path the problem lives (branch, ISP edge, backbone, or SaaS edge), and SD-WAN Manager tells you whether the overlay reacted correctly. Scope test frequency carefully to avoid overloading routers or collectors, and protect agents with proper authentication [Source: https://thousandeyes.github.io/enna-study-guide/platforms-and-architecture/cisco-integration/].

Key Takeaway: ThousandEyes Enterprise Agents run on the Catalyst 8000 itself to measure real end-to-end experience toward SaaS targets; correlating their path-aware findings with SD-WAN Manager and WAN Insights pinpoints whether a problem is in the branch, the underlay, or the application provider — and whether the overlay steered around it.

Troubleshooting

When something breaks, ad hoc poking wastes time. The discipline that consistently resolves Catalyst 8000 SD-WAN issues is to follow the dependency chain from the bottom up: prove the control plane first, then OMP and TLOCs, then tunnels, then BFD, and finally app-route/SLA policy [Source: https://www.cisco.com/c/en/us/support/docs/routers/sd-wan/214509-troubleshoot-control-connections.html] [Source: https://www.cisco.com/c/en/us/support/docs/routers/sd-wan/222302-troubleshoot-common-sd-wan-control-and-d.html]. If a lower layer is broken, every layer above it will look broken too — so always start at the foundation.

Figure 10.1: Bottom-up SD-WAN troubleshooting dependency chain

flowchart TD
    A["Layer 1: Control connections<br/>(DTLS/TLS to vBond/vSmart/vManage)"] --> B["Layer 2: OMP and TLOCs<br/>(overlay routing established)"]
    B --> C["Layer 3: Tunnels<br/>(IPsec/GRE per color)"]
    C --> D["Layer 4: BFD sessions<br/>(liveliness + loss/latency/jitter)"]
    D --> E["Layer 5: App-route / SLA policy<br/>(path selection)"]
    A -. "If broken, every layer above looks broken" .-> E

Control connection troubleshooting

A Catalyst 8000 cEdge cannot do anything useful in the overlay until it has stable control connections to the controllers. Begin there.

Step 1 — Verify current control connections.

Router# show sdwan control connections

This shows the current DTLS/TLS control connections to vBond, vSmart, and vManage [Source: https://www.cisco.com/c/en/us/support/docs/routers/sd-wan/214509-troubleshoot-control-connections.html]. The columns you read are the controller type and system-IP, the state (which should be up; intermediate values include connect, handshake, or certificate-verify), the local and remote public/private IPs and ports, the protocol (DTLS or TLS), and uptime. Quick interpretation:

• No entries, or all down/connect → likely an underlay reachability, DNS, or controller-IP problem.
• Only vBond up, no vSmart/vManage → the device reaches vBond but fails full onboarding (certificate, organization, or policy).
• Frequent flaps with very short uptime → underlay packet loss, MTU, firewall/ALG, or an unstable WAN link.

Step 2 — Check local SD-WAN parameters.

Router# show sdwan control local-properties

This shows the router’s own SD-WAN identity: system-ip, site-id, organization-name, the vbond address, and certificate state [Source: https://www.cisco.com/c/en/us/support/docs/routers/sd-wan/214509-troubleshoot-control-connections.html]. Verify that system-ip matches what the controller expects, site-id is unique in the fabric, organization-name is an exact, case-sensitive match to the value in Manager, the vBond address is correct and reachable, and certificates are installed, valid, and trusted. A mismatched organization name or wrong system-IP causes certificate validation to fail and leaves connections stuck at certificate-verify or connect; a duplicate site-ID makes connections flap or appear only intermittently [Source: https://www.cisco.com/c/en/us/support/docs/routers/sd-wan/222302-troubleshoot-common-sd-wan-control-and-d.html].

Step 3 — Read the history.

Router# show sdwan control connections-history

The history table gives you the exact reason codes and timestamps — Certificate Validation Failure, DTLS negotiation failed, TCP timeout, No route to peer — which are far more actionable than the current state alone [Source: https://www.cisco.com/c/en/us/support/docs/routers/sd-wan/214509-troubleshoot-control-connections.html]. Repeated Certificate Validation Failure points you at certificates, org-name/system-IP mismatches, or clock skew; transport errors point you at firewalls, NAT, MTU, and path stability.

Step 4 — Validate certificates and time. Use show sdwan certificate to confirm certificates are installed, valid, and signed by the same root CA as the controllers, and confirm NTP/clock is synchronized — clock skew can make a valid certificate appear expired or not-yet-valid [Source: https://www.cisco.com/c/en/us/support/docs/routers/sd-wan/214509-troubleshoot-control-connections.html]. Certificate problems, organization-name mismatches, and clock skew are the dominant root causes of control-connection failures.

Step 5 — Confirm underlay reachability. Use classic IOS XE tools — ping, traceroute, and show ip route toward the controller IP shown by show sdwan control connections — and confirm any NAT/firewall in the path permits the relevant DTLS/TLS ports [Source: https://www.cisco.com/c/en/us/support/docs/routers/sd-wan/222302-troubleshoot-common-sd-wan-control-and-d.html].

Once you have made a fix, do not wait for timers — force a clean renegotiation:

Router# clear sdwan control connections
Router# clear sdwan control connection-history

It is also worth confirming OMP itself, because control can be “up” while OMP is not, which leaves the data plane non-functional:

Router# show sdwan omp peers
Router# show sdwan omp tlocs

Figure 10.2: Control connection troubleshooting decision flow

flowchart TD
    Start(["show sdwan control connections"]) --> Q1{"Any connections up?"}
    Q1 -- "None / all down" --> R1["Suspect underlay reachability,<br/>DNS, or controller-IP problem"]
    Q1 -- "Only vBond up" --> R2["Onboarding failure:<br/>certificate, org-name, or policy"]
    Q1 -- "Frequent flaps,<br/>short uptime" --> R3["Underlay loss, MTU,<br/>firewall/ALG, unstable WAN"]
    R1 --> Local["show sdwan control local-properties"]
    R2 --> Local
    R3 --> Local
    Local --> Q2{"system-IP, site-ID,<br/>org-name, vBond,<br/>certs all correct?"}
    Q2 -- "No" --> Fix1["Correct identity / cert mismatch"]
    Q2 -- "Yes" --> Hist["show sdwan control connections-history"]
    Hist --> Q3{"Reason code?"}
    Q3 -- "Certificate Validation Failure" --> Cert["show sdwan certificate<br/>+ verify NTP / clock skew"]
    Q3 -- "TCP timeout / No route" --> Under["ping / traceroute / show ip route<br/>check NAT, firewall, ports"]
    Fix1 --> Clear["clear sdwan control connections"]
    Cert --> Clear
    Under --> Clear
    Clear --> Done(["Confirm OMP:<br/>show sdwan omp peers / tlocs"])

Key Takeaway: Always fix the control plane first. show sdwan control connections shows current state, show sdwan control local-properties catches identity/cert mismatches, and show sdwan control connections-history gives the exact failure reason — with certificate errors, case-sensitive org-name mismatches, duplicate site-IDs, and clock skew as the usual culprits.

Data plane (BFD/IPsec) issues

Once control connections and OMP are stable, BFD sessions for the SD-WAN IPsec tunnels should form automatically. BFD (Bidirectional Forwarding Detection) is enabled by default on SD-WAN connections and monitors both tunnel liveliness and performance, feeding the loss/latency/jitter measurements that drive application-aware routing [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/26x-later/high-availability/high-availability-book-xe/ha.html].

Step 1 — Baseline the sessions.

Router# show sdwan bfd sessions

This shows per-TLOC tunnel state: local and remote system-IP, local TLOC color (such as mpls, biz-internet, public-internet), transport/encap (IPsec or GRE), the state (should be up), and packet counters [Source: https://www.cisco.com/c/en/us/support/docs/routers/sd-wan/222302-troubleshoot-common-sd-wan-control-and-d.html]. A crucial diagnostic shortcut: if BFD is down while control to the same peer is up, the problem is in the data plane — the tunnel, crypto, or underlay — not the control plane.

Figure 10.3: Data-plane (BFD/IPsec) troubleshooting flow

flowchart TD
    Start(["show sdwan bfd sessions"]) --> Q1{"BFD up to peer?"}
    Q1 -- "Up" --> SLA["show sdwan app-route statistics"]
    SLA --> Q4{"Chronic high loss<br/>on a path?"}
    Q4 -- "Yes" --> Steer["Policy may steer away<br/>despite tunnel alive"]
    Q4 -- "No" --> OK(["Data plane healthy"])
    Q1 -- "Down (control up)" --> DP["Data-plane problem:<br/>tunnel, crypto, or underlay"]
    DP --> Tun["show sdwan tunnel"]
    Tun --> Q2{"Tunnel building?"}
    Q2 -- "No" --> Underlay["Troubleshoot underlay for color:<br/>routes, NAT, firewall ports, MTU"]
    Q2 -- "Yes" --> Hist["show sdwan bfd history"]
    Hist --> Q3{"Pattern?"}
    Q3 -- "Repeated timeout / loss" --> Path["Underlay instability or<br/>firewall dropping keepalives"]
    Q3 -- "Stuck down across<br/>many sites / colors" --> Bug["Suspect software defect:<br/>check release notes, plan upgrade"]

Step 2 — Correlate with the tunnel.

Router# show sdwan tunnel

Confirm an IPsec/GRE tunnel exists for each expected color/TLOC, check its up/down state and drop counters, and verify NAT detection and public/private IPs match the control-plane view [Source: https://www.cisco.com/c/en/us/support/docs/routers/sd-wan/222302-troubleshoot-common-sd-wan-control-and-d.html]. If tunnels are not building, BFD will correspondingly stay down.

Step 3 — Read BFD history.

Router# show sdwan bfd history
Router# clear sdwan bfd history

History tells you exactly when sessions transitioned and why (timeout, loss, admin-down where the release reports it), which distinguishes a persistent config problem (sessions never come up) from an intermittent path problem (sessions flap) [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/command/iosxe/qualified-cli-command-reference-guide/m-troubleshooting-commands.html]. Clear the history, reproduce the issue, and re-read for clean events.

Step 4 — Relate BFD to SLA policy.

Router# show sdwan app-route statistics

These per-path loss, latency, and jitter values are derived from BFD and decide path selection [Source: https://www.cisco.com/c/en/us/support/docs/routers/sd-wan/222302-troubleshoot-common-sd-wan-control-and-d.html]. If sessions are up but app-route statistics show chronic high loss on a path, performance policy may be steering traffic away even though the tunnel is technically alive.

Step 5 — Suspect a software defect for “stuck” BFD. There are documented cases on specific IOS XE SD-WAN releases where control connections and configuration are entirely correct, yet all BFD sessions remain stuck down, and the fix is upgrading to a corrected release [Source: https://www.thenetworkdna.com/2023/12/cisco-catalyst-sd-wan-ios-xe-router-bfd.html]. So if control is clean and stable, underlay and IPsec are correct, yet BFD stays down across multiple sites or colors, check the release notes and plan an upgrade to a recommended fixed version.

Worked example — control up, BFD down to one site. show sdwan control connections shows vBond/vSmart/vManage all up, and show sdwan bfd sessions shows most peers up but one remote site/color down. Check show sdwan tunnel for that path; if the tunnel is down, troubleshoot the underlay for that color (routes, NAT, firewall ports, MTU). Review show sdwan bfd history for that peer — repeated timeout/loss points to underlay instability or a firewall dropping keepalives. Verify the color, TLOC-extension, and VPN settings at the remote site match the design. If everything checks out but BFD still misbehaves across multiple routers on the same software train, suspect a software bug and consider an upgrade [Source: https://www.thenetworkdna.com/2023/12/cisco-catalyst-sd-wan-ios-xe-router-bfd.html].

Key Takeaway: After control is stable, validate the data plane with show sdwan bfd sessions, then show sdwan tunnel, then show sdwan bfd history. BFD down while control is up means a data-plane problem; widespread “stuck” BFD on otherwise healthy fabrics can be a software defect that an upgrade resolves.

Useful show and admin-tech commands

Some problems — intermittent flaps, crashes, unexplained tunnel failures — outrun what live show commands and dashboards reveal. For these you collect an admin-tech bundle: a comprehensive snapshot containing configuration, routing tables, control-connection state, log files, core files, and system information [Source: https://www.thenetworkdna.com/2025/03/how-to-generate-admin-tech-in-cisco-sd.html]. Admin-tech bundles can be generated for SD-WAN Manager itself and for any individual device (vSmart, vBond, or a Catalyst 8000 edge), and they are the primary artifact Cisco TAC requests for difficult cases [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/Monitor-And-Maintain/monitor-maintain-book/m-alarms-events-logs.html].

To generate one from SD-WAN Manager:

1. Select Tools > Operational Commands.
2. Choose Generate Admin Tech for Manager to collect controller-side diagnostics, or select an individual router and choose Generate admin-tech file, then click Generate [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/Monitor-And-Maintain/monitor-maintain-book/m-alarms-events-logs.html].
3. Wait for generation — often around 10 minutes depending on system size and log volume — then download the file and attach it to your TAC case [Source: https://www.thenetworkdna.com/2025/03/how-to-generate-admin-tech-in-cisco-sd.html].

Two practical tips. First, generate admin-tech while the problem is occurring, or as close as possible, so the logs capture the relevant events. Second, for control-plane problems (OMP/BFD flaps), collect admin-tech from the affected Catalyst 8000 and from vSmart, vBond, and SD-WAN Manager, since a control issue can originate anywhere along that chain [Source: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/Monitor-And-Maintain/monitor-maintain-book/m-alarms-events-logs.html] [Source: https://www.youtube.com/watch?v=AvGpSMDRpbg].

The table below consolidates the operator’s CLI toolkit for this chapter:

Key Takeaway: When live commands run out of detail, generate an admin-tech bundle via Tools > Operational Commands — while the problem is live and from every device in the control chain — and attach it to your TAC case as the primary diagnostic artifact.

Lifecycle Management

A fabric is never “finished.” Software releases mature and retire, hardware reaches end of sale, and the safe-and-supported version of today becomes the unpatched liability of tomorrow. Lifecycle management is the practice of renewing the platform on a deliberate schedule rather than reacting to an outage or a security advisory.

Software upgrade workflows

Catalyst 8000 platforms (C8200/C8300/C8500/C8000V) run IOS XE in install mode, where a .bin image is unpacked into packages and packages.conf becomes the boot file [Source: https://www.cisco.com/c/en/us/td/docs/routers/C8000V/Configuration/c8000v-installation-configuration-guide/m_installing-the-software-using-install-commands.html]. SD-WAN images are IOS XE variants with the SD-WAN personality enabled, but they follow the same install and rollback mechanisms.

Before any upgrade, capture the current state with show version and show install summary (to confirm install mode and the active package set), back up the running configuration, read the target release notes for caveats and minimum ROMMON, and check free space with dir bootflash: [Source: https://www.cisco.com/c/en/us/support/docs/routers/catalyst-8300-edge-platform/225727-upgrade-software-on-catalyst-8000-edge.html]. These prerequisites prevent the classic failures: “not enough space,” ROMMON incompatibility, or an unsupported hardware module on the new image.

Upgrading via SD-WAN Manager (recommended for SD-WAN deployments). Manager encapsulates best practice for fleets of routers:

1. Stage the image. Go to Maintenance > Software Repository > Software Images, click Add, and upload the .bin (validating checksum). The repository avoids pushing large images repeatedly over the WAN [Source: https://www.scribd.com/document/784316970/SDWAN-Best-practice].
2. Scope the job. Group devices by site/region and role, and follow the Cisco upgrade sequence: upgrade the controllers (SD-WAN Manager, vSmart, vBond) first, then the edges [Source: https://www.lookingpoint.com/blog/upgrading-cisco-sd-wan-vmanage-vsmart-vbond-and-vedge].
3. Execute. Go to Maintenance > Software Upgrade, select the device(s), choose the repository image, optionally enable pre/post checks and a download-only pre-stage, then start or schedule the job [Source: https://www.scribd.com/document/784316970/SDWAN-Best-practice]. Under the hood, Manager pushes the image to bootflash, runs the install-mode unpack and activate, triggers one reload, and monitors control connections after boot.

Upgrading via CLI (install mode). For autonomous routers or fine-grained control, the canonical worked example is:

Router# copy scp://user@server/path/c8000-universalk9_ias.XX.Y.Z.SPA.bin bootflash:
Router# verify /md5 bootflash:c8000-universalk9_ias.XX.Y.Z.SPA.bin
Router# install add file bootflash:c8000-universalk9_ias.XX.Y.Z.SPA.bin activate commit

Here add loads the image into the install repository, activate switches the active package set, and commit makes it the permanent boot set; the router then reloads once into the new image [Source: https://www.cisco.com/c/en/us/td/docs/routers/C8000V/Configuration/c8000v-installation-configuration-guide/m_installing-the-software-using-install-commands.html]. For more control, run the three phases separately — install add file ..., then install activate, then install commit after validation.

Figure 10.4: Install-mode upgrade and rollback state machine

stateDiagram-v2
    [*] --> Committed: Current image
    Committed --> Added: install add file
    Added --> Activated: install activate (reload)
    Activated --> NewCommitted: install commit
    Activated --> Committed: install rollback to committed (reload)
    NewCommitted --> [*]: New image permanent
    NewCommitted --> Committed: install rollback to <id> (reload)

    note right of Added
        Image unpacked into
        install repository
    end note
    note right of Activated
        New package set running,
        not yet permanent
    end note

Minimizing downtime. True zero-downtime ISSU is limited on the single-RP branch routers common in Catalyst 8000 deployments [Source: https://www.samuraj-cz.com/en/article/cisco-ios-26-ios-xe-upgrade-standalone-switch-stack-and-issu/]. Instead, minimize disruption through high-availability design — deploy branches in pairs (for example, dual C8300s) with dual SD-WAN tunnels so traffic shifts to the peer during a reload — fast-converging protocols, and draining traffic from a router via SD-WAN policy before upgrading it in small, verified batches [Source: https://www.scribd.com/document/784316970/SDWAN-Best-practice].

Key Takeaway: Catalyst 8000 routers upgrade in install mode via install add file ... activate commit (one reload); prefer SD-WAN Manager’s repository and staged upgrade jobs for fleets, always run prerequisite checks, upgrade controllers before edges, and drain traffic in HA pairs to minimize downtime.

Backup and recovery

Every upgrade needs a safety net, and IOS XE install mode provides two independent ones: software rollback and configuration rollback.

Software rollback reverts to a previously committed image:

Router# show install rollback
Router# install rollback to committed

You can also roll back to a specific stored instance with install rollback to <id>; rollback restores the earlier package set and reloads once [Source: https://www.cisco.com/c/en/us/td/docs/routers/C8000V/Configuration/c8000v-installation-configuration-guide/m_installing-the-software-using-install-commands.html]. Because SD-WAN Manager drives the same underlying install operations, this safety net is available whether you upgraded through Manager or the CLI.

Configuration rollback is separate and equally important. IOS XE supports checkpoints and archive-based workflows: save a checkpoint before the change, keep the new config as the baseline after validation, and if something regresses, use configure replace or the rollback features to restore the prior config [Source: https://www.reddit.com/r/networking/comments/df5for/cisco_config_staging_and_rollback/]. Combining software rollback with configuration rollback lets you revert both image and config to a known-good state — the seatbelt and the airbag working together.

For the worst case, where an image is corrupted and the router will not boot, recovery uses ROMMON with USB or TFTP to load a known-good IOS XE image, after which you re-enter install mode [Source: https://www.cisco.com/c/en/us/td/docs/routers/C8000V/Configuration/c8000v-installation-configuration-guide/m_installing-the-software-using-install-commands.html].

Key Takeaway: Treat rollback as a planned capability, not an afterthought — install rollback to committed reverts the image, configuration archives and configure replace revert the config, and ROMMON with USB/TFTP recovers a router that will not boot.

End-of-life planning and roadmap

Software ages on a published schedule, and ignoring it is how organizations end up running unpatched, unsupported code in production. Cisco’s Software Lifecycle Support Statement for IOS XE defines how long each release receives maintenance and security updates; from IOS XE 26.1.1, Cisco plans roughly two releases per year with defined lifecycle stages [Source: https://www.cisco.com/c/en/us/products/collateral/ios-nx-os-software/ios-xe-26/bulletin-c25-2378701.html].

The stages, and what they mean for you:

Sources for lifecycle stages: [Source: https://www.cisco.com/c/en/us/products/collateral/ios-nx-os-software/ios-xe-26/bulletin-c25-2378701.html] [Source: https://www.router-switch.com/faq/what-breaks-when-ios-xe-is-no-longer-updated.html]. As a concrete data point, some 17.x trains already carry EoL dates — for example, IOS XE 17.17 has an EoL date of March 31, 2026 [Source: https://eosl.date/eol/product/cisco-ios-xe/].

Figure 10.5: IOS XE release lifecycle stages

timeline
    title IOS XE Release Lifecycle Progression
    Active maintenance : Bug fixes, security patches, limited new features : Preferred state for production
    Maintenance-only / extended : Critical bug fixes and security updates only : Plan your successor release
    End of Software Maintenance (EoSM) : No routine fixes : Migrate now; severe issues may go unaddressed
    End of Support (EoS) : No TAC support or updates : Running here means unpatched vulnerabilities

A practical lifecycle strategy:

1. Maintain an inventory mapping each Catalyst 8000 to its current IOS XE release and image type (SD-WAN or autonomous), hardware model, and serial — exportable directly from SD-WAN Manager.
2. Look up status and dates using Cisco’s lifecycle resources: release status (Active / Maintenance / EoSM), version EoL/EoS dates, and the hardware platform EoL index [Source: https://www.cisco.com/c/en/us/support/eol/index.html].
3. Standardize on a currently recommended “safe/preferred” or extended-support release.
4. Plan with lead time — leave months before EoSM/EoS to test and roll out a successor, aligned to maintenance windows [Source: https://eosl.date/eol/product/cisco-ios-xe/].

Running past EoSM/EoS leaves you exposed to unpatched vulnerabilities and risks incompatibility with newer controllers and management tools [Source: https://www.router-switch.com/faq/what-breaks-when-ios-xe-is-no-longer-updated.html].

Worked example — fleet migration off an aging release. A deployment of C8300 branch routers in SD-WAN mode sits on an IOS XE SD-WAN version nearing EoSM, and you need to move to a supported successor. (1) Assess by exporting inventory and image versions from SD-WAN Manager and confirming lifecycle status and EoL/EoS timelines. (2) Select and lab-test a recommended target release, upgrading a pilot C8300 via SD-WAN Manager and validating VPN, QoS, voice, and application steering. (3) Stage the image into the software repository and use download-only jobs to pre-position it on branch routers. (4) Roll out in batches — five pilot branches in week 1, larger staggered groups in weeks 2-4 — draining traffic from each dual-router branch via SD-WAN policy before upgrading and watching job logs and router health. (5) Keep rollback ready with install rollback to committed and configuration archives in case a critical issue surfaces [Source: https://www.cisco.com/c/en/us/td/docs/routers/C8000V/Configuration/c8000v-installation-configuration-guide/m_installing-the-software-using-install-commands.html] [Source: https://www.scribd.com/document/784316970/SDWAN-Best-practice].

Key Takeaway: Anchor upgrade planning to Cisco’s published IOS XE lifecycle stages (Active → Maintenance → EoSM → EoS), keep an accurate inventory mapped to EoL/EoS dates, standardize on a supported release, and migrate in tested batches with months of lead time before maintenance ends.

Chapter Summary

This chapter completed the operational lifecycle of a Catalyst 8000 edge fabric, moving from seeing to fixing to renewing.

On monitoring, you saw how Cisco Catalyst SD-WAN Manager acts as the single pane of glass — fabric-wide dashboards, per-device health panels, and an alarms/events trail that ties incidents to precise timestamps. Model-driven telemetry adds high-resolution streaming visibility over gRPC/gNMI to catch what coarse charts miss, and ThousandEyes Enterprise Agents running on the routers themselves measure true end-to-end user experience, with WAN Insights closing the loop back into the overlay.

On troubleshooting, you internalized the one rule that makes everything else efficient: follow the dependency chain from the bottom up — control connections, then OMP/TLOCs, then tunnels, then BFD, then SLA policy. You have the exact commands for each layer (show sdwan control connections, show sdwan control local-properties, show sdwan bfd sessions, show sdwan tunnel, and their history and clear variants), the knowledge that BFD-down-while-control-up means a data-plane problem, and the admin-tech workflow for escalating the hard cases to TAC.

On lifecycle management, you learned the install-mode upgrade path (install add file ... activate commit), the dual safety nets of software and configuration rollback, the high-availability patterns that minimize upgrade downtime, and the discipline of mapping every router to Cisco’s published IOS XE lifecycle stages so you migrate before — not after — support ends.

Forward-looking synthesis and next steps. As the closing chapter of this guide, it is worth seeing how these operational practices and everything that came before form a single arc. The platforms and interfaces you studied early on, the SD-WAN overlay and policy you built in the middle chapters, and the security you layered on top all converge here, in the daily work of keeping the network healthy and current. The clear trajectory across the industry is toward proactive, data-driven operations: telemetry feeding analytics, analytics feeding automation, and automation closing the loop — WAN Insights recommending a path change, an upgrade pipeline staging and validating releases, a lifecycle inventory flagging a router before its software retires. The natural next steps for a practitioner are to operationalize what you have learned: stand up a telemetry collector and define your critical subscriptions; pilot ThousandEyes agents on your most business-critical branches; codify your troubleshooting runbooks around the command sequences in this chapter; and build a living lifecycle inventory tied to Cisco’s EoL roadmap so upgrades become a scheduled routine rather than an emergency. A Catalyst 8000 fabric that is well-monitored, methodically troubleshot, and deliberately renewed is not just operational — it is resilient, and ready for whatever Tuesday afternoon brings.

Key Terms