This chapter is from the book
This chapter is from the book
This chapter is from the book
Cisco Next-Generation Multiservice Routers
For next-generation multiservice networks, routing platforms born and bred on the service pull of IP networking have the advantage. The greatest customer demand is for IP services. Networks built on IP are naturally multiservice-capable, given IP embellished data, VoIP, and video over IP convergence capabilities.
IP routing architecture has reached the hallowed five 9s of availability status, and representative platforms are faster, more scalable, and more service rich than any networking technology that has come before. Innovations such as MPLS have created the flexibility to combine both conventional and contemporary networking approaches, achieving more customer-service granularity in the process. The combination of distributed processing architectures, IP and hardware acceleration in programmable silicon, virtualization architecture, and continuous system software operations now deliver high-end, service provider IP routing platforms that are constant, flexible, affordable, and secure.
For high-end service provider multiservice routing, the notable products are the Cisco CRS-1 Carrier Routing System, the Cisco IOS XR Software, and the Cisco XR 12000/12000 Series Routers.
Cisco CRS-1 Carrier Routing System
Once you turn on the Cisco CRS-1 Carrier Routing System, you might never turn it off. Unlike many routers that have preceded the CRS-1 design, the CRS-1 is scalable and simple, continuous and adaptable, flexible and high performance. None of these individual characteristics of the CRS-1 compromises another, leading to new achievements in nondisruptive scalability, availability, and flexibility of the system. Using the CRS-1 Carrier Routing System, providers can visualize one network with many services and limitless possibilities.
Using a new Cisco IOS XR Software operating system that is also modular and distributed, the CRS-1 is the first carrier IP routing platform that can support thousands of interfaces and millions of IP routes, using a pay-as-you-grow architectural strategy. The CRS-1 blends some of the best of computing, routing, and programmable semiconductor and software architectures for a new, high-end routing system that you can use for decade-plus lifecycles.
Using the CRS-1's concurrent scalability, availability, and performance, you can use the CRS-1 to consolidate service provider point-of-presence (POP) designs, collapsing core, peering, and aggregation layers inside the covers of one system. Previous routing platforms had limitations in the number of peers, interfaces, or processing cycles, leading to network POP designs that layered functionality based on the performance constraints of the routing platforms. With the CRS-1, these limitations are removed—hardware works in concert with software for extensible convergence of network infrastructure and services. The CRS-1 represents the next-generation IP network core and is the foundation for IP/MPLS provider core consolidation.
CRS-1 Hardware Design
The CRS-1 hardware system design uses the primary elements of line card and fabric card shelves. Each type of shelf comprises a standard telecommunications rack from a dimensional footprint.
Line Card Shelf
Line card shelves support the routing processors, integrated fabric cards and the line card slots, each of which is capable of 40 Gbps performance. Known collectively as a Line Card Chassis, the chassis comes in either an 8-slot version or a 16-slot version.
Two Route Processors are installed per chassis, one for active and one for hot standby. The Route Processors have their own dedicated slots and don't subtract from the 8 or 16 potential line cards slots of either chassis. Each Line Card Chassis contains up to 8 fabric cards in the rear of the chassis to support the Benes switching fabric for a single shelf system configurations. Each line card is composed of rear-facing Interface Modules and front-facing Modular Services Cards connected together via a midplane design. The Line Card Chassis is where route processing, forwarding, and control-plane intelligence of the system resides.
Within each Line Card Chassis are 2 Route Processors, up to 16 Interface Modules, each pairing with 16 Modular Services Cards, and 8 fabric cards. Redundant fan trays, power supplies, and cable management complete the distinctive elements within Line Card Chassis.
Each Route Processor is made up of a symmetrical multiprocessing architecture based on a Dual PowerPC CPU complex with at least 4 GB of DRAM, 2 GB of Flash memory, and a 40 GB micro hard drive. One of the Route Processors operates in active mode with the other in hot standby. The Route Processors, along with system software, can provide nonstop forwarding (NSF) and stateful switchover (SSO) functions without losing packets. Another plus of the CRS-1 architecture is that any Route Processor can control any line card slot, on any Line Card Chassis in a multishelf system. Using features of the Cisco IOS XR Software operating system, Route Processors and line cards can be formed across the system chassis to create logical routers within the physical CRS-1 overall system. Any time that supplementary processing power is needed, the architecture supports the addition of distributed Route Processors, providing two additional Dual PowerPC CPU complexes with their associated DRAM, Flash, and hard drive.
To create a line card, a combination of Interface Modules and Modular Services Cards are used. The Interface Modules, also referred to as Physical Layer Interface Modules (PLIMs) contain the physical interface ports and hardware interface-specific logic. Interface Modules for the CRS-1 exist for OC-768c/STM-256x, OC-192c/STM-64c, OC-48c/STM-16c, and 10 Gigabit Ethernet. The Interface Modules, installed in the rear card cage of the Line Card Chassis, connect through the midplane to Modular Services Cards in the front card cage of the chassis.
The Cisco Modular Services Cards are made up of a pair of Cisco Silicon Packet Processors (SPPs), each of which is an array of 188 programmable Reduced Instruction Set Computer (RISC) processors. These SPPs are deployed two per Modular Services Card, with one for the input direction and one for output packet processing. The SPP is another key innovation, as the SPP architecture achieves 40 Gbps line rates with multiple services, offering new features through in-service software upgrades to the SPP. The Interface Module and the Modular Services Card work together as a pair to form a complete line card slot. The Modular Services Card interfaces with the fabric cards, using the switching fabric to reach other line cards or the Route Processor memory.
Fabric Chassis
The Fabric Chassis is used to extend the CRS-1 into a CRS-1 Multishelf System. Up to 8 Fabric Chassis can interconnect as many as 72 Line Card Chassis to create the maximum CRS-1 Multishelf System. The Fabric Chassis is used as a massively scalable stage 2 of the three-stage Benes switching fabric in a multishelf system configuration.
A switching fabric is a switch backplane, and many of the Cisco products use various types of switching fabrics to move packets between ingress interfaces and Route Processor memory and out to egress interfaces. For example, a crossbar fabric is a popular fabric used in many Cisco products, such as the 12000 series and the 7600 series. For hundreds or even thousands of interface ports, a crossbar switching mechanism becomes too expensive and scheduling mechanisms too complex.
Therefore, the CRS-1 implements a three-stage, dynamically self-routed, Benes topology cell-switching fabric. This fabric is a multistage buffered switching fabric that represents the lowest-cost N x N cell-switching matrix that avoids internal blocking. The use of a backpressure mechanism within the fabric limits the use of expensive off-chip buffer memory, instead making use of virtual output queues in front of the input stage. Packets are converted to cells, and these cells are used for balanced load distribution through the switch fabric. The cells are multipath routed between stages 1 and 2 and again between stages 2 and 3 to assist with the overall goal of a nonblocking switching architecture. The cells exit stage 3 into their destination line card slots where the Modular Services Cards reassemble these cells into the proper order, forming properly sequenced packets. The Benes topology switching fabric is implemented in integrated fabric cards for single shelf systems and additionally implemented as standalone Fabric Chassis in a multishelf system configuration. Each standalone Fabric Chassis can contain up to 24 fabric cards for stage 2 operation.
A CRS-1 Single-Shelf system will use integrated fabric cards within the Line Card Chassis that include all three stages within the card. In a CRS-1 Multishelf System, from one to eight CRS-1 Fabric Chassis are used to form stage 2 of the switching fabric, with stage 1 operating on the fabric card of the egress line card shelf and stage 3 operating on the ingress line card shelf across the fabric.
Figure 3-8 shows a conceptual diagram of the CRS-1 switching fabric. Physically, the Cisco CRS-1 fabric is divided into eight planes over which packets are divided into fixed-length cells and then evenly distributed. Within the planes, the three fabric stages—S1, S2, and S3—dynamically route cells to their destination slots, where the Modular Services Cards reassemble cells in the proper order to form properly sequenced packets.
)
Figure 3-8 One Plane of the Eight-Plane Cisco CRS-1 Switching Fabric
Together the Route Processors, fabric cards, Interface Modules, and Modular Services Cards work with the IOS XR operating system to create a routing architecture that is scalable from 640 Gbps to 92 Tbps (terabits per second) of performance. These capacities are accomplished through various configurations of a CRS-1 Multishelf System or a CRS-1 Single-Shelf System. The overall CRS-1 architectural design is conceptualized in Figure 3-9.
Cisco CRS-1 Multishelf System
The Cisco CRS-1 Multishelf Systems are constructed using a combination of Line Card Chassis and Fabric Chassis. Up to 72 Line Card Chassis can be interconnected with 8 Fabric Chassis to create a multishelf system with as many as 1,152 line card slots, each capable of 40 Gbps, yielding approximately 92 Tbps (full duplex) of aggregate performance capacity. Cisco CRS-1 Multishelf Systems can start with as few as 2 Line Card Chassis and 1 Fabric Chassis and grow as demand occurs.
)
Figure 3-9 Cisco CRS-1 Hardware Architecture
Within a multishelf system, any Route Processor can control any line card on any Line Card Chassis in the system. For example, a Route Processor in Line Card Chassis number 1 can be configured to control a line card in Line Card Chassis number 72 using the Fabric Chassis as an internal connectivity path. Route Processors and distributed Route Processors are responsible for distributing control plane functions and processing for separation, performance, or logical routing needs.
Using a Cisco CRS-1 Multishelf System, providers can achieve the following configurations:
- 2 to 72 Line Card Chassis
- 1 to 8 Fabric Chassis
- Switching capacity from 640 Gbps to 92 Tbps (full duplex)
- Support for up to 1,152 line cards at 40 Gbps each
- 1,152 OC-768c/STM-256c POS ports
- 4,608 OC-192c/STM-64c POS/DPT ports
- 9,216 10 Gigabit Ethernet ports
- 18,432 OC-48c/STM-16c POS/DPT ports
Cisco CRS-1 16-Slot Single-Shelf System
The CRS-1 Single-Shelf Systems come as either a 16-slot or an 8-slot Line Card Chassis. Single-shelf systems use integrated Switch Fabric Cards (SFCs), installed in the rear card cage of the Line Card Chassis rather than using a standalone Fabric Chassis. In a single-shelf system configuration, the integrated SFCs perform all three stages of the Benes topology switching fabric operation. Using a Cisco CRS-1 16-Slot Single-Shelf System, providers can achieve the following configurations:
- 16-slot Line Card Chassis with integrated fabric cards
- Switching capacity to 1.28 Tbps (full duplex)
- Support for up to 16 line cards at 40 Gbps each
- 16 OC-768c/STM-256c POS ports
- 64 OC-192c/STM-64c POS/DPT ports
- 128 10 Gigabit Ethernet ports
- 256 OC-48c/STM-16c POS/DPT ports
Cisco CRS-1 8-Slot Single-Shelf System
The CRS-1 Single-Shelf Systems also come in an 8-slot Line Card Chassis. The 8-slot Line Card Chassis is one half as tall as a 16-slot Line Card Chassis. As previously mentioned, single-shelf systems use the integrated SFCs, installed in the rear card cage of the Line Card Chassis, performing all three stages of the Benes topology switching fabric operation. Using a Cisco CRS-1 8-Slot Single-Shelf System, providers can achieve the following configurations:
- 8-slot Line Card Chassis with integrated fabric cards
- Switching capacity to 640 Gbps (full duplex)
- Support for up to 8 line cards at 40 Gbps each
- 8 OC-768c/STM-256c POS ports
- 32 OC-192c/STM-64c POS/DPT ports
- 64 10 Gigabit Ethernet ports
- 128 OC-48c/STM-16c POS/DPT ports
Cisco IOS XR Software
The Cisco IOS XR Software is likely to be one of the most important technology innovations of this decade. Benefiting from over 20 years of IOS development and experience, the Cisco IOS XR answers the following questions:
- "Why can't a router platform be divided into separate physical and logical partitions as the computer industry has done with mainframes for many years?" Now it can.
- When presented with the question, "Why can't a router's control plane be separated to individually manage, restart, and upgrade software images without risk to other partitions?" With IOS XR, now you can.
- When the inquiry is made as to "when will a router support five nines of reliability?" With IOS XR in use, now it does.
IOS XR answers these questions and more with massive scalability; a high-performance, distributed processing, multi-CPU optimized architecture; and continuous system operation. With IOS XR in a CRS-1 Multishelf System, distributed processing intelligence can take full advantage of hardware interface densities and symmetric multiprocessing power, scaling up to 92 Tbps per multishelf system. IOS XR is built on a QNX microkernel operating system with memory protection that places strict logical boundaries around subsystems to ensure independence, isolation, and optimization. Only the essential operating functions reside in the kernel to strengthen this key element of the overall software system.
Through the ability to distribute processes and subsystems anywhere across CRS-1 hardware resources, the IOS XR can dedicate processing, protected memory, and control functions to these resources—creating not only logical routers, but resource-allocated physical routers as well. This leads to the ability to partition operations such that a production routing system and a development routing system can reside on the same physical system. This can become an opportunity to market to a sophisticated customer both a production networking service for mission-critical applications, as well as a development networking partition where new features can be developed and tested without the consequences of impacting mission-critical applications. Or a provider can run multiple MPLS administrative domains on the same physical system, each with attributes and software characterized to a leading edge, edge, or lagging edge type of network service, applying more granularity to customer risk and choice. The separation architecture of IOS XR blended with hardware platforms provides flexibility in IP network design for providers.
With IOS XR, multiple partitions can mean multiple software versions running on the same physical system chassis. IOS software levels are distributed in a modular fashion, allowing for software patches and bug fixes in one partition without affecting others. This takes on an in-service upgrade approach, as each partition process can be restarted without affecting the other running systems and their respective routing topology.
In today's networks, security and reliability are mutual. Perhaps one of the greatest benefits of the IOS XR's isolatable architecture is the ability to resist malicious attacks, such as TCP/IP-based denial of service and distributed denial of service threats. Even if a TCP/IP subsystem were to be compromised, a compromised TCP subsystem would run outside of the IOS XR system kernel, so the IOS XR system kernel and other protected subsystem processes would continue to operate. The Cisco IOS XR Software architecture is conceptualized in Figure 3-10.
)
Figure 3-10 Cisco IOS XR Software Architecture
The Cisco IOS XR Software assists with making the latest high-end routing systems more scalable, flexible, reliable, and secure. The Cisco IOS XR Software is perhaps the prime catalyst for next-generation IP/MPLS networks that can now operate on a worldwide scale. For a full listing of features and functions, examine the various Cisco CRS-1 and IOS XR information found at http://www.cisco.com/go/crs.
Cisco XR 12000/12000 Series Routers
The Cisco XR 12000 Series Routers are so named because they combine the innovative features of the Cisco IOS XR Software with the superior heritage of the Cisco 12000 Series routing platforms. The Cisco XR 12000/12000 Series Routers are optimally positioned for the next-generation core and edge of provider networks, with a strength in multiservice edge consolidation. The XR 12000s are optimized to run the Cisco IOS XR Software, while the 12000s are the original 12000 series running the Cisco IOS software.
Using the Cisco IOS XR Software with the distributed architecture of the XR 12000, the XR 12000 routers achieve both logical and physical routing functionality that can operate independently within a single XR 12000 chassis. A private MPLS VPN service could be completely isolated from a public Internet service for security but also operationally separate. For example, an anomaly affecting the public Internet service might result in a need to restart that service within the router; however, this action wouldn't affect the private MPLS VPN service running as a separate process. There are four primary elements that comprise the XR 12000 architecture:
- General Route Processor
- Switch fabric
- Intelligent line cards
- Operating software
XR 12000/12000 Architecture
All generic routers use a general Route Processor to provide control plane, data plane, and management plane functions. As line speeds and densities increase, this Route Processor must be able to keep up with the data forwarding rate, while also maintaining control and management functions simultaneously. At higher line rates, centralized processor architectures encounter timing sensitivities that put constraints on parallel feature processing. Distributed processing architectures, as in the XR 12000/12000 series, remove these constraints and leverage multiprocessing for aggregate switching performance gains. The XR 12000/12000 routers are optioned with a premium routing processor known as the Performance Route Processor P2 (PRP-2). The PRP-2 is capable of more than one million route prefixes and 256,000 multicast groups. It assists the 12000 routers with reaching up to 1.2 Tbps of aggregate switching performance in conjunction with an appropriate quantity and speed of the intelligent line cards.
In addition to the Cisco IOS XR Software benefits, the distribution of multiple processors within the XR 12000 chassis allows for an extension and separation of the control plane across multiple service instances. This provides control and management plane independence, helping facilitate logical and physical independence. These distributed processors are manifested in IP Services Engines (ISEs) with a particular ISE personalization representing the central intelligence of each line card.
ISEs are Layer 3-forwarding, CEF-enabled packet processors built with programmable, application-specific integrated circuits (ASICs) and optimized memory matrices. The primary benefit to the ISE technology is the ability to run parallel, IP feature processing at the network edge—at line rate. The programmability of the ISEs is key to investment protection, as new features can be added without a hardware upgrade. ISEs are architected for 2.5 Gbps, 10 Gbps, and 40 Gbps operation and are often optimized toward core or edge functions. The ISEs have been proceeding through various technology enhancements over the past several years and are classified relative to functionality. ISE functional classifications, such as the following, are by engine type:
- ISE engine 0—Known internally as the OC-12/BMA, this original ISE engine 0 uses an R5K CPU. Most features are implemented in software. An example of an ISE engine 0 is the 4-port OC-3 ATM line card. QoS features are rather limited.
- ISE engine 1—Known internally as the Salsa/BMA48, this engine was improved using a new ASIC (Salsa), allowing IP lookup to be performed in hardware. An example of an ISE engine 1 is the 2-port OC-12 Dynamic Packet Transport (DPT) line card. QoS features are rather limited.
- ISE engine 2—Known internally as the Perf48, this engine added new ASICs to perform hardware lookup for IP/MPLS switching. On-card packet memory was increased to 256 MB or 512 MB. New hardware-based class of service features were added, such as weighted random early detection (WRED) and Modified Deficit Round Robin (MDRR). An example of an ISE engine 2 is the 3-port Gigabit Ethernet line card.
- ISE engine 3—Internally referred to as the Edge engine, engine 3 is a completely rearchitected Layer 3 engine. Engine 3 accommodates an OC-48 worth of bandwidth and integrates additional ASICs to improve QoS and access control list (ACL) features that can be performed at line rate. An example of an ISE engine 3 is the 1-port OC-48 POS ISE line card. There is also an engine 3 version of the 4-port OC3 ATM card mentioned earlier.
- ISE engine 4—Referred to as the Backbone 192 engine, this engine is optimized and accelerated to support an OC-192 line rate. An example of an ISE engine 4 is the 1-port OC-192 POS line card.
- ISE engine 5—Optimized for 10 Gbps line rates with full feature sets including multicast replication. An example of an ISE engine 5 is the SIP-600 SPA Interface Processor-600 line card.
Depending on an ISE's functional legacy, an ISE might not be supported by new features in Cisco IOS software or the Cisco IOS XR Software. It is always wise to consult Cisco support tools to determine hardware platform, ISE engine type, and software feature compatibility when designing with these components.
The XR 12000/12000 multigigabit switch fabric works in combination with a passive chassis backplane, interconnecting all router components within an XR 12000/12000 router chassis. The active switching fabric is resident on pluggable cards known as SFCs and clock scheduler cards (CSCs), and these SFCs/CSCs are installed in a lower card shelf that interconnects with the XR 12000 backplane. This allows the SFCs/CSCs to be field upgraded easily. For example, changing a router to support 40 Gbps per line card slot from a 10 Gbps per line card slot can be accomplished through a replacement of the SFCs/CSCs with appropriate SFCs/CSCs that can clock and switch 40 Gbps-enabled ISE line cards and the PRP-2. This allows a XR 12000/12000 router to grow to as much as 1.28 Tbps of aggregate switching capacity. Another performance-enhancing feature of the XR 12000 switch fabric is that any IP multicast packet replication (for example, IP video) is now performed by the switch fabric itself, rather than burdening the general Route Processor (PRP-2).
The Cisco XR 12000 Series Routers are capable of running the Cisco IOS XR Software previously described. This software extends continuous system operation, performance scalability, and logical and physical virtualization features to the XR 12000 series routing platforms.
Cisco XR 12000/12000 Capacities
The Cisco XR 12000/12000 Series Routers comprise a scalable range of capacity from 30 Gbps to 1,280 Gbps (1.28 Tbps). Multiservice routers are commonly categorized by card slot quantity, throughput capacity per slot, and aggregate switching fabric capacity (full duplex or bidirectional). You can determine these three items via the Cisco model number without referencing any documentation. The model number convention defines the first two digits (12XXX) as the 12000 series family of routers. An XR-capable chassis will be prefixed with an XR (XR-12XXX).
The third digit of the 12000 model number represents the full-duplex (FDX) line rate capacity per card slot where XX0XX equals 2.5 Gbps (which is 5 Gbps FDX), XX4XX equals 10 Gbps (20 Gbps FDX), and XX8XX equals 40 Gbps (80 Gbps FDX).
The fourth and fifth digits of the 12000 model number convention define the total number of chassis card slots, where 12X04 equals four card slots, 12X06 equals six card slots, 12X10 equals 10 card slots, and 12X16 equals a 16-card slot router chassis.
To determine the gross-effective aggregate switching capacity of a particular model, you can multiply the line rate per card slot by the number of card slots, but this is where it can get confusing. Vendor literature often discusses line rate capabilities of the vendor's products using industry-familiar line rates of 2.5 Gbps (OC-48/STM-12), 10 Gbps (OC-192/STM-64), and 40 Gbps (OC-768/STM-256) services. On closer introspection, that line rate is used in a total aggregate capacity calculation for the router, but the line rate is doubled to reflect a full-duplex mode of operation. Often forgotten is that a 10 Gbps line rate is capable of that speed bidirectionally, both in the transmit and receive directions simultaneously. The calculation of theoretical total capacity becomes the full-duplex line rate (for example, 10 Gbps becomes 20 Gbps FDX) times the number of card slots.
Continuing with the Cisco model number convention, you can examine the third digit to determine the full-duplex line rate per card slot (for example, 4 = 10 Gbps half duplex [HDX] = 20 Gbps FDX) and multiply times the number of total card slots indicated by the fourth and fifth digits of the model number. A model with the number 12410 would calculate as 20 Gbps x 10 cards = 200 Gbps of total aggregate switching capacity for the 12410 platform. A model 12816 would calculate to 80 Gbps x 16 slots = 1,280 Gbps or 1.28 Tbps. This is gross-effective switching capacity, and the actual net-effective capacity will depend on the number of general-purpose processors (for example, PRP-2) configured for the system, as these subtract from the available card slots in most of the systems.
Figure 3-11 shows the relative positioning of the Cisco XR 12000/12000 Series Routers based on gross-effective capacities. As the figure shows, most models have a growth path for executing a pay-as-you-grow strategy.
)
Figure 3-11 Cisco XR 12000/12000 Series Router Capacities
The XR 12000/12000 series router product line includes additional features worthy of mention. The routers use the Cisco I-Flex design, which is implemented as intelligent, programmable interface processors with modular port adapters. This design combines both shared port adapters (SPAs) with SPA interface processors (SIPs) to improve line card slot economics and service density. The SIPs use the IP Services Engine (ISE) technology and are packaged into a SIP-400 or SIP-600 line card for the 12000 platform. The SIP-600 supports 10 Gbps per slot with two single- or double-height SPAs, and the SIP-400 supports 2.5 Gbps per slot and up to four single-height SPAs. A number of different SPAs are available to connect high-speed interfaces. The combination of the SPAs/SIPs creates interface flexibility, portability, and density for the XR 12000/12000 router platforms.
The platforms have enhanced fabrics that now support Building Integrated Timing Source (BITS) and single-router Automatic Protection Switching (SR APS). BITS allows for a centralized timing distribution for multiservice edge applications, particularly where the 12000 is used to aggregate traffic from ATM access networks. These ATM networks have relied on BITS, and the feature is essential to allow migration of ATM access networks onto XR 12000/12000-based IP/MPLS core networks. The SR APS feature enables true APS through the 12000 system platforms. Adding APS to the fabric and the support of a backpressure mechanism in the fabric scheduler eliminates timing slips when switching between active and standby cards, leveraging the fabric mirroring function and locking the timing to BITS. The fabric's backpressure support keeps the routers from dropping packets if an active card is removed.