virtualizacion de centro de datos avanzados
TRANSCRIPT
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
1/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 1
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
2/102
© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 2
Advanced Data Center VirtualizationBR KD C T-3831 –C arlos P ereira, D ata C enter C S E, LA TA M
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
3/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 3
Recuerde siempre…
1. Apagar su teléfono celular mientras dure la sesión.
2. Completar su evaluación y entregarla a la asistente de sala.
3. Ser puntual en todas las actividades de entrenamiento,almuerzos y eventos sociales para lograr un desarrollo óptimo
de la agenda.
4. Completar la evaluación general incluida en su material y
entregarla el miércoles 12 de Noviembre durante la tarde.
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
4/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 4
Before We Get Started
1. Intermediate level session focused on data centervirtualization technologies and solutions, includingboth front-end and back-end networks as well asserver virtualization
2. Prerequisites: being familiar with the basic LAN andSAN design models as well as server virtualizationtechnologies
3. Other recommended sessions
BRKDCT-2866: Data Center Architecture Strategy and Planning
BRKAPP-2005: Deploying Wide Area Application Services
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
5/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 5
Agenda
1. Data Center VirtualizationOverview
2. Front-End Data CenterVirtualization
Core Layer
VDC
Aggregation Layer
VSS
vPC
Server Load Balancing
Security Services
Access Layer
3. Server VirtualizationNexus 1000v
4. Back-End VirtualizationSAN
HBA
Unified IO (FCoE)
Storage
5. End-to-End ManagementVFrame Data Center
F r o n
t - E n
d
Virtual SANs/Unified IO
Virtual Storage
Virtual Network Services
VirtualFirewall
Context1
VirtualSSL
Context3
Virtual Machines
Front-End Virtualization
VirtualFirewallContext
1
VirtualFirewallContext
1
VirtualSLBContext
29
VirtualSSLContext
3
VirtualSSLContext
175
VSSVLAN VRF VPNsVDC
vHBAVSANs FCoECNA
B a c k
- E n
d
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
6/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 6
Biz Requirements
VirtualizationBusiness agility at minimal OPEX
1. Ever increasing business requirements
Fulfillment with legacy technologies is too expensive (OPEX and CAPEX)
Next Generation technologies required as enablers
Technology enables Innovative Biz Processes and fosters Business creativity
2. Virtualization technologies become baseline technologies over time
3. Richer infrastructure / services at comparable relative OPEX
time
OPEX InitialDeployment
Phase
Legacy Technologies
(projected OPEX)
Existing OPEX baseline
Virtualization Technologies?
?
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
7/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 7
Virtual
Merged New
Company
What Is Network Virtualization?
1. Virtualization: One to many
2. One network supports many virtual networks
Data Center Front-End Network/LAN
Outsourced
IT Department
Virtual Virtual
Segregated Department
(Regulatory Compliance)
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
8/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 8
Data Center Network
Out-of-Band Management Network
Backup Network
Guest/Partner Network
Security Network
What Is Network Virtualization?
1. Virtualization: Many to one
2. One network consolidates many physical networks
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
9/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 9
ConsolidatedData Center
“Network Virtualization” in the Data Center One Term, Many Contexts
1. Virtual connectivity services
IP/MPLS, L3 VPN, VRFs
L2 VPNs, VFIs, PW
2. Virtualized front-end
VLANs, PVLANs, VRF lite, VDC
Virtual intelligent services(Firewall, SLB, SSL, L4–7, etc.)
3. Compute virtualization
Clustering, GRID, virtualizationsoftware (hypervisor-based)
4. Virtualized storage
Virtual HBAs, CNAs
Virtual SANs (VSANs)
Network-hosted storagevirtualization software
Storage
Area
Network
Storage
Servers
Front-
End
Network S e r v
i c e
M o
d u
l e s
S e r v
i c e
M o
d u
l e s
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
10/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 10
DC Core
CBS 3100Blade
Cisco Catalyst49xxRack
Nexus 7000End-of-Row
Gigabit Ethernet
10 Gigabit Ethernet
10 Gigabit DCE
4/8Gb Fiber Channel
Nexus 5000Rack
DC Access
Nexus 700010GbE Agg
Cisco Catalyst6500DC Services
MDS 9500Storage
CiscoCatalyst 6500End-of-Row
Storage
IP+MPLS WAN
Agg Router
10GbE and 4Gb FC Server Access
CBS 3100MDS 9124eBlade
10GbE and 4/8Gb FC Server Access
10Gb FCoE Server Access
10 Gigabit FCoE/DCE
1GbE Server Access
Nexus 700010GbE Core
Cisco Catalyst 6500
10GbE VSS AggDC Services
DC Aggregation
Virtualized Data Center Infrastructure
FC
WAN
SAN A/BMDS 9500
Storage Core
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
11/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 11
Agenda
1. Data Center VirtualizationOverview
2. Front-End Data CenterVirtualization
Core Layer
VDC
Aggregation Layer
VSS
vPC
Server Load Balancing
Security Services
Access Layer
3. Server VirtualizationNexus 1000v
4. Back-End VirtualizationSAN
HBA
Unified IO (FCoE)
Storage
5. End-to-End ManagementVFrame Data Center
F r o n
t - E n
d
Virtual SANs/Unified IO
Virtual Storage
Virtual Network Services
VirtualFirewall
Context1
VirtualSSL
Context3
Virtual Machines
Front-End Virtualization
VirtualFirewall
Context1
VirtualFirewall
Context1
VirtualSLB
Context29
VirtualSSL
Context3
VirtualSSL
Context175
VSSVLAN VRF VPNsVDC
vHBAVSANs FCoECNA
B a c k
- E n
d
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
12/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 12
VRF OverviewWhat Is a VRF (Virtual Routing and Forwarding)?
1. VRFs allow dividing up your routing
table into multiple virtual tables
2. Routing protocol extensions allowbinding a process/address family to
a VRF
3. Interfaces are bound to a VRF usingip vrf forwarding
router eigrp 1
network 10.1.1.0 0.0.0.255
!
router ospf 1 vrf orange
network 10.2.1.0 0.0.0.255 area 0
!
router bgp 65000
address-family ipv4 vrf blue
!
ip route vrf green 0.0.0.0 0.0.0.0 …
Global Routing Table
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
13/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 13
VRF OverviewRoute Targets
1. Import/export routes to/from MP-BGP updates
2. Globally significant—creates the VPN
3. Allows hub and spoke connectivity (central services)
VRF Export 3:3
Import 3:3
Export 2:2
Import 1:1
Export 3:3
Import 3:3
Export 2:2
Import 1:1VRF
VRF
VRFExport 3:3
Import 3:3Import 2:2
Export 1:1
VRF VRFRed: Any-to-Any
Blue: Hub-and-Spoke
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
14/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 14
Shared Services Extranet VPN
Multiple-Box Extranet Implementation
1. Central services routesimported into both VRF red andblue (1:1)
2. Central VRF imports routes forblue and red subnets (3:3, 2:2)
1. No routes exchanged betweenblue/red
2. No transitivity: imported routesare not “reexported”
à Blue and red remain isolated
VRFExport 3:3
Import 1:1
Export 2:2
Import 1:1
Export 3:3
Import 1:1
Export 2:2
Import 1:1VRF
VRF
VRF
Import 3:3
Import 2:2
Export 1:1
VRF
Shared
ServicesBidirectional Communication
Between All VRFs and
Central Services VRF
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
15/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 15
Agenda
1. Data Center VirtualizationOverview
2. Front-End Data CenterVirtualization
Core Layer
VDC
Aggregation Layer
VSS
vPC
Server Load Balancing
Security Services
Access Layer
3. Server VirtualizationNexus 1000v
4. Back-End VirtualizationSAN
HBA
Unified IO (FCoE)
Storage
5. End-to-End ManagementVFrame Data Center
F r o n
t - E n
d
Virtual SANs/Unified IO
Virtual Storage
Virtual Network Services
VirtualFirewall
Context1
VirtualSSL
Context3
Virtual Machines
Front-End Virtualization
VirtualFirewall
Context1
VirtualFirewall
Context1
VirtualSLB
Context29
VirtualSSL
Context3
VirtualSSL
Context175
VSSVLAN VRF VPNsVDC
vHBAVSANs FCoECNA
B a c
k - E n
d
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
16/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 16
Virtual Device Contexts at Nexus 7000VDC ArchitectureVirtual Device Contexts Provides Virtualization at the Device Level AllowingMultiple Instances of the Device to Operate on the Same Physical Switch atthe Same Time
Kernel
Infrastructure
Protocol Stack (IPv4/IPv6/L2)
L2 Protocols
VDC1
VLAN Mgr
Nexus 7000 Physical Switch
VDCn
Protocol Stack (IPv4/IPv6/L2)
L3 Protocols
UDLD
VLAN Mgr UDLD
LACP CTS
IGMP 802.1x
RIB
OSPF GLBP
BGP HSRP
EIGRP VRRP
PIM SNMP
RIB
L2 Protocols
VLAN Mgr
L3 Protocols
UDLD
VLAN Mgr UDLD
LACP CTS
IGMP 802.1x
RIB
OSPF GLBP
BGP HSRP
EIGRP VRRP
PIM SNMP
RIB
…
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
17/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 17
Virtual Device Contexts
VDC Fault Domain
Kernel
Infrastructure
Protocol StackVDCA
Physical Switch
VDC A
P r o c e s s A B C
P r o c e s s D E F
P r o c e s s X Y Z
…
Protocol StackVDCB
VDC B
P r o c e s s A B C
P r o c e s s D E F
P r o c e s s X Y Z
…
Fault Domain
Process “DEF” in
VDC B Crashes
Process DEF in VDC
A Is Not Affected and
Will Continue to Run
Unimpeded
A VDC Builds a Fault Domain Around All Running Processes Within ThatVDC—Should a Fault Occur in a Running Process, It Is Truly Isolated fromOther Running Processes and They Will Not Be Impacted
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
18/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 18
Virtual Device Contexts
VDC Configuration
A VDC Is Created in the Following Manner—This Example Creates a VDC Called Networkers
switch# conf t
switch(config)# vdc NETWORKERS
switch(config-vdc)# show vdc
vdc_id vdc_name state mac
------ -------- ----- ----------
1 switch active 00:18:ba:d8:4c:3d
2 NETWORKERS active 00:18:ba:d8:4c:3e
switch(config-vdc)# show vdc detail
vdc id: 1
vdc name: switch
vdc state: activevdc mac address: 00:18:ba:d8:4c:3d
vdc ha policy: RESET
vdc id: 2
vdc name: NETWORKERS
vdc state: active
vdc mac address: 00:18:ba:d8:4c:3e
vdc ha policy: BRINGDOWN
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
19/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 19
Virtual Device Contexts
VDC and Interface Allocation
32-Port
10GE
Module
VDC
A
VDCB
VDC
C
VDCC
Ports Are Assigned on a per VDC
Basis and Cannot Be Shared
Across VDCs
Once a Port Has Been Assigned to aVDC, All Subsequent Configuration Is
Done from Within That VDC…
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
20/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 20
Linecard 1 Linecard 2 Linecard 3
V D C
3 0
V D C
2 0
V D C
2 0
V D C
2 0
Virtual Device Contexts
VDC Resource Utilization (Layer 2)
Switch Fabric
MAC Table MAC Table MAC Table
V D C
1 0
V D C
1 0
V D C
3 0
1/1 1/2 1/3 1/4 2/1 2/2 2/3 2/4 3/1 3/2 3/3 3/4
MAC Address A
MAC “A” MAC “A”
X
MAC “A” Is Propagated to Linecard 2 and 3 but OnlyLinecard 2 Installs MAC Due to Local Port Being In VDC 10
Layer 2 Learning with Multiple Active VDCs Also Has an Impact on ResourceUtilization—MAC Addresses Learnt in a VDC Are Only Propagated to OtherLinecards When That Linecard Has a Port in That VDC
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
21/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 21
Virtual Device Contexts
VDC Resource Utilization (Layer 3)
Linecard 1 Linecard 2 Linecard 3 Linecard 4 Linecard 5 Linecard 6 Linecard 7 Linecard 8
64K 64K 64K 64K 64K 64K 64K 64K
128K 128K 128K 128K 128K 128K 128K 128K
FIB TCAM FIB TCAM FIB TCAM FIB TCAM FIB TCAM FIB TCAM FIB TCAM FIB TCAM
ACL TCAM ACL TCAM ACL TCAM ACL TCAM ACL TCAM ACL TCAM ACL TCAM ACL TCAM
When Only the Default VDC Is Active, the FIB and ACL TCAM on EachLinecard Is Primed with Forwarding Prefixes and Policies Associated withThat Default VDC as Shown Below
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
22/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 22
Virtual Device Contexts
VDC Resource Utilization (Layer 3)
VDC Number Number of Routes Number of ACEs Allocated Linecards
10 100K 50K Linecard 1 and 2
20 10K 10K Linecard 1, 2, 3, 5
30 90K 40K Linecard 3 and 5
When Physical Port Resources Are Split Between Multiple VDCs, Then OnlyLinecards That Have Ports Associated with a Given VDC Have Local TCAMsPrimed with FIB and Policy Information
Let’s See How This Setup Impacts TCAM Resource Allocation on the SameChassis Assuming the Following Breakup Shown Below
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
23/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 23
Virtual Device Contexts
VDC Resource Utilization (Layer 3)
Linecard 1 Linecard 2 Linecard 3 Linecard 4 Linecard 5 Linecard 6 Linecard 7 Linecard 8
64K 64K 64K 64K 64K 64K 64K 64K
128K 128K 128K 128K 128K 128K 128K 128K
FIB TCAM FIB TCAM FIB TCAM FIB TCAM FIB TCAM FIB TCAM FIB TCAM FIB TCAM
ACL TCAM ACL TCAM ACL TCAM ACL TCAM ACL TCAM ACL TCAM ACL TCAM ACL TCAM
VDC 10 VDC 20 VDC 30
FIB and ACL TCAMResources Are MoreEffectively Utilized
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
24/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 24
VDC Use Case Examples
Partitioning – Security/Admin Boundaries
Appliance Model Service Module Model
Network OpsInfosec
VDC
Infosec
Network Ops
§ Some Infosec departments are still
reluctant about collapsed infrastructure
§ Concerns around change management
§ Infrastructure misconfiguration could
bypass policies
§ Ideally they want to have separately
managed and controlled infrastructure.
§ Not cost effective in larger deployments.
§ VDCs provide data and control plane
separation§ Extremely low possibility of configuration
bypassing security path
§ Separate administrative domains for
tight change control
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
25/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 25
Agenda
1. Data Center VirtualizationOverview
2. Front-End Data CenterVirtualization
Core Layer
VDC
Aggregation Layer VSS
vPC
Server Load Balancing
Security Services
Access Layer
3. Server VirtualizationNexus 1000v
4. Back-End VirtualizationSAN
HBA
Unified IO (FCoE)
Storage
5. End-to-End ManagementVFrame Data Center
F r o
n t - E n
d
Virtual SANs/Unified IO
Virtual Storage
Virtual Network Services
VirtualFirewall
Context1
VirtualSSL
Context3
Virtual Machines
Front-End Virtualization
VirtualFirewall
Context1
VirtualFirewall
Context1
VirtualSLB
Context29
VirtualSSL
Context3
VirtualSSL
Context175
VSSVLAN VRF VPNsVDC
vHBAVSANs FCoECNA
B a c
k - E n
d
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
26/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 26
Common Data Center challenges
Traditional Data Center Designs Are Requiring Ever Increasing Layer 2 Adjacencies Between Server Nodes Due to Prevalence of VirtualizationTechnology. However, They Are Pushing the Limits of Layer 2 Networks,Placing More Burden on Loop-Detection Protocols Such as Spanning Tree…
L2/L3 Core
L2
Distribution
L2 Access
Dual-Homed Serversto Single Switch,Single Active Uplinkper VLAN (PVST), L2Reconvergence
Single Active Uplinkper VLAN (PVST), L2
Reconvergence,Excessive BPDUs
FHRP, HSRP, VRRPSpanning TreePolicy Management
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
27/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 27
Virtual Switch System at Data Center
A Virtual Switch-Enabled Data Center Allows for Maximum Scalability soBandwidth Can Be Added When Required, but Still Providing a LargerLayer 2 Hierarchical Architecture Free of Reliance on Spanning Tree…
L2/L3 Core
L2
Distribution
L2 Access
Dual-HomedServers, SingleActive Uplink perVLAN (PVST), FastL2 Convergence
Dual Active Uplinks,Fast L2 Convergence,
Minimized L2 ControlPlane, Scalable
Single Router Node,Fast L2 Convergence,Scalable Architecture
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
28/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 28
Introduction to Virtual Switch
Concepts
Virtual Switch System Is a New Technology Break Through for theCisco Catalyst 6500 Family
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
29/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 29
Virtual Switch Architecture
Forwarding Operation
Virtual Switch Domain
Switch 1—Control Plane Active Switch 2—Control Plane Hot Standby
Virtual Switch Domain
Switch 1—Data Plane Active Switch 2—Data Plane Active
In Virtual Switch Mode, While Only One Control Plane Is Active,Both Data Planes (Switch Fabrics) Are Active, and as Such, EachCan Actively Participate in the Forwarding of Data
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
30/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 30
EtherChannel Concepts
Multichassis EtherChannel (MEC)
Regular EtherChannel on
Single Chassis
Multichassis EtherChannel Across
Two VSL-Enabled Chassis
Virtual Switch Virtual Switch
LACP, PAGP, or ON EtherChannelModes Are Supported…
Prior to Virtual Switch, EtherChannels Were Restricted to Reside Within theSame Physical Switch. In a Virtual Switch Environment, the Two PhysicalSwitches Form a Single Logical Network Entity—Therefore EtherChannelsCan Now Also Be Extended Across the Two Physical Chassis
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
31/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 31
Virtual Switch System at Data Center
Benefits
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
32/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 32
Nexus 7000 Virtualization
Virtual Port Chann els (vPC)
aggx aggx+1
L2
L3
NX-OS 4.1Dec/2008
Increase usable bandwidth,
by eliminating STPblocked ports
ü Separate physical
switches independent
control and data plane
ü Transparent to hosts or
switches
ü Neighbors only need
LACP support.
Increase usable bandwidth,
by eliminating STP
blocked ports
ü Separate physical
switches independent
control and data plane
ü Transparent to hosts or
switches
ü Neighbors only needLACP support.
vPC: Avoiding Spanning TreevPC: Avoiding Spanning TreevPC: Avoiding Spanning Tree
vPC vPC
Server
vPCvPC
vPC vPC
corex corex+1
access
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
33/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 33
vPC and VSS Comparison
Nexus 7000 vPC(Virtual Port Channels)
Catalyst 6500 VSS(Virtual Switching System)
Multi-Chassis Port Channel Yes Yes
Loop-free Topology (no blockingports)
Yes Yes
STP as a “fail-safe” protocol only Yes Yes
Switch Control PlaneTwo Independent Nodes,
both activeSingle Logical Node
Switch Redundancy (sup failover) Intra-chassis Inter-chassis
Control Plane Protocols Instances per Node Single instance
Switch ConfigurationCommon Configs
(w/ consistency checker)Combined Configs
Maximum Physical Nodes 2 2
ISSU Support YesQ3CY08
(12.2(33)SXI)
Inter-switch Link Hardware32 Port 10GE Module
Current HardwarePFC3C mode, Sup 70
10G, 6708, 6716
NX-OS 4.1Dec/2008
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
34/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 34
Agenda
1. Data Center VirtualizationOverview
2. Front-End Data CenterVirtualization
Core Layer
VDC
Aggregation Layer VSS
vPC
Server Load Balancing
Security Services
Access Layer
3. Server VirtualizationNexus 1000v
4. Back-End VirtualizationSAN
HBA
Unified IO (FCoE)
Storage
5. End-to-End ManagementVFrame Data Center
F r o
n t - E n
d
Virtual SANs/Unified IO
Virtual Storage
Virtual Network Services
VirtualFirewall
Context1
VirtualSSL
Context3
Virtual Machines
Front-End Virtualization
VirtualFirewall
Context1
VirtualFirewall
Context1
VirtualSLB
Context29
VirtualSSL
Context3
VirtualSSL
Context175
VSSVLAN VRF VPNsVDC
vHBAVSANs FCoECNA
B a c
k - E n
d
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
35/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 35
Storage10GbE and 4Gb FC Server Access10GbE and 4/8Gb FC Server Access10Gb FCoE Server Access
1GbE Server Access
CBS 3100Blade
Cisco Catalyst49xxRack
Nexus 7000End-of-Row
Gigabit Ethernet
10 Gigabit Ethernet
10 Gigabit DCE
4/8Gb Fiber Channel
Nexus 5000Rack
DC Access
Nexus 700010GbE Agg
Cisco Catalyst6500DC Services
MDS 9500Storage
CiscoCatalyst 6500End-of-Row
IP+MPLS WAN
Agg Router
CBS 3100MDS 9124eBlade
10 Gigabit FCoE/DCE
Nexus 700010GbE Core
Cisco Catalyst 6500
10GbE VSS AggDC Services
DC Aggregation
FC
WAN
SAN A/BMDS 9500Storage Core
DC Core
One-Arm Service SwitchesEmbedded Service Modules
Aggregation Services Design Options
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
36/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 36
One Physical Device
Multiple Virtual Systems
(Dedicated Control and Data Path)
ACE Virtual Partitioning
System Separation for Server Load Balancing and SSL
1. Single configuration file
2. Single routing table
3. Limited RBAC
4. Limited resource allocation
1. Distinct context configurationfiles
2. Separate routing tables
3. RBAC with contexts,roles, domains
4. Management and dataresource control
5. Independent application rulesets
6. Global administration andmonitorin
25% 25% 20%15%15%100%
Cisco Application Infrastructure ControlTraditional Device
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
37/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 37
Guaranteed
Rates
Guaranteed
Memory
ACE Virtual PartitionsResource Control
1. Bandwidth
2. Data connections/sec
3. Management connections/sec
4. SSL bandwidth5. Syslogs/sec
1. Access lists
2. Regular expressions
3. # Data connections
4. # Management connections5. #SSL connections
6. # Xlates
7. # Sticky entries
§ Guaranteed resource levels for each context with support foroversubscription
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
38/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 38
Firewall Service Module (FWSM)Virtual Firewalls
1. e.g., Three customers à three security contexts—scales up to 250
2. VLANs can be shared if needed (VLAN 10 on the right-hand side example)
3. Each context has its own policies (NAT, access-lists, fixups, etc.)
4. FWSM supports routed (Layer 3) or transparent (Layer 2) virtual firewalls at thesame time
Core/Internet
Cisco
Catalyst
6500
FW SM
VFW VFW VFW
MSFC
Core/Internet
Cisco
Catalyst
6500
FW SM
VFW VFW VFW
MSFC
VLAN 10 VLAN 20 VLAN 30
VLAN 11 VLAN 21 VLAN 31
VLAN 10
VLAN11 VLAN 21 VLAN 31
A B C A B C
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
39/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 39
FWSM—Virtual Firewall Resource Limiter
1. In system mode, classes can be defined
2. Individual contexts are then mapped to classes
3. Within a class, limits can be applied to specific resources suchas: (use “show resource types” for up-to-date list)
Rate Limited
Absolute Limits
§ Limits specified as integer or %; 0 means no limit
§ Resources can be oversubscribed: e.g., class assigns max 10% ofresources, but 50 contexts are mapped to it
§ Conns CPS§ Fixups Fixups/sec§ Syslogs Syslogs/sec
Conns Connections XlatesHosts Hosts MAC-entriesIPSec IPSec Mgmt Tunnels ALL
SSH SSH Sessions
Telnet Telnet Sessions
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
40/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 40
Data Center Virtualized ServicesCombination Example
v5
v105
v6 v7
v107
v2081
v2082
v2083...
v206 v207
v206
BU-4BU-2 BU-3
v105
v108
BU-1
1
2
3
4
* vX = VLAN X
**BU = Business Unit
VRF
VRF
VRFVRFVRF
v208
“Front-End” VRFs (MSFC)
Firewall Module Contexts
ACE Module Contexts
“Back-End” VRFs (MSFC)
Server Side VLANs
v207
3
4
v8
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
41/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 41
Cisco ACE
andCisco FWSM
ESX Server
Virtual Machines
Bank
Apps
Micro
softOracle
Microsoft
Outlook
Virtual Machines
Bank
Apps
Micro
softOracle
App Has
Capacity
Available
Ideal
Isolation
Online Bank
Application
(SSL Offloading
Required)
Virtualized ServicesCisco ACE and FWSM Virtualized
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
42/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 42
Agenda
1. Data Center VirtualizationOverview
2. Front-End Data CenterVirtualization
Core Layer
VDC
Aggregation Layer VSS
vPC
Server Load Balancing
Security Services
Access Layer
3. Server VirtualizationNexus 1000v
4. Back-End VirtualizationSAN
HBAUnified IO (FCoE)
Storage
5. End-to-End ManagementVFrame Data Center
F r o n
t - E n
d
Virtual SANs/Unified IO
Virtual Storage
Virtual Network Services
VirtualFirewall
Context1
VirtualSSL
Context3
Virtual Machines
Front-End Virtualization
VirtualFirewallContext
1
VirtualFirewallContext
1
VirtualSLBContext
29
VirtualSSLContext
3
VirtualSSLContext
175
VSSVLAN VRF VPNsVDC
vHBAVSANs FCoECNA
B a c
k - E n
d
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
43/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 43
On Failover, Src MAC Eth1 = Src MAC Eth0IP Address Eth1 = IP Address Eth0
Eth1: StandbyEth0: Active
SFT—Switch Fault Tolerance
IP=10.2.1.14MAC =0007.e910.ce0f
On Failover, Src MAC Eth1 = Src MAC Eth0IP Address Eth1 = IP Address Eth0
Eth1: StandbyEth0: Active
AFT—Adapter Fault Tolerance
H e a r t
b e a
t s
H e a r t
b e a
t s
One Port Receives, All Ports Transmit
Incorporates Fault Tolerance
One IP Address and Multiple MAC Addresses
Eth1-X: ActiveEth0: Active
ALB—Adaptive Load Balanci
H e a r t b e a
t s
IP=10.2.1.14MAC =0007.e910.ce0f
IP=10.2.1.14
MAC =0007.e910.ce0f
IP=10.2.1.14
MAC =0007.e910.ce0e
Default GW10.2.1.1
HSRP
Default GW10.2.1.1
HSRP
Default GW10.2.1.1
HSRP
Increasing HA in the Data CenterCommon NIC Teaming Configurations
Note: NIC manufacturer drivers are changing and may operate differently. Also, server OShave started integrating NIC teaming drivers which may operate differently.
Note: You can bundle multiple links to allow generating higher throughputs between serversand clients.
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
44/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 44
Virtual Switch SystemDeployment Scenario at Data Center Access Layer
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 44BRKDCT-383114488_04_2008_c1
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
45/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 45
LAN
Nexus 5000 Ethernet Host Virtualizer 1. Eliminates need for spanning
tree protocol on uplink bridgeports
Reduces CPU load on upstreamswitches
2. Allows multiple active uplinksfrom nexus 5000 switch tonetwork
Doubles effective bandwidthvs. STP
3. Prevents loops by pinning a
MAC address to only oneport
4. Completely transparent tonext hop switch
Ethernet Host Virtualizer
Nexus5000
Active-Active
MACB
MACA
MACB
MACA
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
46/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 46
Cisco Virtual Blade Switching (VBS)
1. Up to 8 Swi tches acts as Single VBS Swi tch
Dist r ibuted L2/ MAC learning
Central ized L3 learnin g
2. Each sw i t ch cons i s t s of
Swi tch Fabr ic
Por t As i cs (downl i nk & up l i nk por t s )
3. One Master Swi tch per VBS
1:N Resi l iency fo r Master
L2/L3 reconv ergence is sub 200 msec
4. High Speed VBS Cable (64 Gbps)
5. Example Deployment:
16 servers per enclosure X2 GE ports per server X
4 enclosures per rack = 128GE
– 2 x 10GE uplinks = 20GE
– 128GE / 20GE = 6.4:1 oversubscription
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
47/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 47
Cisco Catalyst Virtual Blade Switch
w ith Non-VSS Aggregat ion Aggregation Layer Access Layer (Virtual Blade Switch)
Single Switch / Node(for Spanning Tree or
Layer 3 or Management)
Spanning-Tree Blocking
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
48/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 48
Cisco Catalyst Virtual Blade Switch
w ith Non-VSS Aggregat ion Aggregation Layer
Access Layer (Virtual Blade Switch)
Single Switch / Node(for Spanning Tree or
Layer 3 or Management)
Spanning-Tree Blocking
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
49/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 49
Cisco Catalyst Virtual Blade Switch
w ith VSS Aggregat ion Aggregation Layer Access Layer (Virtual Blade Switch)
Single Switch / Node(for Spanning Tree or
Layer 3 or Management)
All Links Forwarding
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
50/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 50
Cisco Catalyst Virtual Blade Switch
w ith Non-VSS Aggregat ion
Aggregation Layer
(VSS or vPC) Access Layer (Virtual Blade Switch)
Single Switch / Node (forSpanning Tree or Layer 3
or Management)
All Links Forwarding
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
51/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 51
Agenda
1. Data Center VirtualizationOverview
2. Front-End Data CenterVirtualization
Core Layer
VDC
Aggregation Layer VSS
vPC
Server Load Balancing
Security Services
Access Layer
3. Server VirtualizationNexus 1000v
4. Back-End VirtualizationSAN
HBAUnified IO (FCoE)
Storage
5. End-to-End ManagementVFrame Data Center
F r o n
t - E n
d
Virtual SANs/Unified IO
Virtual Storage
Virtual Network Services
VirtualFirewall
Context
1
VirtualSSL
Context
3
Virtual Machines
Front-End Virtualization
VirtualFirewallContext
1
VirtualFirewallContext
1
VirtualSLBContext
29
VirtualSSLContext
3
VirtualSSLContext
175
VSSVLAN VRF VPNsVDC
vHBAVSANs FCoECNA
B a c
k - E n
d
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
52/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 52
Hypervisor Hypervisor
Full Virtualization
Software-Based Virtualization
(Examples)
1. VMware ESXserver
2. Microsoft HyperV
3. Xen (with AMD-SVM or Intel VM-T)
4. Virtuallron(hardware-assisted)
Para-Virtualization Application Virtualization
Examples
§ Xen (with traditionalhardware)
§ Oracle VM server
Examples
§ VMware server
§ VMware workstation
Examples
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
53/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 53
VMware ESX Architecture in a Nutshell
ESX Server Host
VirtualMachines
…
ProductionNetwork
MgmtNetwork
VM KernelNetwork
OS OS OS
Console
OS
App. App. App.
VM Virtualization Layer
Physical Hardware
CPU
M e m o
r y
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
54/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 54
VMware Networking Components
VMs
vmnic0
vmnic1
vNIC
vNIC
Virtual Ports
VM_LUN_0007
VM_LUN_0005
vSwitch0
vSwitch
VMNICS =
Uplinks
Per ESX Server Configuration
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
55/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 55
vSwitch Overview
VM1 VM2Service
Console
VMkernel
VMkernel
NIC VSwitch AVSwitch B
ESX
Server
Physical
Switches
Physical NIC’s
Virtual NIC’s
XNo Loop
XNo Loop
In ESX
Without a bridging VM
XNo Trunk
Btwn vSwitch
Software
implementation of
an Ethernet switch
How is it like a
switch:
-MAC addr forwarding
VLAN segmentation
How is it different:
-No need to learn
MAC addresses – it
knows the address of
the connecting vNIC’s
-No participation in
spanning tree
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
56/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 56
Introducing Cisco Virtual Network Link
Policy Based VM
Connectivity
Policy Based VMPolicy Based VM
ConnectivityConnectivityNon-Disruptive
Operational Model
NonNon--DisruptiveDisruptive
Operational ModelOperational ModelMobility of Network &
Security Properties
Mobility of Network &Mobility of Network &
Security PropertiesSecurity Properties
Virtualizing the Network DomainVirtualizing the Network DomainVirtualizing the Network Domain
Two Complimentary Models to Address Evolving Customer RequirementsTwo Complimentary Models to Address Evolving Customer Requirements
•• Cisco switch for VMW ESXCisco switch for VMW ESX
•• Compatible with any switchingCompatible with any switching
platformplatform
•• Leverages Virtual Center for serverLeverages Virtual Center for server
admin; Cisco CLI for networkadmin; Cisco CLI for network
adminadmin
••Scalable, hardware based, highScalable, hardware based, high
performance solutionperformance solution
••Standards driven approach toStandards driven approach to
delivering hardware based VMdelivering hardware based VM
networkingnetworking
••Combines VM & physical networkCombines VM & physical network
operations into 1 managed nodeoperations into 1 managed node
VMW ESXVMW ESXVMW ESX
VM
#4
VMVM
#4#4VM
#3
VMVM
#3#3
Server Server
VM
#2
VMVM
#2#2VM
#1
VMVM
#1#1
Initiator Initiator Initiator
Nexus 5000Nexus 5000
Nexus 5000 with VN-Link
(Hardware Based)
Nexus 5000 with VNNexus 5000 with VN--LinkLink
(Hardware Based)(Hardware Based)
VMW ESXVMW ESXVMW ESX
VM#1
VMVM
#1#1
VM#4
VMVM#4#4
VM#3
VMVM#3#3
Server Server
VM#2
VMVM#2#2
Nexus 1000VNexus 1000VNexus 1000V
NICNICNIC NICNICNIC
LAN
Nexus1000V
NexusNexus1000V1000V
Cisco Nexus 1000V
(Software Based)
Cisco Nexus 1000VCisco Nexus 1000V
(Software Based)(Software Based)
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
57/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 57
VMW ESXVMW ESXVMW ESX
Server 2Server 2
VMW ESXVMW ESXVMW ESX
Server 1Server 1
Cisco Nexus 1000V
Industry Firs t 3 rd Party Distr ibuted Virtual Switch
VM
#5
VMVM
#5#5VM
#8
VMVM
#8#8VM
#7
VMVM
#7#7VM
#6
VMVM
#6#6VM
#4
VMVM
#4#4VM
#3
VMVM
#3#3VM
#2
VMVM
#2#2VM
#1
VMVM
#1#1
VMware vSwitchVMware vSwitchVMware vSwitch VMware vSwitchVMware vSwitchVMware vSwitchNexus 1000VNexus 1000VNexus 1000VNexus 1000VNexus 1000VNexus 1000VNexus 1000V DVSNexus 1000V DVSNexus 1000V DVS
VM
#8
VMVM
#8#8VM
#7
VMVM
#7#7VM
#6
VMVM
#6#6VM
#4
VMVM
#4#4VM
#3
VMVM
#3#3VM
#2
VMVM
#2#2VM
#5
VMVM
#5#5
§ Nexus 1000V providesenhanced VM switchingfor VMware ESX
§ Features Cisco VN-Link:§Policy Based VM Connectivity
§Mobility of Network & SecurityProperties
§Non-Disruptive OperationalModel
§ Ensures proper visibility& connectivity duringVMotion
Enabl ing A ccelerat ion of Server Vir tual izat ion Benefi ts
VM
#1
VMVM
#1#1
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
58/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 58
Cisco Nexus 1000V Architecture
Virtual Center Virtual Center
VMW ESXVMW ESXVMW ESX
Server 1Server 1
VMware vSwitchVMware vSwitchVMware vSwitchVMW ESXVMW ESXVMW ESX
Server 2Server 2
VMware vSwitchVMware vSwitchVMware vSwitchVMW ESXVMW ESXVMW ESX
Server 3Server 3
VMware vSwitchVMware vSwitchVMware vSwitch
VM
#1
VMVM
#1#1VM
#4
VMVM
#4#4VM
#3
VMVM
#3#3VM
#2
VMVM
#2#2VM
#5
VMVM
#5#5VM
#8
VMVM
#8#8VM
#7
VMVM
#7#7VM
#6
VMVM
#6#6VM
#9
VMVM
#9#9VM
#12
VMVM
#12#12VM
#11
VMVM
#11#11VM
#10
VMVM
#10#10
Nexus 1000V
VSM
Nexus 1000VNexus 1000V
VSMVSM
VEMVEMVEM VEMVEMVEM VEMVEMVEMNexus 1000V DVSNexus 1000V DVSNexus 1000V DVS
Virtual Supervisor Module (VSM)
§ Virtual or Physical appliancerunning Cisco OS (supports HA)
§ Performs management,monitoring, & configuration
§ Tight integration with VMwareVirtual Center
Virtual Ethernet Module (VEM)
§ Enables advanced networkingcapability on the hypervisor
§ Provides each VM with dedicated“switch port”
§ Collection of VEMs = 1 DVS
Cisco Nexus 1000V Enables:
§ Policy Based VM Connectivity
§ Mobility of Network & SecurityProperties
§ Non-Disruptive Operational Model
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
59/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 59
Cisco Nexus 1000V
Policy Based VM
Connectivity
Policy Based VMPolicy Based VM
ConnectivityConnectivityNon-Disruptive
Operational Model
NonNon--DisruptiveDisruptive
Operational ModelOperational ModelMobility of Network &
Security Properties
Mobility of Network &Mobility of Network &
Security PropertiesSecurity Properties
VN-Link: Virtualizing the Network DomainVNVN--Link: Virtualizing the Network DomainLink: Virtualizing the Network Domain
VMW ESXVMW ESXVMW ESX
ServerServer
Virtual Center Virtual Center
VMW ESXVMW ESXVMW ESX
Server Server
Cisco Nexus 1000VCisco Nexus 1000VCisco Nexus 1000V
VM
#1
VMVM
#1#1VM
#4
VMVM
#4#4VM
#3
VMVM
#3#3VM
#2
VMVM
#2#2VM
#5
VMVM
#5#5VM
#8
VMVM
#8#8VM
#7
VMVM
#7#7VM
#6
VMVM
#6#6
VM Connection Policy§ Defined in the network
§ Applied in Virtual Center
§ Linked to VM UUID
Defined Policies
WEB AppsWEB Apps
HRHR
DBDB
ComplianceCompliance
Faster VM Deploym ent Faster VM Deployment
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
60/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 60
Policy Based VM
Connectivity
Policy Based VMPolicy Based VM
ConnectivityConnectivityNon-Disruptive
Operational Model
NonNon--DisruptiveDisruptive
Operational ModelOperational ModelMobility of Network &
Security Properties
Mobility of Network &Mobility of Network &
Security PropertiesSecurity Properties
VN-Link: Virtualizing the Network DomainVNVN--Link: Virtualizing the Network DomainLink: Virtualizing the Network Domain
VMW ESXVMW ESXVMW ESX
ServerServer
VMW ESXVMW ESXVMW ESX
Server Server
Cisco Nexus 1000VCisco Nexus 1000VCisco Nexus 1000V
VM
#5
VMVM
#5#5VM
#8
VMVM
#8#8VM
#7
VMVM
#7#7VM
#6
VMVM
#6#6VM
#4
VMVM
#4#4VM
#3
VMVM
#3#3VM
#2
VMVM
#2#2VM
#1
VMVM
#1#1
VM
#4
VMVM
#4#4VM
#3
VMVM
#3#3VM
#2
VMVM
#2#2VM
#1
VMVM
#1#1
VN-Link Property Mobility•• VMotion for the networkVMotion for the network
•• Ensures VM securityEnsures VM security
•• Maintains connection stateMaintains connection state
Virtual Center Virtual Center
VMs Need To MoveVMs Need To Move•• VMotionVMotion•• DRSDRS
•• SW Upgrade/PatchSW Upgrade/Patch
•• Hardware FailureHardware Failure
Cisco Nexus 1000VRicher Network Services Richer Network Services
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
61/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 61
Policy Based VM
Connectivity
Policy Based VMPolicy Based VM
ConnectivityConnectivityNon-Disruptive
Operational Model
NonNon--DisruptiveDisruptive
Operational ModelOperational ModelMobility of Network &
Security Properties
Mobility of Network &Mobility of Network &
Security PropertiesSecurity Properties
VN-Link: Virtualizing the Network DomainVNVN--Link: Virtualizing the Network DomainLink: Virtualizing the Network Domain
VMW ESXVMW ESXVMW ESX
ServerServer
Virtual Center Virtual Center
VMW ESXVMW ESXVMW ESX
Server Server
Cisco Nexus 1000VCisco Nexus 1000VCisco Nexus 1000V
VM
#1
VMVM
#1#1VM
#4
VMVM
#4#4VM
#3
VMVM
#3#3VM
#2
VMVM
#2#2VM
#4
VMVM
#4#4VM
#3
VMVM
#3#3VM
#2
VMVM
#2#2VM
#1
VMVM
#1#1
Network Benefits§
Unifies network mgmt & ops§ Improves operational security
§ Enhances VM network features
§ Ensures policy persistence
§ Enables VM-level visibility
Server Benefits
§ Maintains existing VM mgmt
§ Reduces deployment time
§ Improves scalability
§ Reduces operational workload
§ Enables VM-level visibility
Cisco Nexus 1000VIncrease Operational Eff ic iency Increase Operational Eff ic ienc y
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
62/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 62
Cisco Nexus 1000V – VM Security
ServerServer
Private VLANPrivate VLAN
•• Promiscuous portPromiscuous port•• Isolated portIsolated port
•• Community portCommunity port
Server Server
I
Server Server
I
Cisco Nexus 1000VCisco Nexus 1000VCisco Nexus 1000V
VM
#1
VMVM
#1#1VM
#4
VMVM
#4#4VM
#3
VMVM
#3#3VM
#2
VMVM
#2#2VM
#4
VMVM
#4#4VM
#3
VMVM
#3#3VM
#2
VMVM
#2#2VM
#1
VMVM
#1#1VM
#4
VMVM
#4#4VM
#3
VMVM
#3#3VM
#2
VMVM
#2#2VM
#1
VMVM
#1#1
VMW ESXVMW ESX VMW ESXVMW ESX VMW ESXVMW ESX
II II
Security FeaturesSecurity Features•• Access Control List Access Control List
•• Port SecurityPort Security
•• DHCP SnoopingDHCP Snooping•• IP Source GuardIP Source Guard
•• Dynamic ARP InspectionDynamic ARP Inspection
PP CCCC
Cisco TrustSecCisco TrustSec•• Admission control: 802.1X Admission control: 802.1X
•• HopHop--byby--hop crypto:hop crypto:
802.1AE802.1AE
•• Security Group TagSecurity Group Tag
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
63/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 63
Agenda
1. Data Center VirtualizationOverview
2. Front-End Data CenterVirtualization
Core Layer
VDC
Aggregation Layer
VSS
vPC
Server Load Balancing
Security Services
Access Layer
3. Server VirtualizationNexus 1000v
4. Back-End VirtualizationSAN
HBAUnified IO (FCoE)
Storage
5. End-to-End ManagementVFrame Data Center
F r
o n
t - E n
d
Virtual SANs/Unified IO
Virtual Storage
Virtual Network Services
VirtualFirewall
Context
1
VirtualSSL
Context
3
Virtual Machines
Front-End Virtualization
VirtualFirewallContext
1
VirtualFirewallContext
1
VirtualSLBContext
29
VirtualSSLContext
3
VirtualSSLContext
175
VSSVLAN VRF VPNsVDC
vHBAVSANs FCoECNA
B a c
k - E n
d
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
64/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 64
Virtual Storage Area Network
Deployment
1. Consolidation of SAN islands
Increased utilization of fabric ports with just-in-time provisioning
2. Deployment of large fabrics
Dividing a large fabric in smaller VSANs
Disruptive events isolated per VSAN
RBAC for administrative tasks
Zoning is independent per VSAN
3. Advanced traffic management
Defining the paths for each VSAN
VSANs may share the same EISL
Cost effective on WAN links4. Resilient SAN extension
5. Standard solution(ANSI T11 FC-FS-2 section 10)
SAN Islands
Department A
Department B Department C
Virtual SANs
(VSANs)
Department A
Department B
Department C
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
65/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 65
VSAN Advantages for Consolidation
OLTP
SAN Islands
Overlay Isolated Virtual
Fabrics (VSANs) on Same
Physical Infrastructure
E-Mail
Backup Backup VSAN
E-Mail VSANOLTP VSAN
Consolidated SANs
Attribute
More Number of SAN Switches Fewer
No Share Disk/Tape Yes
No Share DR Facilities Yes
Complex SAN Management Simple
Very hardSupport Virtualization
and MobilityEasy
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
66/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 66
VSAN Technology
1. Hardware-based isolation oftagged traffic belonging todifferent VSANs
2. Create independent instanceof fiber channel services foreach newly created VSAN—services include:
Fibre Channel
Services for
Blue VSAN
Fibre Channel
Services forRed VSAN
Fibre Channel
Services for
Blue VSANFibre Channel
Services forRed VSAN
Cisco MDS 9000
Family with VSANService
VSAN Header Is
Added at Ingress
Point Indicating
Membership
No Special
Support Required
by End Nodes
Trunking
E_Port
(TE_Port)
Trunking
E_Port
(TE_Port)
Enhanced ISL (EISL)
Trunk Carries
Tagged Traffic from
Multiple VSANs
VSAN Header Is
Removed at
Egress Point
The Virtual SANs Feature Consists
of Two Primary Functions
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
67/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 67
Inter-VSAN Routing
1. Similar to L3interconnectionbetween VLAN
2. Allows sharing of
centralized storageservices such as tapelibraries and diskswithout mergingseparate VSANs
3. Network address
translation allowinterconnection ofVSANs without apredefined addressingschema
Engineering
VSAN_1
Marketing
VSAN_2
HR
VSAN_3
HR
VSAN_3
Marketing
VSAN_2
Blade Server with Integrated
MDS 9100 Switch
Engineering
VSAN_1
Tape
VSAN_4
(Access via IVR)
Blade Server
VSAN_1
(Access via IVR)
IVR
IVR
IVR
VSAN-Specific
Disk
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
68/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 68
Agenda
1. Data Center VirtualizationOverview
2. Front-End Data CenterVirtualization
Core Layer
VDC
Aggregation Layer
VSS
vPC
Server Load Balancing
Security Services
Access Layer
3. Server VirtualizationNexus 1000v
4. Back-End VirtualizationSAN
HBAUnified IO (FCoE)
Storage
5. End-to-End ManagementVFrame Data Center
F r o n
t - E n
d
Virtual SANs/Unified IO
Virtual Storage
Virtual Network Services
VirtualFirewall
Context
1
VirtualSSL
Context
3
Virtual Machines
Front-End Virtualization
VirtualFirewallContext
1
VirtualFirewallContext
1
VirtualSLBContext
29
VirtualSSLContext
3
VirtualSSLContext
175
VSSVLAN VRF VPNsVDC
vHBAVSANs FCoECNA
B a
c k - E n
d
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
69/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 69
N-Port ID Virtualization (NPIV)
1. Mechanism to assignmultiple N_Port_IDs toa single N_Port
2. Allows all the accesscontrol, zoning, portsecurity (PSM) beimplemented onapplication level
3. Multiple N_Port_IDs
are so far allocated inthe same VSAN
Application Server
File
Services
N_PortID-3
Web
N_PortID-2
E-Mail
N_PortID-1
F_PortF_Port F_Port
E-Mail
VSAN_3
Web
VSAN_2
File and Print
VSAN_1
E_Port
E_Port
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
70/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 70
NPIV Configuration Example
1. npiv enable
2. Notice that a F-port supportsmultiple logins
NPIV Is Enabled Switchwide with the
Command:
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
71/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 71
NPIV Usage Examples
‘Intelligent Pass-Thru’Virtual Machine Aggregation
FC FC FC FC
NP_Port
F_PortF_Port
FC FC FC FC
FC
NPIV-Enabled HBA
NPV Edge
Switch
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
72/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 72
FC
Storage Array
(LUN Mapping and Masking)MDS9000
Zone FC Name Server
pWWN-P
Single Login on a Single Point-to-Point Connection
Virtual Servers Share a Physical HBA
1. A zone includes the physical HBAand the storage array
2. Access control is demanded to storagearray “LUN masking and mapping”, it isbased on the physical HBA pWWN andit is the same for all VMs
3. The hypervisor is in charge of the
mapping, errors may be disastrous
H W
H y p e r v i s o r
V i r t u a l
S e r v e r s
pWWN-P
Mapping
FC
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
73/102
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
74/102
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
75/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 75
VMotion LUN Migration with NPIV
HBAs
with NPIV
VM1 VM2 VM3
Centralized management ofVMs and resources
Redeploy VMs and support
live migration
No need to reconfigure zoning
or LUN masking
Dynamically reprovision VMs
without impact to existinginfrastructure
WWPN1
WWPN2
WWPN3
FCFC
STATUS1 2 3 4 5 6 7 8 9 10 11 12 13 1 4 15 1 6
WS-X9016
1/2GbpsFCModule
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
76/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 76
Blade Switch/Top-of-RackDomain ID Explosion
1. Domain ID used foraddressing, routing, andaccess control
2. One domain ID per
SAN switch
3. Theoretically 239 domainID, practically much lesssupported
4. Limits SAN fabricscalability
Tier 1 Tier 2 Tape Farm
Blade SwitchesIncrease Domain
IDs, Increase
Fabrics
MDS
9500
Theoretical
Maximum: 239
Domain IDsper SAN
Blade Switch
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
77/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 77
Cisco MDS Network Port Virtualization
1. Eliminates edge switchDomain ID
2. Edge switch acts as anNPIV host
3. Simplifies server andSAN management andoperations
4. Increases fabricscalability
Tier 1 Tier 2 Tape Farm
NPV-EnabledSwitches Do
Not Use
Domain IDs
Supports
Up to 100 EdgeSwitches
MDS
9500
Edge Switch
Acts as a
NPIV HostNPV NPV
Blade Switch
NPV NPV NPV NPV
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
78/102
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
79/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 79
Agenda1. Data Center Virtualization
Overview
2. Front-End Data CenterVirtualization
Core Layer
VDC
Aggregation Layer
VSS
vPC
Server Load Balancing
Security Services
Access Layer
3. Server VirtualizationNexus 1000v
4. Back-End VirtualizationSAN
HBA
Unified IO (FCoE)
Storage
5. End-to-End ManagementVFrame Data Center
F r o n
t - E n
d
Virtual SANs/Unified IO
Virtual Storage
Virtual Network Services
VirtualFirewall
Context1
VirtualSSL
Context3
Virtual Machines
Front-End Virtualization
VirtualFirewallContext
1
VirtualFirewallContext
1
VirtualSLBContext
29
VirtualSSLContext
3
VirtualSSLContext
175
VSSVLAN VRF VPNsVDC
vHBAVSANs FCoECNA
B a
c k - E n
d
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
80/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 80
Unified I/O (FCoE)Fewer HBA/NICs per Server
CNA
CNA
FC HBA
FC HBA
NIC
NIC
SAN (FC)
SAN (FC)
LAN (Ethernet)
LAN (Ethernet)
SAN (FCoE)
LAN (Ethernet)
CNA = Converged Network Adapter
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
81/102
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
82/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 82
Virtual LinksAn example
VL1
VL2
VL3
LAN/IP Gateway
Storage Gateway
VL1 – LAN Service – LAN/IP
VL3 – Delayed Drop Service - IPC
VL2 - No Drop Service - Storage
Up to 8 VL’s per physical link
Ability to support QoS queues within the lanes
DCECNA
DCECNA
DCECNA
Campus Core/
Internet
Storage Area
Network
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
83/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 83
Fiber Channel over Ethernet:How It Works
1. Direct mapping of fiber channel over Ethernet
2. Leverages standards-based extensions to Ethernet toprovide reliable I/O delivery
Priority flow control
Data Center Bridging CapabilityeXchange Protocol (DCBCXP)
MAC
PHY
FCoE Mapping
FC-0
FC-1
FC-2
FC-3
FC-4
FC-2
FC-3
FC-4
FC Frame
Ethernet
Header
Ethernet
Payload
Ethernet
FCS
S O F
E O F
C R C
(a) Protocol Layers (b) Frame Encapsulation
10GE LosslessEthernet
Link
FCoE Traffic
Other Networking
Traffic
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
84/102
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
85/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 85
SAN BSAN ALAN
FCoE
Ethernet
FC
Today
Unified I/O Use Case
Management
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
86/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 86
SAN BSAN ALAN
FCoE
Ethernet
FC
Unified I/O Use Case
Unified I/O
1. Reduction of server adapters
2. Fewer cables
3. Simplification of accesslayer and cabling
4. Gateway-free implementation—fits in installed base of existingLAN and SAN
5. L2 multipathing access—distribution
6. Lower TCO
7. Investment protection
(LANs and SANs)8. Consistent operational model
9. One set of ToR switches
Unified I/O
FCoE
Switch
Management
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
87/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 87
CNA: I/O Consolidation Adapter 1. Off the shelf NIC and HBA ASICs from: Qlogic, Emulex
Dual 10 GbE/FCoE ports
2. Support for native driversand utilities
Customer certified stacks
3. Replaces multiple adaptersper server
4. Consolidates 10 GbE andFCon a single interface
5. Minimum disruption inexisting customerenvironments
10 GbE/FCoE
PCIe Bus
Designed Multiplexer and FCoE Offload Protocol Engine
FC10 GbE
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
88/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 88
FCoE Software Stack1. Supported on Intel Oplin
10 GbE Adapters
Software upgraded turns10 GbE adapter into FCoE adapter
2. Software implementation
Initiator and target mode
FCP, FC class 3
Fully supports Ethernet pauseframes (per priority pause)
3. Supported OS
Linux: Red Hat and SLES
Windows
4. “Free” access to the SAN
L2 Ethernet NIC
S o f t w
a r e
H a r d w a r e
FCoE Software Stack
Website: www.Open-FCoE.org
Announcement is: http://lkml.org/lkml/2007/11/27/227
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
89/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 89
CNAs: View from Operating System
1. Standard drivers
2. Same management
3. Operating system
sees:2 x 10 Gigabit
Ethernet adapter
2 x 4 Gbps fiberchannel HBAs
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
90/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 90
IO Consolidation
1. virtual-ethernet interface (veth)
Paired with host’s Ethernet device
Configuration point for allEthernet features
2. virtual-fc interface (vfc)
Paired with host’s HBA device
Configuration point for allfiber channel features
3. virtual-interface-group (vig)
Logical representation of a switch port
Consists of one veth and one vfc
Configured online or offline
Bound to physical switch port fordeployment
EtherChannel post FCS
vig
vethvfc
Ethernet
Forwarding
Fiber
Channel
Forwarding
mux
Ethernet
Connecting LAN and SAN on a
Single Physical Link
SAN A SAN B LAN
SCSI IP
eth0host0
mux
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
91/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco NetworkersColombia 2008 91
Agenda1. Data Center Virtualization
Overview
2. Front-End Data CenterVirtualization
Core Layer
VDC
Aggregation Layer
VSS
vPC
Server Load Balancing
Security Services
Access Layer
3. Server VirtualizationNexus 1000v
4. Back-End VirtualizationSAN
HBA
Unified IO (FCoE)
Storage
5. End-to-End ManagementVFrame Data Center
F
r o n
t - E n
d
Virtual SANs/Unified IO
Virtual Storage
Virtual Network Services
VirtualFirewall
Context1
VirtualSSL
Context3
Virtual Machines
Front-End Virtualization
VirtualFirewallContext
1
VirtualFirewallContext
1
VirtualSLBContext
29
VirtualSSLContext
3
VirtualSSLContext
175
VSSVLAN VRF VPNsVDC
vHBAVSANs FCoECNA
B a c
k - E n
d
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
92/102
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
93/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 93
SAN
Fabric
Storage Volume Virtualization
1. A SCSI operation from the host is mapped in one or moreSCSI operations to the SAN-attached storage
2. Zoning connects real initiator and virtual target or virtualinitiator and real storage
3. Works across heterogeneous arrays
Virtual Volume2
Virtual
Target 1
VSAN_10
Virtual Volume
1
Virtual
Target 2
VSAN_20
Virtual
Initiator
VSAN_30
Virtual
Initiator
VSAN_30
Initiator
VSAN_20
Initiator
VSAN_10
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
94/102
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
95/102
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
96/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 96
Data Center Virtualization
StorageServersSecurity LAN SLB LAN SAN
FC
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
97/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 97
Application Service ProvisioningDesign, Orchestration, and Deployment
Service
Delivery
Chain
Switch
Config
VLAN, Port
SVIs, etc.
Zones,
VSANs,
LUNs,
Volumes
CPU
Memory,
IO, etc.
VIPs, LB
Policies,
Probes
Firewall,
Context,
Policies,
etc.
Service
Policies
Automated Failover Policy-Based Resource Optimization
Service MaintenanceManagement Integration API
Automate
Boot OS/Application
Server VSANsL4–L7VLANs
Service
Components
Firewall
VLAN_A VLAN_B VLAN_D VLAN_E VSAN_Z
Partition_1 Partition_1
Server
Boot
Image, VM,
Application
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
98/102
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
99/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008 99
Q and A
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
100/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008
10
0
Recommended Reading
1. Continue your Networkerslearning experience withfurther reading from CiscoPress
2. Check the RecommendedReading flyer for suggestedbooks
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
101/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008
10
1
MUCHAS GRACIAS !!!!!
Recuerde Completar su Formulario de Evaluación !
-
8/16/2019 Virtualizacion de Centro de Datos Avanzados
102/102
©2008 Cisco Systems, Inc. All rights reserved. Cisco PublicCisco Networkers Argentina 2008
10
2