FICON
101
CCIE
SAN
Study
Group
By Ozden Karakok – May 2011
FICON © 2010 Cisco Systems, Inc. All rights reserved. 1
Objective & Agenda
At the end of this session, attendees will be able to answer what is FICON is all about!
FICON Concepts
• Terminology
• Topologies
• Addressing
• Frames
FICON Applications
MDS Requirements
• FICON Manager
• FICON Control Unit Process (FCD)
Configuring FICON
• SAN-OS
• Mainframe
SAN-OS 3.0 Update
Troubleshooting FICON
Summary
FICON © 2010 Cisco Systems, Inc. All rights reserved. 2
FICON Terminology
FICON © 2010 Cisco Systems, Inc. All rights reserved. 3
IBM Mainframe Systems
Current IBM mainframe systems:
S/390 G5 OS/390
VM
S/390 G6
VSE/ESA
zSeries 800 MVS
zSeries 890
zSeries 900 z/OS
z/VM
zSeries 990 Linux
z9 Ent. Class
z9 Business C.
z10 Ent. Class
z10 Business C.
FICON © 2010 Cisco Systems, Inc. All rights reserved. 4
Introduction to FICON
FICON Protocol Layer
• FICON is Fibre
Channel FC-4 layer
Single Byte protocol
ULP SCSI – 3/4 Command Code
(Upper Level Protocol) Sets • It is at the same
layer as FCP
protocol
SCSI— FC-SB2/FC-SB3
FC - 4 Command Set
• Fibre Channel
Mapping (FCP) (FICON) –”Single Byte
Command Code”
FC - 3 mapping protocol
Common Services is described in FC-
SB3, FC-SB4
FC - 2 Framing Protocol/Flow control • FC-SB2 is an older
version. FC-SB3-4
FC - 1 is superset.
Encode/Decode (8/10)
FC - 0 Ref: www.t11.org
Physical Variant
FICON © 2010 Cisco Systems, Inc. All rights reserved. 5
FICON Terminology
A Channel is a physical entity/element of the processor, acting as
the interface between the CPU and the CU
– A Channel Image instantiates multiple, logical channels behind a
physical channel
A Channel Path includes the interface adapter, physical fiber pair,
and up to two intervening FICON switches/directors between the
channel and the CU
A Channel Path Identifier (CHPID) is a single hexadecimal byte
value identifying each channel path to the system
FICON © 2010 Cisco Systems, Inc. All rights reserved. 6
FICON Terminology
Outboard I/O processors called Control Units (CUs)
– CUs can be stand-alone or integrated in peripheral devices
– CUs are identified by a two-byte hexadecimal CUNUMBER
– A Control Unit Image instantiates multiple, logical CUs behind
a physical CU
A Device Address (aka Unit Address) is a decimal value
that selects a specific device on a given CU image
FICON © 2010 Cisco Systems, Inc. All rights reserved. 7
FICON Terminology
A “Channel” is an entity
typically mapped to an N-port
in a host computer which
performs access to I/O devices
Channel
M IBM
A Control Unit There can be multiple
Channel
I Image “Channel Images” behind one
Images N port, each of which represent
N a separate logical channel.
F
R
A
M N Port A “Control Unit” is the interface
E to I/O devices such as printer,
disks etc.
N Port
Legend
Physical There can be multiple “Control
Unit images” behind each
Logical Control Unit (CU) “Control Unit”
FICON © 2010 Cisco Systems, Inc. All rights reserved. 8
FICON Terminology
A Logical Partition (LPAR) is a subset of mainframe
resources, dedicated to implementing a particular
programming environment
– Each LPAR functions independently of other LPARs and is
protected against the actions of poorly-behaved LPARs
– Resources such as CHPIDs (Channel Path Identifier) may be
shared between LPARs
FICON © 2010 Cisco Systems, Inc. All rights reserved. 9
MIF
LPARs can share channels
Multiple Image Facility (MIF)
MIF
LPAR 1
Channel N_Port CU
N_Port
LPAR 2
N_Port CU
LPAR n
FICON © 2010 Cisco Systems, Inc. All rights reserved. 10
FICON Terminology
An Output Port is the FICON director or switch port that connects
to the CU
– In a switched point-point environment, it can be a single byte
hexadecimal value identifying the director/switch port number
connected to the CU
– FICON port numbering schemes are proprietary to the switch vendor/
model; typically different than what’s used for FC config
– The port number may also be added with the switch domain ID to
form a two-byte Output Port value if supported by the CU
– In a cascaded environment, Output Port must be a two-byte value
derived from the non-Entry Switch
FICON © 2010 Cisco Systems, Inc. All rights reserved. 11
FICON Cascaded Topology
Output Ports
Entry Switch Non-entry switch
CH =
Initiator
CU = Target
Switch ID = Domain ID Portaddress = int fc X/Y
FICON © 2010 Cisco Systems, Inc. All rights reserved. 12
FICON Terminology
The Input Output Control Program (IOCP) is a
mainframe application that defines the Input Output
Control Data Set (IOCDS), which the mainframe uses
to define & identify control units, devices, channel
paths, channels and their relationships to each other.
I/O definition file (IODF)
All of the terms we’ve just covered are required IOCP/
IOCDS parameters
FICON © 2010 Cisco Systems, Inc. All rights reserved. 13
Objective & Agenda
FICON Concepts
• Terminology
• Topologies
• Addressing
• Frames
FICON Applications
MDS Requirements
• FICON Manager
• FICON Control Unit Process (FCD)
Configuring FICON
• SAN-OS
• Mainframe
SAN-OS 3.0 Update
Troubleshooting FICON
Summary
FICON © 2010 Cisco Systems, Inc. All rights reserved. 14
FICON Topologies
A Point-to-Point (direct connection)
A Switched Point-Point Environment results when there is one
FICON director/switch in the channel path between the channel
and CU
A Cascaded Environment results when two FICON directors/
switches are connected in series along the channel path
between the channel and CU
An Entry Switch is the FICON director or switch connected to the
channel via the channel path
– In a cascaded environment, the Entry Switch is first switch
encountered in the channel path from the channel to the CU
– The Entry Switch is identified by a single hexadecimal byte equal
to it’s FC domain ID
FICON © 2010 Cisco Systems, Inc. All rights reserved. 15
FICON Topologies (Contd)
Point to Point
FICON In Point to Point topology, channel
Channel and control unit are directly
connected. There is no switch
between them.
FICON
Control
Unit
FICON © 2010 Cisco Systems, Inc. All rights reserved. 16
FICON Topologies (Contd)
Single Switch Fabric Path
FICON FICON Configuration
Channel Channel
In Single Switch Fabric path
configuration, all channels and
control units are attached to a
single switch
This is called ‘switched point-to-
point’ configuration.
FICON FICON FICON
Control Control Control
Unit Unit Unit
FICON © 2010 Cisco Systems, Inc. All rights reserved. 17
FICON Topologies (Contd)
Two Switch Fabric Path
FICON FICON Configuration
Channel Channel
In Two Switch Fabric path
configuration there may be more
than 2 switches in the fabric, but
only one hop between any pair of
channel and control units.
Multiple ISLs between connected
switches are allowed. However, the
fabric should perform load
balancing on these ISLs.
FICON FICON FICON
Control Control Control
Unit Unit Unit
FICON © 2010 Cisco Systems, Inc. All rights reserved. 18
FICON Addressing
8 Static in Link Address
Link
FICON fabric FICON defines concept of link
address (usually 0) address.
Each channel and control unit have
one unique link address.
Domain Area Port In older versions, link address used
to be single byte and could only be
used in single switch environment
Single Byte Link address to FCID mapping FC-SB3 defines two byte link
addresses which allow more than
one switch in FICON path.
16 Static in Link addresses are mapped to FC-
FICON fabric ID as depicted in figure.
Link address (usually 0)
Domain Area Port
Two Byte Link address to FCID mapping
(for Cascaded environments)
FICON © 2010 Cisco Systems, Inc. All rights reserved. 19
HCD – Point to Point
FICON © 2010 Cisco Systems, Inc. All rights reserved. 20
HCD – Cascaded
FICON © 2010 Cisco Systems, Inc. All rights reserved. 21
Using DM to identify Port Addresses
9513 - 16 ports per slot 9509 - 32 ports per slot
Notice FICON port
numbers are displayed
FICON © 2010 Cisco Systems, Inc. All rights reserved. 22
Objective & Agenda
FICON Concepts
• Terminology
• Topologies
• Addressing
• Frames
FICON Applications
MDS Requirements
• FICON Manager
• FICON Control Unit Process (FCD)
Configuring FICON
• SAN-OS
• Mainframe
SAN-OS 3.0 Update
Troubleshooting FICON
Summary
FICON © 2010 Cisco Systems, Inc. All rights reserved. 23
FICON Channel Initialization
Same Well
Known Fibre
CH to Switch Channel
Addresses
CH to CU
FICON © 2010 Cisco Systems, Inc. All rights reserved. 24
FICON Control Unit Initialization
Can register
Defined both SCSI and
In FC-FS FC-SB FC-4
types
port-wwn (vendor) :50:05:07:63:00:c2:04:2b (IBM)
node-wwn :50:05:07:63:00:c0:04:2b
Defined class :2,3
node-ip-addr :0.0.0.0
In FC-GS ipa :ff ff ff ff ff ff ff ff
fc4-types:fc4_features:scsi-fcp fcsb2-ch-cu fcsb2-cu-ch
[snip]
FICON © 2010 Cisco Systems, Inc. All rights reserved. 25
QSA – Query Security Attributes
Used by Channel to verify Fabric Binding and Insistent
Domain IDs is configured on MDS
Only used on Cascade environment
• QSA = 0x7E 00 00 00
• Revision field = 0x00 00
00 01
FICON © 2010 Cisco Systems, Inc. All rights reserved. 26
QSA Accept
QSA Accept sent if Security Features enabled
Bit Meaning
0 Fabric Binding
1 Static DomainID
2 Soft Zoning
3 Hard Zoning
4 SLAP
5 SRP
6 MS CT
0000 0011 Bit 0 & 1 are on
Authentication
7654 3210 BIT numbers 7-31 Reserved
FICON © 2010 Cisco Systems, Inc. All rights reserved. 27
ESA – Exchange Security Attributes
Used between Switches (ISL) to validate QSA information
Passed during ISL link initialization
Address by and to MDS Fabric Controller 0xFF FF FD
ESA frame command code is 0x32 00 00 00
272.210534 ff.ff.fd -> ff.ff.fd 0xa 0xffff SW_ILS 0x32000000! ESA
272.210594 ff.ff.fd -> ff.ff.fd 0xa 0xa FC Link Ctl, ACK1! Class F Ack
272.241828 ff.ff.fd -> ff.ff.fd 0xa 0xa SW_ILS SW_ACC (0x32000000)! ESA Accept
272.242168 ff.ff.fd -> ff.ff.fd 0xa 0xa FC Link Ctl, ACK1!
Class F Ack
FICON © 2010 Cisco Systems, Inc. All rights reserved. 28
RNID – Request Node Id Data
• Used to determine characteristics of node at other end
FICON © 2010 Cisco Systems, Inc. All rights reserved. 29
RNID Accept
Accept is broken into two sections:
Common ID Field - shows
pWWN and nWWN.
Ignored by FICON
Specific ID Data – shows
SB-2, FCID, Name,
Manufacturer, Model…..
FICON © 2010 Cisco Systems, Inc. All rights reserved. 30
LIRR, RLIR, DRLIR
LIRR (Link Incident Record Registration) used to
register for RLIR (Registered Link Incident Record) -
similar to SCR
RLIR used by N-Ports to send link incidents to
registered ports - similar to RSCN
DRLIR (Distribute Registered Link Incident Records)
used to send incidents to entire fabric
Switch receives DRLIR and send RLIR to registered N-
Ports
LIRR sent to all Switches in path and Control Unit
Sent to 0xFF FF FA (MDS Management Server)
FICON © 2010 Cisco Systems, Inc. All rights reserved. 31
ELP – Establish Logical Path
LPE – Logical Path Established
• ELP - Sent from
Channel to Control
Unit to verify optional
parameters and verify
logical path
• LPE – Sent from
Control Unit to
Channel as
verification of logical
path and pass
optional parameters
• LACK – Close
exchange
FICON © 2010 Cisco Systems, Inc. All rights reserved. 32
Frame Format
FC-SB-3 Frame Format
Bytes: 4 24 32 0 - 8160 4 4
S FC FICON E
Idles O Data Payload CRC O Idles
F Header Header F
Bit 0 7 8 15 16 23 24 31
Reserved CH_ID Reserved CU_ID
Device Address Reserved SB-3 Header
IUI DH_FLGS CCW Number
Reserved Token IU Header
Command / Data / Status / Control / Link Header
Reserved IU Count DIB Data Count DIB Header
LRC
FICON © 2010 Cisco Systems, Inc. All rights reserved. 33
FICON Read Operation
FICON FICON FCP FCP
Channel CU Initiator Target
Command DIB (READ) FCP_READ
Command DIB (RSP) FCP_DATA
Data DIB FCP_DATA
Data DIB FCP_RSP
Status DIB
FICON Read Operation FCP Read Operation
FICON © 2010 Cisco Systems, Inc. All rights reserved. 34
FICON Write Operation
FICON FICON FCP FCP
Channel CU Initiator Target
Command DIB (WRITE) FCP_WRITE
Command DIB (RSP) FCP_XFER_RDY
Data DIB FCP_DATA
Data DIB FCP_DATA
Status DIB FCP_RSP
FICON Write Operation FCP Write Operation
FICON © 2010 Cisco Systems, Inc. All rights reserved. 35
Flow Control
FICON uses standard FC-2 BB_Credit flow control
FICON also uses IU Pacing:
Similar to Fibre Channel EE_Credits
IU Pacing credit is sent to the channel by the CU
Indicates the maximum number of IUs a channel can send
before a command response IU is expected
IU Pacing
CRR CMR
Channel CU
Frames Frames
N_Port F_Port E_Port E_Port F_Port N_Port
R_RDYs R_RDYs
FC-2 BB_Credit FC-2 BB_Credit
FICON © 2010 Cisco Systems, Inc. All rights reserved. 36
Objective & Agenda
FICON Concepts
• Terminology
• Topologies
• Addressing
• Frames
FICON Applications
MDS Requirements
• FICON Manager
• FICON Control Unit Process (FCD)
Configuring FICON
• SAN-OS
• Mainframe
SAN-OS 3.0 Update
Troubleshooting FICON
Summary
FICON © 2010 Cisco Systems, Inc. All rights reserved. 37
Linux Support
IBM zSeries supports Linux LPARs:
Allows Linux to access ESCON/FICON storage
Linux also supports open systems FCP:
Same adapters used for FICON and FCP
Different firmware versions
Applications:
CPU-intensive
workloads FICON FC
Server/storage FICON FC
consolidation LINUX
NPIV (like MIF for FCP
FCP channels) FC
z/OS FICON FC FC
FICON FC
FC
FC
zSeries server
FICON © 2010 Cisco Systems, Inc. All rights reserved. 38
Disaster Recovery Applications
Data replication over FICON:
Backup, snapshot, and disaster tolerance
Local and remote replication
Globally Dispersed Parallel Sysplex (GDPS) w/
IBM XRC
IBM PtP VTS
GDPS w/ IBM PPRC (runs over FCP)
HDS HXRC
FICON © 2010 Cisco Systems, Inc. All rights reserved. 39
Disaster Recovery Applications
XRC - eXtended Remote Copy
FICON © 2010 Cisco Systems, Inc. All rights reserved. 40
Channel Extension
Channel extension is different than distance extension:
Distance extension simply means extending FICON over a
WAN
Channel extension involves protocol translation
Channel extenders proxy some commands—e.g. writes—to
increase performance over distance
MDS does not do channel extension in SAN-OS 3.X
FICON © 2010 Cisco Systems, Inc. All rights reserved. 41
Objective & Agenda
FICON Concepts
• Terminology
• Topologies
• Addressing
• Frames
FICON Applications
MDS Requirements
• FICON Manager
• FICON Control Unit Process (FCD)
Configuring FICON
• SAN-OS
• Mainframe
SAN-OS 3.0 Update
Troubleshooting FICON
Summary
FICON © 2010 Cisco Systems, Inc. All rights reserved. 42
MDS FICON Requirements
Feature Explanation
Predictable and persistent Since Link addresses are mapped to FC-ID, FICON requires FC-Ids to
FCID allocation be predictable and persistent.
FICON also requires uniform “port” byte (last byte in FC-ID) in FICON
fabric.
Support for new LIRR, RLIR, Link Incident Record (LIR) ELS reports link level events in the fabric
and DRLIR ELS to the channel N-port.
Examples of link incidents are loss of signal or synchronization,
primitive sequence timeout, Invalid primitive sequence, NOS
received
Concept similar to RSCN
Report Node Identification RNID is an ELS command, which can have common and specific
(RNID) ELS Node data.
FICON defines a specific node data field format.
Large Buffer to Buffer Credit FICON requires 60 B2B credit for 100 Km and 6 B2B credit for 10 Km.
Allocation MDS 9000 FICON supports up to 255 B2B Credits per Port +
extended buffer credits
FICON © 2010 Cisco Systems, Inc. All rights reserved. 43
MDS FICON Requirements
Feature Explanation
Insistent Domain ID Since Link addresses are mapped to FC-ID, FICON requires FC-Ids to be
predictable and hence Domain ID to be static in cascaded FICON
switches.
In-order Delivery FICON requires in-order delivery for class 2 and class 3 frames.
Fabric Binding and EFMD ELS Fabric Binding feature prevents unauthorized switches joining the fabric
based on switch WWNs.
Exchange Fabric Membership Data (EFMD) is used to distribute this
information fabric wide.
Security Attributes (QSA and FICON requires Fabric Binding and Insistent Domain ids to be enabled
ESA) in the entire fabric.
Query Security Attribute (QSA) is used by Channels to ensure these
features are turned on.
Exchange Security Attribute (ESA) is used by switches to validate these
attributes during E port bringup.
FICON © 2010 Cisco Systems, Inc. All rights reserved. 44
Objective & Agenda
FICON Concepts
• Terminology
• Topologies
• Addressing
• Frames
FICON Applications
MDS Requirements
• FICON Manager
• FICON Control Unit Process (FCD)
Configuring FICON
• SAN-OS
• Mainframe
SAN-OS 3.0 Update
Troubleshooting FICON
Summary
FICON © 2010 Cisco Systems, Inc. All rights reserved. 45
What is FICON manager?
FICON Manager
Running process that manages functionality required
to support FICON in a VSAN, doing the necessary
checks for enabling FICON and holding all FICON
specific configuration information.
FICON © 2010 Cisco Systems, Inc. All rights reserved. 46
FICON Manager
FICON Manager is responsible for:
- Providing CMI interface for configuration
- Serialization of FICON config
- Maintain and apply FICON configuration files
- Port Number Allocation and assignment
- Port Address Allocation and assignment
- Consistency Checks during enabling of FICON
- Maintaining port numbers in FICON VSANs
- Interact with ACL Manager for necessary ACLs during port bringup
- Exchanging new ELS During E-port bringup
FICON © 2010 Cisco Systems, Inc. All rights reserved. 47
FICON Manager Role
CLI
FCD
Mainframe SNMP GUI
(CUP)
FICON
Manager
FC2 Others
Vsan Port
Manager Manager
FICON © 2010 Cisco Systems, Inc. All rights reserved. 48
FICON Configuration Files
Up to 16 configuration files per FICON VSAN
- Restricted to 8 alphanumeric characters
Shared lock for file access
Proprietary IBM format IPL contains the following
attributes for each implemented port:
- Port Address Name
- Host Control
- Block
- Prohibit Dynamic Connectivity Mask
- ‘show ficon vsan <x>’ will show saved IPL files
FICON © 2010 Cisco Systems, Inc. All rights reserved. 49
Port Number and Port Address
Implemented/Unimplemented Port Number Slot 91xx 92xx 9506 9509 9513
Refers to any available port in the chassis
Reflected in the table to the right 1 0 - 39 0 - 31 0 - 31 0 - 31 0 - 15
Installed/Uninstalled Port
Only ports participating in FICON VSANs are
2 32 - 63 32 - 63 32 - 63 16 - 31
considered ‘installed’
i.e. If port 4 is in a FICON VSAN on a 16 port line 3 64 - 95 64 - 95 32 - 47
card in a chassis, then ports 0-3, 5-31 are
considered ‘uninstalled’ 4 96 - 127 96 - 127 48 - 63
Port number is the physical location on the switch
Port address is:
5 -- -- 64 - 79
- Port name
- Block/Unblock state
6 -- -- 80 - 95
- Connectivity attributes 7 128 - 159 --
- Same as port number unless port “swapped”
256 possible port number/addresses for 9509 (8 8 160 - 191 --
bits, some unimplemented)
0xFE and 0xFF reserved
9 192 - 223 96 - 111
An FCIP tunnel must be explicitly bound to a port 10 112 - 127
number for FICON
- Available port numbers depends on the slot the
IPS blade is located in 11 128 - 143
A Port Channel must be explicitly bound to a port
number for FICON
12 144 - 159
- Available port numbers depends on chassis ‘show
ficon first-available port-number’ reveals next port-
13 160 - 175
number free for allocation
Max
40 64 128 224 528
Physical*
Port
40-55 64-79 128-143 224-249 224 - 249
Channels
FICON © 2010 Cisco Systems, Inc. All rights reserved. 50
Using DM to identify Port Addresses
9513 - 16 ports per slot 9509 - 32 ports per slot
Notice FICON port
numbers are displayed
FICON © 2010 Cisco Systems, Inc. All rights reserved. 51
FICON States
Various states for FICON
STATE Meaning
Enabled User has configured FICON on a given VSAN.
Disabled User configures a non-FICON vsan.
Up FICON enabled and all prerequisites met
FICON enabled, but one or more prerequisites not
Down satisfied
FICON © 2010 Cisco Systems, Inc. All rights reserved. 52
FICON Prerequisites
Before FICON is ‘up’, FICON manager must check
prerequisite conditions:
In-order delivery enabled
Fabric Binding
Insistent Domain ID
No conflicting persistent FCIDs
- Does not allow user configuration for reserved range:
D.0.0 thru D.MaxPa.0 where:
D is domain
MaxPa is max Port Addr implemented in FICON
FICON © 2010 Cisco Systems, Inc. All rights reserved. 53
FICON Port Initialization
Blocked/Unblocked State
- A blocked port is configured ‘down’ by Port manager, at request of FICON
manager
- Blocked ports will continue to transmit an Off-Line State (OLS) primitive
sequence.
- Similar to shut/no shut, as blocked port must be reinitialized
- Exception is TE port:
- Removal of the VSAN from the VSAN allowed list
- Port Channels
- Members get port channel configuration (block/unblock)
Prohibit State
- FICON manager programs ACLs to allow communication between certain
ports and prohibit others
- Similar concept to zoning
- Port remains ‘up’
FICON © 2010 Cisco Systems, Inc. All rights reserved. 54
FICON Port Initialization ELS
Replies to Query Security Attributes (QSA)
- For Fx, Responsible for sending:
- Fabric Binding
- Insistent Domain ID
- In-order delivery
Sends Exchange Security Attributes (ESA)
- Upon bring up of (T)E Ports
Sends Request Node Identification (RNID)
- Sent after QSA reply
FICON © 2010 Cisco Systems, Inc. All rights reserved. 55
FICON Port Swap Support
Port Swap feature
- Must be supported for FICON directors
- Enables swapping configuration on two physical ports
- FICON manager swaps all config except Port #
- Ports left in “shut” state following…’no shut’ required
FICON FICON
Configuration Configuration
Including Including
Port Address, Port Address,
Block State, etc. Block State, etc.
Physical Port A Physical Port B
FICON © 2010 Cisco Systems, Inc. All rights reserved. 56
Objective & Agenda
FICON Concepts
• Terminology
• Topologies
• Addressing
• Frames
FICON Applications
MDS Requirements
• FICON Manager
• FICON Control Unit Process (FCD)
Configuring FICON
• SAN-OS
• Mainframe
SAN-OS 3.0 Update
Troubleshooting FICON
Summary
FICON © 2010 Cisco Systems, Inc. All rights reserved. 57
FICON Control Unit Port (CUP)
CUP protocol provides inband
management of switch from
mainframe
In order to be classified as
“FICON director”, CUP protocol
needs to supported.
CUP specification is proprietary
to IBM and is available only
through special license
agreement with them.
CUP can configure a port to be
prohibited from another port,
blocked, etc.
Complimentary to SNMP, CLI,
SANOS 3.0(2) View of FICON IPL File and is serialized through CMI
FICON Manager holds all
configuration and FICON specific
information; FCD acts as a client
FICON © 2010 Cisco Systems, Inc. All rights reserved. 58
FICON Control Device (FCD)
FCD process is responsible for:
- Acquiring persistent pWWN for Control Device
- Uses Key-Based Chassis Specific (KCS) WWNs
- Uses VSAN based nWWN
- Acquiring FCID from FPort server
- 0xFE is well defined address required
- FCID will be <domain, 0xFE, 00> for Control Device
- Device allegiance based Serialization of FICON config
- Support for both SB-3 configuration and query commands
FICON © 2010 Cisco Systems, Inc. All rights reserved. 59
FICON Port Configuration
Block Block all communication from a port. In this state port
is down and will not send any information.
Prohibit Each port (identified by port address) can block any
other port. This is done via ACLs.
Port Address Each port (address) can have a 24 character port
Name address name.
Port Swap Reassigns port address, connectivity attributes and
port address name of ports by swapping.
Service This is set when FICON Director determines that some
Required service is required on the port. Port is kept offline.
Cleared when required service performed.
Port Statistics The MDS needs to maintain detailed statistics on each
port. The statistics are reported based on port
numbers and not port addresses
FICON © 2010 Cisco Systems, Inc. All rights reserved. 60
HCD - CUP
FICON © 2010 Cisco Systems, Inc. All rights reserved. 61
FCD Control Device
An example of an FCD Control Device on a FICON vsan:
switch2# show ficon control-device sb3 vsan 1977
Control Unit Image:0x80b9df4
VSAN:1977 CU:0x4dfe00 CUI:0 CUD:0 CURLP:(nil)
ASYNC LP:(nil) MODE:0 STATE:1 CQ LEN:0 MAX:0
PRIMARY LP: VSAN:0 CH:0x0 CHI:0 CU:0x0 CUI:0
ALTERNATE LP: VSAN:0 CH:0x0 CHI:0 CU:0x0 CUI:0
Control Unit Image:0x80bcbec
VSAN:1978 CU:0x4efe00 CUI:0 CUD:0 CURLP:(nil)
ASYNC LP:(nil) MODE:0 STATE:1 CQ LEN:0 MAX:0 Domain ID = 77 (0x4d)
PRIMARY LP: VSAN:0 CH:0x0 CHI:0 CU:0x0 CUI:0 Well known port = 0xFE
ALTERNATE LP: VSAN:0 CH:0x0 CHI:0 CU:0x0 CUI:0
pWWN assigned
switch2# show flogi database vsan 1977
---------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
---------------------------------------------------------------------------
sup-fc0 1977 0x4dfe00 22:00:00:05:30:00:46:e0 27:b9:00:05:30:00:46:df
Total number of flogi = 1.
FICON © 2010 Cisco Systems, Inc. All rights reserved. 62
Objective & Agenda
FICON Concepts
• Terminology
• Topologies
• Addressing
• Frames
FICON Applications
MDS Requirements
• FICON Manager
• FICON Control Unit Process (FCD)
Configuring FICON
• SAN-OS
• Mainframe
SAN-OS 3.0 Update
Troubleshooting FICON
Summary
FICON © 2010 Cisco Systems, Inc. All rights reserved. 63
CLI Configuration of FICON
An example of using the ‘setup ficon’ script:
switch1# setup ficon
--- Ficon Configuration Dialog ---
This setup utility will guide you through basic Ficon Configuraton on the system.
Press Enter if you want to skip any dialog. Use ctrl-c at anytime to skip all remaining dialogs.
Would you like to enter the basic configuration dialog (yes/no)[yes]:yes
Enter vsan [1-4093] :3
vsan 3 does not exist, create it? (yes/no)[yes]:yes
Enable ficon on this vsan? (yes/no)[yes]:yes
The following options required for ficon will be enabled:
In-order-delivery
Fabric binding
Configure domain-id for this ficon vsan (1-239) :3
Would you like to configure ficon in cascaded mode: (yes/no)[no]:yes
Configure peer wwn (hh:hh:hh:hh:hh:hh:hh:hh): 20:00:00:0d:ec:0e:5b:80
swwn from
Configure peer domain (1-239) :4 remote switch
Would you like to configure additional peers: (yes/no)[no]: # show wwn switch
Enable SNMP to modify port connectivity parameters? (yes/no)[yes]:
Disable Host from modifying port connectivity parameters? (yes/no)[no]:
Enable active=saved? (yes/no)[yes]:no
Would you like to configure additional ficon vsans (yes/no)[yes]:no
The following configuration will be applied:
vsan database
vsan 3
in-order-guarantee vsan 3
fcdomain domain 3 static vsan 3
fabric-binding enable
fabric-binding database vsan 3
swwn 20:00:00:0d:ec:0e:5b:80 domain 4
fabric-binding activate vsan 3
zone default-zone permit vsan 3
ficon vsan 3
no active equals saved
FICON © 2010 Cisco Systems, Inc. All rights reserved. 64
CLI Configuration of FICON
A FICON port-channel configuration example:
switch1# conf t
Enter configuration commands, one per line. End with CNTL/Z.
MDS9506(config)# int port-channel 1
Remember that PortChannels need to be assigned FICON port numbers, so in
the next command we are determining what the next available FICON port number is.
switch1(config-if)# do show ficon first-available port
Port number 128(0x80) is available
Next step is to bind the FICON port number, set the port type to an ISL (E_Port),
and set it up so that only VSANs 1 and 10 can trunk across the PortChannel.
switch1(config-if)# ficon portnumber 0x80
switch1(config-if)# switchport mode e
switch1(config-if)# switchport trunk allowed vsan 1
switch1(config-if)# switchport trunk allowed vsan add 10
Next we move interfaces fc1/9 and fc2/1 into PortChannel 1
switch1(config-if)# int fc1/9
switch1(config-if)# channel-group 1 force
fc1/9 added to port-channel 1 and disabled
please do the same operation on the switch at the other end of the
port-channel,then do "no shutdown" at both ends to bring them up
switch1(config-if)# int fc2/1
switch1(config-if)# channel-group 1 force
fc2/1 added to port-channel 1 and disabled
please do the same operation on the switch at the other end of the
port-channel,then do "no shutdown" at both ends to bring them up
FICON © 2010 Cisco Systems, Inc. All rights reserved. 65
CLI Configuration of FICON
A FICON port-channel configuration example continued:
Next we no shut interfaces fc1/9, fc2/1, and PortChannel 1.
switch1(config-if)# int fc1/9, fc2/1
switch1(config-if)# no shut
switch1(config)# int port-channel 1
switch1(config-if)# no shut
Next we repeat the configuration steps on the switch2.
switch2(config)# int port-channel 1
switch2(config-if)# do show ficon first-available port-number
Port number 64(0x40) is available
swich2(config-if)# ficon portnumber 0x40
switch2(config-if)# switchport trunk allowed vsan 1
switch2(config-if)# switchport trunk allowed vsan add 10
swich2(config-if)# int fc1/1
switch2(config-if)# channel-group 1 force
fc1/1 added to port-channel 1 and disabled
please do the same operation on the switch at the other end of the
port-channel, then do "no shutdown" at both ends to bring them up
switch2(config-if)# int fc1/9
switch2(config-if)# channel-group 1 force
fc1/9 added to port-channel 1 and disabled
please do the same operation on the switch at the other end of the
port-channel,
then do "no shutdown" at both ends to bring them up
switch2(config)# int fc1/1, fc1/9
switch2(config-if)# no shut
switch2(config)# interface port-channel 1
switch2(config-if)# no shut
switch2(config-if)# end
FICON © 2010 Cisco Systems, Inc. All rights reserved. 66
CLI RLIR
MDS-1# show rlir ?
erl Show Established Registration List
history Show link incident history
internal Show RLIR internal
recent Show recent link incident
statistics Show RLIR Statistics
MDS-1# show rlir internal vsan 3
Internals for VSAN: 3
------------------------
VSAN state : Active
Ficon State : Up
Fabric Name : 20:03:00:05:30:00:51:1f
R_A_TOV : 10000 ms
Local Domain : 0x3(3)
Remote Domains : None
MDS-1# show rlir statistics vsan 3
Statistics for VSAN: 3
-------------------------
Number of LIRR received = 0
Number of LIRR ACC sent = 0
Number of LIRR RJT sent = 0
Number of RLIR sent = 0
Number of RLIR ACC received = 0
Number of RLIR RJT received = 0
Number of DRLIR received = 0
Number of DRLIR ACC sent = 0
Number of DRLIR RJT sent = 0
Number of DRLIR sent = 0
Number of DRLIR ACC received = 0
Number of DRLIR RJT received = 0
FICON © 2010 Cisco Systems, Inc. All rights reserved. 67
CLI RNID and Port attributes
MDS-1# show ficon vsan 5 portaddress 0x21
Port Address 33(0x21) is up in vsan 5
Port number is 33(0x21), Interface is fc1/3
Port name is
Port is not admin blocked
Prohibited port addresses are 250-253,255(0xfa-0xfd,0xff)
Admin port mode is FX
Port mode is F
Peer is type 002064 model 104 manufactured by IBM
MDS-1# show ficon vsan 5 portaddress brief installed
-----------------------------------------------------------------------------
Port Port Interface Admin Status Oper FCID
Address Number Blocked Mode
-----------------------------------------------------------------------------
0x40 0x40 port-channel 5 off up TE --
0xfe 0xfe -- off up -- --
FICON © 2010 Cisco Systems, Inc. All rights reserved. 68
CLI FICON File Configuration
switch1# show run
ficon vsan 2
file IPL
switch1(config)# ficon vsan 2
switch1(config-ficon)# ?
FICON Configuration:
active Configure automatic saving of ficon configuration
code-page Ficon configuration for codepage on a VSAN
do EXEC command
end Exit from configure mode
exit Exit from this submode
file Ficon file configuration on a VSAN
host Enable host control of ficon
no Negate a command or set its defaults
portaddress Ficon configuration for portaddress on a VSAN
snmp Enable snmp control of ficon
switch1(config-ficon)# portaddress 1
switch1(config-ficon-portaddr)# ?
FICON config for portaddress:
block Block a range of portaddress
do EXEC command
end Exit from configure mode
exit Exit from this submode
name Give name for a portaddress
no Negate a command or set its defaults
prohibit Prohibit communication with a range of portaddress
FICON © 2010 Cisco Systems, Inc. All rights reserved. 69
Creating FICON VSAN from DM
FICON © 2010 Cisco Systems, Inc. All rights reserved. 70
Creating FICON VSAN from FM
FICON © 2010 Cisco Systems, Inc. All rights reserved. 71
Creating/Modifying FICON Files from DM
FICON © 2010 Cisco Systems, Inc. All rights reserved. 72
FICON Configuration File (IPL)
Edit IPL file, Block – Prohibit – Name Ports
FICON © 2010 Cisco Systems, Inc. All rights reserved. 73
Creating Port Channel through FM
Now select the Port Channel Wizard
Determine the ISLs to add to the Port Channel
FICON © 2010 Cisco Systems, Inc. All rights reserved. 74
Creating Port Channel through FM
(cont’d)
Select a Port number for the logical PC
Fill in Port Channel Information Port Channel is now created
FICON © 2010 Cisco Systems, Inc. All rights reserved. 75
Showing Port Channel through DM
Select Interface->Port Channels
All Port channel information is displayed
FICON © 2010 Cisco Systems, Inc. All rights reserved. 76
Swapping FICON Ports through DM
Select Ports to swap Select FICON->Swap Selected Ports
FICON © 2010 Cisco Systems, Inc. All rights reserved. 77
Swapping FICON Ports through DM
(contd)
Notice the swapped ports
FICON © 2010 Cisco Systems, Inc. All rights reserved. 78
FICON VSAN Options
• Offline SW = Enables ability for Host to disable FICON or offline MDS
• Sync Time = Enables ability for Host to sync MDS time with Host
• Port Control – Host = Enables ability for Host to Name, Prohibit, Block MDS
ports
• Port Control – SNMP = Enables ability for Device Manager to Name, Prohibit,
Block MDS ports
• Active=Saved = Enables ability for IPL (ficon running configuration) to be
saved automatically and immediately
FICON © 2010 Cisco Systems, Inc. All rights reserved. 79
FICON Load Balancing
Packets of a flow (SID,DID) or an exchange (SID, DID,
OXID) are received in the order they were sent. Some
applications can not handle Out Of Order exchanges
FICON is one such application. In fact FICON uses two
unique exchanges, one for the entire outbound exchange
(For example a write chain) with an OXID allocated by
the CH. The CU sends back a command response with a
different OXID representing the inbound exchange
FICON © 2010 Cisco Systems, Inc. All rights reserved. 80
FICON Load Balancing
The FICON load balancing tool has been enhanced in
the Fabric Manager 3.0 release.
FICON © 2010 Cisco Systems, Inc. All rights reserved. 81
Objective & Agenda
FICON Concepts
• Terminology
• Topologies
• Addressing
• Frames
FICON Applications
MDS Requirements
• FICON Manager
• FICON Control Unit Process (FCD)
Configuring FICON
• SAN-OS
• Mainframe
SAN-OS 3.0 Update
Troubleshooting FICON
Summary
FICON © 2010 Cisco Systems, Inc. All rights reserved. 82
Configuring Devices on Mainframe
Consider the following example and IOCP definition
FICON © 2010 Cisco Systems, Inc. All rights reserved. 83
Configuring Devices on Mainframe
Configuring CHPID 89 online HCD
SDSF
TSO Command Input
FICON © 2010 Cisco Systems, Inc. All rights reserved. 84
Configuring Devices on Mainframe
Varying devices online
SDSF
TSO Command Input
HCD
Vary devices D40 through D74 online. I/O errors occur
because the volumes at these addresses are not
initialized.
V D40-D74,ONLINE
IEF503I UNIT 0D50 I/O ERROR
IEF503I UNIT 0D51 I/O ERROR
...
Following command automatically executed based on
the range that was varied on in previous command.
D U,,,0D40,00052,L=OZI-Z VARY RANGE DISPLAY
IEE457I 10.55.06 UNIT STATUS 198
UNIT TYPE STATUS VOLSER VOLSTATE
0D40 3390 O PPRD40 PRIV/RSDNT
0D41 3390 O PPRD41 PRIV/RSDNT
…
0D50 3390 OFFLINE /RSDNT
FICON © 2010 Cisco Systems, Inc. All rights reserved. 85
Displaying Devices on Mainframe
Notice the online state for HCD
configured CHPID 86
TSO Notice the availability of devices
configured in HCD
HCD
HCD
TSO
FICON © 2010 Cisco Systems, Inc. All rights reserved. 86
Displaying Devices on Mainframe
Displaying CHPID 86 and 89 Matrix
52 Devices
Unit = 3390
Begins at D40
CUP Device
Unit = 2032
at well-known
0xFE
FICON © 2010 Cisco Systems, Inc. All rights reserved. 87
Displaying Devices on Mainframe
Displaying Path Matrix for Device D40
CHPID with corresponding
Destination Link Address
Entry Link Address
FICON © 2010 Cisco Systems, Inc. All rights reserved. 88
Objective & Agenda
FICON Concepts
• Terminology
• Topologies
• Addressing
• Frames
FICON Applications
MDS Requirements
• FICON Manager
• FICON Control Unit Process (FCD)
Configuring FICON
• SAN-OS
• Mainframe
SAN-OS 3.0 Update
Troubleshooting FICON
Summary
FICON © 2010 Cisco Systems, Inc. All rights reserved. 89
New FICON Port Numbering scheme
Port-numbers are user-configurable
Duplicates allowed
- Port-number 1 can be assigned to fc1/1 and fc2/1
Assigned on per-slot or logical basis
ficon slot <x> assign port-numbers <range>
ficon logical-port assign port-numbers <range>
Default configuration
- No change for 1, 2, 6 and 9 slot chassis
- 13 slot chassis will have 11 slots w/ 16 port-numbers each
HA compatibility
FICON © 2010 Cisco Systems, Inc. All rights reserved. 90
Configuring Port-Numbers
Consider the following 9509 with LC modules in slots 2, 3, 4, and 8
switch1# sh module
Mod Ports Module-Type Model Status
--- ----- -------------------------------- ------------------ ------------
2 24 1/2/4 Gbps FC Module DS-X9124 ok
3 8 IP Storage Services Module DS-X9308-SMIP ok
4 16 1/2 Gbps FC Module DS-X9016 ok
5 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 active *
6 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 ha-standby
8 4 10 Gbps FC Module DS-X9704 ok
Finding out what is assigned – this example show the defaults for a 9509
switch1# show ficon port-numbers assign
ficon slot 1 assign port-numbers 0-31
ficon slot 2 assign port-numbers 32-63
ficon slot 3 assign port-numbers 64-95
ficon slot 4 assign port-numbers 96-127
ficon slot 7 assign port-numbers 128-159
ficon slot 8 assign port-numbers 160-191
ficon slot 9 assign port-numbers 192-223
ficon logical-port assign port-numbers 224-249
FICON © 2010 Cisco Systems, Inc. All rights reserved. 91
Configuring Port-Numbers
Unreserve FICON port-numbers from an unused slot, slots 1, 7, & 9 in this case
switch1# show ficon port-numbers assign
ficon slot 1 assign port-numbers 0-31
ficon slot 2 assign port-numbers 32-63
ficon slot 3 assign port-numbers 64-95
ficon slot 4 assign port-numbers 96-127
ficon slot 7 assign port-numbers 128-159
ficon slot 8 assign port-numbers 160-191
ficon slot 9 assign port-numbers 192-223
ficon logical-port assign port-numbers 224-249
switch(config)# no ficon slot 1 assign port-numbers 0-31
switch(config)# no ficon slot 7 assign port-numbers 128-159
switch(config)# no ficon slot 9 assign port-numbers 192-223
Reserve port-numbers for slot 2, a 24 port card
switch(config)# ficon slot 2 assign port-numbers 128-151
FICON © 2010 Cisco Systems, Inc. All rights reserved. 92
Configuring Port-Numbers
The output should reflect the current status for all the ports
switch(config)# do show ficon vsan 2 portaddress brief | inc fc2
0x80 0x80 fc2/1 off not installed -- --
0x81 0x81 fc2/2 off not installed -- --
0x82 0x82 fc2/3 off not installed -- --
0x83 0x83 fc2/4 off not installed -- --
0x84 0x84 fc2/5 off not installed -- --
0x85 0x85 fc2/6 off not installed -- --
0x86 0x86 fc2/7 off not installed -- --
0x87 0x87 fc2/8 off not installed -- --
0x88 0x88 fc2/9 off not installed -- --
0x89 0x89 fc2/10 off not installed -- --
0x8a 0x8a fc2/11 off not installed -- --
0x8b 0x8b fc2/12 off not installed -- --
0x8c 0x8c fc2/13 off not installed -- --
0x8d 0x8d fc2/14 off not installed -- --
0x8e 0x8e fc2/15 off not installed -- --
0x8f 0x8f fc2/16 off not installed -- --
0x90 0x90 fc2/17 off not installed -- --
0x91 0x91 fc2/18 off not installed -- --
0x92 0x92 fc2/19 off not installed -- --
0x93 0x93 fc2/20 off not installed -- --
0x94 0x94 fc2/21 off not installed -- --
0x95 0x95 fc2/22 off not installed -- --
0x96 0x96 fc2/23 off not installed -- --
0x97 0x97 fc2/24 off not installed -- --
FICON © 2010 Cisco Systems, Inc. All rights reserved. 93
Configuring Port-Numbers
Also the running config will show the newly reserved ports
Switch-9509(config)# do sh run | incl assign
no ficon slot 1 assign port-numbers
ficon slot 2 assign port-numbers 128-151 force
no ficon slot 7 assign port-numbers
no ficon slot 9 assign port-numbers
FICON © 2010 Cisco Systems, Inc. All rights reserved. 94
Configuring Port-Numbers
Assign logical port-numbers for FCIP interfaces and Port-channels
switch(config)# ficon logical-port assign port-numbers 16-31,224-249
Logical ports are shown in a show running config
Switch-9509(config)# do sh run | incl assign
no ficon logical-port assign port-numbers
no ficon slot 1 assign port-numbers
ficon slot 2 assign port-numbers 128-151 force
no ficon slot 7 assign port-numbers
no ficon slot 9 assign port-numbers
ficon logical-port assign port-numbers 16-31, 224-249 force
FICON © 2010 Cisco Systems, Inc. All rights reserved. 95
Objective & Agenda
FICON Concepts
• Terminology
• Topologies
• Addressing
• Frames
FICON Applications
MDS Requirements
• FICON Manager
• FICON Control Unit Process (FCD)
Configuring FICON
• SAN-OS
• Mainframe
SAN-OS 3.0 Update
Troubleshooting FICON
Summary
FICON © 2010 Cisco Systems, Inc. All rights reserved. 96
Fabric Binding Troubleshooting
First, verify that identical Fabric Binding Databases are active on both
switches
Debug information:
debug fabric-binding all (collect from both switches)
Activate the fabric binding database on both switchs while the debug is running.
fabric-binding activate vsan X
Flap the port-channel or ISL interface (Only if customer can take the outage)
Collect the following
show fabric-binding statis vsan X (x=vsan 500)
show fabric-binding efmd statistics vsan X
show fabric-binding database vsan X
show fabric-binding database act vsan X
show fabric-binding internal global
show fabric-binding internal info vsan X
show fabric-binding internal event
show fabric-binding internal efmd event interface fcx/x or port-channel x
FICON © 2010 Cisco Systems, Inc. All rights reserved. 97
CUP Troubleshooting
Collect the following:
show tech-support details
show ficon control-device sb3
show ficon control-device internal info
show ficon control-device internal event-history vsan x (x=ficon vsan)
show ficon control-device internal event-history errors
show ficon control-device internal event-history msgs
show ficon control-device internal event-history vsan <x>
sh ficon internal event-history errors
sh ficon internal event-history msgs
sh ficon internal event-history vsan <x>
From the Mainframe:
MVS SYSLOG before, during, and after the failure.
Output from D IOS,FICON
FICON © 2010 Cisco Systems, Inc. All rights reserved. 98
Basic Troubleshooting
Troubleshoot port problems (performance, failing) like any other case. Assume the switch is the problem
and rule it out!
attach mod x
term len 0
show hardware internal debug-info interface fc x/x
clear asic-cnt list-all-devices
clear asic-cnt all
clear asic-cnt device-id 3
From Supervisor
show flogi internal event interface fcx/y (for both CH and CU interfaces)
show port internal event-history interface fcx/y
show ficon internal event-history vsan x interface fcx/y.
show ficon internal event-history vsan 1
show ficon internal event-history msgs
show ficon internal event-history errors
show ficon internal info
FICON © 2010 Cisco Systems, Inc. All rights reserved. 99
Basic Troubleshooting
The list of commands that are useful from SAN-OS(later NX-OS) CLI:
Setup ficon
ficon slot X assign port-numbers 48-63
show ficon port-numbers assign slot 3
show ficon port-numbers assign logical-port
show ficon vsan X portaddress brief
show ficon vsan vsan-id file name filename portaddress
show ficon vsan 20 portaddress 8 counters
ficon swap portnumber old-port-number new-port-number
ficon swap interface old-interface new-interface
clear ficon vsan 1 allegiance
ficon vsan 2 apply file SampleFile
ficon vsan vsan-id offline
ficon vsan vsan-id online
show ficon control-device sb3 vsan X
show flogi database vsan X
FICON © 2010 Cisco Systems, Inc. All rights reserved. 100
Basic Troubleshooting continued
Gather the following Mainframe displays before and
after any failures or recreates
D M=DEV(nnnn)
D M=CHP(mmm)
Collect the MVS SYSLOG before, during, and after
the failure.
Collect any messages that may appear on Hardware
Console, for example, "LINK LEVEL ERROR -
FICON PLOGI ELS ERROR",
FICON © 2010 Cisco Systems, Inc. All rights reserved. 101
Summary / Review
FICON uses the Single-Byte Command Code Sets-3 (SB-3) and
Single-Byte Command Code Sets-4 (SB-4)
Three types of FICON topologies: Point-to-point, Switched point-to-
point (via a FICON Director) ,Cascaded FICON Directors (through
two FICON Directors)
For FICON, there is a maximum of 1 hop from CH to CU
Allow for fabric-binding between FICON fabrics
Utilize ‘setup ficon’ command
Utilize GUI (FM - DM) for basic troubleshooting / configuration
FICON © 2010 Cisco Systems, Inc. All rights reserved. 102
References
FICON Planning and Implementation Guide:
http://www.redbooks.ibm.com/abstracts/sg246497.html?Open
MDS FICON Configuration Guide:
http://www.cisco.com/en/US/docs/switches/datacenter/
mds9000/sw/4_1/configuration/guides/cli_4_1/ficon.html
MDS Fabric Manager Configuration Guide
http://www.cisco.com/en/US/docs/switches/datacenter/
mds9000/sw/4_1/configuration/guides/fm_4_1/ficon.html
FICON © 2010 Cisco Systems, Inc. All rights reserved. 103
Question and Answer…
Thanks for attending this session and Good Luck in your CCIE Journey
FICON © 2010 Cisco Systems, Inc. All rights reserved. 104
FICON © 2010 Cisco Systems, Inc. All rights reserved. 105