Data Center Applications Standards Reference Guide: White Paper
Data Center Applications Standards Reference Guide: White Paper
Introduction 3
InfiniBand 8
www.commscope.com 2
Scope
This technical paper provides an overview of the most prevalent networking and storage applications in
modern data centers. Additionally, it offers information about the different structured cabling systems
capable of running these applications and also functions as an application reference book for network
cabling designers in data centers.
Layers
Multilayer Switch
Core
10/40/100 Gigabyte Ethernet
Aggregation
Fibre Channel
Edge
Layer 2 Switch
The applications can be grouped into: IEC 11801 (ed. 2.2—Generic Cabling for Customer Premises) it
defines the cabling systems for copper and fiber applications in
Application Functional data centers. These standards are used as references for linking
Networking area the mentioned data center applications to standards-compliant
Ethernet
(red marked) cabling systems.
Storage area
Fibre Channel
(yellow marked)
2.1 Copper cabling systems
High-performance server cluster and storage
InfiniBand
(blue marked) ISO/IEC 11801 ed. 2.2 defines the following performance
classifications for balanced cabling:
Note: IP convergence is becoming more and more popular in Data centers, resulting in
deployment of Fibre Channel over Ethernet (FCoE) and InfiniBand over Ethernet (IoE)
applications. Although they are not listed here, they will be covered in later chapters. Class Specified up to
Class D 100 MHz
Class E 250 MHz
Class E A 500 MHz
Class F 600 MHz
Class FA 1000 MHz
www.commscope.com 3
The following copper connector definitions have been made for the 2.3.1 Balanced cabling
equipment outlet (EO)
The main distribution cabling shall be designed to provide a
minimum of Class EA channel performance as specified in ISO/IEC
Category 6A unscreened IEC 60603-7-41 11801.
Copper Category 6A
Minimum modal bandwidth
MHz*km Minimum OM3,
Multimode fiber
Effective recommended OM4
Overfilled launch
modal LC (1-2 fibers)
bandwidth
bandwidth Fiber connector
MPO (>2 fibers)
Wavelength 850 nm 1300 nm 850 nm
Category Nominal core diameter µm Table 5: ANSI/TIA 942-A requirements
www.commscope.com 4
(OM2) and 62.5/125 µm (OM1) fibers, the international cabling eight fibers (40GBASE-SR4) or 20 fibers (100GBASE-SR10)
standards had to define a new laser-optimized 50/125 µm fiber terminated in the multifiber connector MPO.
(OM3) featuring a much more precise fiber core index profile. The
much higher effective modal bandwidth of that fiber allows longer IEEE 802.3bm was published in March 2015 and defines the
link lengths meeting building and data center requirements. second generation of 100 GbE using four full-duplex data streams
of 25 Gbit/s. This results in using eight fibers terminated in the
There are three 10GBE multimode fiber applications used in data multifiber connector MPO.
centers: 10GBASE-LX4, 10GBASE-LRM and 10GBASE-SR. All are
dual-fiber applications for transmit and receive. Table 7 gives a complete overview about all 40/100 GBE
applications.
3.3.2. 10 Gigabit Ethernet over copper 3.4.1. 40GBASE-SR4 (40GBE) and 100GBASE-SR4
10GBE over copper (10GBASE-T), defined as IEEE 802.3an, was (100GBE)
similarly challenging for copper cabling systems as IEEE 802.3ae
has been for fiber-optic systems. Because of the link length The following graphics illustrate the concept of the parallel data
limitations for Class E/Category 6 UTP systems to 37 meters, transmission on multifiber links using the MPO connector and the
the cabling standards had to define the new cabling performance respective pinout for 40GBASE-SR4 (40GBE) and 100GBASE-SR4
Class EA, which is the minimum cabling requirement in data (100GBE).
center cabling. Class EA cabling supports 100-meter channels for
10GBASE-T.
Table 6: Application vs. media vs. link length for 10-gigabit Ethernet
Channel length Channel length Channel length Channel length Channel length Channel length
40GBASE-SR4 100GBASE-SR10 100GBASE-SR4 40GBASE-LR4 100GBASE-LR4 100GBASE-ER4
Table 7: Channel length definitions for 40/100GBE depending on application and fiber type
01
data center cabling standard for cabled optical fiber in data centers
is OM3. Other fiber types are listed for reference only.
12
www.commscope.com 5
Network application Maximum channel insertion loss (dB)
Multimode Singlemode
850 nm 1300 nm 1310 nm
IEEE 802-3: 10BASE-FL and FB 12.5 (6.8)** – –
IEEE 802-3: 1000BASE-SX 2.6 (3.56)** – –
IEEE 802-3: 1000BASE-LX – 2.35 4.56
ISO/IEC 8802-3: 100BASE-FX 11.0 (6.0) –
IEEE 802.3: 10GBASE-LX4 2.00 6.20
IEEE 802.3: 10GBASE-SR/SW 1.60 (62.5)/1.80 (OM2 50)/2.60 (OM3) – –
IEEE 802.3: 10GBASE-LR/LW – – 6.20
IEEE 802.3: 40GBASE-LR4 – – 6.70
IEEE 802.3: 100GBASE-LR4 – – 6.3
IEEE 802.3: 100GBASE-ER4 – – 18.0
IEEE 802.3: 40GBASE-SR4 1.9 (100m OM3)/1.5 (150m OM4)* – –
IEEE 802.3: 100GBASE-SR10 1.9 (100m OM3)/1.5 (150m OM4)* ––
IEEE 802.3: 100GBASE-SR4 1.9 (70m OM3)/1.9 (100m OM4) – –
* While all listed applications allocate 1.5 db insertion loss for splices and connections within a cabling channel, 40 and 100 GBE on OM4 requires a lower insertion loss
of 1 db for all splices and connections in a channel, requiring an engineered link using a FO cabling system with highest performing connector technology.
** Values shown are for 62.5/125 µm. Values in parenthesis represent 50/125 µm.
Table 8: Channel budget for Ethernet applications depending on application and fiber type
01
24
QSFP-40G-SR-BD QSFP-40G-SR-BD
Figure 4: 100GBASE-SR10 full duplex operation on 20 fibers Duplex Multi-mode Fiber with LC
Connectors at both ends
10 fibers RX
10 fibers TX
www.commscope.com 6
The channel loss budget for 40 Gb/s BiDi is 2 dB. This results in the 5.2. Fibre Channel over fiber
following cabling specifications:
Fibre Channel is primarily deployed over fiber-optic cabling systems.
Wavelength Cable Core size Modal bandwidth Cable
type (MHz x km) distance Channel length is a function of fiber type and specific Fibre Channel
500 (OM2) 30m application. The following table provides a complete overview.
850 to 900 50.0
MMF 2000 (OM3) 100m*
nm microns
4700 (OM4) 125m* Channel lengths / m
32
* Connector loss budget for OM3 fiber is 1.5 dB 1 Gbps 2 Gbps 4 Gbps 8 Gbps 16 Gbps
Fiber type Gbps
FC FC FC FC FC
** 125 meters over OM4 fiber is with an engineered link with 1 dB FC
budget for connector loss OM3,
860 500 380 150 100 70
50/125 μm
Table 9: Channel lengths for 40 Gb/s BiDi
OM4, min. min.
400 190 125 100
50/125 μm 860 500
In addition to the BiDi application, other nonstandard
OS1/OS2,
implementations—such as 40GBASE-eSR4—have been implemented. 9/125μm
10000 10000 10000 10000 10000 10000
40GBASE-eSR4 transceivers provide an extended range performance
compared with standard 40GBASE-SR4 transceivers.
Note: Fibre Channel specifies these lengths assuming an insertion loss of 1.5 dB (MM)
5. Fibre Channel applications (INCITS, T11) and 2 dB (SM) for all connections and splices in the channel. See 5.5 for deviating
multimode channel lengths depending on different connection/splice losses.
Fibre Channel (FC) is a gigabit-speed network technology primarily Table 11: Channel length definitions for Fibre Channel over fiber
used for storage networking. Fibre Channel is standardized in depending on application and fiber type
the T11 Technical Committee of the InterNational Committee for
Information Technology Standards (INCITS), an American National 5.3. Fibre Channel over Ethernet (FCoE)
Standards Institute (ANSI)–accredited standards committee. It started
primarily in the supercomputer field, but has become the standard Fibre Channel over Ethernet (FCoE) is an extension of the Fibre
connection type for storage area networks (SAN) in data centers. Channel storage protocol that uses Ethernet as its physical
transmission technology. FCoE combines Fibre Channel and
The following shows the FCIA (Fibre Channel Industry Association) Ethernet to provide end users with a “converged” network option
technology roadmap for this application. for storage SAN connectivity and LAN traffic. Combined with
enhancements to Ethernet, FCoE allows data centers to consolidate
5.1 Fibre Channel—Overview
Fibre channel speed roadmap—(FC V20)
T11 spec technically
Product naming Throughput (MBps) Line rate (GBaud) Market availability (years)‡
completed (years)‡
1GFC 200 1.0625 1996 1997
2GFC 400 2.125 2000 2001
4GFC 800 4.25 2003 2005
8GFC 1600 8.5 2006 2008
16GFC 3200 14.025 2009 2011
32GFC 6400 28.05 2013 2016
128GFC 25600 4X28.05 2014 2016
64GFC 12800 56.1 2017 2019
256GFC 51200 4X56.1 2017 2019
128GFC 25600 TBD 2020 Market Demand
256GFC 51200 TBD 2023 Market Demand
512GFC 102400 TBD 2026 Market Demand
1TFC 204800 TBD 2029 Market Demand
“FC” used throughout all applications for Fibre Channel infrastructure their I/O and network infrastructures into a converged network.
and devices, including edge and ISL interconnects. Each speed maintains FCoE is simply a transmission method in which the Fibre Channel
backward compatibility at least two previous generations (e.g., 8GFC frame is encapsulated into an Ethernet frame at the server. The
backward compatible to 4GFC and 2GFC) server encapsulates Fibre Channel frames into Ethernet frames
before sending them over the LAN, and de-encapsulates them
* Line rate: All “…GFC” speeds listed above are single-lane serial stream
when FCoE frames are received. Server input/output (I/O)
I/Os. All “…GFCp” speeds listed above are multi-lane I/Os
consolidation combines the network interface card (NIC) and host
‡ Dates: Future dates estimated bus adapter (HBA) cards into a single converged network adapter
Table 10: Fibre Channel speed roadmap (CNA). Fibre Channel encapsulation requires use of 10-gigabit
(source: Fibre Channel Industry Association) Ethernet transmission electronics.
www.commscope.com 7
Fibre Channel speed roadmap (V20)
Fibre Channel speed roadmap—FCoE
Spec technically completed
Product naming Throughput (MBps) Equivalent line rate (GBaud) Market availability (year)
(year)*
10GFCoE 2400 10.52 2002 2009
40GFCoE 9600 41.25 2010 2013
100GFCoE 24000 10X10.3125 2010 Market Demand
100GFCoE 24000 4X25.78125 2015 Market Demand
400GFCoE 96000 8X51.5625 2017 Market Demand
Channel Budget / dB
Fiber type 1 Gbps FC 2 Gbps FC 3 Gbps FC 4 Gbps FC 5 Gbps FC 6 Gbps FC
OM3, 50/125 μm 4.62 3.31 2.88 2.04 1.86 1.87
OM4, 50/125 μm 4.62 3.31 2.95 2.19 1.95 1.87
OS1/OS2, 9/125μm 7.8 7.8 7.8 6.4 6.4 6.21
Table 13: Channel budget for Fibre Channel over fiber depending on Although InfiniBand was developed to address I/O performance,
application and fiber type
InfiniBand is widely deployed within high-performance compute
The multimode channel lengths given in 5.2 are based on an (HPC) clusters and storage networks due to the high bandwidth
allocation of 1.5 dB insertion loss of all connectors and splices and low latency transport characteristics it offers.
within a channel. However, a connector/splice loss in a channel
deviating from the 1.5 dB results in different maximum channel
lengths. The following table gives an overview.
4G FC 400-SN max operating distance and loss budget for different connection losses 8G FC 800-SN max operating distance and loss budget for different connection losses
Distance (m) / Loss budget (dB) Distance (m) / Loss budget (dB)
Connection loss Connection loss
Fiber type Fiber Type
3.0 dB 2.4 dB 2.0 dB 1.5 dB 1.0 dB 3.0 dB 2.4 dB 2.0 dB 1.5 dB 1.0 dB
M5F (OM4) 200 / 3.72 300 / 3.49 370 / 3.34 400 / 2.95 450 / 2.63 M5F (OM4) 50 / 3.18 120 / 2.83 160 / 2.58 190 / 2.19 220 / 1.80
M5E (OM3) 150 / 3.54 290 / 3.45 320 / 3.16 380 / 2.88 400 / 2.45 M5E (OM3) 35 / 3.13 110 / 2.80 125 / 2.45 150 / 2.04 180 / 1.65
16G FC 1600-SN max operating distance and loss budget for different 32G FC 3200-SN max operating distance and loss budget for different
connection losses connection losses
Distance (m) / Loss budget (dB) Distance (m) / Loss budget (dB)
Connection loss Connection loss
Fiber type Fiber type
3.0 dB 2.4 dB 2.0 dB 1.5 dB 1.0 dB 3.0 dB 2.4 dB 2.0 dB 1.5 dB 1.0 dB
M5F (OM4) N/A 50 / 2.58 100 / 2.36 125 / 1.95 150 / 1.54 M5F (OM4) 20 / 3.04 65 / 2.64 80 / 2.36 100 / 1.86 110 / 1.48
M5E (OM3) N/A 40 / 2.54 75 / 2.27 100 / 1.86 120 / 1.43 M5E (OM3) 15 / 3.03 45 / 2.64 60 / 2.24 70 / 1.87 80 / 1.41
Table 14: Channel length for Fibre Channel over fiber depending on application,
fiber type and connection/splice loss
www.commscope.com 8
The following graph shows the InfiniBand Roadmap taken from the
Cabling System Overview
home page of the InfiniBand Trade Association
Designed for the requirements of modern data centers
(www.infinibandta.org).
Computer Clustering
Infiniband solutions
Cable Routing
SDR—Single data rate FDR—Fourteen data rate HDR—High data rate Fiber cable management
for MM and SM
DDR—Double data rate EDR—Enhanced data rate NDR—Next data rate
Capacity Planning
QDR—Quad data rate Data Center Infrastructure Management
The SDR application for multimode (IB 1x-SX) and all singlemode
applications (IB 1x-LX) uses two fibers with LC connectors for
transmission while all other applications starting with DDR use the
multifiber MPO connector.
Application Connector type
IB 1x-SX 2 x LC
IB 4x-SX 1 x MPO 12f
IB 8x-SX 2 x MPO 12f
IB 12x-SX 2 x MPO 12f
IB 1x-LX 2 x LC
IB 4x-LX 2 x LC
Channel lengths / m
IB 1x-SX IB 4x-SX IB 8x-SX IB 12x-SX IB 1x-LX IB 4x-LX
Fiber type
SDR/DDR/QDR SDR/DDR SDR/DDR SDR/DDR SDR/DDR/QDR SDR
OM3, 50/125 μm 500/200/300 200/150 200/150 200/150 N/A N/A
OM4, 50/125 μm* 500/200/300 200/150 200/150 200/150 N/A N/A
OS1/OS2, 9/125μm N/A N/A N/A N/A 10 km for all 10 km
* IB physical spec does not mention OM4. Hence OM4 is treated as OM3 in this table.
Table 16: InfiniBand channel lengths depending on applications and fiber types
* IB physical spec does not mention OM4. Hence OM4 is treated as OM3 in this table.
www.commscope.com 9
Everyone communicates. It’s the essence of
the human experience. How we communicate
is evolving. Technology is reshaping the way
we live, learn and thrive. The epicenter of this
transformation is the network—our passion.
Our experts are rethinking the purpose, role
and usage of networks to help our customers
increase bandwidth, expand capacity, enhance
efficiency, speed deployment and simplify
migration. From remote cell sites to massive
sports arenas, from busy airports to state-of-
the-art data centers—we provide the essential
expertise and vital infrastructure your business
needs to succeed. The world’s most advanced
networks rely on CommScope connectivity.
commscope.com
Visit our website or contact your local CommScope representative for more information.