Power Distribution II

Transcript

Slide 1
Welcome to Power Distribution II. This course focuses on Rack Powering Requirements for North American
Data Centers.

Slide 2: Welcome
For best viewing results, we recommend that you maximize your browser window now. The screen controls
allow you to navigate through the eLearning experience. Using your browser controls may disrupt the
normal play of the course. Click the Notes tab to read a transcript of the narration.

Slide 3: Learning Objectives

At the end of this course, you will be able to:
• Identify the importance of power distribution to the rack
• Determine voltage requirements
• Define power density requirements
• Discuss rack powering requirements with regards to
• Redundancy
• Overload protection
• Connectors
• Harmonics
• De-rating
• Cabling
• Current monitoring, and Consistency

Slide 4: Introduction
As we discussed in previous power courses, it is imperative that critical components within the data center
have an adequate and steady supply of power.

Power Distribution II Page |1

© 2023 Schneider Electric. All rights reserved. All trademarks provided are the property of their respective owners.
Information Technology (IT) refreshes in the data center and network room typically occur every 24 to 36
months. As equipment is changed, the power requirement, the voltage requirement, the redundancy
requirement, and the connector requirement often change as well.

As rack enclosures have become the standard means for housing and organizing computing and
communication systems, the power distribution system for the rack enclosure must adapt to these changing
requirements.

Slide 5: Introduction
Failing to do so, may result in millions of dollars in losses due to downtime. Maintaining availability resides
in the ability to provide adequate power distribution. Adequate power distribution is essential if operators
want to avoid many instances of equipment failure, downtime, software and data corruption.

This course will explore rack powering requirements for data centers. Let’s begin with a brief overview of
power transmission from the utility to where it is used in the data center.

Slide 6: Power Distribution to the Rack

The most common approach today is to design, engineer, and install power solutions specific to a rack
enclosure. Should the power requirements for that rack enclosure change, an alternative power solution
must be designed, engineered, and installed. While this approach can comprehend any unique power
requirement, it involves significant planning, engineering, and rewiring.

Rack enclosures are usually fed from a common power distribution panel within the data center or network
room. In many instances, this panel cannot be de-energized in order to adapt a rack enclosure(s) power
distribution system (i.e. install another breaker). The result is known as “hot work” and not only introduces a
very serious safety hazard, but a high degree of risk of creating a fault in the circuit being worked on and / or
dislodging / faulting adjacent wiring circuits. Such errors result in undesirable downtime.

Ideally, the rack enclosure power system would be adaptable to any realistically possible combination of
equipment, on demand, without the need to perform any work that would present a safety hazard or that
might adversely affect system availability.

Power Distribution II Page |2

© 2023 Schneider Electric. All rights reserved. All trademarks provided are the property of their respective owners.
Now let’s explore some rack enclosure power requirements.

Let’s begin with voltage.

Slide 7: Voltage Requirements

In North America, data centers are provided with both 120V and 208V power. Typical IT equipment voltage
requirements and power cords are represented on this chart:

This complex situation suggests the need for the rack power distribution system to provide both 120V and
208V. However, it is possible to consider using a system restricted to a single voltage, either 120V or 208V.

Let’s explore these two possibilities now.

Slide 8: Voltage Requirements

The selection of 120V as a single voltage standard for a data center in North America seems most
convenient because almost 95% of equipment is provided with a 120V power cord. Unfortunately, the 5% of
equipment that accepts only 208V is often the most important and mission critical equipment, such as large
routers and blade servers. Therefore, it is not realistic to design a rack power distribution system based
solely on 120V except for very small network rooms.

Power Distribution II Page |3

© 2023 Schneider Electric. All rights reserved. All trademarks provided are the property of their respective owners.
The selection of 208V as a single voltage standard for data centers in North America assures compatibility
with over 97% of equipment, including the most critical equipment. However, the supplied 120V power
cords of the majority of the equipment must be replaced by the user with the appropriate 208V cord,
requiring that the user keep appropriate replacement cords on-hand.

For some equipment it is also necessary to switch the power supply from 120V to 208V operation with a
selector switch; the failure to activate this switch on such equipment can lead to catastrophic failure when
powered by 208V.

(The 3% of equipment that only operates from 120V can be excluded from the data center, because in
almost all cases these devices are small accessory components that have acceptable and readily available
substitutes that will operate on 208V.)

Nevertheless, providing 120V in the rack power system can be a very significant convenience, eliminating
many plug incompatibilities. For this reason, nearly all existing data centers in North America provide both
120V and 208V and virtually none have standardized on 208V exclusively.

Slide 9: Voltage Requirements

The rack environment is generally single-phase. The quantity of rack mounted IT equipment manufactured
that requires 3-phase power is very small, some brands of blade servers being notable examples.
Occasionally, a pre-configured Original Equipment Manufacturer (OEM) rack enclosure is wired using an
internal power distribution unit (PDU) that takes in 3-phase power and provides 3 branches of single-phase
power to the single-phase IT loads.

It is important to note that these IT loads are actually single-phase. Despite the absence of 3-phase loads,
3-phase power distributed to racks could provide benefits to a high density data center. One key benefit is
that 3-phase distribution to the rack has the ability to power 120V and 208V loads from the same branch
circuit.

Power Distribution II Page |4

© 2023 Schneider Electric. All rights reserved. All trademarks provided are the property of their respective owners.
Now let’s explore the topic of power density.

Slide 10: Power Density

Power densities within the rack enclosure can vary greatly, depending upon the equipment installed. In the
extreme low load case, a rack enclosure may only have passive patch panels or a few internetworking
switches with a power draw of less than 100W. In the extreme high load case, a rack enclosure may be
completely filled with high-density servers for a total load of 20kW or more.

In addition to supplying the total rack power requirement, the rack power system must also be able to
provide the required power to an individual device.

Sending multiple branch circuits to a rack may appear to provide the total power requirement, but the power
requirement of an individual large load may exceed the capability of any of the branches.
For example, sending any number of 20 Amp branch circuits to a rack where a single piece of equipment
requires 30 Amps is insufficient. Another example is a blade chassis with a 30 Amp plug that may be
initially populated with only a few blades and use 5 Amps on a 30 Amp circuit.

Slide 11: Power Density

Some customers may think they can put multiple blade chassis' on a single 30 Amp circuit, but as they
populate the chassis they overload the circuit. In cases like these, it is recommended that only one load
device be attached to each branch circuit.

Slide 12: Power Density-Design Value

The appropriate design value for average rack power level is a subject of considerable controversy. A
survey of usage patterns in corporate data centers, network rooms, and communication rooms for the year

Power Distribution II Page |5

© 2023 Schneider Electric. All rights reserved. All trademarks provided are the property of their respective owners.
2019 identified the distribution of per-rack power consumption shown in here in this graph. This graph
shows the frequency of occurrence of racks configured to different power levels. The frequency of
occurrence goes down with increasing power level.

Power Distribution II Page |6

© 2023 Schneider Electric. All rights reserved. All trademarks provided are the property of their respective owners.
Similar trend for usage in the year 2022 (based on technology / client trends) is also illustrated in the graph.
This indicates that the average power per rack is increasing over time. It is possible to configure IT
equipment today that would exceed a 50kW per-rack power requirement if fully populated into a rack
enclosure. While possible to achieve, we did not find this occurrence to be frequent in real-world
installations (5% of data centers). The data collected indicates that the average power density per rack
enclosure will rise significantly. However, power densities (more than) > 30kW will remain a smaller fraction
of the installed base.

Power Distribution II Page |7

© 2023 Schneider Electric. All rights reserved. All trademarks provided are the property of their respective owners.
Now let’s move on to rack powering requirements.

Slide 13: Rack Powering Requirements-Power

An analysis of the underlying data of the distribution of rack power indicates the following:
• The very low loads are mainly rack enclosures with wiring patch panels, switches, and hubs
• Loads in the 1-4kW range are composing 25% of data centers
• Loads in the 5-9kW range are mainly rack enclosures that are populated with typical equipment but
with significant unfilled rack space
• Loads in the 19-19kW range are partially loaded with 1U servers, or contain a mix of technologies
• Loads in the 20kW + range are rare but, according to customers, are going to become more
common with the recent density increases resulting from server technology advancements

The mean value for rack power consumption in corporate computing environments is around 5-9kW.
However, organizations were found to have variable mean power consumption within their overall rack
enclosure environment. These organizations take different approaches as to how much equipment they
pack into a rack enclosure. Some leave large amounts of space within the rack enclosure unused while
others may pack equipment as tightly as possible. Therefore, the market-wide mean rack power
consumption is not necessarily a good predictor of average rack power consumption within an organization.

Slide 14: Rack Powering Requirements-Power

It is important to understand that electrical codes specify that branch circuit ratings are dictated by the load
and cannot be arbitrarily chosen by the user.

In the USA, virtually all branch circuits in rack enclosures are rated for 20 amps, but note that 30 amp
circuits are also sent to the rack. It is important to understand that these circuits are feeder circuits and not
branch circuits, because they require additional circuit breakers in the rack. Very few rack devices can
directly utilize a 30A branch circuit.

Those that can leverage the 30 Amp circuit are usually blade servers, routers, free-standing servers or
storage devices.

Power Distribution II Page |8

© 2023 Schneider Electric. All rights reserved. All trademarks provided are the property of their respective owners.
Slide 15: Rack Powering Requirements-Power
The maximum power available to the rack enclosure depends on the number and type of branch circuits
provided within the rack enclosure. Clearly the number of branch circuits will need to be greater than one to
support the power density of current and future IT technology.
• A single 120V branch circuit can supply the load requirement for the most common rack enclosures
today, but this will not be true in the future.
• Two 120V branch circuits, or one 208V branch circuit, can supply the load requirement for
approximately 80% of rack enclosures today, but only 40% of racks in the future.
• Three 120V branch circuits in North America can supply the load requirement for over 92% of rack
enclosures today, and still over 85% in the future.

Ironically, the inability to provide sufficient branch circuits to a rack enclosure does not prevent operation of
the system. If the rack has insufficient power distribution capacity, the power drawn by a rack enclosure can
be reduced by removing equipment from it and moving it to another rack enclosure. The consequence is
that a reduction in space utilization occurs, however, for an occasional rack enclosure this is not a serious
problem.
A practical design baseline in North America is three 120V branch circuits per rack enclosure or two 208V
branch circuits, with the ability to add additional circuits easily as needed.

Now let’s take a moment to explore redundancy requirements.

Slide 16: Rack Powering Requirements-Redundancy

The keys to increasing the availability of a computing system lies in providing redundancy and/or fault
tolerance in the power system. In high availability environments, a common way to provide redundancy is to
supply two independent power paths to each piece of computing equipment. The equipment in question
accepts the two power feeds via independent paralleled power sources. This parallel power infrastructure is
sized such that the equipment will continue to operate if supported by only one power path. This system
provides the following advantages:
• If a power supply fails, the system continues to operate
• If one power feed fails, due to equipment malfunction the system continues to operate
• If one power feed fails due to user error, the system continues to operate

Power Distribution II Page |9

© 2023 Schneider Electric. All rights reserved. All trademarks provided are the property of their respective owners.
 • If the power supply fails in a way which faults the power feed and trips breakers, the equipment
sharing the breaker is not affected
• If one power feed needs to be shut down for maintenance or upgrade, the system continues to
operate
• For this approach to be effective, the following requirements must be met:

The protected equipment must support dual power feeds and operate with one feed faulted
The loading of breakers within each power path must always be less than 50% of trip rating during normal
conditions, so that the increase in load that will accompany failure of the alternate path does not cause
breakers to trip. This also helps prevent tripping of the alternate path due to low line voltage conditions.

Slide 17: Rack Powering Requirements-Redundancy

Meeting these redundancy requirements can be very difficult. Some computing equipment is only available
with a single power cord. There is also equipment manufactured with three power cords, where any two are
needed for proper operation. These types of equipment cannot operate with the loss of one power feed. In
these cases an Automatic Transfer Switch (ATS) can be used which generates a single feed from two inputs.

Such an ATS may be deployed centrally or it may be deployed in a distributed manner by placing small rack
mount ATS in the rack enclosure with the protected equipment.

An adaptable rack enclosure power system would be able to support a single or dual path environment or a
hybrid of both single and dual equipment. In addition, it is necessary to provide current monitoring to ensure
that all circuits are loaded below 50% capacity in order to prevent breaker tripping during a loss of one
power path.

Now let’s investigate the requirements for overload protection.

Slide 18: Rack Powering Requirements-Overload Protection

One of the most misunderstood concepts of power distribution is the over-current protection of branch
circuits. Each branch circuit in a rack enclosure must be supplied by an independent circuit breaker and a

Power Distribution II P a g e | 10

© 2023 Schneider Electric. All rights reserved. All trademarks provided are the property of their respective owners.
typical rack enclosure will need multiple branch circuits. This rendering illustrates the common methods of
power distribution to the rack, showing different branch circuit configurations. Let’s take a closer look at
each of these configurations.

In this figure, a single branch circuit supplies a rack. For a 20 Amp system, this arrangement is limited to
2.4 kW max capacity for 120V or 4.2 kW max capacity for 208V. To achieve higher rack power multiple
branch circuits are required. There are two options for providing multiple branch circuits to a rack enclosure,
so let’s discuss those next.

(Image on next page)

Slide 19: Rack Powering Requirements-Overload Protection

The arrangements of these configurations are capable of providing the same power, but with a different
wiring and circuit breaker arrangement. Note that in this figure, the conduit or “whip” contains multiple
branch circuits.
These alternatives give rise to significant advantages and disadvantages, so let’s explore those next.

Power Distribution II P a g e | 11

© 2023 Schneider Electric. All rights reserved. All trademarks provided are the property of their respective owners.
Slide 20: Rack Powering Requirements-Overload Protection
This summary suggests there is a significant advantage to avoiding the need to generate branch circuits in
the rack enclosure.
• Most users who send 30 Amp or larger circuits to racks do not understand that these circuits are
feeder circuits and not branch circuits. When 30 amp whips are used, the typical 20 amp branch
circuits required to supply receptacles must be provided by branch-rated breakers located within
the racks.
• An adaptable rack power system would eliminate the need for hard wiring, cascaded breakers, and
breaker coordination analysis, which would suggest a preferred approach of multiple branch
circuits per rack. Ideally, multiple branches would be provided using a single multi-conductor cable
to the rack in order to simplify wiring to the rack.

Because connectors are an integral part of the power distribution system, let’s take some time to investigate
them now.

Slide 21: Rack Powering Requirements-Connectors

Over 99.99% of all AC powered equipment used in rack enclosures are connected via a power cord and
plug (a negligible fraction is hard-wired). While most have plugs, these plug types can vary greatly from one
product to another.

In North America, the approximate statistical breakdown of frequency of occurrence for plug types is
illustrated here.

Power Distribution II P a g e | 12

© 2023 Schneider Electric. All rights reserved. All trademarks provided are the property of their respective owners.
(Image on next page)

Slide 22: Rack Powering Requirements-Connectors

In the rest of the world, plugs vary by country, which makes this situation more complex. However, unlike
North America, most countries use a single service voltage and therefore power cords are rated for 230V
16A. This means that a single plug type serves most applications from small hubs to enterprise servers.
Therefore, outside of North America, a single country-specific plug type handles 99% of applications.

It is important to point out that 99% of all IT equipment in the data center utilizes a detachable power cord.
This means that the plug type can be changed. OEM suppliers take advantage of this when creating
complete rack systems and often will install cords with IEC 60320 (previously IEC-320) style power plugs on
all the equipment in the rack enclosure, along with an IEC60320 style outlet strip. This has the benefit that a
single configuration can be used worldwide.

Power Distribution II P a g e | 13

© 2023 Schneider Electric. All rights reserved. All trademarks provided are the property of their respective owners.
Slide 23: Rack Powering Requirements-Connectors
The number of receptacles required in a rack enclosure varies dramatically with the installed equipment. A
rack enclosure may contain only a single load as a minimum. Conversely it may be populated with say, 42
thin servers with dual power cords for a total requirement of 84 receptacles.

An adaptable rack enclosure power system would be able to provide power receptacles for all the various
plug types, which might be encountered, as well as two feeds, each containing 42 servers. To accomplish
this, one must provide a large quantity and assortment of receptacles in every rack enclosure, or provide a
number of easily changeable outlet strip options to meet evolving requirements.

Slide 24: Rack Powering Requirements-Harmonics

Historically, computing equipment generated harmonic currents on AC power lines. This led to the need to
incorporate specialized features into power systems such as oversized neutral wiring and K-rated
transformers.

During the 1990’s, regulations placed on the design of computing equipment, combined with the gradual
retirement of older equipment, resulted in the elimination of this as a problem by the year 2000. Today,
lighting and power distribution transformers are the predominant source of harmonic currents in data centers.
Harmonic-rated wiring and transformers are not needed in the rack enclosure power environment.

Slide 25: Rack Powering Requirements-De-rating

Operating with circuits at the rated current limit is not desirable since the circuit breakers are on the edge of
the trip rating and may trip. Therefore the user should not expect to utilize the full current or power rating of
the system, but should apply a de-rating factor. The National Electric Code specifies a 20% de-rating factor
be applied to rack mounted PDUs. This is a practical and recommended de-rating factor.

Some users conservatively specify lower de-rating factors, such as 30% or even 40%. (The power
capacities of the distribution architectures described in this course are full rated values unless otherwise
stated, and therefore the actual values should be computed by applying the desired de-rating factor.)
Now that we’ve reviewed de-rating factors, let’s move on to cabling requirements.

Power Distribution II P a g e | 14

© 2023 Schneider Electric. All rights reserved. All trademarks provided are the property of their respective owners.
(To learn more about de-rating and calculating power requirements for the data center, consider viewing the
DCU course “Calculating Total Power Requirements”)

Slide 26: Rack Powering Requirements-Cabling

Cables to deliver power to the rack enclosures are an essential part of the rack enclosure power system. A
common practice today is to use under floor power cabling. The under floor power cabling method presents
a number of barriers to adaptability, which are explored in the course “Re-examining Raised Floor
Applications”.

In an adaptable rack enclosure power system, the cabling provided to each rack would provide all the
branch circuits and voltages that might ever be required. No changes to the cabling would be required due
to equipment changes in the rack enclosure. It would also be easy and safe to provide the appropriate
power feeds to additional rack enclosures in the future.

Now let’s investigate the importance of current monitoring.

Slide 27: Rack Powering Requirements-Current Monitoring

Rack enclosure power systems are subject to constant load changes due to the installation and removal of
equipment and to the dynamic power draw variation in the installed equipment. These circumstances lead
to a requirement to monitor power flowing in branch circuits in order to prevent failures or hazard due to
overloads.

Let’s wrap up this section, by reviewing why consistency is essential in power distribution.

Slide 28: Rack Powering Requirements-Consistency

Due to the large number of power circuits in the typical data center, there is a significant advantage to
minimizing the different types of power distribution provided (branch circuit ratings, poles per whip, circuit
breaker types and location, etc). Ideally a uniform single type of power feed would be provided to every
rack to maximize flexibility and to reduce human error.

As we have discussed in other courses, human error is a constant threat in a data center and has been the
cause of much downtime.

Power Distribution II P a g e | 15

© 2023 Schneider Electric. All rights reserved. All trademarks provided are the property of their respective owners.
Standardizing on a common power distribution circuit that fulfills the need 97% of the time is just one
method of reducing the risk of human error. With standardized whips users are less apt to become
confused. Parts are minimized and learning curves are accelerated all which lower the risk of a costly
mistake.

(To learn more about the advantages of standardization, consider viewing the DCU course “Standardization
in the Data Center”)

Slide 29: Summary

Let’s wrap up with a summary of the information we have covered today.
• Individual rack enclosure power consumption in the data center or network room varies widely and
is expected to grow in the next few years. Rack enclosure equipment is replaced 5 or more times
during the life of a data center in a piecemeal manner.
• This situation requires a rack enclosure power distribution system that can cope with the changing
requirements. Key requirements of an effective rack power distribution system were described,
which suggest a practical rack enclosure power architecture that can meet the requirements for an
adaptable rack enclosure power system.

Slide 30: Thank You!

Thank you for participating in this course.

Power Distribution II P a g e | 16

© 2023 Schneider Electric. All rights reserved. All trademarks provided are the property of their respective owners.