Computer Communications 40 (2014) 1–21

Contents lists available at ScienceDirect

Computer Communications
journal homepage: www.elsevier.com/locate/comcom

Review

A survey on architectures and energy efficiency in Data Center Networks

Ali Hammadi, Lotfi Mhamdi ⇑
School of Electronic and Electrical Engineering, University of Leeds, LS2 9JT, Leeds, United Kingdom

a r t i c l e i n f o a b s t r a c t

Article history: Data Center Networks (DCNs) are attracting growing interest from both academia and industry to keep
Received 20 May 2013 pace with the exponential growth in cloud computing and enterprise networks. Modern DCNs are facing
Received in revised form 26 November 2013 two main challenges of scalability and cost-effectiveness. The architecture of a DCN directly impacts on
Accepted 29 November 2013
its scalability, while its cost is largely driven by its power consumption. In this paper, we conduct a
Available online 12 December 2013
detailed survey of the most recent advances and research activities in DCNs, with a special focus on
the architectural evolution of DCNs and their energy efficiency. The paper provides a qualitative catego-
Keywords:
rization of existing DCN architectures into switch-centric and server-centric topologies as well as their
Data Center Networks (DCNs)
Architecture
design technologies. Energy efficiency in data centers is discussed in details with survey of existing tech-
Energy efficiency niques in energy savings, green data centers and renewable energy approaches. Finally, we outline poten-
Virtualization tial future research directions in DCNs.
Renewable energy for DCNs Ó 2013 Elsevier B.V. All rights reserved.

1. Introduction and how to keep the DCN power budget manageable including vir-
tualization, network load management and scheduling, etc. (iii)
Recent years are witnessing an unprecedented growth in data congestion handling in DCNs including congestion notification
centers. This is mainly driven by the plethora of services and appli- and avoidance. A typical challenge is the problem of TCP incast,
cations housed by modern data centers, such as web-search, scien- (iv) routing in DCNs with the provision of efficient and cost-effec-
tific computations, social networks, distributed files systems, etc. tive routing mechanisms such as multipath routing concepts. The
Data centers are at the heart of almost every sector at the private, focus of this article is to survey the first two issues of architectural
public and governmental levels. Today’s data centers contain hun- evolution and energy efficiency in DCNs. Readers are referred to [2]
dreds of thousands of servers, interconnected via switches, routers and references therein for issues such as routing, congestion con-
and high-speed links. The design of modern data centers mandates trol, etc.
joint expertise from diverse engineering fields and entails a num- The choice of the architecture of a DCN is of premium impor-
ber of considerations ranging from real estate, geographic location, tance as it impacts on the overall efficiency of the DCN. The archi-
to the data center hardware and software requirements as well as tecture of a DCN, or its topology, directly reflects on its scalability,
its power budget. The ultimate objective in designing a data center cost, fault-tolerance, agility and power consumption [3]. Conven-
is to maximize efficiency while maintaining a low cost. tional DCNs have been designed using a tree-like topology. A typ-
In order to keep pace with the high-demand in services and ical example of this topology is the three-tier topology proposed by
applications, DCNs are continuing to evolve and considerable re- [3] where the tree’s leaves (end-nodes or servers) are connected to
search efforts are being made to address the various challenges ob- Top-of-Rack (ToR) switches and these (ToR) switches are con-
served. The choice of a DCN solution to address one challenge nected to aggregation switches which are in turn connected to core
impacts and often limits the alternatives of how to address other routers at the root of the tree. This topology has soon been shown
issues. Furthermore, DCNs are deployed for various sectors and to suffer numerous drawbacks of scale, capacity, reliability, utiliza-
the solutions (and geometries) differ such as the difference be- tion and power budget [2]. As a result, efforts have been dedicated
tween enterprise DCNs and cloud-service DCNs [1]. Irrespective to address some (or all) of the above encountered problems in the
of the DCNs type, various common challenges for the design of tree-based DCN topology and various DCN architectures have
DCNs have been observed at various levels, including: (i) the archi- appeared [4–15]. These architectures can be classified as server-
tecture of the DCN and its topology, (ii) the energy efficiency issue centric and switch-centric and/or based on their infrastructure
technologies such as electronic versus optical DCNs.
Energy efficiency is a central issue in modern data centers.
⇑ Corresponding author. Tel.: +44 113 343 6919.
DCNs are typically high-capacity networks that are tuned for max-
E-mail addresses: [email protected] (A. Hammadi), [email protected]
imum performance (peak network load or busy-hour), making
(L. Mhamdi).

0140-3664/$ - see front matter Ó 2013 Elsevier B.V. All rights reserved.
http://dx.doi.org/10.1016/j.comcom.2013.11.005
2 A. Hammadi, L. Mhamdi / Computer Communications 40 (2014) 1–21

them extremely power hungry. A typical DCN facility has more outline potential futures research directions in DCNs. Finally, Sec-
than 100 times power density than a typical office building of tion 9 concludes the article.
the same space. In 2007, a report to the U.S. congress by the EPA
estimated that DCNs power usage has doubled between 2000
2. Conventional data center architecture and challenges
and 2006 [16]. This trend is likely to persist between 2006 and
2011, rendering the DCN’s energy budget a major cost. In response
2.1. Conventional data center design
to this, numerous proposals for energy-efficiency at various levels
of the DCN have been studied and proposed. In addition to ad-
The classic data center design architecture [3,18] consists of
vances in energy-efficient server designs [17], other studies have
switches and routers in two or three tier hierarchal structure as
addressed ways to dynamically monitor the DC network traffic
shown in Fig. 1. The hierarchy in the case of three tiers consists
load in order to balance it and optimize the DCN power consump-
of layer-3 with border routers, layer-2 with aggregation switches,
tion. Virtualization is perhaps the most promising approach,
and layer-1 with Top of Rack (ToR) access switches. A ToR switch
whereby DCN resources are logically partitioned (or ‘‘sliced’’) and
usually connects 20–40 servers placed in a rack with 1Gbps link,
provided to users as virtual machines (VMs). Simply put, virtual-
and for redundancy each ToR is connected with two aggregation
ization is a means through which the same amount of processing
switches which in turn connect with the core layers through multi-
can be run on fewer servers by increasing server utilization, there-
ple high speed 10 Gbps links. The aggregation layer provides and
fore allowing more efficient use of DCN resources.
manages many functions and services such as spanning tree pro-
While previous work has surveyed research efforts in DCNs,
cessing, default gateway redundancy, server to server traffic flow,
such the comprehensive work by Zhang and Ansari [2], our work
load balancing, firewall and more. Core routers/switches, running
focuses on the two interrelated issues of DCN architecture and its
10Gbps high-speed links, are at the top of the hierarchy. These
energy efficiency. Our work differs from [2] in both the architec-
are used for traffic going in and out of the data center. The Core
ture of DCNs and their energy efficiency as follows. First, the cur-
routers/switches also run well-known routing algorithms such as
rent work surveys more DCN architectures such as the one-tier
Open Shortest Path First (OSPF) or Enhanced Interior Gateway
DCN architecture, the hybrid electro-optical, the full optical and
Routing Protocol (EIGRP) and can load balance traffic using Cisco
the passive optical-based DCN architectures. Second, the current
Express Forwarding based hashing algorithms between core and
work covers more energy efficiency related research efforts in
aggregation layers [3]. Unfortunately, the hierarchal three tiers
DCNs. We also provide a comparative study (Section 6) on energy
DCN structure suffers various issues as will be discussed next.
efficiency in DCNs. We further provide a survey and a comparative
study of the most adopted testbeds and simulation tools for DCNs.
The remainder of this article is structured as follows. Section 2 2.2. Conventional data center challenges
revisits conventional data center architecture and design. We
describe the multi-tier DCN topology and discuss the main chal- Several challenges [2] and issues appeared with conventional
lenges facing conventional DCNs. In Section 3, we survey most data centers which have led many researches to explore and inten-
recent advances in DCNs and we classify these architectures into sively study alternative designs and approaches to provide
switch-centric and server-centric topologies. We also categorize scalable, fault tolerant and efficient data centers. One of the most
these proposals based on their underlying technologies into important performance handicaps that could lead to congestions
electronic DCNs, full optical DCNs and hybrid electro-optical DCNs. is oversubscription. Oversubscription is the ratio between the
Section 4 provides a qualitative comparison and discussion of the servers’ bandwidth to the total uplink bandwidth at the access
DCN architectures. In Section 5, we conduct a detailed study of layer. Hence, as moving up to aggregation and core layer, the
energy efficiency in DCNs including the various recently proposed number of servers sharing the uplinks increases and hence the
techniques such as virtualization, energy-aware routing, dynamic oversubscription ratio also increases and results in bottlenecks.
voltage/frequency scaling, dynamic power management, renew- Oversubscription limits the server to server capacity where the ra-
able energy, cooling, etc. In Section 6, provides a comparative study tio should be 1:1 so hosts can communicate with their full network
of existing efforts in DCN energy efficiency and optimisation. interface bandwidth. On the other hand, congestion resulting from
Section 7, surveys some of the most adopted testbeds and simula- oversubscription could also lead to overloading switch buffers
tion tools for DCNs. In Section 8, we provide some insights and which will in turn start dropping packets. Hence, another issue

Fig. 1. Conventional data center.

 A. Hammadi, L. Mhamdi / Computer Communications 40 (2014) 1–21 3

arises because of the lack of a mechanism to avoid packet drops at Dcell, Bcube, and FiConn. These topologies and designs are based
congested switches. Moreover, congestion can also occur at a on packet-switched electronic networks, however; hybrid elec-
switches where simultaneous transmission of packets from multi- tro-optical packet switch along with full optical solutions were also
ple senders arrive at the same time, switch gets overloaded and proposed and implemented for low power consumption and high
starts to drop packets leading to TCP timeout and hence a collapse bandwidth.
in TCP throughput, known as TCP incast. Many approaches to mit-
igate the issues of incast and congestion are covered in details in 3.1. Switch centric data center architectures
[19–23]. Other challenges introduced with classical data center
network such as the lack of fault tolerance especially at the upper In this section, the most well-known switch centric data center
levels of the tree due to the low physical connectivity. Hardware designs such as Fat-Tree [4], portland [5], VL2 [6], and one-tier Qfa-
failures in the core or aggregation layers will result in sharp de- bric [25] shall be covered. Such designs rely on switches for inter-
grade of overall network performance. Additionally, poor utiliza- connection and traffic routing. The different design choices in the
tion of resources occurred because of the fact that within layer-2 switch centric class came to resolve many issues that existed with
domain, Spanning Tree Protocol (STP) only uses one path even the conventional data center. These issues, as shall be explained in
though multiple paths exist; in addition to that another issue with subsequent sections, are oversubscription, agility, load balancing
load balancing arises since traffic cannot be evenly distributed over and high power consumption.
paths within core and aggregation layers.
The fast growth of DCNs has triggered the issue of power con- 3.1.1. Fat-Tree
sumption due to the high number of power hungry devices and The Fat-Tree topology [4], depicted in Fig. 2, consists of k pods,
cooling systems. Most of these devices are underutilized, as statis- each of which consisting of 2k edge switches and 2k aggregation
tics has shown that a typical utilization of a data center is only 30% switches. Edge and aggregation switches connected as a clos topol-
[24]. Hence, dynamic reassignment of resources among servers ogy [26] and form a complete bipartite in each pod. Also each pod
running on the data center is an optimal solution to consolidate is connected to all core switches forming another bipartite graph.
most jobs on 30% of the servers while being able to shut down Fat-Tree built with k-port identical switches in all layers of the
the other unused servers and hence save power. The ability to as- topology and each of which supports k3 4
hosts. Fat-Tree IP addresses
sign any server to any service without considering topology is are in the form 10:pod:subnet:hosted. With Fat-Tree topology is-
called Agility. This is another challenge in conventional data cen- sues with oversubscription, costly aggregation and core switches,
ters and will be covered in details in subsequent sections. fault tolerance, and scalability are resolved. Fat-Tree established
a solid topology for researchers to work onto solve other important
issues such as agility through virtualization.
3. Data center architectural evolution In Fat-Tree, the issue of address lookup time was studied and a
two table lookup approach was proposed to replace the longest
Numerous problems in conventional data centers have driven prefix match which is found impractical with data centers having
researchers to propose and design various data center architec- hundreds of thousands entries for routing tables. Two level routing
tures to solve these issues. Data centers can be categorized mainly table and address lookup were implemented using specialized
in two classes, the switch-centric and the server-centric. In switch- hardware; Ternary Content Addressable Memory (TCAM) which
centric, switches are the dominant components for interconnection can store addresses and also perform parallel searches among its
and routing whereas in server-centric, servers with multiple Net- entries. Basically, address lookups are done in two steps; first the
work Interface Cards (NIC) exist and take part in routing and pack- lookup engine does a lookup on the TCAM to find the longest
et forwarding decisions. matching prefix. Then the matched address is used to index the
The conventional data center is a switch-centric design. Other SRAM which holds the information of the IP address and output
examples of switch-centric include VL2 [6], Portland [5], Fat-Tree port to reach the intended destination. It is also worth mentioning
[4], and Monsoon [10]. Server-centric topology also attracted great that routing tables are meant to be static to avoid delays that may
interest by researcher and many designs were proposed such as occur from routing table updates but on the other side CAMs are

Fig. 2. The Fat-Tree topology with k ¼ 4.

4 A. Hammadi, L. Mhamdi / Computer Communications 40 (2014) 1–21

seen as power hungry components and have low storage density virtualization capability to improve energy saving contribution
and also may introduce considerable cost. for green data centers.

3.1.2. VL2 3.1.3. Portland

VL2 was proposed in [6] and considered as a solution to over- The Portland DCN topology, proposed in [5], is similar to VL2
come some of the critical issues in conventional data centers such in that both are based on a Fat-Tree [4] network topology. Port-
as oversubscription, agility and fault tolerance. VL2, shown in land, depicted in Fig. 4, consists of three layers: edge, aggregation
Fig. 3, exploits a uniform high capacity from server to server, sup- and core. It is built out of low cost commodity switches. They
ports VM migration from server to server without breaking the TCP both differ in the way of associating and separating names from
connection and keeping the same address. It is very similar to the locators but both at the end aim at providing agility among ser-
three-tier architecture DCN proposed by Cisco, except that it vices running on multiple machines. Both reduce broadcast by
implements a clos topology (low cost ASICs) between core and intercepting Address Resolution Protocol (ARP) requests and em-
aggregation layers to provide multipath and rich connectivity be- ploy a unicast query through a centralized lookup service. Port-
tween the two top tiers. The architecture design of the VL2 topol- land imposes additional requirements on the switch software
ogy enhances the availability and reliability of the network, and hardware unlike VL2 where implementation only takes place
especially in the presence of link or hardware failures. VL2 employs in the servers’ network stack. A deeper look into the methodology
Valiant Load Balancing (VLB) to evenly load balance traffic flows imposed by Portland for agility and virtualization support shall be
over the paths using Equal Cost Multi Path (ECMP). VL2 also em- covered in details in Section 5.2. For load balancing, Portland and
ploys TCP for end to end congestion control. As additional advan- VL2 employ flow hashing in ECMP; except that VL2 employs VLB
tage is that VL2 can be easily implemented on low cost existing which before forwarding a packet, it randomly selects an inter-
commodity switches since it uses already existing ECMP for packet mediate switch. This was found to be impractical in the case
forwarding and link state routing for topology updates. VL2 will be where two hosts, connected to the same edge switch, want to
revisited later in this article with further details on agility and communicate.

Fig. 3. The VL2 DCN architecture.

Fig. 4. The Portland DCN topology.

 A. Hammadi, L. Mhamdi / Computer Communications 40 (2014) 1–21 5

3.1.4. One-tier fabric architecture applications for different traffic patterns such as one to one, one
Flattening three-tier tree structure to one tier fabric is an to many, one to all and all to all. BCube supports and accelerates
existing solution proposed for modern data center architecture as all types of traffic patterns and provides high network capacity
introduced by Juniper [25]. Juniper Qfabric architecture as shown due to its low diameter. The benefits of BCube design is that it
in Fig. 5 has flattened the data center network and simplified the can provide fault tolerance and load balancing and while requiring
management of the data center by reducing the number of lower cooling and manufacturing cost. BCube, as shown in Fig. 6,
switches. Furthermore, since there is no tree structure, there is can be constructed in a recursive manner starting at BCube0 as
no need for multiple hops traversing between any communicating its basic building block, which is built around n-servers connected
nodes within the network. The location of hosts is not any more an to n-port switch. Then, BCube1 is built out of n-BCube0 each of
issue since the network diameter and the shortest path between which has n-servers. A general BCubek is constructed recursively,
any two communicating nodes is always equal to one and no more as before, based on BCube (k  1). For detailed construction of a
oversubscription or congestion issues are arising and all nodes can general BCubek topology, readers are referred to [8].
benefit from their all line card bandwidth. BCube employs source routing protocol (BSR) when existing
The Qfabric single logical switch has an added value to the DCN routing protocol such as Intermediate System To Intermediate Sys-
since it reduces the complexity, operational cost, cooling cost, tem (IS-IS) and OSPF cannot scale to thousands of servers. BSR can
occupied floor space and power consumption. The Qfabric supports utilize high multipath capacity and also load balance the traffic
high speed server to server connectivity with low latency which automatically. With BSR, the source server controls the selection
makes it an attractive structure for modern data centers hosting of the path without coordination with intermediate servers which
delay sensitive applications. It also smoothen the process of virtu- is only responsible for forwarding received packets based on infor-
alization among servers within the data center leading to great en- mation obtained from the header. BSR probes the network to select
ergy savings. Qfabric could reduce power saving to less than 77% if the best path which eliminates the need of frequent link state
the reduced number of switches, links, cooling systems are consid- broadcasting which is not scalable since the network consists of
ered along with applying other energy saving techniques such as 1000s of servers.
virtualization among data center resources [27]. Consequently,
Qfabric is considered to be a green data center architecture that 3.2.2. DCell
can contribute to reducing carbon footprint in the environment. DCell [7] is another server-centric structure for data center that
can provide desirable properties to overcome issues with scalabil-
3.2. Server-centric data centers ity, fault tolerance and network capacity. As illustrated in Fig. 7,
DCell is a structure with rich physical connectivity among servers
Unlike switch centric designs, server centric designs appeared and switches and replaces expensive core and aggregation
to use servers to act as relay nodes to each others and participate switches with mini low cost switches. However, additional cost
in the traffic forwarding. Server centric schemes such as Bcube introduced because of additional and lengthy wiring communica-
[8], Dcell [7], and Ficonn [9] can provide low diameter compared tion links between switches and servers.
to switch centric schemes, can provide high capacity and support Similar to BCube [8], large DCells are recursively constructed
all types of traffic, especially for the intensive computing applica- from smaller DCells, with DCell0 as the initial building block. A
tions with very low delays. In this section, an overview of Bcube, DClell0 is constructed by connecting n servers to one low cost
Dcell, and Ficonn server centric schemes shall be described along mini-switch with small port count. A DCell1 consists of (n þ 1)
with their properties. DCell0, where every DCell0 is connected to every other DCell0 in
full mesh fashion as depicted in Fig. 7. Servers in a generalized
DCell topology have two interfaces each, one connects to its
3.2.1. BCube
mini-switch and the other interface is connected to another server
BCube [8] is an example of server-centric DCN structure which
in a neighboring DCell0. Any two servers with 2-tuples ½i; j  1 and
consists of servers equipped with multiple network ports connect-
½j; i are connected with a link to every i and every j > i [7]. As an
ing multiple low cost mini switches. In BCube, servers are not only
example, in Fig. 7, server with tuple ½4; 1 shall be connected to
hosts but they also act as relay nodes for each other and take part
½1; 3.
in traffic forwarding through multiple parallel short paths between
DCell is a scalable network structure which can be expanded
any pair of servers. The design is driven by demands for intensive
gradually without a need for re-wiring or changing addresses.
computing and higher bandwidth requirements to support
DCell networks with k equal to 3 can be scaled up to accommodate
millions of servers. DCell employs a distributed fault tolerant rout-
ing algorithm designed to well exploit the DCell structure/topol-
ogy, with fault tolerance capability in the presence of various
types of failures such as links, servers and/or racks [7].

3.2.3. Ficonn
Ficonn [9] employs an interconnection structure using com-
modity servers and switches to establish a scalable data center net-
work. It differs from BCube and DCell by making use of the two
built-in Ethernet ports in the servers to establish connections
and load balance traffic on two outgoing links through a Traffic
Aware Routing algorithm (TAR). The throughput and routing path
length can be severely affected by changing network condition.
The TAR in Ficonn has the capability to monitor link capacities
and, based on information obtained on the status of network con-
dition, it adapts accordingly.
The construction of the Ficonn interconnection can be demon-
Fig. 5. The Qfabric architecture [27]. strated and explained as shown in Fig. 8 where the Ficonn physical
6 A. Hammadi, L. Mhamdi / Computer Communications 40 (2014) 1–21

Fig. 6. The BCube topology.

to provide switching, routing, and interconnection. Such devices

are Tunable Wavelength Converters (TWC), Optical Amplifier, Ar-
rayed-Waveguide Grating (AWG), Micro-Electro-Mechanical Sys-
tems Switches (MEMS), Wavelength Selective Switch (WSS),
Couplers, Splitters and Combiners. Optical interconnect schemes
are mainly classified into two categories, the hybrid scheme where
optical along with the electrical switches are considered in the de-
sign to constitute the fabric interconnection, and the full optical
network where only optical devices are employed. An insight of
each scheme shall be presented in this section with a demonstra-
tion of the architecture and main properties of the most well-
known schemes such as Helios, C-through and Petabit.

3.3.1. Hybrid electro-optical data centers

3.3.1.1. Helios. Helios is a hybrid Electronic/Optical switch architec-
ture for modular data center proposed by Farrington et al. [29] as a
design to reduce the number of switches, number of cables, cost
Fig. 7. The DCell DCN topology.
and power consumption while maintaining full bisectional band-
width at minimum oversubscription ratio. Helios is a two-tier net-
topology consists of FiConn2 with n = 4. FiConn2 is composed of 4 work consisting of ToR and core switches. The ToR switches are
FiConn1, and each FiConn1 is composed of 3 FiConn0. There are electronic packet switches while the core switches are a combina-
three different level-links to constitute the interconnection within tion of optical and electronic switches. The electronic switches are
the topology, level 0 link connects each server with its switch with- used for all to all communication among pods, while the optical
in the same Ficonn0, level 1 or level 2 links connect the second port ones are used for long lived high-bandwidth communication. Each
of the server to either another server within the same Ficonn1 or a ToR switch has two types of transceivers: 10G colorless for con-
server in another Ficonn1 within Ficonn2. necting pods to electronic core switches and Wx10G (where W
Ficonn is found to be scalable since its number of servers can be can be from 1 to 32 and it is the number of wavelength multi-
scaled up and increased exponentially with the increase of levels. plexed) for connecting pods to optical core switches.
The optical circuit switching in Helios relies on MEMS [30] tech-
The number of servers (N) can be calculated from; N ¼ 2ðkþ1Þ 
nology. MEMS consists of crossbar fabric made of mirrors which
ð4nÞ2k , For n > 4 where n is the number of servers in FiConn0. Also can direct light beams from inputs to outputs without decoding
Ficonn has a relative small diameter with an upper bound of or processing packets. Employing MEMS excludes the requirement
2ðkþ1Þ  1, which makes the structure suitable for real time applica- of signal conversion from optical to electronic which results in high
tions. Most attractively, the Ficonn’s cost is much less than other performance and less delays. Furthermore, MEMS consumes less
topologies since it employs less number of switches and most re- power as compared to electronic switching (240 mW vs. 12.5 W
lies on servers and efficient routing algorithms for switching and per port). However, MEMS has an issue with the reconfiguration
packet forwarding. time (few ms) which is seen to be long. A simplified Helios topol-
ogy model consists of 64 pods, each with 1024 hosts and two core
3.3. Optical data centers switches; one for optical circuit switching and the other for packet
switching. Depending on communication patterns, traffic shift and
In 2009 vision and roadmap report estimated a 75% of energy assignment are done statically between core switches through
saving can be obtained if data center infrastructure moves toward control software.
full optical network [28]. Optical interconnect schemes in data cen- The Helios design as depicted in Fig. 9 was based on three main
ters mainly relies on a mixture of active and passive optical devices modules for its control software: Topology Manager (TM), Circuit
 A. Hammadi, L. Mhamdi / Computer Communications 40 (2014) 1–21 7

Fig. 8. The Ficonn topology.

Fig. 9. The Helios hybrid electro-optical DCN topology.

Switch Manager (CSM) and Pod Switch Manager (PSM). Each mod- part, optical circuit switched network is used for rack to rack
ule has a distinct role. The TM is responsible for monitoring and high-speed connectivity. Each rack can have one circuit switched
estimating pods traffic demands between servers. Then, it com- connection at a time to communicate with any other rack in the
putes a new topology with optical switch configuration to sustain network. For changing traffic demands over time, optical switch
high network throughput all the time. The CSM is responsible for can be reconfigured (few milliseconds) to establish new matching
configuring the MEMS after receiving the graph of traffic connec- between different pairs of racks.
tion. The PSM module resides on the pod switches and has a con- The traffic demands are analyzed and hence links are formu-
nection interfacing with the topology manager. The PSM maintains lated by Edmond’s algorithm [32] for best maximum weight
statistical details about traffic sent out from its pods. Based on cal- matching to satisfy dynamic intensive traffic requests among racks.
culation made by the TM for traffic routing decisions, the PSM gets The design relies on optical configuration manager that collects
the information and routes traffic accordingly either through the traffic information from the traffic monitoring systems placed on
colorless transceivers or the WDM transceivers [30]. each host. Based on collected information, configuration manager
establishes circuit switched optical links among racks with respect
3.3.1.2. C-Through. C-Through as depicted in Fig. 10 is a hybrid to the bandwidth requirement among every pair of racks. Once the
packet and circuit switched data center network architecture (Hy- optical switch is configured, the ToR switches are informed about
PaC) introduced in [31]. The design aims at supplying high band- the new set up to route traffic via a special preconfigured VLAN
width to data intensive applications through high speed optical that is dedicated to serve only optical circuits.
circuit switched network that interconnects the DCN’s ToR In addition to C-Through, there were other hybrid designs such
switches. The HyPaC configuration, as can be seen in Figure 10, as the Optical Switching Architecture (OSA) architecture. OSA is a
consists of traditional packet switched DCN tree hierarchy with ac- recent novel scheme for data center network presented in [33].
cess, aggregation and core switches in the top part and in the lower The OSA is closely related to C-through and Helios architectures
8 A. Hammadi, L. Mhamdi / Computer Communications 40 (2014) 1–21

Fig. 10. The C-through hybrid electro-optical DCN topology.

except that OSA avoids using electronic components other than with oversubscription, bottlenecks, latency, wiring complexity
ToR switches. The added value of the OSA design is that it is highly and high power consumption. The Petabit switch flattened the net-
flexible because it can instantaneously adapt its topology and link work by designing one switch that is capable of connecting all
capacities whenever traffic patterns changes. racks within the data center. The design is targeting 10,000 of
100 Gbps ports by using one optical switch that is capable of deliv-
3.3.2. Hybrid electro-WDM PON data centers ering Petabit per second capacity. The structure of the Petabit
Tomkos in [34] proposed a novel design that introduces passive switch as shown in Fig. 12 is composed of a three-stage clos net-
optical network devices (PON) such as Arrayed Wave Guide Rou- work fabric with Input Modules (IMs), Central Modules (CMs)
ters (AWGR) in data centers. The design scheme as shown in and Output Modules (OMs), where each module has an AWGR
Fig. 11 consist of Ethernet ToR electronic switches that are used [36]. Multiple of AWGRs are required for the Petabit switch since
for intra rack communication and WDM PON devices (AWGR) for each AWGR can support few ports (128  128). Although the
inter rack communication. Each server is equipped with Ethernet AWGR is passive and not configurable, the routing path from an
and optical WDM transceivers. WDM PON participates in offload- input to an output and reconfiguration of the switch fabric are
ing inter-rack traffic and eliminating additional processing on managed by TWCs which take care of wavelength conversion and
ToR switches, hence power dissipated by TOR switches is reduced hence traffic can be routed from any input to any output. To
and high throughputs between racks are achieved with low delays. overcome the switch fabric reconfiguration time delay when deal-
Authors reported a 10% power saving through simulation using ing with small packets, Petabit assembles packets in frames of
three different traffic ratios for inter-rack and intra-rack flows. 200 ns size to allow sufficient time for fabric reconfiguration. In
addition, the Petabit switch employs an iterative frame scheduling
3.3.3. Full optical data centers algorithm to coordinate input output traffic assignment. The per-
Petabit [35,36] is a full optical switching solution for Data Cen- formance of Petabit was shown to be improved; with the employ-
ter Networks based on a bufferless optical switch fabric using com- ment of three iterations and speed up of 1.6, the scheduling
mercially available Array Waveguide Grating Router (AWGR) and algorithm achieved 100% throughput, a detailed description of
TWC [37]. The Petabit design objective is to overcome the issues the scheduling algorithm is presented in [36].

Fig. 11. The hybrid electro-WDM PON DCN topology.

 A. Hammadi, L. Mhamdi / Computer Communications 40 (2014) 1–21 9

Fig. 12. The Petabit full optical DCN topology.

Numerous other full optical designs for DCN interconnection switch centric architectures that can support fault tolerance, load
have been presented to provide viable solutions for future data balancing, agility and also overcome high oversubscription ratios.
centers, allowing for high bandwidth interconnection for especially Server centric data centers then came next to use servers as relay
video streaming and cloud computing applications with acceptable nodes for each other and provide an infrastructure with low diam-
reduced latency. Such full optical solutions are like DOS [38], Pro- eter and high capacity in order to support different traffic types for
teus [39] OSMOSIS [40], Space-WL [41], E-RAPID [42], IRIS [43], applications with intensive computing requirements. However, in
Data vortex [44], Polatis [45], OPST [46]. server centric designs, additional wiring cost and complexity are
a result of having servers equipped with more than one port.
4. Comparison and discussion of DCN architectures Advances in the optical networking technologies in providing
optical transceivers, arrayed wave guide routers, wave division
Over the past few years, the emergence of bandwidth intensive multiplexing, tunable lasers and passive optical devices have at-
applications with power consumption concerns has driven the evo- tracted great attention by researchers in academia and industries
lution of data center architectural designs. Fig. 13 depicts a classi- to adopt these technologies to overcome many existing issues in
fication of the most well-known DCN architectures and their the design of electronic switch and server centric data centers.
categorizations. DCNs are mainly classified into two classes: the The driving force for the redesign of data centers to include optical
electronic switch centric and server centric designs and the optical switching along with electronic switches has become an attractive
DCN designs. option because of the advancement of optical technology which
The efforts in the design of electronic data centers have suc- has brought the prices of optical switches and transceivers down
ceeded to mitigate many dilemmas and obstacles in providing and also due to the fact that optical switching can provide high

Fig. 13. Summary of data center architectures.

10 A. Hammadi, L. Mhamdi / Computer Communications 40 (2014) 1–21

bandwidth, low power consumption and less complexity as com- more than 5000. On the other hand, Fat-Tree structure topology
pared to the designs which only include electronic switching was found to have moderate power consumption values between
technology. Dcell and Bcube.
Hybrid schemes such as Helios and C-through are based on
readily commercially available optical components and can be 5.2. Virtualization
implemented by upgrading current data centers. Helios and
C-through are quite similar in the design except that C-through Starting with virtualization since it has given a great attention
uses WDM links. The main drawback of hybrid schemes is that by researchers and being the mostly adopted technique for data
MEMS take few milliseconds to be reconfigured, however, MEMS centers power saving [49]. Virtualization is a method to enable
were found as an attracting solution to replace high power con- services to be moved between servers and have mutiple VMs
suming electronic switches, where MEMs consume 0.24 Watts Machines which can serve different applications multiplexed to
and electronic switches consume 12.5 Watts per port. On the other share one server. Knowing that idle servers consume about 66%
hand, most of full optical data center schemes are based on Semi- of its peak and having in mind that data center resources are
conductor Optical Amplifier (SOA) switches which can replace underutilized since the average traffic load accounts for about
MEMS and sustain negligible reconfiguration time. Unlike hybrid 30% of its resources [24], agility can achieve servers statistical mul-
schemes, full optical schemes require a complete change of current tiplexing and give the illusion to services to make them feel that
data center in order to be implemented. Apparently, optical data they all connected to the same switch. Hence, servers can be placed
center schemes seem to be promising solutions to gradually re- anywhere within the network and be assigned to any service. The
place electronic data center schemes as they tend to provide low migration of virtual machines to consolidate workloads on a set of
power consumption and high bandwidth with low latency. servers and then by shutting down underutilized servers could def-
initely lead to a great power saving in data centers. However, many
barriers [1] like VLANs, access lists (ACLs), broadcast domains, and
5. Energy-efficient data centers
Load Balancers (LB) were standing as an obstacle and prevented
researchers and industries from immediate implementation of
The United States (US) Environmental Protection Agency (EPA)
VM migration (agility) on conventional data centers. The static net-
has reported in 2007 that data center power usage in the US dou-
work assignment between servers and services in conventional
bled between 2000 and 2006 to nearly 61 billion kilowatt-hours,
data centers prevent idle servers from being assigned for over-
which represented about 1.5% of all US electricity consumption
loaded services thus resulting in underutilization of resources [2].
[16]. The increase of power consumption will definitely result in
VL2, for instance, can be implemented on existing hardware and
large carbon foot print and more emission of greenhouse gases
can still provide high load balancing. The main objectives of VL2
which are the main contributors to global warming. The IT equip-
are providing agility among servers via the use of special flat
ment is the most power hungry components in data centers, repre-
addressing scheme that separate server names (AA) from their
sented by the servers, switches, routers and power distribution
locations (LA), then mapping between the AA and LA can be man-
infrastructure [18]. A performance metric for Power Usage Effi-
aged and handled by a directory system.
ciency (PUE) is used to measure how efficient a data center is in
LAs are addresses assigned to switches and interfaces (network
using its power and can be calculated by dividing the total facility
infrastructure) while applications are assigned with permanent
power by the IT equipment power consumption. The value of the
AAs. AAs remain unchanged no matter how servers’ location
PUE can be within the range 1.2 and 2, where a PUE value of 1.2
changes because of the VM migration. Each AA is associated with
would indicate a highly energy efficient data center [47].
LA which is the IP of the ToR switch to which the application server
In the following sections, the followings will be presented: a
is connected. The sender server, before sending, must encapsulate
concentrated investigation in different methods with a detailed
the packets in the outer header with the LA of the destination AA.
overview of energy saving approaches and industry adopted
Once packets arrive to the LA (ToR), the ToR switch encapsulates
techniques for energy efficient data centers such as virtualization,
the packets and sends them to the destination AA [6]. All servers
dynamic frequency and voltage scaling, dynamic network manage-
believe that they all belong to the same subnet, hence when any
ment, efficient green routing, green schedulers, network schedul-
application sends a packet to AA for the first time; the servers’ net-
ing schemes, and rate adaptation, smart cooling and renewable
work stack broadcasts an ARP request [6]. The VL2 agent intercepts
energy.
the request and sends a unicast query message to the directory ser-
ver which replies with the LA of the ToR switch where packets
5.1. Architecture of the data center topology and power consumption should be tunneled.
Portland [5] has proposed another way to solve the agility issue
The energy consumption of different structures of data centers in data centers. Portland, just like VL2, is based on a Fat-Tree net-
with emphasis on energy requirement of data center architecture work topology and consists of three layers: edge, aggregation and
with respect to transmission capability has been studied in [48]. core. Both separate names from locators and reduce broadcast by
Different topologies covering switch centric and server centric intercepting ARP requests and employ a unicast query through a
have been studied and simulated using power consumption values centralized lookup service. Portland assigns Pseudo MAC (PMAC)
of switches available in the market (Cisco and D-link). The power to all end hosts to encode their positions within the topology and
consumption of a server’s port was assumed to be 3 W. The energy it is changed whenever the location of the host is changed. The
consumption of DCell, BCube, Fat-Tree and balanced tree architec- Portland fabric manager is used for centralized lookup services, it
tures with comparable number of servers were studied in [48]. The is used to reduce broadcast overhead from the network and it
result has shown that balanced tree architecture consumes less works in the following manner: The switches intercept the ARP re-
power regardless of the number of servers but it has limited trans- quests for IP to MAC mapping and forward a unicast query to the
mission capacity because it has a server in the root that becomes a fabric manager which then provides the requested information to
throughput bottleneck. DCell and BCube happened to consume the the switch. The switch then forwards it to the requesting end host.
same amount of energy for small sized data centers consisting of In the case where the mapping details are not available, the fabric
about 2500 servers. However, Bcube consumes more energy for manager broadcasts to the core/aggregation/edge/hosts, host
larger data center if the number of servers is to be increased to which will reply with its AMAC which will be rewritten by the
 A. Hammadi, L. Mhamdi / Computer Communications 40 (2014) 1–21 11

egress switch to the appropriate PMAC before forwarding to the on a realistic web 2.0 application hosted on networked virtual ma-
requesting host and the fabric manager [2]. chines with several load samples and experimental results have
According to the EPA study, the servers are found to be the most shown that live migration overhead is acceptable but cannot be
hungry power parts of the ICT components of data centers since disregarded.
they contribute about 40% of total electricity consumption [16]. Having network resources into consideration while migrating
The agility was mainly proposed to reduce the cost of power con- VMs, [54] has discussed the impact of VM live migration on net-
sumption by low utilized servers and to facilitate an efficient use of work resources and how to control the traffic overhead caused
data center resources. By introducing virtualization, the services by live VMs migrations. A network aware scheduling for live
are able to be moved between machines and have the illusion that migration of VMs is introduced in [54]. It categorizes VMs accord-
all the servers assigned to them are connected by the same switch ing to their workload size and duration with an emphasis on hosts’
and hence allow a smooth migration among servers without utilization. Adequate resource and migration scheduling models
changing the IP address or any occurrence of TCP disconnection. for each class were also introduced [54], taking into account band-
Two methods were presented to achieve agility in current data width requirement for the migrations and network topologies to
centers via the implementation of VL2 [6] or Portland [5] designs ensure that load generated from VMs migration will not cause net-
respectively. Researchers [50] have proposed a method to optimize work congestion.
data center resources through dynamic consolidation of VMs on SecondNet [55] is a virtual data center network architecture
few servers while putting the rest on sleep state and hence bring- that can be built on top of many existing data center network
ing substantial energy savings while providing the required Qual- topologies such as Fat-Tree [4], VL2 [6], and Bcube [8]. In Second-
ity of Services (QoS). Net, a central Virtual Data Center (VDC) manages the VM requests
The migration of VMs is optimized by selecting the VMs to be and controls virtual to physical mapping with guaranteed band-
relocated on the basis of heuristics related to utilization thresholds. width reservation. Neighboring servers are grouped into clusters,
By setting up predefined thresholds values and through a continu- so that when VM requests are received, VDC allocation requires
ous monitoring of the servers’ resources utilization, a decision of search in specific cluster instead of searching in the whole net-
migrating VMs can be taken if these thresholds are exceeded. This work, which reduces the time complexity. In addition, grouping
results in a better performance for servers and also in lower power servers into clusters can place communicating VMs in the same
consumption because of the overheating and the cooling system. cluster or within a close distance which is in fact more bandwidth
On the other hand, VMs migration will also take place if servers’ re- efficient. The VDC manager uses spanning tree for signaling and
sources utilization is below certain predefined threshold values, devices also uses the spanning tree to deliver failure messages to
which will allow for shutting off these servers and save the power the manager VDC which in turn changes the routing paths and
consumed by an idle device. The system structure as shown in reallocate VMs if required. Path reallocation can be done in seconds
Fig. 14 [50] consists of a dispatcher, global and local managers. where VM migration takes tens of seconds. Simulation results have
The local manager role in the structure is to monitor the thermal shown that SecondNet provides a guaranteed bandwidth along
status and resources utilization of the network devices [51]. Based with high network utilization.
on the local manager observations, it sends to the global managers A method for VM placement has been proposed in [56] to
the collected information about the utilization of resources and the minimize the distance between VMs with large mutual bandwidth
VMs that have to be migrated, when the global managers become to reduce load at aggregation and core switches and to avoid
responsible for issuing commands for live migration of VMs, resiz- unnecessary consumption of network resources. The authors of
ing the network and hence eliminating servers by switching them [56] defined the Traffic-aware VM Placement Problem (TVMPP)
off. Another measure of VMs migration decisions carried by the lo- as an optimization problem, having traffic matrices and cost
cal manager is the VMs that have intensive communication with among VMs as the input. The heuristic algorithm to solve the
other VMs that are allocated in different physical hosts. TVMPP works in two tier approach, first it partitions the VMs
Workloads corresponding to web applications and online ser- and hosts into clusters, then it matches the VMs and hosts with re-
vices have been simulated in [50]. Results show that dynamic con- spect to traffic matrices and cost among VMs. [56] has investigated
solidation of VMs brings substantial energy savings that is close to four known data center architectures; Tree, Fat-Tree, VL2 and
83% while providing the required QoS. Consequently, this results in Bcube under different traffic patterns. The results have shown that
a user’s enhanced Quality of Experience (QoE) [52]. The VM migra- the benefits of the TVMPP relies on the network architecture. Since
tion in data center can generate a significant amount of overhead the VL2 architecture deploys valiant load balancing, it has shown
traffic passing from one server to another; hence an efficient meth- results with minimal benefits where Bcube shows more benefits.
od for migration should be taken into account to avoid unnecessary Finally, virtualization and live VM migration are widely adopted
congestion of the network links that may occur. A study [53] inves- for resource allocation because of the fact that this technique can
tigated the effect of overhead traffic resulting from live migration lead to considerable energy savings in data centers. Nevertheless,
the effects this could cause to the network performance in terms
of delay and throughput require careful considerations. A study
on Amazon data center to measure the impact of virtualization
on network parameters such as throughput, packet delay, and
packet loss has been conducted in [57]. This study shows that,
although data center is underutilized, virtualization can cause de-
lay variation and throughput instability. Hence, further studies are
required on how to customize applications to achieve good perfor-
mance on virtualized data centers.

5.3. Energy-aware routing in data centers

The objective of energy aware routing is to save power con-

sumption via putting idle devices on sleep or shutting them down
Fig. 14. Virtual data center. and using few network devices to provide routing with no sacrifice
12 A. Hammadi, L. Mhamdi / Computer Communications 40 (2014) 1–21

on network performance. Network devices consume 20%–30% of An ethernet link dissipates 2–4 W when operating at
the energy of the whole data center [58]. In [59], an energy aware 100 Mbps–1 Gbps and can dissipate 10–20 W when operating at
routing model was proposed and described as ERP-1: (G; T; K), 10 Gbps. Hence, lowering the operating data rate could have a dra-
where G is the topology, T is the traffic matrix and K is a predefined matic effect on power saving in data centers [62]. However, a keen
threshold of network throughput. The objective is to find a routing care has to be taken into consideration while lowering down the
for a specific topology where the total number of switches involved rate to keep the overall performance of the network intact which
in the routing can sustain a network throughput that is equal to or can be caused by congested links. The authors in [62] proposed
higher than the defined threshold. two approaches for rate adaptation; optimal and practical
A knapsack problem approach is used to optimize the number approaches. The optimal approach assumes a full knowledge of
of nodes that should be part of the routing for a number of flows future traffic arrival time, link rates and packet delay. Hence, the
while maintaining a throughput level not less than a predefined service curve is chosen to be the optimal for minimizing energy
threshold value [59]. The authors proposed a heuristic routing consumption and shall be the arrival curve which is shifted in time
algorithm made of three modules: Route Generation (RG), by the accepted delay bounds. On the other hand, the practical ap-
Throughput Computation (TC), and Switch Elimination (SE). Basi- proach makes use of the history of previous packet arrival rates
cally, the algorithm first computes the network throughput and chooses the rates among a predefined rate set. For queuing de-
through basic routing. Then, it gradually removes switches until lay estimation, the practical approach relies on link buffer size and
the network throughput approaches the predefined performance the rates. In summary, the rate shall be increased to r iþ1 for if the
threshold. Finally, it powers off or puts on sleep mode the switches current queue size divided by the current rate is bigger than the
that are not involved in the final routing. The output of the delay time. A decrease of rate to r i1 shall be enforced if the queue
heuristic shall consist of a (R, G) tuple where R is energy-aware is empty and the arrival rate is lower than r i1 . The authors pro-
routing chosen for T, and G is a final topology with SE. Three posed an algorithm to monitor the queuing delays in order to adapt
modules are executed repeatedly until the best acceptable the rates accordingly and achieve a steady state without creating
performance threshold is achieved and, in each round, some large queues and delays.
switches and links are eliminated then routing is generated for T
in the updated topology G.
5.6. Dynamic power management (DPM)
5.4. Dynamic voltage/frequency scaling (DVFS)
Dynamic power management (DPM) is a method used in data
centers to reduce power consumptions of some IT infrastructure
The frequency and voltage scaling represents another method
components by switching them off or by lowering the power state
to reduce servers power consumption, where there is a relation be-
when inactive. Such components can be the NICs, access switches,
tween voltage/frequency and the power consumed as described
aggregation switches, and servers as well [63]. Putting network
by: P = V2 ⁄f , (f is the frequency, V is the voltage and P is the
elements to sleep is not a new idea; it has been already imple-
power). The servers’ memory, bus, I/O resources and disks power
mented for microprocessors and smart phones. The idea is to put
consumptions are not affected since they do not rely on the CPU
line cards on sleep one by one then to put route processor and
frequency. Still, a significant saving can be achieved by reducing
switch fabric on sleep if all line cards are on sleep [62]. Measures
power via reducing frequency or voltage supplied to the processing
and considerations for modeling a network sleep state should take
chips [18]. In order to implement the DVFS technique on comput-
care of power draw of sleep state over idle state, transition time in
ing devices such as servers, hardware support for Advanced Config-
and out of a sleep mode, and the method to enter and exit a sleep
uration and Power Interface (ACPI) power management is required.
state [62]. The Wake Up on Arrival (WOA) method was proposed in
The ACPI has four modes of power states: G0 for power-on, G1 for
[64] for green internet is another example deployed for data center
partial sleeping that subdivides into four states, G2 is for soft-off
routers. The routers shall have a sensing circuit that is left powered
except with having the Power Supply Unit (PSU) still supplying
on during the sleep mode, and it senses traffic arrival and hence
power and G3 for power-off state [60].
wakes up routers to forward and then returns to sleep if no more
Researchers in [61] proposed PowerNap, which is a system that
packets are arriving. An issue of lost bits which arrive first to wake
dynamically and rapidly responds to instantaneous workload and
up the router which takes time to transit from sleep to active mode
makes transitions between the active and the idle states. Power-
was also solved by having dummy packets. In [62], an issue of fre-
Nap detects Lower Power Idle (LPI) state that discovers servers that
quent transitions due to small packet sizes was discussed and a
have been idle for long period to put them in low power states and
solution was proposed to overcome this issue by shaping traffic
also to minimize the response time in case of transition between
into bursts. As in Fig. 15 [62], the routers arrange and maintain
states. The authors introduced the concept of Redundant Array of
packets destined to the same egress into bursts and then forward
Inexpensive Load Sharing (RAILS) that facilitates the sharing of
them. This approach is called Buffer and Burst (B&B) and allows
power draw among multiple power supplies and guarantees zero
routers to sleep for longer time and hence save more power.
power draw in idle states which results in saving up to 66% of serv-
ers’ consumed idle powered. The authors claimed that through the
implementation of RAILS and PowerNap power consumptions by
servers can be reduced to 74% [61].

5.5. Rate adaptation in networks

Similar to the servers, DVS can be applied to links and switches

to reduce power consumption. With respect to traffic patterns and
link utilizations, data rate can be reduced by applying DVS on
transceivers and ports. The energy consumed by a switch can be
defined as [18]:
P
P switch ¼ P chasis þ nlinecards  Plinecard þ Ri¼0 nports  P r , where Pr is
the power consumed with respect to rate. Fig. 15. Buffer and Burst B&B.
 A. Hammadi, L. Mhamdi / Computer Communications 40 (2014) 1–21 13

5.7. Data center energy-aware scheduling methods simulation results for the different three schedulers: DENS sched-
uler, green scheduler and round robin scheduler [65].
Different approaches [18] of traffic scheduling in data centers The noticeable increase in energy consumption due to the in-
were studied and proposed to either consolidate workloads on a crease of number of selected servers between the green scheduler
few set of servers or to fairly distribute workload on the servers. and the DENS scheduler can be justified by the necessity of involv-
A tradeoff is always present between energy saving and perfor- ing extra servers and communication resources, detected by DENS
mance, hence the scheduling should always consider delay bounds, methodology to keep the quality of job execution at the desired le-
rate threshold and buffers occupancy in order to avoid degradation vel without congestions. DENS methodology always avoids over-
of performance while achieving a considerable saving in power loaded switches and servers by distributing the traffic load based
consumption in data centers. Traffic workloads are classified into on switches’ queue size and servers’ loads using a special metric.
three main categories as described in [62] with the workload types
in data centers and the best way of scheduling implemented in M ¼ a  fs þ b  fr þ c  fm
each category. The DENS metric M is defined as a weighted combination of
In [65], the authors proposed Data center Energy efficient Net- server-level fs , rack-level fr , and module-level fm functions. The
work aware Scheduling (DENS) whose main objective is to balance coefficients b; a; and c define the impact of each component (serv-
the energy consumption of a data center with performance, QoS ers, racks, and/or modules) on the metric behavior. Higher a values
and traffic demands. DENS achieves this objective via the imple- result in under-utilized racks being selected by the overloaded
mentation of feedback channels between network switches for servers. Higher values of b give priority to computationally loaded
workloads consolidation distribution amendments to avoid any racks with low network traffic activity, and Higher c values favor
congestion or hot spots occurrences within the network which the selection of loaded modules [65].
can definitely affect the overall performance. Congestion notifica-
tion signal by overloaded switches can prevent congestion which
may lead to packet losses and sustain the high data center network 5.8. Dynamic adjustments of active elements via network power
utilization. manager
On the other hand, the green scheduler [65] performs workload
consolidation on minimum possible set of links, switches, and 5.8.1. ElasticTree
servers and then uses DPM to switch off unused servers and The ElasticTree as depicted in Fig. 17, proposed in [58], is a
switches. Finally, round robin scheduler is implemented for Fat-Tree based data center topology that consists of a network
uniform distribution of workload over all servers, which results optimizer which continuously monitors traffic conditions within
in underutilization of resources of data centers. Fig. 16 shows the data center. The optimizer then chooses the set of network ele-
ments that need to be active in order to meet performance and
fault-tolerance requirements and switch off the remaining links
and switches that are not involved in the routing and load process-
ing. Various methods are proposed in [58] to decide which subset
of links and switches to use, such methods can be greedy bin-pack-
er, topology-aware heuristic, or prediction methods. The experi-
ments have shown that savings of 25–40% of the network energy
in data centers can be achieved if ElasticTree was implemented.
The ElasticTree as depicted in Fig. 19, consists of three logical
modules: optimizer, routing and power control. The optimizer role
is to find the minimum subset of the network in order to satisfy
traffic demands with decent performance. Once the optimizer
analyses the traffic matrix, it selects the minimum set of network
that needs to be onto satisfy the demands, then it passes the infor-
Fig. 16. The DENS, green and round Robin Schedulers [65]. mation to the power and routing modules which in turn control

Fig. 17. The Fat-Tree versus ElasticTree.

14 A. Hammadi, L. Mhamdi / Computer Communications 40 (2014) 1–21

Fig. 18. ElasticTree experimental setup for k = 4 (1 NetFPGA emulates 4 servers with 1GE connection each) [58].

the switching on or off of links and select flows routing the traffic patterns within a data center. For low traffic, Pcube turns
respectively. The optimization goal of ElasticTree is to minimize off many links and outputs a topology with few numbers of links.
P
ðlinks + switch power) while meeting the capacity, demand, Fig. 20 demonstrates an example for PCube (2, 4, 3) and PCube
and flow conservation constraints. Power saving can be calculated (2, 4, 4) where both host the same number of servers, but in PCube
by finding the ratio between the power consumed by the elastic- (2; 4; 3) 25% of switches are powered off to save 25% of the energy
Tree and the power consumed by Fat-Tree. Fig. 18 presents the consumed by switches. Further reductions in the number of
experimental setup through test bed to test and evaluate Elastic- switches, by switching off 8 more switches, will result in PCube
Tree as demonstrated in [58]. In the experiments, four NetFPGA (2, 4, 2), contributing to a saving of 50% in switch power.
representing 16 servers were used to generate traffic and a latency In PCube, a dedicated server acting as the network manager is
monitor was used to monitor packet drops and delays. responsible for receiving bandwidth requirement requests from
other servers and based on that it decides on how the structure
5.8.2. Pcube can be transformed. PCube calculates the new routing paths for
In addition to the switch centric ElasticTree, researchers have the modified structure and broadcasts the new topology structure
also investigated server centric data centers to improve power and routing information to all servers. The network manager al-
efficiency through dynamic adjustment of the network structure lows for few seconds before switching off the unnecessary
based on traffic demands. Modular server centric data centers such switches to ensure that all servers received the changes to the
as Bcube can offer high bandwidth with sufficient speed-up to topology and reacted upon it.
support high-performance for one to one, one to all and all to all Both PCube and ElasticTree dynamically and instantaneously
applications. PCube [66] as shown in Fig. 20 is represented as act on traffic demands to select a subset of the network elements
PCube (n,k,q) where n is the number of switch ports, k is the num- and hence reduce power consumption without sacrificing network
ber of levels and q is the number of ports in each server. Pcube is an performance. PCube and ElasticTree are based on the switch cen-
adaptive design with capability of switching off links depending on tric Fat-Tree and server centric BCube respectively. As a compari-
son, PCube (BCube) provides better one-to-all traffic support than
ElasticTree and with respect to power consumption, PCube can
save 50% of power consumed by ElasticTree serving the same num-
ber of servers [66].

5.8.3. Hierarchical energy optimization (HERO)

In data centers with thousands of switches, complexity is an is-
sue in solving the energy optimization problem because of the high
number of variables and constraints needed to formulate the prob-
lem. HERO is a recently introduced work by [67,68] to solve an
optimization problem in a hierarchal way to achieve similar results
for power saving achieved in non-hierarchal models. Hierarchal
HERO to non-hierarchal models ratio of variables and constraints
are 35% and 40% smaller respectively. Hence a great reduction in
time complexity is achieved with HERO.
In HERO, power optimization in data centers is divided into two
levels, the core level and the pod level. In the core level optimiza-
tion, core switches that serve outgoing traffic and aggregation
switches that serve for out of pod traffic must be determined and
be on active mode. While in Pod level optimization, aggregation
switches to serve intra pods traffic are determined and put on
Fig. 19. The Elastic Tree System Diagram. active mode.
 A. Hammadi, L. Mhamdi / Computer Communications 40 (2014) 1–21 15

Fig. 20. PCube.

For a given traffic matrix, a capacity constraint multi-commod- conditioning settings in parts of the data center accordingly. The
ity flow optimization (CMCF) problem is formulated for each level facility infrastructure design along with sensor reporting values
to determine switches and links that should be switched off in the of temperature indicate which parts of the data center is in need
network while assuring connectivity and QoS. Greedy heuristics of cooling, instead of having the cooling system in operation
based on different criteria for switches/links disconnection were continually. The DSC was implemented in Bangalore and achieved
also implemented to find additional nodes and links that can be 40% reduction in cooling energy consumption [74]. Schneider
switched off and hence more power saving is achieved. Simula- Electric [75,76] proposed 15 economizer modes to fully or partially
tions have been carried out for different traffic scenarios and re- bypass the function of the compressor without effecting the
sults have shown that the hierarchical model can achieve similar performance and reliability of the data centers, hence reducing
results of power saving as with the non-hierarchical model with the compressor energy use at full operational mode. Details on
a great reduction in algorithm complexity. the functionalities of the compressor and the proposed technique
can be found in [75,76].
5.9. Energy efficient cooling in data centers A case study [77] conducted by numerical modeling to show
that proper layout of racks can result in efficient utilization of cool-
The most challenging issue in designing a data center is how to ing systems and hence save energy. The study has shown that
come up with a design which can reduce overall power consump- slight changes in the layout and location of the data center racks
tion and has less carbon foot print contribution to the environ- could result in imbalance in cooling load on air conditioning units
ment. According to some studies [69], the power consumption of by 25%. In [78], the authors have presented a method to control the
the global data centers in 2007 was 330 billion kWh. The same temperatures at the racks by employing a thermodynamic model
study claimed if no implementation on green energy methods is that combines air flow control and thermal aware scheduling.
enforced on data centers, data center’s power consumption will ex- The thermal aware scheduler receives information about tempera-
ceed 1000 billion kWh by 2020 which translates to 257 MtCO2 gas tures measurement at the racks outlets and based on that it
emissions [69]. As the size of data centers increases, power density dynamically allocates workloads to the servers. A model called
and heat dissipation from servers and network equipment have heat flow has been proposed in [79] that can be implemented with
also significantly increased. The dissipated heat from these sys- real time thermal aware scheduling to achieve optimal energy effi-
tems along with the elevated equipment temperature must be ciency. The heat flow model is designed to characterize hot air
maintained at acceptable rates for reliable operation and to avoid recirculation based on temperature information collected by dis-
any hardware failures. A break down cost analysis for a data center tributed sensors. C-Oracle [80] and Weatherman [81] introduced
has been provided in [1], stating that the amortized cost for cooling a software infrastructure to predict heat profiles in the data center
and power distribution is about 25% of the total cost. As a result, for dynamic thermal management, thereby obviating the need of
smart energy efficient cooling techniques for data centers have be- manual static configuration of thermal load management systems.
come a major and an attractive challenge. To tackle this issue, in As presented earlier, many techniques for reducing power on
2004 the American Society of Heating, Refrigerating, and Air-Con- idle servers rely on shutting them down and concentrate loads
ditioning Engineers (ASHRAE) published ‘‘Thermal Guidelines for on a subset of the network. This method will cause hot spots and
Data Processing Environments’’ [70] as a guideline document for therefore increase the cooling cost. Also degradation in the re-
designers and equipment manufacturers to standardize the de- sponse time for servers to transit from off to active mode will cause
signs of data center facility to help creating energy efficient data a delay and affect the overall performance. In order to reduce cool-
centers [71]. ing power, the network load has to be spread over many servers
Researchers and industries have investigated smart cooling which will result in low utilization and increase idle power. To
technologies and shown its effectiveness in saving energy in data overcome these problems, two techniques were proposed in [82],
centers. Dynamic Smart Cooling (DSC) [72,73], proposed by HP the PowerTrade technique for joint optimization of idle power
can reduce the cooling power consumption within a data center and cooling power and the SurgeGaurd technique to address and
by 30–60%. DSC combines sensors with control nodes that solve the issue of response time degradation caused by the state
continuously monitor temperatures and consequently adjust air transition of servers.
16 A. Hammadi, L. Mhamdi / Computer Communications 40 (2014) 1–21

5.10. Renewable energy supply for data centers be presented along with a classification of these efforts as depicted
in Table 2.
The driving force of green energy efficient data center is not The architecture of a DCN plays an important role and could
only related to reducing power consumption and electrical bills lead to great savings in power budget if taken into consideration.
but also to minimizing the carbon foot print of DCNs to the envi- Architectural designs such as Portland [5] and VL2 [6] provided a
ronment. In order to achieve these goals, industry and academia solution for agility. The introduction of PONs in the DCNs design
have a great interest in applying renewable green energy sources for rack to rack communication have enhanced the performance
to power data centers by using wind or solar energy and replacing and reduced energy consumption for about 10% [34]. The optical-
brown energy supplied from the electrical grid. In [83,84] a parallel based architecture designs such as Petabit [36] and Proteus [39]
patch job scheduler was proposed to maximize the usage of the were reported to bring huge savings in power consumption that
green supply while meeting the job’s deadlines. If the deadlines could reach almost 75% [28]. Flattening data centers from three
cannot be met, brown energy is used and the workload is sched- tiers and two tiers to one-tier fabric such as QFabric has also re-
uled for times when the energy is cheap. This implementation ported to increase the savings to reach 77% [27]. Numerous other
guarantees more consumption of renewable energy and incurs efforts as presented in Section 6 are also included in the summary
low cost when brown energy is issued. Similarly, the Hadoop comparison as shown in Table 2. Different metrics have been con-
[85] was proposed, a MapReduce framework for data centers, sidered in the comparison such as techniques used, methods of
powered by solar energy and also by electrical grid as a back-up. evaluation and reported results of saving.
Hadoop predicts the amount of solar energy that will be available,
and schedules the MapReduce [86] jobs to maximize the solar en-
7. Green data center simulators
ergy consumption within the deadlines of the jobs.
Maximizing the usage of green energy for multiple data centers
Different simulation tools were developed to test and experi-
which are geographically distributed has also been studied. In fact,
ment the performance and energy saving in cloud computing
for a geographically distributed data center, better opportunities
infrastructures such as GreenCloud [18], CloudSim [93–95] and
can be exposed to implement greener data centers where less
MDCSIM [96]. These simulators differ in many aspects. A qualita-
brown energy is consumed. A framework [87] was proposed for
tive comparison using different evaluation metrics to present key
request distribution within two policies. The policies consider time
differences between the three simulators is presented in Table 3.
zones, variable electricity prices and green energy to leverage data
Important aspects are evaluated in the Table to provide insights
centers powered by green energy and data centers with minimum
on which simulator to use to experiment certain criteria on power
electricity prices. Similarly in [88], a framework is presented for
saving within a data center. The main metrics used for the compar-
request distribution policy to promote capping of brown energy
ative evaluation are; simulation run time, supported workload
under the concept of cap-and-trade [89]. Distributed algorithms
types, platform used, implemented power saving modes and virtu-
for green optimal geographical load balancing have been proposed
alization capability.
in [90]. The distributed algorithms proved that optimal route can
GreenCloud [18], a packet based with TCP/IP support, built on
be computed to route to areas where green energy is available
top of the NS2 [98] network simulator, can determine the total
and therefore reduce the use of the brown energy. A discussion
power consumption by the data center components (servers,
[91] on how to maximize green energy use in data centers through
switches, and links). GreenCloud is coded in C++ with Otcl script-
renewable-aware management made two observations. First,
ing. An advantage of Greencloud is that it has the capability of
request distribution policy should identify data centers powered
computing the power consumption with enabled saving modes
by renewable energy and forward as many user requests as possi-
such as DVS, DNS or by enabling both modes together. However,
ble to these data centers. The second observation is to build re-
GreenCloud has a drawback since its simulation takes a long time,
quest-level power/energy profiles that can classify energy hungry
requires high memory usage and its scalability is limited to only
requests and have them handled by data centers with excess
small data centers [18]. GreenCloud supports the test and evalua-
renewable energy. GreenWare [92] was proposed to maximize
tion of different workload types: Computational Intensive Work-
the use of renewable energy. GreenWare is a middleware system
load, Data Intensive Workload, and Balanced Workload. For more
that can dynamically dispatch incoming service requests among
details on each workload type refer to Table 1.
multiple geographically distributed data centers, based on electric-
The CloudSim [93] simulator, developed at the University of
ity prices and weather conditions.
Melbourne, is found to be one of the best simulators since it has
short simulation times and can run for large data centers scaling
6. Comparison of energy efficient efforts in DCNs to hundreds of thousands of nodes [93]. Unlike GreenCloud, where
algorithms to model and simulate data centers to evaluate power
Both industry and academia have heavily contributed to tackle saving modes such as DVS and DNS are only supported, CloudSim
the issue of power consumption of the exponentially growing allows evaluating other energy saving schemes in data centers
DCNs. In this section, a qualitative comparison to summarize and through resource virtualization and does not evaluate energy
compare different efforts for designing energy efficient DCNs shall consumption by network elements. Hence, its workload is more

Table 1
Data Center Workloads Classification [65].

Computational Intensive Workload (CIW) Data Intensive Workload (DIW) Balanced Workload (BW)
 Load at computing servers and require almost no data  No load at computing servers and require heavy data  Applications having both com-
transfer in the interconnection network fabric of the data transfers. (examples: video file sharing and streaming). puting and data transfer
center.  Continuous feedback between switches and scheduler requirement
 CIW scheduling groups the workload at the minimum set to allow scheduler send loads over non congested  Example of such is Geographic
of servers and routing using the minimum set of routes. routes Information System (GIS).
 No danger of congestion as long as data transfer is low.  Scheduler accounts for servers’
load and the load of links.
 A. Hammadi, L. Mhamdi / Computer Communications 40 (2014) 1–21 17

Table 2
Comparison of energy efficient efforts in data centers.

Ref. Approach Methods/Techniques Evaluation Results

[48] Architecture Structure of topology Simulation Results have shown a tradeoff between transmission
capacity and power consumption. Fat-Tree structure
topology has moderate power consumption values lays
between Dcell and the Bcube.
[27] Architecture One-tier fabric prototyped Qfabric reported to save 77% by reducing power
consumption of links, cooling systems, and switches.
[31,33,29] Architecture Hybrid optical and electrical prototyped Reduction in power consumption when decision is taken
to route traffic through optical switches
[38–46,35,36] Architecture Full Optical Proteus and IRIS are Saving reported to reach 75%
prototyped only
[34] Architecture Hybrid WDM PON Simulation 10% of power saving with negligible delay
[5,6] Virtualization Agility Testbed and simulation Agility implemented on Portland using additional
hardware (fabric manager), and by modifying servers’
network stack for VL2 to allow smooth migration of
services from one machine to another without TCP
disconnection.
[50] Virtualization Dynamic relocation of VMs CloudSim toolkit Simulator 83% of energy saving for web application
[53] Virtualization Live migration Testbed Effect of overhead traffic resulted from live migration
between servers for web-applications are reported to be
acceptable but shall not be disregarded
[54] Virtualization Design of network aware To be implemented on Provided a sketch for how to include migration control
scheduler commercial data center and scheduling to the architecture. Proposed scheduler
operator design to categorized VMs according to workload size
and duration in order to control and avoid congestion.
[55] Virtualization SecondNet Virtual data Testbed and simulation SecondNet can provide guaranteed bandwidth along
center with high network utilization. Path relocation and VMs
migrations can be done in few seconds.
[56] Virtualization Traffic-aware VM Heuristics Minimization of distance between VMs with large
placement mutual bandwidth reduced the load at core and
aggregation switches and unnecessary power
consumption is avoided.
[57] Virtualization Virtualization on Amazon Measurement study on Study has shown considerable saving if virtualization is
data center to study its commercial Amazon EC2 implemented however delay variation and throughput
impact on throughput, cloud service instability will occur.
packet delay, and packet
loss.
[59] Energy-Aware Routing Routing algorithm to Algorithm Algorithm tested on Fat-Tree and Bcube, results have
balance between shown that at low network load, energy-aware routing
throughput and switch algorithm could result in significant saving on energy
elimination consumed by network.
[60] DVFS Equipping servers with ACPI Hardware interface The ACPI has four modes of power states: G0 for power-
on, G1 for partial sleeping, G2 is for soft-off and G3 for
power-off state
[61] DVFS PowerNap and RAILS Analytical modeling and Tested on Web 2.0; PowerNap minimizes idle power and
simulation on real workload transition time, and result in almost 70% of power
traces reduction. RAILS add additional 26% of power savings if
implemented with PowerNap.
[62] Rate adaptation (RA) Optimal RA and Practical RA Agorithms Saving with low impact on performance
[64] DPM Wake up on arrival (WOA) Simulation Saving with some impact on performance caused by
transition delays
[58] DPM ElasticTree Heuristics and testbed with 25-40% saving of the network energy
NetFPGA
[62] DPM Burst and Buffer (B&B) Simulation/ Algorithm 20-75% energy saving with low impact on performance
(low delays).
[67,68] DPM HERO Hierarchal model for Heuristic HERO achieves similar results of power saving as with
power optimization on Fat- the non- hierarchical model with a great reduction on
Tree DC algorithm complexity.
[66] DPM Pcube Simulation PCube can save 50% of power consumed by ElasticTree
serving the same number of servers
[65] Energy-aware scheduler Green scheduler GreenCloud simulator 51.3% of power savings compared to round robin
scheduler
[65] Energy-aware scheduler DENS scheduler GreenCloud simulator 49% of power saving compared to round robin scheduler.
Comparable savings with the Green scheduler with
congestion notification to avoid loss of packets at
switches
[72,73] Energy-efficient cooling Dynamic Smart Cooling Experiments on data center Reduce cooling power by 30–60%. Implemented in
(DSC) Bangalore and achieved 40% of reduction in cooling
power
[75,76] Energy-efficient cooling 15 economizer modes N/A Saving can reach 70% of yearly cooling cost
[77] Energy-efficient cooling Numerical modeling for Numerical modeling Minor change of layout and location of the racks would
rack layouts experiments imbalance in cooling load on AC units by 25%.
[78] Energy-efficient cooling Thermo-dynamic model Simulink simulations with Results with simulations have shown that with the
synthetic and real work- approach of combining air flow control and thermal
load traces (NASA) aware scheduling, the temperatures at the racks can be

(continued on next page)

18 A. Hammadi, L. Mhamdi / Computer Communications 40 (2014) 1–21

Table 2 (continued)

Ref. Approach Methods/Techniques Evaluation Results

controlled in an efficient and stable manner.
[79] Energy-efficient cooling Heat flow model Computational Fluids Heat flow model implemented with real time thermal
Dynamics (CFD) simulation aware scheduling to evaluate temperature and
characterize air recirculation in data centers.
[80] Energy-efficient cooling C-Oracle Software infrastructure Implementation of C-oracle with thermal management
policies shown provision of accurate prediction in order
to manage thermal emergencies.
[81] Energy-efficient cooling Weatherman CFD approach with a Weatherman thermal topology aware with different
prototype model load distribution introduced benefits in minimizing
cooling costs, and in the avoidance of degradation under
thermal emergencies.
[83,84] Renewable energy supply GreenSlot: A parallel patch Experimental hardware and GreenSlot scheduler predicts solar availability and
job scheduler software. Micro data center guarantees more consumption of green energy and
prototyped (Parasol) decrease brown energy consumption cost by 39%
[85] Renewable energy supply GreenHadoop Experimental hardware and Similar to GreenSlot, GreenHadoop predict solar
software availability to increase green energy consumption by up
to 31% and decrease grid power cost by up to 39%
[87] Renewable energy supply Request distribution based Heuristic and optimization Request distribution on Geo-distributed mirror data
on policies for problem centers with policies to leverage data centers powered
geographically distributed by green energy and assure minimum electricity prices
data centers when routed to grid power data centers.
[88] Renewable energy supply Request distribution based Cost aware heuristic policy 24% reduction in brown energy consumption for only a
on cap and trade concept. and cap-trade optimization 10% increase in cost
problem
[90] Renewable energy supply Distributed routing Algorithms Distributed routing algorithms to load balance and to
algorithms favor areas where green energy is available.
[92] Renewable energy supply GreenWare Optimization based on Geographical dispatching of requests based on
linear fractional electricity prices and weather conditions to increase
programming (LFP) usage of green energy without violating cost budget.

Table 3
Comparison of data center green simulators.

GreenCloud CloudSim MDCSim

Execution time Minutes Seconds Seconds
GUI implementation Partial with network animation tool Nam Partial with CloudAnalyst None
Platform/language C++/Tcl on NS2 Java on SimJava C++/Java on CSIM [97]
Developer/owner University of Luxembourg University of Melbourne Pennsylvania state University
Year Announced 2010 2009 2009
Licence Open source Open source Commercial Product
DVFS/DNS modes support enabled Not enabled Not enabled
Packets/event support Packet based simulator Event based simulator Event based simulator
Virtualization N/A Enabled N/A
TCP/IP support Full support N/A N/A
Workload support (CIW), (DIW), and (BW) (CIW) and (DIW) (CIW)

relevant in grid networks since it mostly concentrates on servers, 8. Insights and future research directions in DCNs
the highest energy consuming component in the data center. The
application model implemented by CloudSim works well for com- While considerable research efforts have been devoted to
putationally intensive workloads with no specific completion efficient DCN design, there are numerous open questions to be
deadlines. A Graphical User Interface tool (GUI) called CloudAna- addressed to keep up with the ever growing cloud computing
lyst [93] is built on top of the CloudSim to provide simplicity for infrastructures. Below, we outline some of the future research
using the simulator and visual modeling and to help determining directions in DCNs.
the best approaches for allocating resources. The design of scalable cost effective and energy efficient DCNs
The MDCSim Simulator [96] is a commercial product and is not requires a unified view that takes into account various factors such
available for public use unlike the GreenCloud and CloudSim as the architectural choices, the transport layer and the energy
simulators which are released under open source GPL license. efficiency. This unified vue is perhaps largely missing in current
MDCSim is built on CSim platform and coded using Java and C++. research proposals and still to be seen. In fact, up until now, most
The MDCSim simulator is similar to the CloudSim where both are research efforts target one problem while neglecting the impact of
event-driven simulators which give the advantage of minimizing other important and related issues. An example of this is the archi-
the simulation time and have the capability of simulating large tecture and DCN transport design. On the architectural level, most
data centers. Similar to CloudSim, MDCSim is meant to monitor DCNs rely on traditional enterprise-class networking equipment
and measure energy consumption for servers only and ignores designed for traditional Internet infrastructure, which are not opti-
communication elements such as switches and links. MDCSim mized for the levels of agility, reliability and high utilisation re-
lacks accuracy since it relies on rough estimation by employing a quired by modern DCNs. The transport layer design is seeing
heuristics averaging on the number of the received requests for a similar practices by mainly mapping earlier solutions proposed
given time period [18]. for traditional networks where performance is often traded for
 A. Hammadi, L. Mhamdi / Computer Communications 40 (2014) 1–21 19

generality. Therefore, a combined architecture and transport de- dynamic voltage/frequency scaling, rate adaptation, dynamic
sign may be beneficial for both problems, by taking advantage of power management (DPM), energy-aware scheduling methods
the properties and optimisation goals of each. Most existing DCN and dynamic adjustment of active network elements in DCNs.
topologies exhibit some sort of regularity (e.g., Fat-Tree, BCube) We have also outlined current techniques and practices for en-
which should be taken advantage of in better traffic prediction ergy-efficiency in DCNs and green data centers, including cooling
and VM placement. This would not only translate into scalable techniques and the use of renewable energy in DCNs. We have
DCN design but also in more energy efficient DCNs. As a result, provided a detailed comparison of existing research efforts in
having a unified vue of the DCN as a whole would result in a better DCN energy efficiency as well as the most adopted testbeds and
design of both scalable and cost effective architectures, while pro- simulation tools for DCNs. Finally, we have outlined some insights
viding efficient transport. This requires further research work. for potential futures open research questions in DCNs.
As for energy efficiency, a good architecture design should be
tailored along with efficient algorithms to address not only power Acknowledgement
consumption but also network performance along with cost. More
research efforts are required to design energy aware routing algo- This work was partially supported by an EU Marie Curie Grant
rithms capable of consolidating traffic on routes to avoid underuti- (SCALE: PCIG-GA-2012-322250).
lization of parts of the network with continuous monitoring of
network conditions to avoid bottlenecks and to ensure acceptable References
delays for sensitive delay applications such as video and online
gaming applications. Consolidation of workload on group of serv- [1] A. Greenberg, J. Hamilton, D.A. Maltz, P. Patel, The cost of a cloud: research
problems in data center networks, SIGCOMM Comput. Commun. Rev. 39 (1)
ers would create high utilization on servers which will result in (2008) 68–73, http://dx.doi.org/10.1145/1496091.1496103. URL <http://
more power consumption for server cooling. A study has shown doi.acm.org/10.1145/1496091.1496103> .
that for each watt consumed by a network device or a server, [2] Y. Zhang, N. Ansari, On architecture design, congestion notification, tcp incast
and power consumption in data centers, Commun. Surv. Tutorials, IEEE 15 (1)
one watt is consumed for cooling [34]. Hence setting thresholds (2013) 39–64, http://dx.doi.org/10.1109/SURV.2011.122211.00017.
for servers’ utilization and proper thermal monitoring would en- [3] <https://www.cisco.com/application/pdf/en/us/guest/netsol/ns107/c649/
hance network performance and avoid unnecessary power con- ccmigration09186a008073377d.pdf>
[4] M. Al-Fares, A. Loukissas, A. Vahdat, A scalable, commodity data center
sumption. More efforts are encouraged to further study means of network architecture, in: Proceedings of the ACM SIGCOMM 2008 Conference
monitoring loads and thermal status on servers to avoid overheat- on Data Communication, SIGCOMM’08, ACM, New York, NY, USA, 2008, pp.
ing to avoid network performance degradation. More investigation 63–74, http://dx.doi.org/10.1145/1402958.1402967. URL <http://doi.acm.org/
10.1145/1402958.1402967> .
on the reduction of processing power of DCN switches by promot-
[5] R. Niranjan Mysore, A. Pamboris, N. Farrington, N. Huang, P. Miri, S.
ing the implementation of PONs in data centers for not only rack to Radhakrishnan, V. Subramanya, A. Vahdat, Portland: a scalable fault-tolerant
rack communication but also for intra rack interconnections. layer 2 data center network fabric, SIGCOMM Comput. Commun. Rev. 39 (4)
For geographically distributed data centers, the emphasis (2009) 39–50, http://dx.doi.org/10.1145/1594977.1592575. URL <http://
doi.acm.org/10.1145/1594977.1592575> .
should be on the implementation of content distribution network- [6] A. Greenberg, J.R. Hamilton, N. Jain, S. Kandula, C. Kim, P. Lahiri, D.A. Maltz, P.
ing to cache most popular objects at data centers where most Patel, S. Sengupta, Vl2: a scalable and flexible data center network, Commun.
demands for those objects exist to avoid long routes. This reduces ACM 54 (3) (2011) 95–104, http://dx.doi.org/10.1145/1897852.1897877. URL
<http://doi.acm.org/10.1145/1897852.1897877> .
the number of router ports used which is found to be the most [7] C. Guo, H. Wu, K. Tan, L. Shi, Y. Zhang, S. Lu, Dcell: a scalable and fault-tolerant
consuming elements in long haul networks in addition to the network structure for data centers, in: Proceedings of the ACM SIGCOMM 2008
reduction of number of amplifiers, transponders and optical mux/ Conference on Data Communication, SIGCOMM’08, ACM, New York, NY, USA,
2008, pp. 75–86, http://dx.doi.org/10.1145/1402958.1402968. URL <http://
demux used along the long routes. More efforts are needed to make doi.acm.org/10.1145/1402958.1402968> .
efficient use of renewable green energy to power few sites of geo- [8] C. Guo, G. Lu, D. Li, H. Wu, X. Zhang, Y. Shi, C. Tian, Y. Zhang, S. Lu, Bcube:
graphically distributed data centers. Achieving this requires the a high performance, server-centric network architecture for modular data
centers, SIGCOMM Comput. Commun. Rev. 39 (4) (2009) 63–74, http://
design of efficient energy aware routing algorithms to forward dx.doi.org/10.1145/1594977.1592577. URL <http://doi.acm.org/10.1145/1594977.
classified tagged demands with non sensitive delay applications 1592577> .
to data centers supplied with renewable energy. Optimal DCN [9] D. Li, C. Guo, H. Wu, K. Tan, Y. Zhang, S. Lu, Ficonn: using backup port for server
interconnection in data centers, in: INFOCOM 2009, IEEE, 2009, pp. 2276–
design is a challenge since reduction in power consumption should
2285, http://dx.doi.org/10.1109/INFCOM.2009.5062153.
not jeopardize performance or DCN operational cost. More re- [10] A. Greenberg, P. Lahiri, D.A. Maltz, P. Patel, S. Sengupta, Towards a next generation
search efforts are needed to address these challenges. data center architecture: scalability and commoditization, in: Proceedings of the
ACM Workshop on Programmable Routers for Extensible Services of Tomorrow,
PRESTO’08, ACM, New York, NY, USA, 2008, pp. 57–62, http://dx.doi.org/10.1145/
1397718.1397732. URL <http://doi.acm.org/10.1145/ 1397718.1397732> .
9. Conclusions [11] Y. Liao, D. Yin, L. Gao, Dpillar: Scalable dual-port server interconnection for
data center networks, in: 2010 Proceedings of 19th International Conference
on Computer Communications and Networks (ICCCN), 2010, pp. 1–6, http://
This survey provides a full review of the most recent advances dx.doi.org/10.1109/ICCCN.2010.5560132.
in DCNs with a special emphasis on the architectures and energy [12] K.V. Vishwanath, A. Greenberg, D.A. Reed, Modular data centers: how to design
efficiency in DCNs. We have described the conventional tree-based them?, in: Proceedings of the 1st ACM workshop on Large-Scale system and
application performance, LSAP’09, ACM, New York, NY, USA, 2009, pp 3–10,
DCN architecture and discussed the challenges inherited from this http://dx.doi.org/10.1145/1552272.1552275. URL <http://doi.acm.org/10.1145/
architecture. We then surveyed the architectural evolution in DCNs 1552272.1552275> .
and categorized these architectures as switch-centric, server-cen- [13] J. Mudigonda, P. Yalagandula, M. Al-Fares, J.C. Mogul, Spain: cots data-center
ethernet for multipathing over arbitrary topologies, in: Proceedings of the 7th
tric as well as their underlying technologies into electronic, optical USENIX Conference on Networked Systems Design and Implementation,
and hybrid electro-optical DCNs. The switch-centric architectures NSDI’10, USENIX Association, Berkeley, CA, USA, 2010, p. 18. URL <ttp://
surveyed include the Fat-Tree, VL2 and Portland. The server-centric dl.acm.org/citation.cfm?id=1855711.1855729> .
[14] H. Wu, G. Lu, D. Li, C. Guo, Y. Zhang, Mdcube: a high performance network
architectures include BCube, DCell and FiConn. We have provided a
structure for modular data center interconnection, in: Proceedings of the 5th
quantitative comparison and detailed discussion of current DCN International Conference on Emerging Networking Experiments and
architectures. Technologies, CoNEXT’09, ACM, New York, NY, USA, 2009, pp. 25–36, http://
In parallel to the architectural evolution in DCNs, a detailed sur- dx.doi.org/10.1145/1658939.1658943. URL <http://doi.acm.org/10.1145/1658939.
1658943> .
vey of recent advances in energy efficiency has been conducted. [15] P. Costa, A. Donnelly, G. O’Shea, A. Rowstron, CamCubeOS: a key-based
Techniques such as virtualization, energy-aware routing in DCNs, network stack for 3D Torus cluster topologies, in: The 22nd ACM International
20 A. Hammadi, L. Mhamdi / Computer Communications 40 (2014) 1–21

Symposium on High Performance Parallel and Distributed Computing [40] R. Luijten, W. Denzel, R. Grzybowski, R. Hemenway, Optical interconnection
(HPDC’13), ACM Press, New York, NY, US, 2013, pp. 73–84. networks: the osmosis project, in: Lasers and Electro-Optics Society, 2004.
[16] B.R., Report to congress on server and data center energy efficiency public law LEOS 2004. The 17th Annual Meeting of the IEEE, vol. 2, Nov., pp. 563–564.
109–431, Tech. Rep., 2007. http://dx.doi.org/10.1109/LEOS.2004.1363363.
[17] L.A. Barroso, U. Holzle, The case for energy-proportional computing, 40 (12) [41] O. Liboiron-Ladouceur, I. Cerutti, P. Raponi, N. Andriolli, P. Castoldi, Energy-
(2007) 33–37. efficient design of a scalable optical multiplane interconnection architecture,
[18] D. Kliazovich, P. Bouvry, Y. Audzevich, S. Khan, Greencloud: a packet-level IEEE J. Sel. Top. Quantum Electron. 17 (2) (2011) 377–383 (March–April).
simulator of energy-aware cloud computing data centers, in: Global [42] A. Kodi, A. Louri, Energy-efficient and bandwidth-reconfigurable photonic
Telecommunications Conference (GLOBECOM 2010), 2010, IEEE, 2010, pp. 1– networks for high-performance computing (hpc) systems, IEEE J. Sel. Top.
5, http://dx.doi.org/10.1109/GLOCOM.2010.5683561. Quantum Electron. 17 (2) (2011) 384–395, http://dx.doi.org/10.1109/
[19] Y. Chen, R. Griffith, J. Liu, R.H. Katz, A.D. Joseph, Understanding tcp incast JSTQE.2010.2051419 (March–April).
throughput collapse in datacenter networks, in: Proceedings of the 1st ACM [43] J. Gripp, J. Simsarian, J. LeGrange, P. Bernasconi, D. Neilson, Photonic terabit
Workshop on Research on Enterprise Networking, WREN’09, ACM, New York, routers: The iris project, in: Optical Fiber Communication (OFC), collocated
NY, USA, 2009, pp. 73–82, http://dx.doi.org/10.1145/1592681.1592693. URL National Fiber Optic Engineers Conference, 2010 Conference on (OFC/NFOEC),
<http://doi.acm.org/10.1145/1592681.1592693> . March, pp. 1–3.
[20] Y. Zhang, N. Ansari, On mitigating tcp incast in data center networks, in: [44] O. Liboiron-Ladouceur, A. Shacham, B. Small, B. Lee, H. Wang, C. Lai, A.
INFOCOM, 2011 Proceedings IEEE, 2011, pp. 51–55. http://dx.doi.org/10.1109/ Biberman, K. Bergman, The data vortex optical packet switched
INFCOM.2011.5935217. interconnection network, J. Lightwave Tech. 26 (13) (2008) 1777–1789,
[21] A. Phanishayee, E. Krevat, V. Vasudevan, D.G. Andersen, G.R. Ganger, G.A. http://dx.doi.org/10.1109/JLT.2007.913739.
Gibson, S. Seshan, Measurement and analysis of tcp throughput collapse in [45] The New Optical Data Center, Polatis Data Sheet, Polatis Inc., 2009.
cluster-based storage systems, in: Proceedings of the 6th USENIX Conference [46] IVX8000 Product Datasheet. InTune, Networks, 2010.
on File and Storage Technologies, FAST’08, USENIX Association, Berkeley, CA, [47] <http://www.morganclaypool.com/doi/pdf/10.2200/
USA, 2008, pp. 12:1–12:14. URL <http://dl.acm.org/citation.cfm?id=1364813. s00193ed1v01y200905cac006>
1364825> . [48] L. Gyarmati, T.A. Trinh, How can architecture help to reduce energy
[22] V. Vasudevan, A. Phanishayee, H. Shah, E. Krevat, D.G. Andersen, G.R. Ganger, consumption in data center networking?, in: Proceedings of the 1st
G.A. Gibson, B. Mueller, Safe and effective fine-grained tcp retransmissions for International Conference on Energy-Efficient Computing and Networking,
datacenter communication, SIGCOMM Comput. Commun. Rev. 39 (4) (2009) e-Energy’10, ACM, New York, NY, USA, 2010, pp 183–186, http://dx.doi.org/
303–314, http://dx.doi.org/10.1145/1594977.1592604. URL <http://doi.acm. 10.1145/1791314.1791343. URL <http://doi.acm.org/10.1145/1791314.
org/10.1145/1594977.1592604> . 1791343> .
[23] Y. Zhang, N. Ansari, Fair quantized congestion notification in data center [49] D. Chernicoff, The Shortcut Guide to Data Center Energy Efficiency, Realtime
networks, IEEE Trans. Commun. (99) (2013) 1–10, http://dx.doi.org/10.1109/ Publisher, Newyork, 2009. URL <http://nexus.realtimepublishers.com/
TCOMM.2013.102313.120809. sgdcee.php?ref=gbooks> .
[24] J. Liu, F. Zhao, X. Liu, W. He, Challenges towards elastic power management in [50] A. Beloglazov, R. Buyya, Energy efficient resource management in virtualized
internet data centers, in: 29th IEEE International Conference on Distributed cloud data centers, in: 2010 10th IEEE/ACM International Conference on
Computing Systems Workshops, 2009. ICDCS Workshops’09, 2009, pp. 65–72, Cluster, Cloud and Grid Computing (CCGrid), 2010, pp. 826–831. http://
http://dx.doi.org/10.1109/ICDCSW.2009.44. dx.doi.org/10.1109/CCGRID.2010.46.
[25] Revolutonizing Network Design Flattening the Data Center Network with the [51] J. Shuja, S. Madani, K. Bilal, K. Hayat, S. Khan, S. Sarwar, Energy-efficient data
QFabric Architecture. <http://www.itbiz.com.ua/media/docs/Juniper/QFX/ centers, Computing 94 (12) (2012) 973–994.
The%20Q%Fabric%20Architecture.pdf> [52] J. Zhang, N. Ansari, On assuring end-to-end qoe in next generation networks:
[26] C. Clos, A study of non-blocking switching networks, The Bell Syst. Tech. J. 32 challenges and a possible solution, Commun. Mag., IEEE 49 (7) (2011) 185–
(2) (1953) 406–424. 191, http://dx.doi.org/10.1109/MCOM.2011.5936172.
[27] <http://www.slideshare.net/junipernetworks/qfabric-enabling-the-new-data- [53] W. Voorsluys, J. Broberg, S. Venugopal, R. Buyya, Cost of virtual machine live
center> migration in clouds: a performance evaluation, in: Proceedings of the 1st
[28] Vision and Roadmap: Routing Telecom and Data Centers Toward Efficient International Conference on Cloud Computing, CloudCom’09, Springer-Verlag,
Energy Use. Vision and Roadmap Workshop on Routing Telecom and Data Berlin, Heidelberg, 2009, pp. 254–265, http://dx.doi.org/10.1007/978-3-642-
Centers, 2009. 10665-123. URL <http://dx.doi.org/10.1007/978-3-642-10665-123> .
[29] N. Farrington, G. Porter, S. Radhakrishnan, H.H. Bazzaz, V. Subramanya, Y. [54] A. Stage, T. Setzer, Network-aware migration control and scheduling of
Fainman, G. Papen, A. Vahdat, Helios: a hybrid electrical/optical switch differentiated virtual machine workloads, in: ICSE Workshop on Software
architecture for modular data centers, SIGCOMM Comput. Commun. Rev. 41 Engineering Challenges of Cloud Computing, 2009. CLOUD’09, 2009, pp. 9–14.
(4) (2010). URL <http://dl.acm.org/citation.cfm?id=2043164.1851223> . http://dx.doi.org/10.1109/CLOUD.2009.5071527.
[30] C. Kachris, I. Tomkos, A survey on optical interconnects for data centers, [55] C. Guo, G. Lu, H.J. Wang, S. Yang, C. Kong, P. Sun, W. Wu, Y. Zhang, Secondnet: a
Commun. Surv. Tutorials, IEEE 14 (4) (2012) 1021–1036, http://dx.doi.org/ data center network virtualization architecture with bandwidth guarantees,
10.1109/SURV.2011.122111.00069. in: Proceedings of the 6th International COnference, Co-NEXT’10, ACM, New
[31] G. Wang, D.G. Andersen, M. Kaminsky, K. Papagiannaki, T.E. Ng, M. Kozuch, M. York, NY, USA, 2010, pp. 15:1–15:12, http://dx.doi.org/10.1145/
Ryan, C-through: part-time optics in data centers, SIGCOMM Comput. Commun. 1921168.1921188. URL <http://doi.acm.org/10.1145/1921168.1921188> .
Rev. 41 (4) (2010). URL <http://dl.acm.org/citation.cfm?id= 2043164.1851222> . [56] X. Meng, V. Pappas, L. Zhang, Improving the scalability of data center networks
[32] J. Edmonds, R.M. Karp, Theoretical improvements in algorithmic efficiency for with traffic-aware virtual machine placement, in: INFOCOM, 2010 Proceedings
network flow problems, J. ACM 19 (2) (1972) 248–264, http://dx.doi.org/ IEEE, 2010, pp. 1–9. http://dx.doi.org/10.1109/INFCOM.2010.5461930.
10.1145/321694.321699. URL <http://doi.acm.org/10.1145/321694.321699> . [57] G. Wang, T. Ng, The impact of virtualization on network performance of
[33] K. Chen, A. Singlay, A. Singhz, K. Ramachandranz, L. Xuz, Y. Zhangz, X. Wen, Y. amazon ec2 data center, in: INFOCOM, 2010 Proceedings IEEE, 2010, pp. 1–9.
Chen, Osa: an optical switching architecture for data center networks with http://dx.doi.org/10.1109/INFCOM.2010.5461931.
unprecedented flexibility, in: Proceedings of the 9th USENIX Conference on [58] P.M.B. Heller, S. Seetharaman, Elastictree: Saving Energy in Data Center
Networked Systems Design and Implementation, NSDI’12, USENIX Association, Networks., in: NSDIĎ1d’7, Apr. 2010.
Berkeley, CA, USA, 2012, p. 18. URL <http://dl.acm.org/citation.cfm?id=2228298. [59] M.X.T.U.B.C. Yunfei Shang, Dan Li, Green routing in data center networks:
2228323> . modeling and algorithm design.
[34] C. Kachris, I. Tomkos, Power consumption evaluation of hybrid wdm pon [60] <http://en.wikipedia.org/wiki/Dynamicvoltagescaling>
networks for data centers, in: 2011 16th European Conference on Networks [61] D. Meisner, B.T. Gold, T.F. Wenisch, The powernap server architecture, ACM
and Optical Communications (NOC), 2011, pp. 118–121. Trans. Comput. Syst. 29 (1) (2011) 3:1–3:24, http://dx.doi.org/10.1145/
[35] H. Chao, K.-L. Deng, Z. Jing, Petastar: a petabit photonic packet switch, IEEE J. 1925109.1925112. URL <http://doi.acm.org/10.1145/1925109.1925112> .
Sel. Areas Commun. 21 (7) (2003) 1096–1112, http://dx.doi.org/10.1109/ [62] S. Nedevschi, L. Popa, G. Iannaccone, S. Ratnasamy, D. Wetherall, Reducing
JSAC.2003.815678. network energy consumption via sleeping and rate-adaptation, in:
[36] M.Y.K. Xia, Y.-H. Kaob, H.J. Chao, Petabit optical switch for data center Proceedings of the 5th USENIX Symposium on Networked Systems Design
networks, Tech. Rep., Polytechnic Institute of NYU, 2010. and Implementation, NSDI’08, USENIX Association, Berkeley, CA, USA, 2008,
[37] G. Papadimitriou, C. Papazoglou, A. Pomportsis, Optical switching: switch pp. 323–336. URL <http://dl.acm.org/citation.cfm?id=1387589.1387612> .
fabrics, techniques, and architectures, J. Lightwave Techn. 21 (2) (2003) 384– [63] <http://en.wikipedia.org/wiki/Powermanagement>
405, http://dx.doi.org/10.1109/JLT.2003.808766. [64] M. Gupta, S. Singh, Greening of the internet,, in: Proceedings of the 2003
[38] X. Ye, Y. Yin, S.J.B. Yoo, P. Mejia, R. Proietti, V. Akella, Dos: a scalable optical Conference on Applications, Technologies, Architectures, and Protocols for
switch for datacenters, in: Proceedings of the 6th ACM/IEEE Symposium on Computer Communications, SIGCOMM’03, ACM, New York, NY, USA, 2003, pp.
Architectures for Networking and Communications Systems, ANCS’10, ACM, 19–26, http://dx.doi.org/10.1145/863955.863959. URL <http://doi.acm.org/
New York, NY, USA, 2010, pp. 24:1–24:12, http://dx.doi.org/10.1145/1872007. 10.1145/863955.863959> .
1872037. URL <http://doi.acm.org/10.1145/1872007.1872037> . [65] D. Kliazovich, P. Bouvry, S. Khan, Dens: Data center energy-efficient network-
[39] A. Singla, A. Singh, K. Ramachandran, L. Xu, Y. Zhang, Proteus: a topology aware scheduling, in: Green Computing and Communications (GreenCom),
malleable data center network, in: Proceedings of the 9th ACM SIGCOMM 2010 IEEE/ACM Int’l Conference on Int’l Conference on Cyber, Physical and
Workshop on Hot Topics in Networks, Hotnets-IX, ACM, New York, NY, USA, Social Computing (CPSCom), 2010, pp. 69–75, 2010. http://dx.doi.org/10.1109/
2010, pp. 8:1–8:6, http://dx.doi.org/10.1145/1868447.1868455. URL <http:// GreenCom-CPSCom.31.
doi.acm.org/10.1145/1868447.1868455> .
 A. Hammadi, L. Mhamdi / Computer Communications 40 (2014) 1–21 21

[66] L. Huang, Q. Jia, X. Wang, S. Yang, B. Li, Pcube: improving power efficiency in [84] R. Bianchini, Leveraging renewable energy in data centers: present and future,
data center networks, in: 2011 IEEE International Conference on Cloud in: Proceedings of the 21st International Symposium on High-Performance
Computing (CLOUD), 2011, pp. 65–72. http://dx.doi.org/10.1109/CLOUD. Parallel and Distributed Computing, HPDC’12, ACM, New York, NY, USA, 2012,
2011.74. pp. 135–136, http://dx.doi.org/10.1145/2287076.2287101. URL <http://
[67] Y. Zhang, N. Ansari, Hero: hierarchical energy optimization for data center doi.acm.org/10.1145/2287076.2287101> .
networks, in: 2012 IEEE International Conference on Communications (ICC), [85] I.N. Goiri, K. Le, T.D. Nguyen, J. Guitart, J. Torres, R. Bianchini, Greenhadoop:
2012, pp. 2924–2928. http://dx.doi.org/10.1109/ICC.2012.6363830. leveraging green energy in data-processing frameworks, in: Proceedings of the
[68] Y. Zhang, N. Ansari, Hero: hierarchical energy optimization for data center 7th ACM European Conference on Computer Systems, EuroSys’12, ACM, New
networks, Syst. J. IEEE (99) (2013) 1–10, http://dx.doi.org/10.1109/JSYST.2013. York, NY, USA, 2012, pp. 57–70, http://dx.doi.org/10.1145/2168836.2168843.
2285606. URL <http://doi.acm.org/10.1145/2168836.2168843> .
[69] Make IT Green, Cloud Computing and its Contribution to Climate Change. [86] J. Dean, S. Ghemawat, Mapreduce: simplified data processing on large clusters,
Greenpeace, March 2010, 2010. <http://www.greenpeace.org/usa/en/media- Commun. ACM 51 (1) (2008) 107–113, http://dx.doi.org/10.1145/
center/reports/make-it-green-cloud-computing/> 1327452.1327492. URL <http://doi.acm.org/10.1145/1327452.1327492> .
[70] <http://www.eni.com/green-data-center/itIT/static/pdf/ASHRAE1.pdf> [87] K. Le, R. Bianchini, M. Martonosi, T. Nguyen, Cost-and energy-aware load
[71] R. Schmidt, E. Cruz, M. Iyengar, Challenges of data center thermal distribution across data centers, Proceedings of HotPower.
management, IBM J. Res. Dev. 49 (4.5) (2005) 709–723. [88] K. Le, R. Bianchini, T. Nguyen, O. Bilgir, M. Martonosi, Capping the brown
[72] <http://www.hpl.hp.com/news/2006/oct-dec/power.html> energy consumption of internet services at low cost, in: Green Computing
[73] R.S.M.B.C. Patel, C. Bash, R. Friedrich, Smart cooling of data centers. Conference, 2010 International, 2010, pp. 3–14. http://dx.doi.org/10.1109/
[74] <http://www.bmu.de/fileadmin/bmu-import/files/pdfs/allgemein/application/ GREENCOMP.2010.5598305.
pdf/broschuererechenzentrenenbf.pdf> [89] <http://en.wikipedia.org/wiki/Emissionstrading>
[75] J.B. John Niemann, V. Avelar, Economizer modes of data center cooling [90] Z. Liu, M. Lin, A. Wierman, S.H. Low, L.L. Andrew, Greening geographical load
systems, white paper. balancing, in: Proceedings of the ACM SIGMETRICS Joint International
[76] <http://www.apcmedia.com/salestools/JNIN-7RMNRXR0EN.pdf> Conference on Measurement and Modeling of Computer Systems,
[77] C. Patel, R. Sharma, C. Bash, A. Beitelmal, Thermal considerations in cooling SIGMETRICS’11, ACM, New York, NY, USA, 2011, pp. 233–244, http://
large scale high compute density data centers, in: Thermal and dx.doi.org/10.1145/1993744.1993767. URL <http://doi.acm.org/10.1145/
Thermomechanical Phenomena in Electronic Systems, 2002. ITHERM 2002. 1993744.1 993767>.
The Eighth Intersociety Conference on, 2002, pp. 767–776. http://dx.doi.org/ [91] C. Stewart, K. Shen, Some joules are more precious than others: Managing
10.1109/ITHERM.2002.1012532. renewable energy in the datacenter, in: Pro. of Workshop on Power Aware
[78] N. Vasic, T. Scherer, W. Schott, Thermal-aware workload scheduling for energy Computing and Systems, October 10, 2009.
efficient data centers, in: Proceedings of the 7th international conference on [92] Y. Zhang, Y. Wang, X. Wang, Greenware: greening cloud-scale data centers to
Autonomic computing, ICAC’10, ACM, New York, NY, USA, 2010, pp. 169–174, maximize the use of renewable energy, in: Proceedings of the 12th
http://dx.doi.org/10.1145/1809049.1809076. URL <http://doi.acm.org/ International Middleware Conference, Middleware’11, International
10.1145/1809049.1809076> . Federation for Information Processing, Laxenburg, Austria, Austria, 2011, pp.
[79] Q. Tang, T. Mukherjee, S.K.S. Gupta, P. Cayton, Sensor-based fast thermal 140–159. <http://dl.acm.org/citation.cfm?id=2414338.2414350>.
evaluation model for energy efficient high-performance datacenters, in: IN [93] B. Wickremasinghe, R. Calheiros, R. Buyya, Cloudanalyst: A cloudsim-based
INTŠL CONF. INTELLIGENT SENSING & INFO. PROC. (ICISIP2006), 2006. visual modeller for analysing cloud computing environments and applications,
[80] L. Ramos, R. Bianchini, C-oracle: Predictive thermal management for data in: Advanced Information Networking and Applications (AINA), 2010 24th
centers, in: IEEE 14th International Symposium on High Performance IEEE International Conference on, 2010, pp. 446–452. http://dx.doi.org/
Computer Architecture, 2008. HPCA 2008, 2008, pp. 111–122. http://dx.doi. 10.1109/AINA.2010.32.
org/10.1109/HPCA.2008.4658632. [94] S. Long, Y. Zhao, A toolkit for modeling and simulating cloud data storage: An
[81] J. Moore, J. Chase, P. Ranganathan, Weatherman: Automated, online and extension to cloudsim, in: Control Engineering and Communication
predictive thermal mapping and management for data centers, in: IEEE Technology (ICCECT), 2012 International Conference on, 2012, pp. 597–600.
International Conference on Autonomic Computing, 2006. ICAC’06. 2006, pp. http://dx.doi.org/10.1109/ICCECT.2012.160.
155–164. http://dx.doi.org/10.1109/ICAC.2006.1662394. [95] R. Buyya, R. Ranjan, R. Calheiros, Modeling and simulation of scalable cloud
[82] F. Ahmad, T.N. Vijaykumar, Joint optimization of idle and cooling power in computing environments and the cloudsim toolkit: Challenges and
data centers while maintaining response time, SIGARCH Comput. Archit. News opportunities, in: High Performance Computing Simulation, 2009. HPCS’09.
38 (1) (2010) 243–256, http://dx.doi.org/10.1145/1735970.1736048. URL International Conference on, 2009, pp. 1–11. http://dx.doi.org/10.1109/
<http://doi.acm.org/10.1145/1735970.1736048> . HPCSIM.2009.5192685.
[83] I.N. Goiri, R. Beauchea, K. Le, T.D. Nguyen, M.E. Haque, J. Guitart, J. Torres, R. [96] S.-H. Lim, B. Sharma, G. Nam, E.K. Kim, C. Das, Mdcsim: A multi-tier data center
Bianchini, Greenslot: scheduling energy consumption in green datacenters, in: simulation, platform, in: Cluster Computing and Workshops, 2009.
Proceedings of 2011 International Conference for High Performance CLUSTER’09. IEEE International Conference on, 2009, pp. 1–9. http://
Computing, Networking, Storage and Analysis, SC’11, ACM, New York, NY, dx.doi.org/10.1109/CLUSTR.2009.5289159.
USA, 2011, pp. 20:1–20:11, http://dx.doi.org/10.1145/2063384.2063411. URL [97] <http://www.mesquite.com>
<http://doi.acm.org/10.1145/2063384.2063411> . [98] <http://www.isi.edu/nsnam/ns/>