AnySee: Peer-to-Peer Live Streaming

Xiaofei Liao, Hai Jin, *Yunhao Liu, *Lionel M. Ni, and Dafu Deng
School of Computer Science and Technology *Department of Computer Science
Huazhong University of Science and Technology Hong Kong University of Science and Technology
Wuhan, 430074, China Clear Water Bay, Kowloon, Hong Kong
{xfliao, hjin, dfdeng }@hust.edu.cn {liu, ni}@cs.ust.hk

Abstract — Efficient and scalable live-streaming overlay patience and unplanned interruptions would spoil the
construction has become a hot topic recently. In order to improve entertainment value.
the performance metrics, such as startup delay, source-to-end
delay, and playback continuity, most previous studies focused on In order to improve the above metrics, previous studies [9]
intra-overlay optimization. Such approaches have drawbacks focused on intra-overlay optimization, in which each node
including low resource utilization, high startup and source-to-end joins at most one overlay. With the help of locality-aware
delay, and unreasonable resource assignment in global P2P strategies [10][11] and optimization schemes such as DONet in
networks. Anysee is a peer-to-peer live streaming system and CoolStreaming [12], Narada in ESM [13], QoS of live
adopts an inter-overlay optimization scheme, in which resources streaming P2Ps have significantly improved. However, they
can join multiple overlays, so as to (1) improve global resource still suffer from long delay and unplanned interruptions,
utilization and distribute traffic to all physical links evenly; (2) especially when a large number of peers join the network
assign resources based on their locality and delay; (3) guarantee simultaneously.
streaming service quality by using the nearest peers, even when
such peers might belong to different overlays; and (4) balance the Figure 1 shows an example of intra-overlay optimization
load among the group members. We compare the performance of with two logical streaming overlays. Peers A, B, C and D join
our design with existing approaches based on comprehensive the stream originating at S1 and peers E, F, G, H and K join the
trace driven simulations. Results show that AnySee outperforms stream originating at S2. The number on each edge represents
previous schemes in resource utilization and the QoS of the cost of the link between two nodes. In traditional intra-
streaming services. AnySee has been implemented as an Internet overlay optimization schemes, two multicast trees can be
based live streaming system, and was successfully released in the established as shown in Fig. 1 (a) and (b). There are two
summer of 2004 in CERNET of China. Over 60,000 users enjoy obvious drawbacks. First, such overlay construction is not
massive entertainment programs, including TV programs, globally optimal. Considering peer D in Fig. 1(a), the cost S1
movies, and academic conferences. Statistics prove that this Æ D is 8, while if the path S1 Æ S2 Æ D is used, the cost is
design is scalable and robust, and we believe that the wide only 4. Second, resource utilization of traditional approaches is
deployment of AnySee will soon benefit many more Internet relatively low. Most of the existing protocols are tree based.
users.
Consequently, all leaf nodes fail to contribute any bandwidth or
Keywords — Peer-to-Peer; Live Streaming; Inter-Overlay
CPU cycles to the multicast trees.
Optimization; Distributed Approach; Load Balance; AnySee
(a) Overlay Topology (b)
S1 S2
I. INTRODUCTION
3 3 5 3 5 3
With the improvement of network bandwidth, multimedia
services based on streaming live media, such as IPTV [5], have B A G
C F E
gained much attention recently. Significant progress has been 5 6
made on the efficient distribution of live streams in a real-time 2
manner over a large population of spectators with good QoS D H K
[4]. Due to the practical issues of routers, IP multicast [6] has
not been widely deployed. Therefore, researchers have A 5 D 2 S2 5 E
expended a lot of effort building an efficient streaming overlay 6
multicast scheme based on P2P networks [7], in which 5 3 2 2 5 4
spectators behave as routers for other users. Efficient and 3
scalable live-streaming overlay construction [8] has become a 4 2
3 5 G 3 7 2
hot topic. Different from traditional distributed systems, 2
streaming overlays focus on the following three metrics: B S1 C H F K
startup delay, source-to-end delay, and playback continuity, as (c) Physical Topology
these metrics have a direct bearing on the interactive usability
of a live streaming system. Large delays would exhaust user Figure 1. Intra-overlay optimization: (a) optimal multicast tree rooted at S1;
(b) optimal multicast tree rooted at S2; (c) physical topology

1-4244-0222-0/06/$20.00 (c)2006 IEEE

This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the Proceedings IEEE Infocom.
 We propose an inter-overlay optimization based scheme, to build a scalable multicast tree with high efficiency. Multiple
AnySee, in which resources can join multiple overlays tree protocols, such as MDC [16], emphasize the overall
simultaneously, so as to (1) improve global resource utilization resilience and load balance of the streaming network. The main
of a P2P live streaming network and distribute traffic to all idea is to divide the video of one stream into several parts
physical links evenly; (2) assign resources based on their based on “layer concept” in CoopNet or patching ideas [20].
locality and delay; (3) guarantee streaming service quality by However, the leaving or crash behavior of nodes in the upper
using the nearest peers, even if such peers might belong to layers often causes buffer underflow. They cannot provide
different overlays; and (4) balance the load among the group backup streaming services, and waste any spare resources.
members. After AnySee optimization on the example shown in
To improve the stability of services, mesh-based protocols
Fig. 1, better overlays are constructed as illustrated in Fig. 2.
have been proposed, in which each peer can accept media data
from multiple “parents” as well as providing services for
multiple “children”, such as Coolstreaming, PROMISE [17]
and GNUStream [21]. The resource utilization of a mesh is
higher than that of a tree. Meshes based on Gossip protocol can
find fresh peers in the single mesh with low management
overhead, but not in global P2P networks. Due to the random
selection algorithm, the quality of service cannot be
guaranteed, such as the startup delay. Also, to decrease the
impact of autonomy of peers on streaming services, very large
buffer space, such as used in Coolstreaming, is necessary.
Figure 2. AnySee inter-overlay optimization Zhang proposed a DHT based P2P resource pool, SOMO
[22], [23] to manage global resources and optimize multiple
However, for a distributed approach such as AnySee, to ALM (Application Layer Multicast) sessions, especially
reach the above design goal without global network knowledge computation applications. The main idea of such approaches is
is not trivial. Several key issues, including efficient neighbor to structure all peers strictly [24], ignoring the features of
discovery, resource assignment, overlay construction and specific applications. However the huge maintenance overhead
optimization, must be addressed. makes these approaches far from scalable. Indeed, even if we
have global knowledge of a P2P network, finding an optimal
To prove the effectiveness of AnySee, comprehensive trace assignment of resources is still NP-hard. Based on a completely
driven simulations are conducted based on topologies from real
distributed heuristic, our proposed approach selects streaming
P2P networks [1]. Results show that AnySee outperforms
paths and uses key links or peers as backup providers. Inter-
previous schemes in resource utilization and the QoS of
overlay optimization is conducted in AnySee to complement
streaming services. A well-known public implementation,
traditional intra-overlay strategies.
AnySee v.1.1, was released on June 2004. It has been used to
broadcast live-streaming media, including TV programs,
movies and the Grid and Cooperative Computing (GCC’04) III. ANYSEE DESIGN
international conference in Wuhan, to tens of thousands of end-
users in CERNET (China Education and Research Network). To achieve good performance in P2P live streaming
In the past several months, over 60,000 users, from 40 systems, AnySee faces the following challenges: (1) how to
universities and 20 cities in China, have tested AnySee P2P find paths with low delays, including source-to-end delay and
streaming services. The source-to-end delay, resource startup delay, in a global P2P network; (2) how to maintain the
utilization, and the startup delay were all quite encouraging. service continuity and stability (decreasing the impact of
interruption caused by peers leaving); (3) how to determine the
The rest of this paper is organized as follows. Section II frequency of optimization operations; and (4) how to reduce
discusses the related work. Section III presents the idea of the control overhead caused by the algorithm. We introduce the
inter-overlay optimization of AnySee. Section IV describes our design of AnySee in this section.
simulation methodology and performance analysis. We
describe implementation experiences and show our A. Overview
observations and measurements of AnySee in Section V. We As illustrated in Fig. 3, the basic workflow of AnySee is as
conclude this work in Section VI. follows. First, an efficient mesh-based overlay is constructed.
A location detector based algorithm is employed to match the
overlay with the underlying physical topology [25]. Second,
II. RELATED WORKS the single overlay manager, which is based on traditional intra-
Two types of schemes based on intra-overlay optimization overlay optimization, such as Narada [13] and DONet, deals
were proposed recently: tree-based overlays and mesh-based with the join/leave operations of peers. Third, the inter-overlay
overlays. Borrowing ideas from IP multicast, tree-based optimization manager explores appropriate paths, builds
protocols are simple, efficient, and scalable. There are two backup links, and cuts off paths with low QoS for each end
types of tree-based protocols, including single tree protocols, peer. Fourth, the key node manager allocates the limited
such as ESM, NICE [19] and ZigZag [18], and multiple tree resources, and the buffer manager manages and schedules the
protocols [14][15]. The major issue of single tree protocols is transmission of media data.

This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the Proceedings IEEE Infocom.
 D ecoding/Player dm (id , S , 0 ) P1

Buffer K ey N ode P2
N1
Manager Manager S N2 P3

Single Inter-overlays
N4 P4
O verlay O ptim izations
Manager Manager dm (id , S ,1)
N3 P6

P5
M esh-based O verlay M anager

Figure 3. The system diagram of an AnySee node Figure 4. Roadmap of detector message initaited by S

of peer S. After several operations, peers would connect with

B. Mesh-based Overlay Manager their nearest neighbors.
In AnySee, peers join the mesh-based overlay first. Every
peer, with a unique identifier, first connects the C. Single Overlay Manager
bootstrapping peers and selects one or several peers to
construct logical links. Every peer maintains a group of The single overlay manager is responsible for peers
logical neighbors. The key issue here is to let the mesh-based leaving/joining operations. Before inter-overlay
overlay match with the underlying physical topology [26] . optimization, one peer joins one streaming overlay and
The mesh-based overlay manager, a key component of receives media contents from multiple providers or single
AnySee, uses some strategies, such as an LTM (Location- provider according to intra-overlay optimization schemes. In
aware Topology Matching) technique [25], to optimize the this design, a new attribute is introduced called LastDelay,
overlay, find the latest neighbors, and eliminate slow which is the minimal of all source-to-end delays from the
connections. There are two major operations: flooding-based current node to the streaming source on different paths. With
detection with limited TTL, and updating logical LastDelay, each path to the media source can be measured
connections. and evaluated. When a media block is delivered from the
media source to the node, the single overlay manager
In the first operation, each peer periodically floods a records the timestamp and writes it into the media block’s
message, defined as dm(id, S, TTL), to its neighbors. The header. When a peer receives the media block with the initial
message dm(id, S, TTL) means that the peer initiates a timestamp, it computes the difference of the initial
message with ID value id in TTL hops. Since our purpose is timestamp and the arriving timestamp. The minimal
to find the latest neighbors of peer S, we define TTL=2. To difference is the value of LastDelay. Peers can join or leave
detect the distance of peers, the message body has six parts, the topology according to LastDelay.
including messageID, TTL value, sourceIP (the IP address of
the source peer), sourceTimestamp (the timestamp1 when the D. Inter-overlay Optimization Manager
source forwards the message), DirectIP (the IP address of Generally, each peer maintains one active streaming path
one neighbor within one hop) and DirectTimestamp (the set and one backup streaming path set. Initially all streaming
timestamp when the neighbor within one hop gets the paths are managed by the single overlay manager.
message). Figure 4 shows the roadmap of one message from
S. Obviously, a message is broadcast to direct neighbors and When the number of backup streaming paths is less than
2-hop away neighbors. a threshold, the inter-overlay optimization algorithm is called
to find appropriate streaming paths in the global P2P
In the second step, logical links are updated. With the network with the help of the mesh-based overlay. When one
help of the timestamps on peers, peer P1 compares the active streaming path is cut off due to its poor QoS or peer’s
distance between two paths, S → P1 and S → N1 → P1 . If leaving, a new streaming path is selected from the backup
the former length is larger, the link N1 → P1 would be cut set.
off and the direct path between S and P1 would be
established. All peers would do the same operations as those Basically, a peer P under source S with a streaming rate
rate(S) maintains (1) an active streaming path set with
threshold size δ a ( P, S ) , and (2) a backup streaming path set
1
The clocks of all peers are synchronized based on NTP. with threshold size δ b ( P, S ) . Each streaming path SPi from
Current implementation of NTP version 4.1.1 in public S to P has two parameters: delay (SPi, S, P) is the source-to-
domain can reach the synchronization accuracy down to 7.5
milliseconds [27].

This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the Proceedings IEEE Infocom.
end delay from source S to peer P; rate(SPi, S, P) is the D
D D
D
streaming rate in the last hop of the path. Clearly, we have: 5 2 5
2 10
δa (P, S ) S1
S2 A S2 A
∑i =1
rate ( SPi , S , P ) ≥ rate ( S ) (1)
2 5
End
2 >8 3 5 2 2 3 5

δb (P , S ) δa (P , S )
S1 F G S1 B S1 G S1 B
G
∑i =1
rate ( SPi , S , P ) = p ∑i =1
rate ( SPi , S , P ) (2) End
7 2 2 6
End
>=8
>8 End
2 6
End
>=8
>8

H I C
C H C H
Let µ D ( S ) denote the threshold for the delay, which is >8 >8 2
>=8 >=8
5 2 5
related to the priority of the streams only. H S1 H S1
>=8 End >=8 End
We also design a probing message named ProbM as >8
>8
shown in Fig. 5. This message includes two major parts: (1)
initial information about the message, including sequence (a) LastDelay = 8 (b) LastDelay = 8
number, Seq., initial peer ID, Peer_0, message issuance time, j=3 j=2

Timestamp0, media source ID of the initial peer, Source, Figure 6. Examples of reverse tracing algorithm: (a) each peer forwards to
current LastDelay, and TTL; (2) an array with the size of three neighbors; (b) each peer forwards to two neighbors
TTL to record peer ID and the arriving timestamp of the
message. Considering that 95% of peers in the Gnutella A streaming path is treated as invalid if (1) the source-to-
system could be reached within 7 hops by pure flooding, the end delay is larger than a given threshold µ D ( S ) , or (2) the
maximal TTL is set to 7. direct parent of the end peer on the path leaves. In this
design, we only disconnect the overlay link between the end
There are mainly two tasks for the inter-overlay
peer to its parent node because (1) the other connections on
optimization manager, including backup streaming path set
the path can be reserved to provide support for new
management and active streaming path set management.
incoming peers, and (2) our observations show that large
The major operation in backup streaming path set delays often come from the last connection in the path, and
management is the probing procedure, called reverse tracing (3) frequent disconnections incur a lot of unnecessary traffic.
algorithm. This algorithm starts when the size of backup set
The management of an active streaming path set has
is less than δ b ( P, S ) . Peer_0 sends out a ProbM message to j three operations, including maintaining the states of active
of its neighbors with the recording array empty. Each streaming paths, cutting off invalid paths, and adding new
receiver records the message arrival time and its ID into the active paths from a backup set, which are straightforward.
accepted message body. The receiver will stop forwarding When the total bit rates from active streaming paths are
the message if (1) it finds that the delay from the initial peer lower than rate(S), the manager will check whether a better
Peer_0 to this peer is greater than LastDelay; or (2) the path should be activated to replace the current one.
receiver is the source of this streaming service. Otherwise,
This manager has the following characteristics: (1) it
the message would be forwarded to j random neighbors.
employs a heuristic algorithm, and the system is optimized
After reverse tracing, the media source is able to analyze
step by step; (2) probing procedures have originated from the
the arrived messages with ID Seq., and explore the best path
normal peers, not the source peer, so that the control
from the source to the message issuance peer. Informed by
overhead is balanced to normal peers; (3) the number of
the source, the peer is able to construct the best overlay path
forwarding neighbors, j, balance the tradeoff between the
accordingly. Figure 6 shows an example of the reverse
optimization effectiveness and the overhead; (4) the
tracing algorithm based on the overlay shown in Fig. 1, when
frequency of probing and optimization is dynamic. In
j=3 and j=2, respectively. In this figure, all delays are
AnySee, probing procedure is feedback-driven based on
replaced with the cost of two peers. Peer D sends out a
delay. Here how to set the initial value of the threshold,
message and the possible routes of the message are
illustrated. Some routes are cancelled due to a longer delay µ D ( S ) , is of importance. Logs from AnySee, to be
than LastDelay. Eventually, a good path S1ÆS2ÆD is described in Figures 19-21, show that when peers are
successfully selected. Then LastDelay is updated. As a large watching highly popular movies, they are willing to tolerate
portion of ProM messages are stopped during forwarding a higher delay as much as 30 seconds. It is reasonable that
process, the overhead is acceptable. different programs define different µ D ( S ) .
Seq. Peer_0 Timestamp0 Source LastDelay TTL E. Key Node Manager
Peer_1 Timestamp1 It is of great importance for peers to have an effective
Middle admission control policy when there are too many requests.
Peers
Suppose each peer has N spare connections. According to the
Array
characteristics of requests, each request will fall into one of
Peer_TTL Timestampn M queues with different priorities and popularities. When we
assign the N spare connections to M queues, there are two
Figure 5. Structure of message ProbM

This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the Proceedings IEEE Infocom.
interesting cases. First, some queues are assigned with more  i≠k − 
)) = Max  ∑  ρ i + ρ j  
~
than one connection tunnel, which can be modeled as an Max ( ρ ( N1 , N 2 ,...N M
M/M/m/K queuing system [28]. Second, some queues only  
1≤ i , j ≤ M   (8)
receive one connection, which follows an M/M/1/K queuing M

model.
Subject to ∑N
i =1
i = N 1 ≤ Ni < N

The admission control policy of a peer is designed to The above optimization problem (Eq. (8)) can be divided
make the resources utilization optimal. The problem can be into two parts. First, we enumerate all (M, 1)-partitions (M
described as follows. Suppose there are M queues of queues and each should be allocated at least 1 connection) of
requests. The arriving rate of queue j is λ j , all arriving rates N spare connections such that the best allocation can be
satisfy λ1 < ... < λ j < ... < λM . The service rate to assign one found to maximize ρ ( N1 , N 2 ,...N M ) in Eq. (8). Second, for
connection is µ and each connection processor can buffer k all H partitions of N connections, we can compute all H
results of average resources utilization and select the best
requests (k≥1). Assuming the probability that n requests partition, based on which of the resources utilization is
follow the M/M/m/K queuing model is pn , we have maximal. In the first phase, we can get H, the number of
partitions of N by
 ( m ρ )n


p0 n = 0,1...m − 1 ( N − 1)!
pn =  n ! (3) H = ( MN −−11 ) = ,N≥M (9)
m ρ
m n ( M − 1)!( N − M )!
 m ! p0 n = m , m + 1... K
From Equation (9), the first algorithm complexity is
λ O(N). The second algorithm is to select the maximal one
where ρ = ; we also have from H results. Its complexity is O (C MN −−11 ) = O ( N ) .
mµ
Consequently, this optimization problem has complexity of
  m −1 m ρ i −1
O(N). Considering one normal peer with 10Mbps bandwidth
∑ ( ) (m ρ ) 1 − ρ K − m +1 
m

+  ρ ≠1 and average streaming rate 300Kbps, N should be set less

  i=0 i! m! 1− ρ  (4)
 than 33.
p0 =  −1
  m − 1 ( m )i ( m )m 
∑ + ( K − m + 1 ) ρ =1 F. Buffer Manager
  i = 0 i ! m!  This manager is responsible for receiving valid media
data from multiple providers in the active streaming path set
Thus, the average utilization of N spare connections of and continuously keeping the media playback. AnySee
one peer can be given by: employs a similar heuristic as used in the Coolstreaming
system [12] to fetch expected media segments in a dynamic
−  m m ρ K p0  and heterogeneous network to meet two constraints: the
ρ = ρ (1 − ρ k ) = ρ 1 −  (5)
 m!  playback deadline for each segment and the heterogeneous
streaming bandwidth from partners. As Coolstraming does
One connection processor can buffer k requests, not employ any inter-overlay optimization, peers often fail to
then K = mk . When the probability that n requests are find the closest neighbors to supply services. To keep the
following the M/M/1/K queuing model is pn' , we have media playback continuous, a big buffer must be used. Due
to the effectiveness of the inter-overlay optimization scheme
 (1 − ρ ) ρ n adopted in AnySee, a small buffer space is enough, and
 ρ ≠1 indeed a small buffer often means a shorter startup delay.

pn' =  1 − ρ
K +1
0≤n≤ K (6)
 1 IV. SIMULATION
 K + 1 ρ =1
Before introducing our implementation experiences and
λ the observation about the real AnySee system, we evaluate
and ρ = . Then the average utilization of N spare AnySee with comprehensive simulations and contrast its
µ
performance with a recent live streaming system,
connections of one peer can be given by Coolstraming [12].
~  1− ρ K 
ρ = 1 − p0' = ρ  K +1 
(7) A. Simulation Methodology
 1− ρ  We consider two types of topologies, physical topology
and pK' is the failure probability of requests. Then, the target and logical P2P topology. The physical topology represents a
real topology with Internet characteristics. The logical
can be expressed: topology represents the overlay P2P topology built on top of
the physical topology. All P2P nodes are in a subset of nodes
in the physical topology. The communication cost between

This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the Proceedings IEEE Infocom.
two logical neighbors is calculated based on the shortest the ratio between the used connections to all connections; (2)
physical path between this pair of nodes. Continuity index, representing the playback continuity, is
defined as the number of segments that arrive before
We develop a crawler based on Gnutella protocol [1] and playback deadlines over the total number of the segments.
the source codes are rewritten from Limewire open source
client [2]. The crawler’s main function is to probe the
connections of Gnutella peers. When peers are receiving TABLE I. SIMULATION PARAMETERS
crawler ping messages, they reply with corresponding pong
messages. With the help of forty-five independent threads, Abbreviate Comment
our crawler discovers over fifty thousands peers and their S Number of streaming overlays
connections in one week. In this simulation we use three data
M Number of neighbors
sets, obtained from different time slots. Each trace includes
around 2,000 peering nodes. N Size of one overlay
r Streaming playback rate
For the physical topology, we use BRITE [3] generating C Number of total bandwidth connections
three topologies, each with 5,000 nodes. The average number
of neighbors of each node ranges from 4 to 10. B. Results
The major parameters in our simulations are listed in The first set of simulations is conducted in a stable
Table 1. In each run, peers randomly join one of S streaming environment, in which peers do not leave after joining the
overlays (S=1, 4, 8, 12). Each peer randomly has C overlays. For each simulation setup, we take 100 runs and
connections ranging from 4 (1 Mbps bandwidth) to 40 (10 report the average.
Mbps bandwidth) and maintains at least M neighbors. The
size of each overlay is N (N<500). Each stream is 1800- We first evaluate the QoS of the AnySee service in a
seconds long, and the streaming rate is r, normally 300 Kbps. stable environment. Figure 7 plots the continuity index
Based on the delay values from the trace, we set the against streaming rate, where we contrast AnySee and
bandwidths for peers. For simplicity, the threshold µ D ( S ) is Coolstreaming. When the streaming rate is increased, the
continuity of AnySee is relatively good while the continuity
set to 25 seconds, which is estimated from logs of AnySee of Coolstreaming is degraded. There are two reasons. First,
implementation. The adjustment factor p is set to 1, which AnySee can find more near neighbors from all peering nodes
means we provide one backup streaming path for each active to request services, while Coolstraming is only able to find
streaming path. suppliers from the same overlay. Second, the necessary
buffer size of AnySee is only 40 seconds, while
To better evaluate the performance of AnySee, we use
Coolstreaming needs a 120-second buffer.
the metrics as follows. (1) Resource utilization is defined as

Figure 7. Continuity index V.S. streaming rates when N=400, S=12 and Figure 8. Resources utilization: overlay size V.S. the number of streaming
initial buffer size is 40 seconds overlays when M=12, r=300 Kbps

This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the Proceedings IEEE Infocom.
 Figure 9. Continuity index under dynamic environments when M=5, Figure 10. Resource utilization under dynamic environments when M=5,
N=400, r=300 Kbps and initial buffer size is 40 seconds N=400, and r=300 Kbps

Figure 11. Services map of AnySee in CERNET of China (red center point: Figure 12. System modules of AnySee
HUST, Wuhan)

Figure 8 contrasts resource utilization of AnySee and

Coolstreaming. Seen on the left part of Figure 9, a larger V. IMPLEMENTATION OF ANYSEE
number of streaming overlays has a greater impact on the We have implemented the public free system, AnySee, and
performance of AnySee, but no obvious influence on released two versions (v.1.0 and v.1.1) to provide a scalable
Coolstreaming. This is due to the fact that Coolstreaming does live-streaming service platform based on inter-overlay
not let peers to select better relay paths using other peers in optimization in CERNET of China. From June 2004 to
different overlays. February 2005, there were over 60,000 connections to the
We then conduct simulations when peers are leaving and platform and above 40 universities and 20 cities in China were
joining freely. We define the lifetime of each peer in the in the service map as shown in Fig. 11. The system is
overlay, from 100 seconds to 500 seconds. Peer average implemented with Java and is platform-independent.
lifetime is exponentially distributed with an average of T
seconds. We can see from Figures 9 and 10 that longer A. Architecture Overview
lifetime leads to better service quality and higher resource AnySee system is comprised of four components. They are
utilization. However, when the average lifetime of peers is (1) a rendezvous point (RP), (2) a media source, (3) a monitor,
short, the continuity of Coolstreaming is relatively poor. As and (4) end systems. Each end system contains an IP to
our proposed AnySee has a backup path management design Network Coordinates Database (INCD), which is pre-built and
and the reverse tracing component keeps finding better paths integrated into the end system software.
dynamically, AnySee always outperforms Coolstreaming.

This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the Proceedings IEEE Infocom.
 Figure 12 shows the modules of end systems in AnySee. border delays. Suppose the transporting delay per hop is µ
Every end system (including the Broadcaster) is composed of and the total hops between the source peer and peer A is m, the
several function modules as follows: (1) getting media data total transporting delay of peer A is t A' = µ × m . If the path
(GMD) module is for Broadcaster; (2) sending peer selection
(SPS) module is deployed on all peers except the Broadcaster; from source peer to peer A is lS → A = {lS → a1 , la1 →a2 ,..., lam−1 → A } ,
(3) session for controlling message (SFCM) module is the path has the following properties: (1) the source peer
responsible for exchanging control messages between current would persist all the time and the path lS → a1 would not break
peer and its supplier, and monitoring actions of child peers; (4)
buffer manager (BM) module gets media packets from the down; (2) peer A would also stay in the network and the path
upper-layer, sends them to the HTTP server module, and lam−1 → A would exist; (3) the influence that multiple borders
deletes packets with outdated timestamps in the buffer; (5) break down simultaneously is the accumulation of influence
data transmitter (DT) module fetches media packets from the that multiple borders break down one by one; (4) if one peer
buffer, and transmits packets to underlying peers under flow ai leaves, one new peer ai' would join the tree and replace the
control policy; (6) HTTP server (HS) module creates a virtual
HTTP service at a local machine. After retrieving media data position of ai . Then the total link delay can be computed as
packets from the buffer, HS module sends them to media i = m −1
players such as Windows Media Player, under the HTTP
protocol.
tA = ∑t
i =1
lai →ai +1 ,
and
B. Implementation Experiences tla →a = (1 − Pb ) i −1 × tl
We discuss two interesting issues in AnySee i i +1

then, t A = tl + (1 − Pb ) × tl + ... + (1 − Pb )
m−2
implementation, GID based service scheme and locality-aware × tl ,
buffer management scheme.
after computation,
1) GID based service scheme
1 − (1 − Pb )
m −1

Due to the characteristics of streaming applications, it is

desirable to let every peer to get media services from suppliers
t A = tl ×
Pb
(
= tb × 1 − (1 − Pb )
m −1
) (11)
with low latency and high bandwidth. Many approaches have
been proposed, such as GNP. However, most of them are too Then, we have
complex to be feasible. In AnySee, all peers are in the same
CERNET and the physical network map is well known. It is TA = ε × tb × (1 − (1 − Pb ) m −1 ) + µ × m (12)
efficient that the distance is computed with the help of the
paired IP addresses. Thus, AnySee requires each peer maintain Given the estimation of the above parameters in Eq. (12),
an INCD, from which each peer can have a position, named the maximum buffer size of peer A at the m-th layer is
GID in the global network. The GID value of an end host is a computed which only relates to layer m.
128-bits integer encoded by the 4-layer geometrical
information corresponding to ISPs, cities, campuses, and C. Performance of AnySee
buildings, respectively. Such information is also used by Among all log data collected, we select records from
AnySee to estimate the physical locations of peers. 13/08/2004 to 29/08/2004. Over 7200 users from over 40
2) Locality-aware buffer management scheme universities in 14 cities of China received services with
AnySee. We analyzed the performance of the multiple
As the behavior of peers in upper layers have a larger multicast trees every ten minutes.
impact on QoS than that of peers in the lower layers, AnySee
employs a layer-aware buffer management scheme. Each peer Figure 13 plots the average height of AnySee trees against
computes its appropriate buffer space size according to the tree size. Although the height increases when more peers join
layer number. In AnySee, the buffer size of peer A at the m-th each service tree, the height is always less than 7 even with a
layer is given by: thousand peers included in one tree. Such a property helps
shorten the source-to-end delay as shown in Fig. 14. We can
TA = f ( m ) = ε × t A + t A' (10) see the source-to-end delay is always less than 200 ms. From
the logs of AnySee, the startup delay of most peers is less than
where t A denotes the total link delay, t 'A is the total 20s. We have implemented a simple prototype, which can get
transporting delay, and ε is the average disconnection times media services from a Coolstreaming network, and we observe
of one connection. the startup delays for 50 times. Mostly, the startup delay of
Suppose the probability of a link or node failure is Pb , and Coolstreaming is around 60 seconds. Based on the results
shown in Fig. 14, we set µ to 20ms and define tb = 2 ,
a peer needs tb time to explore a new parent, the border delay
Pb = 0.4 , ε = 2 .
is tl = Pb × tb and the link delay t A is the accumulation of all

This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the Proceedings IEEE Infocom.
 Figure 13. Height V.S. tree size Figure 14. Source-to-end delay V.S. tree size

Figure 15. Maximum number of peers in different Figure 16. Maximum percentage of leaving peers in Figure 17. Average delay in different hot periods
hot periods different hot periods

average delay, but also the percentage of leaving peers. That

D. Users Behavior and System Optimization too many users are leaving the overlay is the signal that the
It is important to know the user behavior, which can help overlay is under heavy burden and needs to be optimized.
us optimize the system. We select three log sets to analyze AnySee defines the parameter “optimization index”, ADL,
the total number of peers, the average delay, and the leaving which is given by
peer percentage in different “hot” periods. We select 7
different hot periods for three programs, Program-a, leaving percentage
ADL=100 ×
Program-b, and Program-c. average delay
Figures 15, 16, and 17 show the maximum number of
peers, maximum percentage of leaving peers, and average TABLE II. ADLS IN HOT PERIODS OF DIFFERENT PROGRAMS
delay of three programs, including Program-a, Program-b,
and Program-c for one hour. The results show that the Num. Program-a Program-b Program-c
overlays with popular movies attract more users to join, but 1 0.5217 0.4706 0.6363
cause larger average delays. From the figures, we have the 2 0.4583 0.5000 0.4167
following interesting observations. First, larger delay is not 3 0.7000 0.5238 0.4545
4 0.7143 0.5455 0.8462
always the major reason that causes people to leave the
5 0.5600 0.6190 0.6429
overlay. For example, the leaving percentage of Program-a is 6 0.6154 0.5217 0.6667
not the largest while its average delay is the longest. Peers 7 0.5862 0.4642 0.6875
have more patience than that imaged by previous researchers Average 0.5937 0.5213 0.6215
if the program is very popular. Second, delays from 20 to 30
seconds will not be the killer for the live streaming services.
Most people will still stay in the overlay even if there is a 30 After computation, three ADLs from different periods are
second delay from the source peer. shown in Table 2. From Table 2, the average ADLs for the
above programs are 0.5937, 0.5213, and 0.6215,
Based on the above observations, AnySee does not respectively. AnySee provides a threshold on ADL to
determine the optimization frequency only according to the determine whether an overlay optimization is necessary.

This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the Proceedings IEEE Infocom.
 VI. CONCLUSION AND FUTURE WORK Infrastructure for Real-Time Applications", in Proceedings of
IEEE INFOCOM, 2003.
Efficient and scalable live-streaming overlay construction [9] A. Myers, T.S.E. Ng, and H. Zhang, "Rethinking the Service
has become a hot topic recently. In order to improve the Model: Scaling Ethernet to a Million Nodes", in Proceedings
metrics, such as startup delay, source-to-end delay, and of ACM SIGCOMM HotNets, 2004.
playback continuity, most previous studies focused on intra- [10] Y. Liu, X. Liu, L. Xiao, L. Ni, and X. Zhang, "Location-
overlay optimization. Such approaches have drawbacks Aware Topology Matching in P2P Systems", in Proceedings
including low resource utilization, high startup and source- of IEEE INFOCOM, 2004
to-end delay, and inefficient resource assignment in global [11] T.S.E. Ng and H. Zhang, "A Network Positioning System for
P2P networks. the Internet", in Proceedings of USENIX, 2004.
[12] X. Zhang, J. Liu, B. Li, and T. P. Yum, "DONET: A Data-
In this paper, we propose an inter-overlay optimization Driven Overlay Network for Efficient Live Media Streaming",
based live streaming scheme. Instead of selecting better in Proceedings of IEEE INFOCOM, 2005.
paths in the same overlay, AnySee peers are able to construct [13] Y. Chu, S. G. Rao, and H. Zhang, "A Case for End System
efficient paths using peers in different overlays. We evaluate Multicast," in Proceedings of ACM SIGMETRICS, 2000.
the performance of AnySee by comprehensive simulations. [14] M. Castro, P. Druschel, A. Kermarrec, A. Nandi, A. Rowstron,
Our experimental results show that AnySee outperforms and A. Singh, "SplitStream: High-bandwidth Content
existing intra-overlay live streaming schemes, such as Distribution in Cooperative Environments", in Proceedings of
Coolstreaming. ACM SOSP, 2003.
[15] D. Kostic, A. Rodriguez, J. Albrecht, and A. Vahdat, "Bullet:
The practical AnySee system has been released for High Bandwidth Data Dissemination Using an Overlay Mesh",
several months and its client code is free to be downloaded in Proceedings of ACM SOSP, 2003.
in CERNET of China. To date, over 60,000 users benefit [16] V. N. Padmanabhan, H. J. Wang, P. A. Chou, and K.
from AnySee to enjoy two international academic Sripanidkulchai, "Distributing Streaming Media Content
conferences, namely GCC’04 (Grid and Cooperative Using Cooperative Networking", in Proceedings of ACM
Computing) and NPC’04 (Network and Parallel Computing), NOSSDAV, 2002.
and other massive entertainment programs. Logs from [17] M. Hefeeda, A. Habib, B. Botev, D. Xu, and B. Bhargava,
AnySee show that users have great patience for live "PROMISE: Peer-To-Peer Media Streaming Using
streaming services with large delay if they have enough Collectcast", in Proceedings of ACM Multimedia, 2003.
interest in the programs. We hope the system can serve more [18] D. Tran, K. Hua, and S. Sheu, "Zigzag: An Efficient Peer-To-
people and attain better quality in the future. Peer Scheme for Media Streaming", in Proceedings of IEEE
INFOCOM, 2003.
We are currently building peer-to-peer video-on-demand [19] S. Banerjee, B. Bhattacharjee, and C. Kommareddy, "Scalable
services for large-scale users based on inter-overlay Application Layer Multicast", in Proceedings of ACM
optimization schemes. We are going to observe more user SIGCOMM, 2002.
behaviors to further improve the system performance. [20] Y. Guo, K. Suh, J. Kurose, and D. Towsley, "P2Cast: P2P
Patching Scheme for VoD Service", in Proceedings of WWW,
2003.
ACKNOWLEDGMENT [21] X. Jiang, Y. Dong, and X. D, B. Bhargava, "GNUSTREAM:
This work was partially supported by China National A P2P Media Streaming System Prototype", in Proceedings of
IEEE ICME, 2003.
Natural Science Foundation (NSFC) under grant
[22] Z. Zhang, Y. Chen, S. Lin, B. Lu, S. Shi, X. Xie, and C. Yuan,
No.60125208, 60433040, 60573129, 60573140, Specialized "P2P Resource Pool and Its Application to Optimize Wide-
Research Fund for the Ph.D Program from Ministry of Area Application Level Multicasting", in Proceedings of
Education under grant No. 20050487040, and Hong Kong International Conference on Parallel Processing Workshops,
RGC Grants HKUST6264/04E, DAG 05/06.EG44 and 2004.
AoE/E-01/99. [23] Z. Zhang, S. Shi, and J. Zhu, "SOMO: Self-Organized
Metadata Overlay for Resource Management in P2P DHT", in
REFERENCES Proceedings of IEEE IPTPS, 2003.
[24] S. Ratnasamy, P. Francis, M. Handley, R. Karp, and S.
[1] The Gnutella protocol specification 0.6, http://rfc- Shenker, "A Scalable Content-Addressable Network", in
gnutella.sourceforge.net. Proceedings of ACM SIGCOMM, 2001.
[2] Limewire, http://www.limewire.com/. [25] Y. Liu, L. Xiao, X. Liu, L.M. Ni, and X. Zhang, "Location
[3] BRITE, http://www.cs.bu.edu/brite/. Awareness in Unstructured Peer-To-Peer Systems", IEEE
[4] J. Liu, B. Li, and Y.-Q Zhang, "Adaptive Video Multicast Transactions on Parallel and Distributed Systems, 2005.
Over the Internet", IEEE Multimedia, 2003. [26] M. Ripeanu and I. Foster, “Mapping Gnutella Network”,
[5] B. Alfonsi, "I Want My IPTV: Internet Protocol Television IEEE Internet Computing, 2002.
Predicted a Winner", IEEE Distributed Systems Online, 2005. [27] NTP: The Network Time Protocol, http://www.ntp.org/.
[6] R. Perlman, "Models for IP Multicast", in Proceedings of [28] L. Kleinrock, Queueing Systems, John Wiley, 1974.
IEEE International Conference on Networks, 2004. [29] Y. Liu, A-H. Esfahanian, L. Xiao, and L. M. Ni,
[7] A. Ganjam and H. Zhang, "Internet Multicast Video "Approaching Optimal Peer-to-Peer Overlays", in
Delivery", IEEE Proceeding, 2005. Proceedings of IEEE MASCOTS, 2005.
[8] S. Banerjee, C. Kommareddy, K. Kar, B. Bhattacharjee, and S.
Khuller, "Construction of an Efficient Overlay Multicast

This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the Proceedings IEEE Infocom.