Downloaded 30 times
![Netflow-v5 Header Format
version number # of flows in packet
router uptime (ms)
export time (sec. since 1970-01-01 00:00:00 UTC)
export time (ns)
sequence number
engine type engine ID [padding]](https://image.slidesharecdn.com/networkflowanalysis-100505193034-phpapp01/75/Network-Flow-Analysis-10-2048.jpg)
![Netflow-v5 Flow Record Format
source IP address
destination IP address
IP address of next-hop router
SNMP ifIndex (in) SNMP ifIndex (out)
total number of packets
total number of octets
router uptime at start of flow (ms)
router uptime at end of flow (ms)
source port destination port
[padding] TCP flags protocol ToS
source AS destination AS
source mask dest. mask [padding]](https://image.slidesharecdn.com/networkflowanalysis-100505193034-phpapp01/75/Network-Flow-Analysis-11-2048.jpg)
Network Flow Analysis
- 1. Network Flow Analysis Mark Meiss Presentation for NaN-Group October 4, 2004
- 2. Overview Data description – TheInternet2 (Abilene) data network – Netflow traffic data Data collection Data analysis – Techniques – Preliminary results Future work
- 3. What is Abilene? Internet2(Abilene) is a nationwide high- speed data network for research and higher education. – Network backbone runs at 10 Gbps – Over 220 member institutions – Peers with over 40 other research networks Abilene uses the same protocols as Internet1 but only carries academic traffic. – This is like the old NSFnet or vBNS
- 5. Why is AbileneInteresting? The Abilene network is a transit network. – It includes both international and domestic traffic. – It offers a good view of server networks. – Commercial transmit networks do not share traffic data. The Abilene network is uncongested. – Statistics will not be biased by packet loss. The Abilene network contains students. – Students are unconcerned about niceties of law. – There is a lot of peer-to-peer and “grey” traffic.
- 6. What is “Netflow”? Inthe early 1990’s, Cisco introduced a new network router architecture. The “line cards” in their new routers contained a hardware hash table for current network connections. Somebody got the bright idea of sending entries from the table onto the network before clearing them from the hash table.
- 7. What is aNetwork Flow? A network flow consists of one or more packets sent from a source (IP, port) to a destination (IP, port) using a certain transport protocol during some time interval. Example: Source: 156.56.103.1, port 80 Dest.: 149.159.250.21, port 6132 Protocol: TCP Packets: 20 The above network flow would be typical for a Web connection.
- 8. Wait a Minute! Don’tTCP connections involve two-way communication? – Yes, so every TCP connection is actually two flows from the point of view of Netflow. UDP and ICMP are stateless, so how can they be aggregated into flows? – We assume that packets with matching 5-tuples during some period of time are part of the same flow. Isn’t it hard for a router to keep up with this? – Yes, so most modern routers sample the flow data at a ratio of about 100:1.
- 9. How is Netflowtransmitted? Most modern routers support the “Netflow v5” format for representing flows. – This includes a variety of additional information about each flow. The router uses UDP to send packets containing between 1 and 30 flow records to a management workstation. – (In this case, the management workstation is sitting on my desk.)
- 10. Netflow-v5 Header Format version number # of flows in packet router uptime (ms) export time (sec. since 1970-01-01 00:00:00 UTC) export time (ns) sequence number engine type engine ID [padding]
- 11. Netflow-v5 Flow RecordFormat source IP address destination IP address IP address of next-hop router SNMP ifIndex (in) SNMP ifIndex (out) total number of packets total number of octets router uptime at start of flow (ms) router uptime at end of flow (ms) source port destination port [padding] TCP flags protocol ToS source AS destination AS source mask dest. mask [padding]
- 12. How Much Datais There? The Abilene routers generate between 700,000,000 and 800,000,000 flows per day. – At 48 bytes per record, that amounts to around 35 GB of data. – Flows come in at a rate of about 3.4 Mbps. – Data compresses at a ratio of about 2.8:1. Most existing tools can’t handle this volume of data.
- 13. What’s the Motivation? Okay,so I’m storing egregious amounts of data and making my hard drive whimper… what for?
- 14. Flow Data asa Behavioral Network Think of a single flow as defining an edge from a source node to a destination node. The resulting network describes the Internet as it’s actually being used. – Many possible biases are eliminated. – A lot of dynamic information is included. Most structural analysis of the Internet has (necessarily) focused on its physical structure. Imagine a Google based on data about where people actually go!
- 15. Behavioral Anomaly Detection Mymain interest is in recognizing different types of behavior based on flow data. – Can I determine whether a port is running a peer-to-peer application? – Can I see the spread of a new worm across the network? – Can I determine what kind of behavior is the prelude to an attack? – Can I find new peer-to-peer applications before the word is out?
- 16. Preliminary Results I wishthis section had more, but I’m really just getting off the ground… The size of data has been a major challenge. – The network formed by a day of flow data has about 29.7 million nodes and 128 million edges. – Just finding a way of converting a set of captured flows to a sparse matrix representation has been difficult.
- 17.
- 20. Determining Clients andServers Every network connection involves two hosts: – The client is the system that initiates the connection. – The server is the system that accepts the connection. Because of sampling, we’re as likely to see the client-to-server side as the server-to-client side. – This makes the direction basically meaningless. We can guess which is which using the port information. – The more common port number indicates the server. – The less common port number indicates the client.
- 23. Strength Distribution This isthe distribution of the total number of octets in and out of each node. Special problem for client/server version of the network – If we direct all flows from server to client, what do we do when we only have a volume for the opposite direction? – For now, I treat the network as being undirected for studying strength.
- 26. AS Numbers An “autonomoussystem” is the basic building block of the Internet. – An AS is responsible for its own interior routing. – An AS is usually a large organization. For example, IU has its own AS, as does AT&T.
- 31. Top 10 ASeson Internet2 By degree By strength 1. Hotmail 1. Abilene 2. Microsoft 2. University of Oregon 3. Microsoft-Europe 3. Hotmail 4. North Carolina (NCREN) 4. Microsoft 5. Michigan (MERIT) 5. North Carolina (NCREN) 6. University of Washington 6. UCSD 7. MIT 7. UCLA 8. UC-Berkeley 8. Michigan (MERIT) 9. UMass 9. University of Washington 10. China (CERNET) 10. UMass
- 32.
- 37. Top 10 TCPPorts on Internet2 By degree By strength 1. Web 1. Web 2. Gnutella 2. iperf 3. MS Messenger 3. iperf 4. SQL Server 4. Usenet 5. Web (Encrypted) 5. RTP (Streaming) 6. Gnutella 6. iperf 7. Mail 7. SSH 8. Web Tunneling (8082) 8. BitTorrent 9. BitTorrent 9. Port 388 ?!? 10. Usenet 10. FTP
- 38. Where Do IGo Next? Start to look at the dynamics of the network. Focus on individual ports. Examine clustering coefficients. Attempt to filter out spoofed traffic. Consider the server-only and client-only networks. – This will involve treating flows as edges in a bipartite graph. Cluster nodes, ASes, and ports.
- 39. Thank You! Any thoughts,questions, comments, complaints, or observations are all welcome!