Logging : How much is too much? Network Security Monitoring Talk @ hasgeek
- 1. Hasgeek #rootconf Run Up Chennai Logging: How much is too much ? By Vivek Rajagopalan CTO, Unleash Networks http://trisul.org May-03-2014
- 2. Introduction ● Unleash Networks ● Network Analytics / Protocol Analytics ● Shrinkwrap products Unsniff Network Analyzer and Unbrowse SNMP ● Latest offering is Trisul Network Analytics http://www.unleashnetworks.com http://www.trisul.org links
- 3. Scope of talk ● Types of logging ● Focus on Network logging ● Evolution of monitoring ● Challenges with volumes, correlation & encryption ● How Trisul addresses some of the challenges ● Open Source tools – Security Onion / Bro
- 4. Logging ● Records “data” over time ● Logger originally referred to hardware sensors (record temp, humidity) ● Data ; Raw data? Samples? Snapshots ? Events ? Alerts ? ● Time is always one of the co-ordinates
- 5. Event logs vs Network Logs ● Events specific incidents that occurred a particular time instant. – Tools in space Splunk, Logstash+Kibaba,Elastic Search, Fluentd ● “Sensors” generating these events may have to monitor/sample
- 6. Network Logs ● Our focus for todays talk – Continuous stream of packets – Can every packet be an event? @ 1Mpps ? Increasingly common now ● Traffic monitoring – Sample bytes/sec -> SNMP based – Flow start/end -> Netflow based – Signature matching -> IDS alerts – IDS alerts + above -> NSM (network security mon) ● New : Reassemble and extract objects from N/W
- 7. Evolution of traffic monitoring ● Closely related to hard disk prices and new tech 1996 - $295 / GB ~ 13,000 INR 2014 - $0.04 / GB - 4 cents ~ 3 INR
- 8. Evolution of traffic monitoring ● Early days circle mid-late 90's ● Simple SNMP based counters usually in/out per interface ● Heavy use of roll ups, 5 min counters rolled up into hourly then daily to save space ● Almost no flow based logging – early days of IDS LIFE WAS SIMPLE & SLEEK Just one type of Network data was being stored and processed. Bandwidth utilization was king. Simple Time Series Traffic
- 9. Flows and alerts ● 2000's saw beginnings of IDS (Intrusion Detection) ● Martin Roesch released Snort in late 1998 ● Netflow started appearing without taxing the router CPUs (CEF) ● Traffic was still restricted to SNMP, rollups were still being used. MORE DATA TYPES IDS Alerts were being logged, as were Netflow Records. Traffic still rudimentary and Summarized. Very little integration. STARTS TO GET MESSY – BUT OKAY
- 10. More flows ; early PCAPs ● Mid 2000's – coming of age of Wireshark was a turning point. ● People instantly saw the value in deep packet analysis ● Netflow v9 added a number of extra meters ● Snort rapidly gained adoption – could also log packets like tcpdump ● Storage though cheaper needed new models other than RDMBS PACKETS ! Could store a few packets for context. Netflow based Rudimentary security , IDS – all still not in one place PUSHING IT
- 11. The scene today ● Network Security Monitoring approach ● explained in Practice of Network Security Monitoring book by Richard Bejtlich – Store “all the things” coined by ? - covers URL logs, SSL certificates, raw packet dumps, DNS logs, flows – basically anything you can extract. – Tools like Trisul do “full coverage” metering – not just bytes/in out. Every entity – hosts/ apps/ VLANs/ business assets/ traffic ASN/ monitored for hundreds of data points – Emerging sketching algorithms for fast streams
- 12. The scene today ● So – we added dozens of new types of data – what does the moped look like now ? Next page
- 13. Too much logging ?
- 14. Advantages of logging it all ● Detecting anomalous usage trends – even though bandwidth is less of a problem today ! ● Automated algorithms can be run on past data with newly acquired intel ● Security is a big focus – hacking these days are no longer about teenagers bragging (APT) ● Due to huge volumes & 10G speeds – need to extract “things” for speed (Indexing) ● Network based monitoring is a true record – no need to instrument applications for logging
- 15. Network logging techniques ● To log everything from SSL certs to DNS packets needs CPU power – To reassemble and extract log at high speeds 1G+ needs CPU and special network hardware – To write at high speeds need fask disk arrays ● Logging packets ● Why? Its a must-have to aid incident response and investigation – Hunting – is an emerging technique to use intelligence and dive into the past to discover breaches – Can optimize costs by flow cutoffs
- 16. Challenges ● Traditional storage dont scale – Need custom data stores ; – Even traditional big data stores like Hadoop may not be enough for packets ; – Typical scale 4-5B flows/day/instance ● Traditional algorithms vs Streaming algorithms – Need to process at stream rate (say 1-2M pps) – so one-pass algorithms are used – Cardinality counting (HyperLogLog) – Top-N ● Encrypted traffic on the increase but SSL messages can be logged; or private keys can be used
- 17. Need tools to find & automate ● Traffic stats, 100's of metrics, packets, flows, alerts, DNS, SSL, Cipher Types, HTTP Codes, Countries, BGP ASNs, etc etc – how to pull this together ? Messy ? – Ultimately you've got to be able to find stuff !
- 18. Deployment Basic ● Passive probes – Use SPAN ports (low speeds) or use Network Taps (high speeds) router monitor internet
- 19. Deployment in Cloud ● By the Cloud Provider ● Use an instance of the monitor for each rack at physical level ● Use same network tap technique ● Virtual Switches – Example : VSphere5 – provides Intra Host and Inter Host VM port spanning – Can offer as a service
- 20. Tools ● Open source tools are maturing ● Security Onion by Doug Burks – Linux Distro based on Ubuntu 12.04 64-bits – Integrates a lot of separate applications ; would take serious effort to install them individually – Bro – logs a ton of data from the Network – Snort / Suricata – logs alerts – SGUIL – central alert browser – Netsniff-NG – collects & stores packets – ELSA, Snorby, SqueRT reporting tools
- 21. Trisul Network Analytics ● Linux based – anyone can download and install ● Available on 1G, 10G appliances for both storing and monitoring at line rate through our partners ● At heart a deep traffic monitor around which flows, alerts, resources, and packets are integrated ● Tools to process and automate analysis ● Lua interface to add your own analysis ● Quick overview in next 5 slides – will move fast !
- 22. What this deep monitoring looks like – Not one but many alert points
- 23. Alerts integrated with Traffic ● Need real time – seconds not minutes ● Tools/viz to sift through alerts ● Cut through to other types – flows/traffic/pac kets
- 24. Tools to visualize and explore – Detect patterns – since we have all the flows
- 25. Analyze the past ● Dozens of traffic counter groups (52), hundreds of meters ● Advanced metering like TLS/SSL Cert Authorities seen
- 26. Extensibility ● Using the UI is tedious – need automation ● Need tools like Diff – compare today vs yesterday vs baseline – Need intelligence – Badfellas, Quantcast,Alexa tops ● LUA interface to count and alert on custom rules – see latest example of TLS Heartbleed (we got some press) ● Interface to write custom analytics on existing data
- 27. Cloud use cases ● Top financial brokerage cloud uses this technique ● Our customer is in the business of proving brokerage apps to their clients in the cloud ● Use Trisul to “monitor it all” down to packet level ● Runs at “rack level” then partitioned to support cloud customers ● Stores all data for 10 months ; packets cropped – achievable with 16TB disk and 24CPU IBM server
- 28. Summary ● Network logging has come a long way ● Primary direction is log everything then provide automated tools to go off and mine the data armed with intelligence ● Tune and fine tune strategy to balance costs ● There is no such thing as TOO MUCH LOGGING ! THANK YOU ! OPEN FOR QUESTIONS