Take My Logs. Please!
26 likes13,519 views
The document discusses logging practices at Etsy, highlighting the importance of performance, operability, and simplicity in logging. It covers various log formats, error logging, and methods for analyzing and visualizing log data, including custom fields for segmentation and aggregation. The document emphasizes the value of extensive logging and data visualization for operational insights.
![$GLOBALS['timer'] = microtime(true) * 1000000;](https://image.slidesharecdn.com/velocityolc2011-takemylogsplease-111026122347-phpapp01/85/Take-My-Logs-Please-21-320.jpg)
![$GLOBALS['timer'] = microtime(true) * 1000000;
register_shutdown_function('pageStats');
function pageStats() {
}](https://image.slidesharecdn.com/velocityolc2011-takemylogsplease-111026122347-phpapp01/85/Take-My-Logs-Please-22-320.jpg)
![$GLOBALS['timer'] = microtime(true) * 1000000;
register_shutdown_function('pageStats');
function pageStats() {
$timer_end = microtime(true) * 1000000;
$diff = $timer_end - $GLOBALS['timer'];
}](https://image.slidesharecdn.com/velocityolc2011-takemylogsplease-111026122347-phpapp01/85/Take-My-Logs-Please-23-320.jpg)
![$GLOBALS['timer'] = microtime(true) * 1000000;
register_shutdown_function('pageStats');
function pageStats() {
$timer_end = microtime(true) * 1000000;
$diff = $timer_end - $GLOBALS['timer'];
apache_note('php_microsec', $diff);
apache_note('php_bytes',
memory_get_peak_usage());
}](https://image.slidesharecdn.com/velocityolc2011-takemylogsplease-111026122347-phpapp01/85/Take-My-Logs-Please-24-320.jpg)
![Quick graphs
grep "GET /listing/" access.log |
perl -pe "s/.*[.*d{4}:(d{2}):(d{2}):d{2}.*]/1:2/" |
awk '{print $1, $(NF-1)}' > /tmp/pagetimes.dat
gives you...](https://image.slidesharecdn.com/velocityolc2011-takemylogsplease-111026122347-phpapp01/85/Take-My-Logs-Please-34-320.jpg)
![Quick graphs
# GNUPLOT
set terminal png
set output 'listings.png'
set yrange [0:2000]
set xdata time
set timefmt "%d/%B/%Y:%H:%M:%S"
set format x "%H:%M"
plot '/tmp/pagetimes.dat' using 1:2 with points](https://image.slidesharecdn.com/velocityolc2011-takemylogsplease-111026122347-phpapp01/85/Take-My-Logs-Please-36-320.jpg)
![web0054 [Fri Mar 04 16:27:48 2011] [error]
[sql_long_query] [mk04gw1p71] Query exceeded
5 sec: SELECT * FROM ...](https://image.slidesharecdn.com/velocityolc2011-takemylogsplease-111026122347-phpapp01/85/Take-My-Logs-Please-41-320.jpg)
![web0054 [Fri Mar 04 16:27:48 2011] [error]
[sql_long_query] [mk04gw1p71] Query exceeded
5 sec: SELECT * FROM ...](https://image.slidesharecdn.com/velocityolc2011-takemylogsplease-111026122347-phpapp01/85/Take-My-Logs-Please-42-320.jpg)
![web0054 [Fri Mar 04 16:27:48 2011] [error]
[sql_long_query] [mk04gw1p71] Query exceeded
5 sec: SELECT * FROM ...](https://image.slidesharecdn.com/velocityolc2011-takemylogsplease-111026122347-phpapp01/85/Take-My-Logs-Please-44-320.jpg)
![web0001 [04:28:54 2011] [error] [client 10.101.x.x] Help me, Rhonda.
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Heeeeeeellllllllllllllppp
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!
web0001 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!
web0201 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!
web0034 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web1101 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0201 [04:28:54 2011] [error] [client 10.101.x.x] You've been eaten by a gr
web0055 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!!
web0002 [04:28:54 2011] [warning] [client 10.101.x.x] Sky is falling.
web0089 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling.
web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!
web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!
web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0034 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0087 [04:28:54 2011] [fatal] [client 10.101.x.x] Sky is falling.
web0002 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!
web0201 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!
web0077 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!
web0355 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo
web0052 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0003 [04:28:54 2011] [error] [client 10.101.x.x] You've been eaten by a gr
web0066 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!!
web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Sky is falling
web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling.
web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!
web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!
web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!](https://image.slidesharecdn.com/velocityolc2011-takemylogsplease-111026122347-phpapp01/85/Take-My-Logs-Please-47-320.jpg)
![web0054 [Fri Mar 04 16:27:48 2011] [error]
[login] [mk04gw1p71] User login failed.
Reason: wrong password for ...](https://image.slidesharecdn.com/velocityolc2011-takemylogsplease-111026122347-phpapp01/85/Take-My-Logs-Please-50-320.jpg)
![^.+ [.+] [(?P<log_level>.+)]](https://image.slidesharecdn.com/velocityolc2011-takemylogsplease-111026122347-phpapp01/85/Take-My-Logs-Please-51-320.jpg)
![if (fields['log_level'] == “fatal”):
self.fatals += 1
elif (fields['log_level'] == “error”):
self.errors += 1
elif (fields['log_level'] == “warning”):
self.warnings += 1
...](https://image.slidesharecdn.com/velocityolc2011-takemylogsplease-111026122347-phpapp01/85/Take-My-Logs-Please-52-320.jpg)
Take My Logs. Please!
- 1. Take my logs. Please. Mike Brittain Director of Engineering, Infrastructure Etsy.com [email protected] @mikebrittain
- 2. (hello?)
- 3. This sounds boooooorrrrring... No, no... hang in there!
- 5. 25 MM uniques/month 150 Countries $300 MM+ sales last year
- 6. Apache, PHP, MySQL, PostgreSQL, Memcache, Gearman, Solr, etc.
- 12. App logging (Apache access and error logs)
- 13. “Common” LogFormat "%h %l %u %t "%r" %>s %b
- 14. “Combined” LogFormat "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-agent}i""
- 15. mod_log_config %f Filename requested # of keepalive requests served %k on this connection Time taken to serve the request, %T in seconds
- 16. mod_log_config %f Filename requested # of keepalive requests served %k on this connection Time taken to serve the request, %D in microseconds
- 17. mod_log_config %f Filename requested # of keepalive requests served %k on this connection Time taken to serve the request, %D in microseconds Contents of “note” foobar from %{foobar}n another module
- 19. “Steroids” LogFormat %{True-Client-IP}i %l %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %V %{user_id}n %{shop_id}n %{uaid}n %{ab_selections}n %{request_uid}n %{api_consumer_key}n %{api_method_name}n %{php_bytes}n %{php_microsec}n %D
- 20. “Steroids” LogFormat %{True-Client-IP}i %l %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %V %{user_id}n %{shop_id}n %{uaid}n %{ab_selections}n %{request_uid}n %{api_consumer_key}n %{api_method_name}n %{php_bytes}n %{php_microsec}n %D
- 21. $GLOBALS['timer'] = microtime(true) * 1000000;
- 22. $GLOBALS['timer'] = microtime(true) * 1000000; register_shutdown_function('pageStats'); function pageStats() { }
- 23. $GLOBALS['timer'] = microtime(true) * 1000000; register_shutdown_function('pageStats'); function pageStats() { $timer_end = microtime(true) * 1000000; $diff = $timer_end - $GLOBALS['timer']; }
- 24. $GLOBALS['timer'] = microtime(true) * 1000000; register_shutdown_function('pageStats'); function pageStats() { $timer_end = microtime(true) * 1000000; $diff = $timer_end - $GLOBALS['timer']; apache_note('php_microsec', $diff); apache_note('php_bytes', memory_get_peak_usage()); }
- 25. What about “%D”?
- 26. “Steroids” LogFormat %{True-Client-IP}i %l %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %V %{user_id}n %{shop_id}n %{uaid}n %{ab_selections}n %{request_uid}n %{api_consumer_key}n %{api_method_name}n %{php_bytes}n %{php_microsec}n %D
- 27. “Steroids” LogFormat %{True-Client-IP}i %l %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %V %{user_id}n %{shop_id}n %{uaid}n %{ab_selections}n %{request_uid}n %{api_consumer_key}n %{api_method_name}n %{php_bytes}n %{php_microsec}n %D
- 28. “Steroids” LogFormat %{True-Client-IP}i %l %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %V %{user_id}n %{shop_id}n %{uaid}n %{ab_selections}n %{request_uid}n %{api_consumer_key}n %{api_method_name}n %{php_bytes}n %{php_microsec}n %D
- 29. “Steroids” LogFormat %{True-Client-IP}i %l %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %V %{user_id}n %{shop_id}n %{uaid}n %{ab_selections}n ... easy_reg=1; personalize_widget=0; icon_in_cornflower_blue=1;
- 30. Coming soon... %{locale}n (i18n) %{platform}n (desktop vs. mobile)
- 31. Coming soon... %{locale}n (i18n) %{platform}n (desktop vs. mobile) OPS-1805, OPS-1827 etsy.com/careers
- 32. Using something else? time, http method, request uri, response code, referer, user-agent, response time, response memory, custom segmentation fields...
- 33. Quick averages grep "GET /listing/" access.log | awk '{sum=sum+$(NF-1)} END {print sum/NR}'
- 34. Quick graphs grep "GET /listing/" access.log | perl -pe "s/.*[.*d{4}:(d{2}):(d{2}):d{2}.*]/1:2/" | awk '{print $1, $(NF-1)}' > /tmp/pagetimes.dat gives you...
- 35. Quick graphs # /tmp/pagetimes.dat 18:37 251.0 18:38 252.1 18:39 253.5 18:40 251.0 18:45 250.0 and then...
- 36. Quick graphs # GNUPLOT set terminal png set output 'listings.png' set yrange [0:2000] set xdata time set timefmt "%d/%B/%Y:%H:%M:%S" set format x "%H:%M" plot '/tmp/pagetimes.dat' using 1:2 with points
- 37. Quick graphs
- 38. Error logs PHP + Apache errors in one file Simple logging interface
- 39. Error logs Levels: error, info, debug Namespace: perf, sql, __class__
- 40. Logger::error("Query exceeded 5 sec: $query", “sql_long_query”);
- 41. web0054 [Fri Mar 04 16:27:48 2011] [error] [sql_long_query] [mk04gw1p71] Query exceeded 5 sec: SELECT * FROM ...
- 42. web0054 [Fri Mar 04 16:27:48 2011] [error] [sql_long_query] [mk04gw1p71] Query exceeded 5 sec: SELECT * FROM ...
- 43. $ grep "16:27:48" access.log | wc -l 1527
- 44. web0054 [Fri Mar 04 16:27:48 2011] [error] [sql_long_query] [mk04gw1p71] Query exceeded 5 sec: SELECT * FROM ...
- 45. iow error.log -> request_uid -> access.log request uri, ab selections, user id, locale, platform, api key, etc.
- 46. Filtering tail -f error.log | grep -v “sql_long_query” | ...
- 47. web0001 [04:28:54 2011] [error] [client 10.101.x.x] Help me, Rhonda. web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo! web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh! web0001 [04:28:54 2011] [error] [client 10.101.x.x] Heeeeeeellllllllllllllppp web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo! web0001 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh! web0201 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh! web0034 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web1101 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web0201 [04:28:54 2011] [error] [client 10.101.x.x] You've been eaten by a gr web0055 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!! web0002 [04:28:54 2011] [warning] [client 10.101.x.x] Sky is falling. web0089 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling. web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh! web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh! web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web0034 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web0087 [04:28:54 2011] [fatal] [client 10.101.x.x] Sky is falling. web0002 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo! web0201 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh! web0077 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh! web0355 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo web0052 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web0003 [04:28:54 2011] [error] [client 10.101.x.x] You've been eaten by a gr web0066 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!! web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Sky is falling web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling. web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh! web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh! web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
- 48. Trending fatals errors warnings
- 49. Logster Run by cron Maintains a cursor on log files Simple parsing & aggregation Output to Ganglia or Graphite github.com/etsy
- 50. web0054 [Fri Mar 04 16:27:48 2011] [error] [login] [mk04gw1p71] User login failed. Reason: wrong password for ...
- 52. if (fields['log_level'] == “fatal”): self.fatals += 1 elif (fields['log_level'] == “error”): self.errors += 1 elif (fields['log_level'] == “warning”): self.warnings += 1 ...
- 53. MetricObject("fatals", (self.fatals / self.duration), "per sec") MetricObject("errors", (self.errors / self.duration), "per sec") MetricObject("warning", (self.warnings / self.duration), "per sec")
- 54. fatals errors warnings
- 55. Logster Signed-in vs. Signed-out
- 56. github.com/etsy
- 57. Log a plethora of data. Don’t be afraid to use one file.
- 58. Use custom fields to segment data.
- 59. Correlate errors to specific requests.
- 61. Convert rates to trend lines.
- 62. Take my logs. Please!
- 63. Thank you. codeascraft.etsy.com github.com/etsy Mike Brittain Director of Engineering, Infrastructure Etsy.com [email protected] @mikebrittain