Apache Spark Performance is too hard. Let's make it easier
11 likes•2,994 views
The document discusses Spark performance challenges and solutions aimed at simplifying debugging and enhancing application performance. It introduces the Pepperdata Application Profiler and highlights key features, including code execution plans, time series views, and cluster weather monitoring to track resource consumption across multiple applications. The presentation also mentions available tools and heuristics to identify known issues while proposing an integrated approach for assessing application performance comprehensively.
Apache Spark Performance is too hard. Let's make it easier
- 1. Sean Suchter CTO @ Pepperdata Spark performance is too hard, let’s make it easier
- 2. Pepperdata does performance (for Big Data) 15 Thousand Production Nodes 50 Million Jobs/Year 200 Trillion Performance Data Points
- 3. Today’s talk will cover… • How code translates to execution • How to find common, known problems • For the rest of the problems… – Why debugging performance problems is hard – Data elements needed for complete view of application performance from separate tools – Bringing these elements together in a single tool
- 4. Brief terminology about Spark • An app contains multiple jobs • A job contains multiple stages • A stage contains multiple tasks • Executors run tasks
- 5. Example App A word count app: val textFile = sc.textFile("hdfs:/dict.txt") val counts = textFile.flatMap(line => line.split(" ")) .map(word => (word, 1)) .reduceByKey(_ + _) counts.saveAsTextFile("hdfs:/wordcounts.txt") 1. Declares input from external storage 2. Specifies transformations 3. Triggers an action
- 6. Distributed Architecture Spark executes a job using multiple machines. Spark Driver process Spark Executor 1 process Spark Executor 2 process Spark Executor N process Sends tasks
- 7. Stages Image source. val textFile = sc.textFile("hdfs:/dict.txt") val counts = textFile.flatMap(line => line.split(" ")) .map(word => (word, 1)) .reduceByKey(_ + _) counts.saveAsTextFile("hdfs:/wordcounts.txt")
- 8. Shuffle and Re-partitioning Image source.
- 9. Stages and Tasks in Example Job Task 0 Task 1 Task n Task n+m Task n+1 Task n+2
- 10. Debugging known problems The easier case…
- 13. Intro: Dr Elephant (MapReduce)
- 14. What does Dr. Elephant do? • Performance monitoring and tuning service • Finds common mistakes, indicates best practices 14
- 17. 3 Classes of Spark Heuristics • Configuration Settings • Simple Alarms on Stage/Job Failure • Data-Dependent Tuning Suggestions 17
- 18. Configuration Heuristic • Display some basic config settings for your app • Complain if some settings not explicitly set • Recommend configuring an external shuffle service (especially if dynamic allocation is enabled) • These recommendations won’t change over multiple runs of an application 18
- 19. Stages and Jobs Heuristics • Simple alarms showing stage and job failure rates • Good for seeing when there’s a problem 19
- 20. Executors Heuristic • Looks at the distribution across executors of several different metrics • Outliers in these distributions probably indicate: – Suboptimal partitioning. – One or more slow executors due to external circumstances (cluster weather) 20
- 21. Partitions Heuristic • Ideally data for each task will fit into the RAM available to that task. • Sandy Ryza (once from Cloudera) has an excellent blog on Spark tuning: (observed shuffle write) * (observed shuffle spill memory) * (spark.executor.cores) (observed shuffle spill disk) * (spark.executor.memory) * (spark.shuffle.memoryFraction) * (spark.shuffle.safetyFraction) http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/ 21
- 22. More Heuristics? Yes, please! Dr. Elephant is open source. https://github.com/linkedin/dr-elephant 22
- 23. Is there an enterprise version?
- 24. Pepperdata Application Profiler • Benefits to our users: – Provide simple answers to simple questions – Combination of metrics for experts – Simple actionable insights for all users – Pepperdata support • Why stay close to open source? – Heuristics 24
- 26. Debugging novel problems The harder case…
- 27. 2 reasons this is hard
- 28. Reason #1 Same external symptom (“too slow”), but many possible causes: • code • data • configuration • cluster weather
- 29. Reason #2 Existing tools provide limited visibility • Spark Web UI is the most popular – Good view of query execution plan (job/stages/DAG) – Limited view of aggregate performance data • Time series – Ganglia, Ambari, CM, etc provide time series data for cluster (but not specific to Spark apps) – Spark Sink metrics can be fed to InfluxDb/others, yielding partial Spark app metrics • Code execution not connected to resource consumption • Load from other apps unaccounted
- 30. 3 data elements form a complete picture of Spark application performance 1. Code execution plan – Indicates which block of code is being executed, where 2. Time series view – Visual of resource consumption of application – Outliers in resource usage very easy to detect 3. Cluster weather – A view of all applications that run on the cluster
- 31. Spark Web UI First half of solution
- 32. Logical code execution plan from Spark: Jobs / Stages / DAG
- 33. Physical execution plan from Spark: Executors / Tasks
- 34. Time series view Second half of solution
- 35. Time series view of resource consumption for the App
- 36. Bring them together Best of both worlds
- 37. Code Analyzer = execution plan + time series
- 38. GC across all Stages of App
- 39. Let’s examine GC activity in Stage 4
- 40. Executor skew increased Stage duration 2x
- 41. Executor 6 does twice as much work: possible solution increase number of partitions
- 42. What if it’s not your fault? Cluster weather
- 43. How does cluster weather impact your app ?
- 44. No apparent reason for delay from Spark Web UI
- 45. Time series shows slower run of app with much lower resources
- 46. View cluster weather for slower run of app
- 47. Cluster weather reveals reason for CPU constraints on slower app
- 48. Cluster weather reveals reason for memory constraints on slower app
- 49. Cluster weather reveals reason for HDFS constraints on slower app
- 50. Code Analyzer for Apache Spark • Free during Early Access starting today • Early Access is for development teams • To learn more visit booth #101 • [email protected] pepperdata.com/products/code-analyzer
- 51. Other performance tools mentioned • Dr Elephant – github.com/linkedin/dr-elephant • Application Profiler – www.pepperdata.com/products/application-profiler/
- 52. To recap • Use heuristics to find known problems • Execution plan + time series = powerful visualization • Knowing cluster weather can prevent time wasted debugging performance “issues” that aren’t the app’s fault
- 53. Spark Summit Talk Plugs Tuesday 11:40AM Connect Code to Resource Consumption to Scale Your Production Spark Applications (Vinod @ Pepperdata) Tuesday 12:50PM Kubernetes SIG Big Data Birds-of-a-Feather session (many) Tuesday 3:20PM Apache Spark on Kubernetes (Anirudh @ Google, Tim @ Hyperpilot) Wednesday 11:00AM HDFS on Kubernetes – Lessons Learned (Kimoon @ Pepperdata) Wednesday 11:00AM Dr Elephant for Monitoring and Tuning Apache Spark Jobs on Hadoop (Carl @ LinkedIn, Simon @ Pepperdata)