Running Apache Spark on Kubernetes: Best Practices and Pitfalls
4 likes3,201 views
Since initial support was added in Apache Spark 2.3, running Spark on Kubernetes has been growing in popularity
![[Spark Summit 2017 NA] Apache Spark on Kubernetes](https://cdn.slidesharecdn.com/ss_thumbnails/apachesparkonkubernetespublic1-170929072840-thumbnail.jpg?width=560&fit=bounds)
More Related Content
What's hot (20)
Similar to Running Apache Spark on Kubernetes: Best Practices and Pitfalls (20)
More from Databricks (20)
Recently uploaded (20)
![Glary Utilities Pro 5.157.0.183 Crack + Key Download [Latest]](https://cdn.slidesharecdn.com/ss_thumbnails/artificialintelligence17-250529071922-ef6fe98e-thumbnail.jpg?width=560&fit=bounds)
Running Apache Spark on Kubernetes: Best Practices and Pitfalls
- 1. Running Spark on Kubernetes: Best Practices and Pitfalls Jean-Yves Stephan, Co-Founder & CEO @ Data Mechanics Julien Dumazert, Co-Founder & CTO @ Data Mechanics
- 2. Who We Are Jean-Yves “JY” Stephan Co-Founder & CEO @ Data Mechanics [email protected] Previously: Software Engineer and Spark Infrastructure Lead @ Databricks Julien Dumazert Co-Founder & CTO @ Data Mechanics [email protected] Previously: Lead Data Scientist @ ContentSquare Data Scientist @ BlaBlaCar
- 3. Who Are You? Poll: What is your experience with running Spark on Kubernetes? ● 61% - I’ve never used it, but I’m curious about it. ● 24% - I’ve prototyped using it, but I’m not using it in production. ● 15% - I’m using it in production. This slide was edited after the conference to show the results for the poll. You can see and take the poll at https://www.datamechanics.co/spark-summit-poll
- 4. Agenda A quick primer on Data Mechanics Spark on Kubernetes Core Concepts & Setup Configuration & Performance Tips Monitoring & Security Future Works Conclusion: Should you get started?
- 5. Data Mechanics - A serverless Spark platform ● Applications start and autoscale in seconds. ● Seamless transition from local development to running at scale. ● Tunes the infra parameters and Spark configurations automatically for each pipeline to make them fast and stable. https://www.datamechanics.co
- 6. Customer story: Impact of automated tuning on Tradelab For details, watch our SSAI 2019 Europe talk How to automate performance tuning for Apache Spark ● Stability: Automatic remediation of OutOfMemory errors and timeouts ● 2x performance boost on average (speed and cost savings)
- 7. GatewayData engineers Data scientists We’re deployed on k8s in our customers cloud account
- 8. Spark on Kubernetes: Core Concepts & Setup
- 9. Where does Kubernetes fit within Spark? Kubernetes is a new cluster-manager/scheduler for Spark. ● Standalone ● Apache Mesos ● Yarn ● Kubernetes (since version 2.3)
- 10. Spark on Kubernetes - Architecture Source
- 11. Two ways to submit Spark applications on k8s ● “Vanilla” way from Spark main open source repo ● Configs spread between Spark config (mostly) and k8s manifests ● Little pod customization support before Spark 3.0 ● App management is more manual ● Open-sourced by Google (but works on any platform) ● Configs in k8s-style YAML with sugar on top (configmaps, volumes, affinities) ● Tooling to read logs, kill, restart, schedule apps ● Requires a long-running system pod spark-on-k8s operatorSpark-submit
- 12. App management in practice spark-on-k8s operatorSpark-submit # Run an app $ spark-submit --master k8s://https://<api-server> … # List apps k get pods -label "spark-role=driver" NAME READY STATUS RESTARTS AGE my-app-driver 0/1 Completed 0 25h # Read logs k logs my-app-driver # Describe app # No way to actually describe an app and its parameters… # Run an app $ kubectl apply -f <app-manifest>.yaml # List apps $ k get sparkapplications NAME AGE my-app 2d22h # Read logs sparkctl log my-app # Describe app $ k get sparkapplications my-app -o yaml apiVersion: sparkoperator.k8s.io/v1beta2 kind: SparkApplication arguments: - gs://path/to/data.parquet mainApplicationFile: local:///opt/my-app/main.jar ... status: applicationState: state: COMPLETED ...
- 13. Dependency Management Comparison ● Lack of isolation ○ Global Spark version ○ Global Python version ○ Global dependencies ● Lack of reproducibility ○ Flaky Init scripts ○ Subtle differences in AMIs or system ● Full isolation ○ Each Spark app runs in its own docker container ● Control your environment ○ Package each app in a docker image ○ Or build a small set of docker images for major changes and specify your app code using URIs KubernetesYARN
- 14. Spark on Kubernetes: Configuration & Performance Tips
- 15. A surprise when sizing executors on k8s Assume you have a k8s cluster with 16GB-RAM 4-core instances. Do one of these and you’ll never get an executor! ● Set spark.executor.cores=4 ● Set spark.executor.memory=11g
- 16. k8s-aware executor sizing What happened? → Only a fraction of capacity is available to Spark pods, and spark.executor.cores=4 requests 4 cores! Compute available resources ● Estimate node allocatable: usually 95% ● Measure what’s taken by your daemonsets (say 10%) → 85% of cores are available Configure Spark spark.executor.cores=4 spark.kubernetes.executor.request.cores=3400m Node capacity Resources reserved for k8s and system daemons Node allocatable Resources requested by daemonsets Remaining space for Spark pods! More configuration tips here
- 17. Dynamic allocation on Kubernetes ● Full dynamic allocation is not available. When killing an exec pod, you may lose shuffle files that are expensive to recompute. There is ongoing work to enable it (JIRA: SPARK-24432). ● In the meantime, a soft dynamic allocation is available from Spark 3.0 Only executors which do not hold active shuffle files can be scaled down. spark.dynamicAllocation.enabled=true spark.dynamicAllocation.shuffleTracking.enabled=true
- 18. Cluster autoscaling & dynamic allocation k8s can be configured to autoscale if pending pods cannot be allocated. Autoscaling plays well with dynamic allocation: ● <10s to get a new exec if there is room in the cluster ● 1-2 min if the cluster needs to autoscale Requires to install the cluster autoscaler on AKS (Azure) and EKS (AWS). It is natively installed on GKE (GCP). K8s cluster Spark application Spark application Cluster autoscaling Dynamic allocation Dynamic allocation
- 19. Overprovisioning to speed up dynamic allocation To further improve the speed of dynamic allocation, overprovision the cluster with low-prio pause pods: ● The pause pods force k8s to scale up ● Spark pods preempt pause pods’ resources when needed Cluster autoscaler doc about overprovisioning. K8s cluster Spark application Cluster autoscaling Dynamic allocation Spark application Dynamic allocation Pause pod
- 20. Further cost reduction with spot instances Spot (or preemptible) instances can reduce costs up to 75%. ● If an executor is killed, Spark can recover ● If the driver is killed, game over! Node selectors and affinities can be used to constrain drivers on non-preemptible nodes. Non-preemptible node Driver Driver Preemptible node Exec Preemptible node Exec Preemptible node Exec
- 21. I/O with an object storage Usually in Spark on Kubernetes, data is read and written to an object storage. Cloud providers write optimized committers for their object storages, like the S3A Committers. If it’s not the case, use the version 2 of the Hadoop committer bundled with Spark: The performance boost may be up to 2x! (if you write many files) spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2
- 22. Improve shuffle performance with volumes I/O speed is critical in shuffle-bound workloads, because Spark uses local files as scratch space. Docker filesystem is slow → Use volumes to improve performance! ● emptyDir: use a temporary directory on the host (by default in Spark 3.0) ● hostPath: Leverage a fast disk mounted in the host (NVMe-based SSD) ● tmpfs: Use your RAM as local storage (⚠ dangerous) Performance We ran performance benchmarks to compare Kubernetes and YARN. Results will be published on our blog early July 2020. (Sneak peek: There is no performance penalty for running on k8s if you follow our recommendations)
- 23. Spark on Kubernetes: Monitoring & Security
- 24. Monitor pod resource usage with k8s tools Workload-agnostic tools to monitor pod usages: ● Kubernetes dashboard (installation on EKS) ● The GKE console Issues: ● Hard to reconcile with Spark jobs/stages/tasks ● Executors metadata are lost when the Spark app is completed GKE console Kubernetes dashboard
- 25. Spark history server Setting up a History server is relatively easy: ● Direct your Spark event log file to S3/GCS/Azure Storage Account with the spark.eventLog.dir config ● Install the Spark history server Helm chart on your cluster What’s missing: resource usage metrics!
- 26. “Spark Delight” - A Spark UI replacement Note: This slide was added after the conference. Sorry for the self-promotion. We look forward to the feedback from the community! ● We’re building a better Spark UI ○ better UX ○ new system metrics ○ automated performance recommendations ○ free of charge ○ cross-platform ● Not released yet, but we’re working on it! Learn more and leave us feedback.
- 27. Export Spark metrics to a time-series database Spark leverages the DropWizard library to produce detailed metrics. The metrics can be exported to a time-series database: ● InfluxDB (see spark-dashboard by Luca Canali) ● Prometheus ○ Spark has a built-in Prometheus servlet since version 3.0 ○ The spark-operator proposes a Docker image with a Prometheus java agent for older versions Use sparkmeasure to pipe task metrics and stage boundaries to the database Luca Canali, spark-dashboard
- 28. Security Kubernetes security best practices apply to Spark on Kubernetes for free! Access control Strong built-in RBAC system in Kubernetes Spark apps and pods benefit from it as native k8s resources Secrets management Kubernetes secrets as a first step Integrations with solutions like HashiCorp Vault Networking Mutual TLS, Network policies (since v1.18) Service mesh like Istio
- 29. Spark on Kubernetes: Future Works
- 30. Features being worked on ● Shuffle improvements: Disaggregating storage and compute ○ Use remote storage for persisting shuffle data: SPARK-25299 ○ Goal: Enable full dynamic allocation, and make Spark resilient to node loss (e.g. spot/pvm) ▪ Better Handling for node shutdown ▪ Copy shuffle and cache data during graceful decomissioning of a node: SPARK-20624 ▪ Support local python dependency upload (SPARK-27936) ▪ Job Queues and Resource Management
- 31. Spark on Kubernetes: Should You Get Started?
- 32. ● Native Containerization ● A single cloud-agnostic infrastructure for your entire tech stack with a rich ecosystem ● Efficient resource sharing guaranteeing both resource isolation and cost efficiency ● Learning curve if you’re new to Kubernetes ● A lot to setup yourself since most managed platforms do not support Kubernetes ● Marked as experimental (until 2.4) with missing features like the External Shuffle service. ConsPros We chose Kubernetes for our platform - should you? For more details, read our blog post The Pros and Cons of Running Apache Spark on Kubernetes
- 33. Checklist to get started with Spark-on-Kubernetes ● Setup the infrastructure ○ Create the Kubernetes cluster ○ Optional: Setup the spark operator ○ Create a Docker Registry ○ Host the Spark History Server ○ Setup monitoring for Spark application logs and metrics ● Configure your apps for success ○ Configure node pools and your pod sizes for optimal binpacking ○ Optimize I/O with proper libraries and volume mounts ○ Optional: Enable k8s autoscaling and Spark app dynamic allocation ○ Optional: Use spot/preemptible VMs for cost reduction ● Enjoy the Ride ! Our platform helps with this, and we’re happy to help too!
- 34. The Simplest Way To Run Spark https://www.datamechanics.co Thank you!
- 35. Appendix
- 36. Cost reduction with cluster autoscaling Configure two node pools for your k8s cluster ● Node pool of small instances for system pods (e.g. ingress controller, autoscaler, spark-operator ) ● Node pool of larger instances for Spark applications Since node pools can scale down to zero on all cloud providers, ● you have large instances at your disposal for Spark apps ● you only pay for a small instance when the cluster is idle!