Visualizing big data in the browser using spark

Mar 25, 201512 likes5,813 views

This document discusses using Spark to enable interactive visualization of big data in the browser. Spark can help address challenges of manipulating large datasets by caching data in memory to reduce latency, increasing parallelism, and summarizing, modeling, or sampling large datasets to reduce the number of data points. The goal is to put visualization back into the normal workflow of data analysis regardless of data size and enable sharing and collaboration through interactive and reproducible visualizations in the browser.

Visualizing big data in the browser
using Spark
Hossein Falaki @mhfalaki
Spark Summit East – March 18, 2015

Exploratory Visualization
Put visualization back in the normal workflow of data analysis
regardless of data size.
2
“Critical part of data analysis”
—William S. Cleveland
• Interactive
• Collaborative
• Reproducible

Expository Visualization
3
Communication is often the bottleneck in data science,  
and a graph is worth a thousand words.
• Control over details
• Shareable

Requirements
• Interactive
• Collaborative
• Shareable
• Reproducible
• Control over details
4
Use visualization libraries
}
Use the browser
}

Visualization as programming
• For complex tasks point and click may not be enough
• Best expressed with a grammar (API)
• Scripts are reproducible
• Control over all details
• Data scientists are already familiar with these tools
5
D3.js, Three.js, matplotlib, ggplot, Bokeh, Vincent, …

Do it in the browser
• Output of these tools can be readily used on the web  
(PNG, SVG, Canvas, WebGL)
• No need to transfer data and results
• Browser is conducive to collaboration (e.g., Notebooks)
• Separating data manipulation from rendering enables users
to freely choose the best tool for each job
6

Challenges with big data visualization
1.Manipulating large data can take a long time
2.We have more data points than pixels
7
Apache Spark can help solve both problems

Challenges
1. Manipulating large data can take a long time
8
> Memory
> CPU

Reducing latency: caching
Take advantage of memory and storage hierarchy
9
• Serialized storage levels (for memory)
• Memory & GC tuning

Reducing latency: parallelism
10
Increase number of CPUs
> Get more executors with Mesos or Yarn
> Click a button to increase cluster size in DBC
• Control level of parallelism for map and reduce tasks
• Configure spark locality if needed

Challenges
1. Manipulating large data can take a long time
2. We have more data points than possible pixels
11
> Summarize
> Model
> Sample

More data than pixels? Summarize
• Extensively used by BI tools
> Aggregation
> Pivoting
• Most data scientists’ nightly jobs
summarize data
12

More data than pixels? Model
MLLib supports a large (and growing)
set of distributed algorithms
• Clustering: k-means, GMM, LDA
• Classification and regression:  
LM, DT, NB
• Dimensionality reduction: SVD, PCA
• Collaborative filtering: ALS
• Correlation, hypothesis testing
13

More data than pixels? Sample
Extensively used in statistics
Spark offers native support for:
• Approximate and exact sampling
• Approximate and exact stratified
sampling
Approximate sampling is faster  
and is good enough in most cases
14

Summary
Using Spark we can extend interactive visualization of large data
Reduce interaction latency to seconds
> Cache data in memory
> Increase parallelism
To visualize millions of points in the browser
> Summarize
> Model
> Sample
16

Visualizing big data in the browser
using Spark

This session will cover a series of problems that are adequately solved with Apache Spark, as well as those that are require additional technologies to implement correctly. Here’s an example outline of some of the topics that will be covered in the talk: Problems that are perfectly solved with Apache Spark: 1) Analyzing a large set of data files. 2) Doing ETL of a large amount of data. 3) Applying Machine Learning & Data Science to a large dataset. 4) Connecting BI/Visualization tools to Apache Spark to analyze large datasets internally. By Vida Ha at Spark Summit East 2016.

Spark Application Carousel: Highlights of Several Applications Built with SparkDatabricks

Spark's Role in the Big Data Ecosystem (Spark Summit 2014)Databricks

This document summarizes the growth and development of the Spark project. It notes that Spark has grown significantly over the past year in terms of contributors, companies involved, and lines of code. Spark is now one of the most active projects within the Apache Hadoop ecosystem. The document outlines major new additions to Spark including Spark SQL for structured data, MLlib for machine learning algorithms, and Java 8 APIs. It discusses the vision for Spark as a unified platform and standard library for big data applications.

Strata NYC 2015 - Supercharging R with Apache SparkDatabricks

R is the favorite language of many data scientists. In addition to a language and runtime, R is a rich ecosystem of libraries for a wide range of use cases from statistical inference to data visualization. However, handling large or distributed data with R is challenging. Hence R is used along with other frameworks and languages by most data scientist. In this mode most of the friction is at the interface of R and the other systems. For example, when data is sampled by a big data platform, results need to be transferred to and imported in R as native data structures. In this talk we show an alternative, and complimentary, approach to SparkR for integrating Spark and R. Since SparkR was released in version 1.4 of Apache Spark distributed data remains inside the JVM instead of individual R processes running on workers. This approach is more convenient when dealing with external data sources such as Cassandra, Hive, and Spark’s own distributed DataFrames. We show two specific techniques to remove the data transfer friction between R and JVM: collecting Spark DataFrames as R data frames and user space filesystems. We think this model complements and improves the day-to-day workload of many data scientists who use R. Spark’s interactive query processing, especially with in-memory datasets, closely matches the R interactive session model. When integrated together Spark and R can provide state of the art tools for the entire end-to-end data science pipeline. We will show how such a pipeline works in real world use cases in a live demo at the end of the talk.

Spark what's new what's comingDatabricks

Lessons from Running Large Scale Spark WorkloadsDatabricks

Introduction to Spark (Intern Event Presentation)Databricks

Transitioning from Traditional DW to Apache® Spark™ in Operating Room Predict...Databricks

The prevailing issue when working with Operating Room (OR) scheduling within a hospital setting is that it is difficult to schedule and predict available OR block times. This leads to empty and unused operating rooms leading to longer waiting times for patients for their procedures. In this three-part session, Ayad Shammout and Denny will show: 1) How we tried to solve this problem using traditional DW techniques 2) How we took advantage of the DW capabilities in Apache Spark AND easily transition to Spark MLlib so we could more easily predict available OR block times resulting in better OR utilization and shorter wait times for patients. 3) Some of the key learnings we had when migrating from DW to Spark.

Real-Time Spark: From Interactive Queries to StreamingDatabricks

This document summarizes Michael Armbrust's presentation on real-time Spark. It discusses: 1. The goals of real-time analytics including having the freshest answers as fast as possible while keeping the answers up to date. 2. Spark 2.0 introduces unified APIs for SQL, DataFrames and Datasets to make developing real-time analytics simpler with powerful yet simple APIs. 3. Structured streaming allows running the same SQL queries on streaming data to continuously aggregate data and update outputs, unifying batch, interactive, and streaming queries into a single API.

Spark Summit San Francisco 2016 - Ali Ghodsi KeynoteDatabricks

This document discusses Apache Spark and analytics in the cloud. It summarizes that there is a gap between the growth of data and ability to perform real-time analytics. It introduces Databricks as a cloud-hosted platform that can democratize big data by providing an integrated workspace with automated Apache Spark management and production-ready applications. Databricks also provides the first end-to-end security solution for Apache Spark to address challenges in securing analytics.

Optimizing Delta/Parquet Data Lakes for Apache SparkDatabricks

Matthew Powers gave a talk on optimizing data lakes for Apache Spark. He discussed community goals like standardizing method signatures. He advocated for using Spark helper libraries like spark-daria and spark-fast-tests. Powers explained how to build better data lakes using techniques like partitioning data on relevant fields to skip data and speed up queries significantly. He also covered modern Scala libraries, incremental updates, compacting small files, and using Delta Lakes to more easily update partitioned data lakes over time.

Enabling Exploratory Analysis of Large Data with Apache Spark and RDatabricks

R has evolved to become an ideal environment for exploratory data analysis. The language is highly flexible - there is an R package for almost any algorithm and the environment comes with integrated help and visualization. SparkR brings distributed computing and the ability to handle very large data to this list. SparkR is an R package distributed within Apache Spark. It exposes Spark DataFrames, which was inspired by R data.frames, to R. With Spark DataFrames, and Spark’s in-memory computing engine, R users can interactively analyze and explore terabyte size data sets. In this webinar, Hossein will introduce SparkR and how it integrates the two worlds of Spark and R. He will demonstrate one of the most important use cases of SparkR: the exploratory analysis of very large data. Specifically, he will show how Spark’s features and capabilities, such as caching distributed data and integrated SQL execution, complement R’s great tools such as visualization and diverse packages in a real world data analysis project with big data.

Spark Summit San Francisco 2016 - Matei Zaharia Keynote: Apache Spark 2.0Databricks

The next release of Apache Spark will be 2.0, marking a big milestone for the project. In this talk, I’ll cover how the community has grown to reach this point, and some of the major features in 2.0. The largest additions are performance improvements for Datasets, DataFrames and SQL through Project Tungsten, as well as a new Structured Streaming API that provides simpler and more powerful stream processing. I’ll also discuss a bit of what’s in the works for future versions.

Practical Large Scale Experiences with Spark 2.0 Machine Learning: Spark Summ...Spark Summit

Spark 2.0 provided strong performance enhancements to the Spark core while advancing Spark ML usability to use data frames. But what happens when you run Spark 2.0 machine learning algorithms on a large cluster with a very large data set? Do you even get any benefit from using a very large data set? It depends.  How do new hardware advances affect the topology of high performance Spark clusters. In this talk we will explore Spark 2.0 Machine Learning at scale and share our findings with the community. As our test platform we will be using a new cluster design, different from typical Hadoop clusters, with more cores, more RAM and latest generation NVMe SSD’s and a 100GbE network with a goal of more performance, in a more space and energy efficient footprint.

Not your Father's Database: Not Your Father’s Database: How to Use Apache® Sp...Databricks

This session will cover a series of use cases where you can store your data cheaply in files and analyze the data with Apache Spark, as well as use cases where you want to store your data into a different data source to access with Spark DataFrames. Here’s an example outline of some of the topics that will be covered in the talk: Use cases to store in file systems for use with Apache Spark: - Analyzing a large set of data files. - Doing ETL of a large amount of data. - Applying Machine Learning & Data Science to a large dataset. - Connecting BI/Visualization tools to Apache Spark to analyze large datasets internally.

Spark Under the Hood - Meetup @ Data Science LondonDatabricks

The document summarizes a meetup on Apache Spark hosted by Data Science London. It introduces the speakers - Sameer Farooqui, Doug Bateman, and Jon Bates - and their backgrounds in data science and Spark training. The agenda includes talks on a power plant predictive modeling demo using Spark and different approaches to parallelizing machine learning algorithms in Spark like model, divide and conquer, and data parallelism. It also provides overviews of Spark's machine learning library MLlib and common algorithms. The goal is for attendees to learn about Spark's unified engine and how to apply different machine learning techniques at scale.

Jump Start into Apache® Spark™ and DatabricksDatabricks

These are the slides from the Jump Start into Apache Spark and Databricks webinar on February 10th, 2016. --- Spark is a fast, easy to use, and unified engine that allows you to solve many Data Sciences and Big Data (and many not-so-Big Data) scenarios easily. Spark comes packaged with higher-level libraries, including support for SQL queries, streaming data, machine learning, and graph processing. We will leverage Databricks to quickly and easily demonstrate, visualize, and debug our code samples; the notebooks will be available for you to download.

New directions for Apache Spark in 2015Databricks

This document discusses new directions for Apache Spark in 2015, including improved interfaces for data science, external data sources, and machine learning pipelines. It also summarizes Spark's growth in 2014 with over 500 contributors, 370,000 lines of code, and 500 production deployments. The author proposes that Spark will become a unified engine for all data sources, workloads, and environments.

New Developments in SparkDatabricks

Building a Dataset Search Engine with Spark and Elasticsearch: Spark Summit E...Spark Summit

Elasticsearch provides native integration with Apache Spark through ES-Hadoop. However, especially during development, it is at best cumbersome to have Elasticsearch running in a separate machine/instance. Leveraging Spark Cluster with Elasticsearch Inside it is possible to run an embedded instance of Elasticsearch in the driver node of a Spark Cluster. This opens up new opportunities to develop cutting-edge applications. One such application is Dataset Search. Oscar will give a demo of a Dataset Search Engine built on Spark Cluster with Elasticsearch Inside. Motivation is that once Elasticsearch is running on Spark it becomes possible and interesting to have the Elasticsearch in-memory instance join an (existing) Elasticsearch cluster. And this in turn enables indexing of Datasets that are processed as part of Data Pipelines running on Spark. Dataset Search and Data Management are R&D topics that should be of interest to Spark Summit East attendees who are looking for a way to organize their Data Lake and make it searchable.

Distributed ML in Apache SparkDatabricks

Spark streaming State of the Union - Strata San Jose 2015Databricks

Announcing Databricks Cloud (Spark Summit 2014)Databricks

Enabling exploratory data science with Spark and RDatabricks

R is a favorite language of many data scientists. In addition to a language and runtime, R is a rich ecosystem of libraries for a wide range of use cases from statistical inference to data visualization. However, handling large datasets with R is challenging, especially when data scientists use R with frameworks or tools written in other languages. In this mode most of the friction is at the interface of R and the other systems. For example, when data is sampled by a big data platform, results need to be transferred to and imported in R as native data structures. In this talk we show how SparkR solves these problems to enable a much smoother experience. In this talk we will present an overview of the SparkR architecture, including how data and control is transferred between R and JVM. This knowledge will help data scientists make better decisions when using SparkR. We will demo and explain some of the existing and supported use cases with real large datasets inside a notebook environment. The demonstration will emphasize how Spark clusters, R and interactive notebook environments, such as Jupyter or Databricks, facilitate exploratory analysis of large data.

Spark streaming state of the unionDatabricks

In this talk at 2015 Spark Summit East, the lead developer of Spark streaming, @tathadas, talks about the state of Spark streaming: Spark Streaming extends the core Apache Spark API to perform large-scale stream processing, which is revolutionizing the way Big “Streaming” Data application are being written. It is rapidly adopted by companies spread across various business verticals – ad and social network monitoring, real-time analysis of machine data, fraud and anomaly detections, etc. These companies are mainly adopting Spark Streaming because – Its simple, declarative batch-like API makes large-scale stream processing accessible to non-scientists. – Its unified API and a single processing engine (i.e. Spark core engine) allows a single cluster and a single set of operational processes to cover the full spectrum of uses cases – batch, interactive and stream processing. – Its stronger, exactly-once semantics makes it easier to express and debug complex business logic. In this talk, I am going to elaborate on such adoption stories, highlighting interesting use cases of Spark Streaming in the wild. In addition, this presentation will also showcase the exciting new developments in Spark Streaming and the potential future roadmap.

HUG France Feb 2016 - Migration de données structurées entre Hadoop et RDBMS ...Modern Data Stack France

Migration de données structurées entre Hadoop et RDBMS par Louis Rabiet (Squid Solution) Avec l'extraction de données stockées dans une base de données relationnelle à l'aide d'un outil de BI avancé, et avec l'envoi via Kafka des données vers Tachyon, plusieurs sessions Spark peuvent travailler sur le même dataset en limitant la duplication. On obtient grâce à cela une communication à coût contrôlé entre la base de données d'origine et Spark ce qui permet de réintroduire de manière dynamique les données modifiées avec MLlib tout en travaillant sur des données à jour. Les résultats préliminaires seront partagés durant cette présentation.

Spark Summit EU 2015: Revolutionizing Big Data in the Enterprise with SparkDatabricks

Spark provides a unified platform for processing big data from multiple sources and in different formats. It allows for just-in-time processing of data without needing to wait for ETL into a data warehouse. This provides lower latency and makes it easy to combine data. Spark also unifies batch, streaming, and machine learning functionality into a single engine. This was demonstrated on a large online service company that leverages Spark for interactive queries, machine learning, and combining data from various sources for analytics and predictive services.

What's new in pandas and the SciPy stack for financial usersWes McKinney

Wes McKinney discusses updates and planned improvements to Python packages for financial analysis, including pandas, NumPy, IPython, Cython, matplotlib, and statsmodels. Major changes include a redesign of pandas' DataFrame internals, hierarchical indexing, time series functionality in statsmodels, and performance optimizations. McKinney aims to make pandas the foundation for rich statistical computing and leverage the best of other languages in Python.

Big Data visualization with Apache Spark and Zeppelinprajods

Sparkly Notebook: Interactive Analysis and Visualization with Sparkfelixcss

More Related Content

What's hot (20)