Azure Databricks - An Introduction 2019 Roadshow.pptx

Editor's Notes

• #2: When it comes to ease of use, Spark again happens to be a lot better than Hadoop. Spark has APIs for several languages such as Scala, Java and Python, besides having the likes of Spark SQL. It is relatively simple to write user-defined functions. It also happens to boast an interactive mode for running commands. Hadoop, on the other hand, is written in Java and has earned the reputation of being pretty difficult to program, although it does have tools that assist in the process. (To learn more about Spark, see How Apache Spark Helps Rapid Application Development.) In-Memory Technology One of the unique aspects of Apache Spark is its unique "in-memory" technology that allows it to be an extremely good data processing system. In this technology, Spark loads all of the data to the internal memory of the system and then unloads it on the disk later. This way, a user can save a part of the processed data on the internal memory and leave the remaining on the disk. Spark also has an innate ability to load necessary information to its core with the help of its machine learning algorithms. This allows it to be extremely fast. Spark’s Core Spark’s core manages several important functions like setting tasks and interactions as well as producing input/output operations. It can be said to be an RDD, or resilient distributed dataset. Basically, this happens to be a mix of data that is spread across several machines connected via a network. The transformation of this data is created by a four-step method, comprised of mapping the data, sorting it, reducing it and then finally, joining the data. Following this step is the release of the RDD, which is done with support from an API. This API is a union of three languages: Scala, Java and Python. Spark’s SQL Apache Spark’s SQL has a relatively new data management solution called SchemaRDD. This allows the arrangement of data into many levels and can also query data via a specific language. Graphx Service Apache Spark comes with the ability to process graphs or even information that is graphical in nature, thus enabling the easy analysis with a lot of precision. Streaming This is a prime part of Spark that allows it to stream large chunks of data with help from the core. It does so by breaking the large data into smaller packets and then transforming them, thereby accelerating the creation of the RDD. MLib – Machine Learning Library Apache Spark has the MLib, which is a framework meant for structured machine learning. It is also predominantly faster in implementation than Hadoop. MLib is also capable of solving several problems, such as statistical reading, data sampling and premise testing, to name a few.
• #4: Azure Databricks features – Enhance your teams’ productivity Get started quickly by launching your new Spark environment with one click. Share your insights in powerful ways through rich integration with PowerBI. Improve collaboration amongst your analytics team through a unified workspace. Innovate faster with native integration with rest of Azure platform. Build on the most compliant and trusted cloud Simplify security and identity control with built-in integration with Active Directory. Regulate access with fine-grained user permissions to Azure Databricks’ notebooks, clusters, jobs and data. Build with confidence on the trusted cloud backed by unmatched support, compliance and SLAs. Scale without limits Operate at massive scale without limits globally. Accelerate data processing with the fastest Spark engine.
• #5: Azure Databricks features – Enhance your teams’ productivity Get started quickly by launching your new Spark environment with one click. Share your insights in powerful ways through rich integration with PowerBI. Improve collaboration amongst your analytics team through a unified workspace. Innovate faster with native integration with rest of Azure platform. Build on the most compliant and trusted cloud Simplify security and identity control with built-in integration with Active Directory. Regulate access with fine-grained user permissions to Azure Databricks’ notebooks, clusters, jobs and data. Build with confidence on the trusted cloud backed by unmatched support, compliance and SLAs. Scale without limits Operate at massive scale without limits globally. Accelerate data processing with the fastest Spark engine.
• #7: Databricks, founded by the team that created Apache Spark – unified analytics platform that accelerates innovation by unifying data science, engineering & business. 75% of the code committed to Apache Spark comes from Databricks Unified Runtime Create clusters in seconds, dynamically scale them up and down. They’ve made enhancements to Spark engine to make it 10x faster than open source Spark Serverless- Auto-configured multi-user cluster, Reliable sharing with fault isolation Unified Collaboration Overall – a simple & collaborative environment that enables your entire team to use Spark & interact with your data simultaneously DE – Improve ETL performance, zero management clusters. Execute production code from within notebooks DS - For data scientists, easy data exploration in notebooks Business SME – interactive dashboards empower teams to create dynamic reports Enterprise Security Encryption Fine grained Role-based access control (files, clusters, code, application, dashboard) Compliance Rest APIs DE – DBIO, SPARK, API’s , JOBS DS – Spark and Serverless, Interactive Data Science Data Products - Everything Creators of Spark Training People Number of Customers Ingest Workflow Schedule / Run / Monitor Execute Troubleshoot Debug Production Jobs --------- Ingest, ETL, Scheduling, Monitoring
• #8: In modern data warehousing, data may be collected from various sources including flat files that are uploaded to storage like Data Lake Store or Azure Storage, or the source data may come from applications that are writing to one more transaction databases. These “ingest stores” form the source for the data that is processed and ultimately served to applications and dashboards either in a data warehouse or by an analytic store that gets its data from the data warehouse. It is worth observing that the modern data warehouse can serve two functions. It can participate in the data processing in addition to being the data store serving analytic clients.
• #10: Note that there are multiple options for all functional categories. In this deck the focus is on: Stream Ingestion: The options are Azure Event Hubs, Azure IoT Hub and Apache Kafka on HDInsight Stream Analytics i.e. querying, filtering and transforming streaming data: The options are Azure Stream Analytics, Azure Databricks Structured Streaming and Apache Storm on HDInsight
• #12: Azure Databricks features – Enhance your teams’ productivity Get started quickly by launching your new Spark environment with one click. Share your insights in powerful ways through rich integration with PowerBI. Improve collaboration amongst your analytics team through a unified workspace. Innovate faster with native integration with rest of Azure platform. Build on the most compliant and trusted cloud Simplify security and identity control with built-in integration with Active Directory. Regulate access with fine-grained user permissions to Azure Databricks’ notebooks, clusters, jobs and data. Build with confidence on the trusted cloud backed by unmatched support, compliance and SLAs. Scale without limits Operate at massive scale without limits globally. Accelerate data processing with the fastest Spark engine.