SQL on Hadoop for the Oracle Professional

Editor's Notes

• #2: Hadoop is a “huge” topic. With one day, we had to choose what to explain. Where it’s awesome, and where it’s not-so-awesome.`
• #3: Gluent is all about enterprise data sharing and transparent data virtualization.
• #4: Why are we excited about hadoop?
• #5: The evolution of big data. How did we get here and why does this training even exist? 10 yrs ago, what was the biggest DB? 5 TB Today, that’s not “exciting”. If you work on ERP systems, easy to think that all data lives in tables and is easy to store. Not always true, different businesses - different departments - have different needs.
• #6: Why not just load up storage array?
• #7: Data moved to computation for processing Records must move from storage to process Access to storage is the bottleneck Running out of space? Expand SAN Becomes expensive. SAN storage arrays are not cheap. Two common problems in db performance work Bad sql statements (optimizer issues) Storage issues. Usually not config issue, but pipes between database nodes and san storage is very small.
• #8: Describe NameNode & DataNodes at high-level NameNode - where does my file - file blocks - live? DataNode - allows access to locally stored data. Data doesn’t move for computation. Processing occurs where the data lives Running out of space? Add new nodes on commodity hardware. Physically Data nodes on commodity, inexpensive “pizza boxes”. Looks like one big filesystem. Get deeper on HDFS later on.
• #9: Name node can be compared to ASM instance in Exadata. (TODO) Not just distributed file system, but distributed computation. Code is distributed as well.
• #10: Data file is chunked into blocks 3 replicas of each block will be created If a node goes down, NameNode ensures each block still has 3 replicas Can also rebalance if needed.
• #11: Software scalability - past, you had to buy many large, big iron hardware systems. Now, open source software. Hadoop ecosystem is organized around open data formats. Makes your system future proof. One data, many engines. DW on Hadoop, not stuck with Hive SQL - you can use any engine you like. Switching engines when using something like Oracle - export, load into new engine. Takes time.
• #12: Workload processing - SQL-like processing in an Oracle example. RDBMS / enterprise database Read data. Filter down to what you need. Perform simple processing. More complex processing. Exadata (similar to Netezza) Push filtering to storage level. Disk I/O (read) and filtering. Less data moves from storage to DB. Still perform aggregations, etc in DB. Hadoop Not just SQL engine, but parallel processing framework that will push application code close to where the data lives. All operations/logic occurs in the same place. Exadata one step closer to hadoop, but not quite.
• #13: Load data once, read it many times Best for large, sequential file scans. Not great for random. MapR has own file system, MapR-FS, multi-level namenodes. (..more info needed..) Kudu can solve this problem as well - doesn’t use HDFS Cannot modify file, only insert on end. If could, uncompress file, update, recompress? Not good anyways.
• #14: You can really put any type of file there, but main formats are these. Typically start with CSV for tutorials. Sequence files, binary-like files Avro - serializing rows and columns Columnar formats - used when extracting data for analysis. Faster to extract/analyze. Stored in big chunks, organized in columns, often compressed.
• #16: 10 yrs ago, yes. HDFS and MapReduce in v1 Now, MR rarely used in new dev. -starting with Hadoop, run SQL against it. It’s very good.
• #17: Hive, the first SQL engine developed at Facebook. Converted SQL to MR under the hood to run against distributed filesystem. Now, many engines! Even Hive evolved to use Spark or Tez rather than MR. Hive will be described further later on.
• #18: Hadoop has a large ecosystem. Similar to redhat linux. Different hadoop vendors. Not vendor driven, community driven Ecosystem table - open and show how many components
• #19: Show a little about how it works Discuss some fundamentals and where Hadoop fits and doesn’t work as well. The big data hype was that it would solve all of your problems and make lots of money. Works for the folks like Facebook/Yahoo. But we don’t want all of our SQL developers to learn Java and mapreduce, etc. The fact you can actually run SQL on Hadoop - that’s the killer app for big data and Hadoop. For example, moving a lot of data from the expensive DW Or using hadoop as a data sharing back-end.
• #20: Impala syntax is pretty powerful. UDFs that can be written in C++ DEMO: show Impala, add files to HDFS, create tables, etc. show Hue as well.
• #21: On the front-end, it looks a lot like a database. SQL statements are similar, etc. On the backend, it’s a bit different. Every single node has its own local disk - there is no SAN storage. Hadoop does sequential reads/writes, not random I/O - so I/O bottleneck is not there. What about processing? I/O throughput is great, now the CPU might be the bottleneck. So in the ideal MPP world Run a select statement across all data Filter and group by. Group by state, not massive amount of records Thrift server picks one node to connect. The “query master”. Query master takes SQL and sends it to all nodes on the cluster Each node only reads local data (usually) - no network bottlenecks Query master dishes out work so unique data is read on each node. Each node does scanning and filtering on their own. Total sum, for aggregate, can be pushed down to the nodes as well. Each node result set is sent to query master for final aggregation and returned to Hive But, life is not always ideal
• #22: Scalability of aggregations Additive - like a Sum, you can sum at lower levels (each node) then aggregate again at the query master. Works similar to Oracle Parallel Query, at a different level. Each parallel slave can do a sum on its own, then later stage in the query plan it sums the sums. But what about count distinct? Or Median? Example - how many different products per customer transaction in a store? Distincts are not additive. That’s where the scale-out “dream” breaks down. We’ll get back to aggregations
• #23: Now we’ve added distinct to the count So nodes must exchange data to get to this final distinct count. Each node can order the local dataset, in parallel. Final merge join in the query master, performed serially. Nodes sharing data is called a ‘shuffle’ in order to see the full picture.
• #24: Fundamentals about joins Oracle - using shared storage, it doesn’t scale to a thousand nodes The benefit, all nodes “see” the storage and can scan through all of the nodes to return the data. Hadoop has a tradeoff. 9 nodes here, with each having a portion of the data. How do we do a join? Cannot just “join” node1 to node4 and hope all of the data is represented. First, read all data from small table and broadcast into memory on all of the nodes that will process the query. Not usually a problem for smaller dimension tables broadcast to join a large fact. But, it could become a problem. I relate this to the issue of change data capture in ETL.?? maybe
• #25: The biggest remaining scalability problem for Hadoop. Big fact to fact joins - probably won’t run well. For small table - optimizer will broadcast to each node. But for a large table, the optimizer will need to partition the join. We can shuffle - read through the first node and run a hash function on the key. Then based on results, send each row to different nodes - selectively. Partition join Then all nodes will do the same. Unfortunately, that’s only half of the story Shuffling must happen on the other side table as well. If right side is sales and left is customers, we need to do the same. Then, the local join can occur. Lots of CPU and network traffic. If Hadoop worked like teradata - not needed. Teradata will contain data of same IDs in same node. Better big table joins that way. But, you have to redistribute data when adding another node.
• #26: How do we workaround these issues? We can pre-join large fact tables, during load or after load and store the copy. A form of denormalization. Just scan through the columns you need in columnar formats Aprox count distinct - similar Oracle operation Uses a bitmap for storing an approximate number of distinct values per customer or state or whatever. The magic, they are additive. Each node can count distinct value, then can still add in final aggregation in query master. Gives roughly correct results. Within 1.5% of correct value - so must be able to tolerate deviation. Perfect for exploring data, but then use more intensive query.
• #27: Hive metastore Even used by other sql engines on hadoop. It’s like a data dictionary for Hadoop. How do we know which files in which directory to go after when running “SELECT *” in Hive? Hive metastore. Impala, Presto, even Spark, can plug into Hive Metastore. Different from how SQL Sever and Oracle work. Cannot share a data dictionary across these two engines. It’s like having multiple data engines for the same database. DEMO: Hive and Impala both show the same metadata!
• #32: Visual of next slide, due to animations hiding content
• #33: Hive is Hortonworks Impala is Cloudera Drill is now sort of MapR Presto, created by Facebook but Teradata now contributes Very neat part of open source, companies like Facebook, Yahoo creating these - but even companies like teradata contributing. Use cases A little background. Hive used to be slow for short queries because it always used MapReduce behind the scenes. Fixed on some platforms. Hive LLAP - low latency, so no slow startup overhead. Plus can use Spark or Tez behind the scenes. Impala - Faster analytics. Hive is more functional, SQL syntax is more rich, and has partition-wise joins, which Impala doesn’t have. But, it’s hadoop. You can use Hive for your ETL and complex transformations with user defined functions in a batch job, then use Impala for your BI users - giving them faster response time. “Use the best tool for the job” Drill is really good for exploratory analytics with it’s use of JSON files - and no need for a “relational” type schema. Presto - more modern version of Hive. Preference for Hadoop vendor will drive decision of what to use. Hortonworks - Hive or Spark SQL Cloudera - Impala or Hive TODO - remove alternative vendor based products like Drill / Presto?
• #34: Today? No! too much to migrate, some things cannot be migrated easily at all. But as far as capabilities of hadoop in the next 4-5 years, some cases yes. OLTP Mongo DB and Cassandra already doing it - for simple OLTP systems. Complex - not at all. RDBMS is here to stay. DW Traditional data warehouses, quite possible, but might take some time to take over Teradata, etc. But, not quite ready yet. Big table joins are still an issue. But it’s coming. Companies like Netflix are using a cloud-based DW, with parquet files in S3. Big Data - yes. Unless the systems with messages or event feeds started before the advent of big data, these IoT projects are going straight into Hadoop. ETL and DI Yeah, probably. A homegrown system, tough to convert to Hadoop. But ETL tools like Informatica or ODI, just point to Hadoop storage and processing apps. It won’t happen overnight, but it’s movig that way. Cloud does introduce an interesting shift. Services such as Google BigQuery might not even be known for traditional enterprises, using ERP systems. But Marketing firms, they just dump data to the cloud and then run analytics against them. Onsite - that’s more of where hadoop will rule.
• #35: Already discussed some of these. Hadoop can improve as well. Just to be realistic - we’re not selling Hadoop licenses or anything! Subquery under OR condition - not supported by Hive or Impala even. Could rewrite this query, but still an issue. You might not be able to just migrate all data and switch from Oracle PL/SQL to HiveQL. DEMO - #3EU 3:00 - Same query in Impala
• #37: Case study using HortonWorks on open source software?
• #38: https://hortonworks.com/customers/arizona-state-university/
• #39: https://hortonworks.com/customers/arizona-state-university/
• #40: Gluent case study
• #41: TODO add quote?
• #42: FYI - cost savings about 5 mill (existing) vs 2 mill (gluent)
• #44: TODO https://www.slideshare.net/AmazonWebServices/how-to-build-a-data-lake-with-aws-glue-data-catalog-abd213r
• #45: Introducing…Gluent Data Platform Support big 3 Hadoop vendors, on-premises and in the cloud - Many customers are cloud-first - recent customer example - Or shifting to the cloud. They look for help from us to do so
• #46: No risk implementation