Abstract image of a woman sitting on books and studying on a laptop

OC Big Data Monthly Meetup #5 - Session 1 - Altiscale

Big Data Joe™ Rossi, profile picture
Big Data Joe™ Rossi

The document discusses debugging Hive within a Hadoop environment provided by Altiscale, outlining the architecture, debugging tools, and common issues encountered when using Hive, such as memory errors and task failures. It covers the operational strategies for utilizing Hadoop logs and tools to troubleshoot and improve performance, along with case studies illustrating specific issues like stuck jobs and missing directories. The presentation emphasizes the service provider's role in enhancing reliability and managing Hadoop tasks efficiently.

Technology
1 of 55
Download to read offline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
DEBUGGING HIVE WITH HADOOP IN THE CLOUD Soam Acharya, David Chaiken, Denis Sheahan, Charles Wimmer Altiscale, Inc. #OCBigData @ 20140917T1845-0700
WHO ARE WE? • Altiscale: Infrastructure Nerds! • Hadoop As A Service • Rack and build our own Hadoop clusters • Provide a suite of Hadoop tools o Hive, Pig, Oozie o Others as needed: R, Python, Spark, Mahout, Impala, etc. • Monthly billing plan: compute (YARN), storage (HDFS) • https://www.altiscale.com • @Altiscale #HadoopSherpa
TALK ROADMAP • Our Platform and Perspective • Hadoop 2 Primer • Hadoop Debugging Tools • Accessing Logs in Hadoop 2 • Hive + Hadoop Architecture • Hive Logs • Hive Issues + Case Studies o Hive + Interactive (DRAM Centric) Processing Engines • Conclusion: Making Hive Easier to Use
OUR DYNAMIC PLATFORM • Hadoop 2.0.5 => Hadoop 2.2.0 => Hadoop 2.4.1 • Hive 0.10 => Hive 0.12 => Stinger (Hive 0.13 + Tez) • Hive, Pig and Oozie most commonly used tools • Working with customers on: Spark, Impala, 0xdata, Flume, Camus/Kafka, …
ALTISCALE PERSPECTIVE • What we do as a service provider… o Performance + Reliability: Jobs finish faster, fewer failures o Instant Access: Always-on access to HDFS and YARN o Hadoop Helpdesk: Tools + experts ensure customer success o Secure: Networking, SOC 2 Audit, Kerberos o Results: Faster Time-to-Value (TTV), Lower TCO • Operational approach in this presentation… o How to use Hadoop 2 cluster tools and logs to debug and to tune Hive o This talk will not focus on query optimization
QUICK PRIMER – HADOOP 2 !!!Hadoop!2!Cluster! Name!Node! ! Node!Managers! +!! Data!Nodes! Hadoop!Slave! Hadoop!Slave! Hadoop!Slave! Resource!Manager! ! Secondary!NameNode! ! Hadoop!Slave!
QUICK PRIMER – HADOOP 2 YARN • Resource Manager (per cluster) o Manages job scheduling and execution o Global resource allocation • Application Master (per job) o Manages task scheduling and execution o Local resource allocation • Node Manager (per-machine agent) o Manages the lifecycle of task containers o Reports to RM on health and resource usage
HADOOP 1 VS HADOOP 2 • No more JobTrackers, TaskTrackers • YARN ~ Operating System for Clusters o MapReduce is implemented as a YARN application o Bring on the applications! (Spark is just the start…) • Should be Transparent to Hive users
HADOOP 2 DEBUGGING TOOLS • Monitoring o System state of cluster: ! CPU, Memory, Network, Disk ! Nagios, Ganglia, Sensu! ! Collectd, statd, Graphite o Hadoop level ! HDFS usage ! Resource usage: • Container memory allocated vs used • # of jobs running at the same time • Long running tasks
HADOOP 2 DEBUGGING TOOLS • Hadoop logs o Daemon logs: Resource Manager, NameNode, DataNode o Application logs: Application Master, MapReduce tasks o Job history file: resources allocated during job lifetime o Application configuration files: store all Hadoop application parameters • Source code instrumentation
OC Big Data Monthly Meetup #5 - Session 1 - Altiscale
ACCESSING LOGS IN HADOOP 2 • To view the logs for a job, click on the link under the ID column in Resource Manager UI.
ACCESSING LOGS IN HADOOP 2 • To view application top level logs, click on logs. • To view individual logs for the mappers and reducers, click on History.
ACCESSING LOGS IN HADOOP 2 • Log output for the entire application.
ACCESSING LOGS IN HADOOP 2 • Click on the Map link for mapper logs and the Reduce link for reducer logs.
ACCESSING LOGS IN HADOOP 2 • Clicking on a single link under Name provides an overview for that particular map job.
ACCESSING LOGS IN HADOOP 2 • Finally, clicking on the logs link will take you to the log output for that map job.
ACCESSING LOGS IN HADOOP 2 • Fun, fun, donuts, and more fun…
HIVE + HADOOP 2 ARCHITECTURE • Hive 0.10+ !!!Hadoop!2!Cluster! Hive!CLI! Hive! Metastore! JDBC/ODBC! Hiveserver! AlaCon,! KeFle,!…!
HIVE LOGS • Query Log location • From /etc/hive/hive-site.xml: <property>" <name>hive.querylog.location</name>" <value>/home/hive/log/${user.name}</value>" </property>" " SessionStart SESSION_ID="soam_201402032341" TIME="1391470900594"" "
HIVE CLIENT LOGS • /etc/hive/hive-log4j.properties: o hive.log.dir=/var/log/hive/${user.name} 2014-05-29 19:51:09,830 INFO parse.ParseDriver (ParseDriver.java:parse(179)) - Parsing command: select count(*) from dogfood_job_data" 2014-05-29 19:51:09,852 INFO parse.ParseDriver (ParseDriver.java:parse(197)) - Parse Completed" 2014-05-29 19:51:09,852 INFO ql.Driver (PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=parse start=1401393069830 end=1401393069852 duration=22>" 2014-05-29 19:51:09,853 INFO ql.Driver (PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=semanticAnalyze>" 2014-05-29 19:51:09,890 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(8305)) - Starting Semantic Analysis" 2014-05-29 19:51:09,892 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(8340)) - Completed phase 1 of Semantic Analysis" 2014-05-29 19:51:09,892 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1060)) - Get metadata for source tables" 2014-05-29 19:51:09,906 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1167)) - Get metadata for subqueries" 2014-05-29 19:51:09,909 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1187)) - Get metadata for destination tables" "
HIVE METASTORE LOGS • /etc/hive-metastore/hive-log4j.properties: o hive.log.dir=/service/log/hive-metastore/${user.name} 2014-05-29 19:50:50,179 INFO metastore.HiveMetaStore (HiveMetaStore.java:logInfo(454)) - 200: source:/10.252.18.94 get_table : db=default tbl=dogfood_job_data" 2014-05-29 19:50:50,180 INFO HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(239)) - ugi=chaiken ip=/10.252.18.94 cmd=source:/10.252.18.94 get_table : db=default tbl=dogfood_job_data " 2014-05-29 19:50:50,236 INFO metastore.HiveMetaStore (HiveMetaStore.java:logInfo(454)) - 200: source:/10.252.18.94 get_table : db=default tbl=dogfood_job_data" 2014-05-29 19:50:50,236 INFO HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(239)) - ugi=chaiken ip=/10.252.18.94 cmd=source:/10.252.18.94 get_table : db=default tbl=dogfood_job_data " 2014-05-29 19:50:50,261 INFO metastore.HiveMetaStore (HiveMetaStore.java:logInfo(454)) - 200: source:/10.252.18.94 get_table : db=default tbl=dogfood_job_data"
HIVE ISSUES + CASE STUDIES • Hive Issues o Hive client out of memory o Hive map/reduce task out of memory o Hive metastore out of memory o Hive launches too many tasks • Case Studies: o Hive “stuck” job o Hive “missing directories” o Analyze Hive Query Execution o Hive + Interactive (DRAM Centric) Processing Engines
HIVE CLIENT OUT OF MEMORY • Memory intensive client side hive query (map-side join) Number of reduce tasks not specified. Estimated from input data size: 999" In order to change the average load for a reducer (in bytes):" set hive.exec.reducers.bytes.per.reducer=<number>" In order to limit the maximum number of reducers:" set hive.exec.reducers.max=<number>" In order to set a constant number of reducers:" set mapred.reduce.tasks=<number>" java.lang.OutOfMemoryError: Java heap space! at java.nio.CharBuffer.wrap(CharBuffer.java:350)" at java.nio.CharBuffer.wrap(CharBuffer.java:373)" at java.lang.StringCoding$StringDecoder.decode(StringCoding.java: 138)"
HIVE CLIENT OUT OF MEMORY • Use HADOOP_HEAPSIZE prior to launching Hive client • HADOOP_HEAPSIZE=<new heapsize> hive <fileName>" • Watch out for HADOOP_CLIENT_OPTS issue in hive-env.sh! • Important to know the amount of memory available on machine running client… Do not exceed or use disproportionate amount. $ free -m" total used free shared buffers cached" Mem: 1695 1388 306 0 60 424" -/+ buffers/cache: 903 791" Swap: 895 101 794" !!
HIVE TASK OUT OF MEMORY • Query spawns MapReduce jobs that run out of memory • How to find this issue? o Hive diagnostic message o Hadoop MapReduce logs
HIVE TASK OUT OF MEMORY • Fix is to increase task RAM allocation… set mapreduce.map.memory.mb=<new RAM allocation>; " set mapreduce.reduce.memory.mb=<new RAM allocation>;" • Also watch out for… set mapreduce.map.java.opts=-Xmx<heap size>m; " set mapreduce.reduce.java.opts=-Xmx<heap size>m; " • Not a magic bullet – requires manual tuning • Increase in individual container memory size: o Decrease in overall containers that can be run o Decrease in overall parallelism
HIVE METASTORE OUT OF MEMORY • Out of memory issues not necessarily dumped to logs • Metastore can become unresponsive • Can’t submit queries • Restart with a higher heap size: export HADOOP_HEAPSIZE in hcat_server.sh • After notifying hive users about downtime: service hcat restart"
HIVE LAUNCHES TOO MANY TASKS • Typically a function of the input data set • Lots of little files
HIVE LAUNCHES TOO MANY TASKS • Set mapred.max.split.size to appropriate fraction of data size • Also verify that hive.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat"
CASE STUDY: HIVE STUCK JOB From an Altiscale customer: “This job [jobid] has been running now for 41 hours. Is it still progressing or has something hung up the map/reduce so it’s just spinning? Do you have any insight?”
HIVE STUCK JOB 1. Received jobId, application_1382973574141_4536, from client 2. Logged into client cluster. 3. Pulled up Resource Manager 4. Entered part of jobId (4536) in the search box. 5. Clicked on the link that says: application_1382973574141_4536" 6. On resulting Application Overview page, clicked on link next to “Tracking URL” that said Application Master
HIVE STUCK JOB 7. On resulting MapReduce Application page, we clicked on the Job Id (job_1382973574141_4536). 8. The resulting MapReduce Job page displayed detailed status of the mappers, including 4 failed mappers 9. We then clicked on the 4 link on the Maps row in the Failed column. 10. Title of the next page was “FAILED Map attempts in job_1382973574141_4536.” 11. Each failed mapper generated an error message. 12. Buried in the 16th line: Caused by: java.io.FileNotFoundException: File does not exist: hdfs://opaque_hostname:8020/ HiveTableDir/FileName.log.date.seq !
HIVE STUCK JOB • Job was stuck for a day or so, retrying a mapper that would never finish successfully. • During the job, our customers’ colleague realized input file was corrupted and deleted it. • Colleague did not anticipate the affect of removing corrupted data on a running job • Hadoop didn’t make it easy to find out: o RM => search => application link => AM overview page => MR Application Page => MR Job Page => Failed jobs page => parse long logs o Task retry without hope of success
HIVE “MISSING DIRECTORIES” From an Altiscale customer: “One problem we are seeing after the [Hive Metastore] restart is that we lost quite a few directories in [HDFS]. Is there a way to recover these?”
HIVE “MISSING DIRECTORIES” • Obtained list of “missing” directories from customer: o /hive/biz/prod/* • Confirmed they were missing from HDFS • Searched through NameNode audit log to get block IDs that belonged to missing directories. 13/07/24 21:10:08 INFO hdfs.StateChange: BLOCK* NameSystem.allocateBlock: /hive/biz/prod/ incremental/carryoverstore/postdepuis/ lmt_unmapped_pggroup_schema._COPYING_. BP-798113632-10.251.255.251-1370812162472 blk_3560522076897293424_2448396{blockUCState=UNDER_C ONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[10.251.255.177:50 010|RBW], ReplicaUnderConstruction[10.251.255.174:50010|RBW], ReplicaUnderConstruction[10.251.255.169:50010|RBW]]}"
HIVE “MISSING DIRECTORIES” • Used blockID to locate exact time of file deletion from Namenode logs: 13/07/31 08:10:33 INFO hdfs.StateChange: BLOCK* addToInvalidates: blk_3560522076897293424_2448396 to 10.251.255.177:50010 10.251.255.169:50010 10.251.255.174:50010 " • Used time of deletion to inspect hive logs
HIVE “MISSING DIRECTORIES” QueryStart QUERY_STRING="create database biz_weekly location '/hive/biz/ prod'" QUERY_ID=“usrprod_20130731043232_0a40fd32-8c8a-479c-ba7d- 3bd8a2698f4b" TIME="1375245164667" : QueryEnd QUERY_STRING="create database biz_weekly location '/hive/biz/ prod'" QUERY_ID=”usrprod_20130731043232_0a40fd32-8c8a-479c-ba7d- 3bd8a2698f4b" QUERY_RET_CODE="0" QUERY_NUM_TASKS="0" TIME="1375245166203" : QueryStart QUERY_STRING="drop database biz_weekly" QUERY_ID=”usrprod_20130731073333_e9acf35c-4f07-4f12-bd9d-bae137ae0733" TIME="1375256014799" : QueryEnd QUERY_STRING="drop database biz_weekly" QUERY_ID=”usrprod_20130731073333_e9acf35c-4f07-4f12-bd9d-bae137ae0733" QUERY_NUM_TASKS="0" TIME="1375256014838"
HIVE “MISSING DIRECTORIES” • In effect, user “usrprod” issued: At 2013-07-31 04:32:44: create database biz_weekly location '/hive/biz/prod' At 2013-07-31 07:33:24: drop database biz_weekly • This is functionally equivalent to: hdfs dfs -rm -r /hive/biz/prod"
HIVE “MISSING DIRECTORIES” • Customer manually placed their own data in /hive – the warehouse directory managed and controlled by hive • Customer used CREATE and DROP db commands in their code o Hive deletes database and table locations in /hive with impunity • Why didn’t deleted data end up in .Trash? o Trash collection not turned on in configuration settings o It is now, but need a –skipTrash option (HIVE-6469)
HIVE “MISSING DIRECTORIES” • Hadoop forensics: piece together disparate sources… o Hadoop daemon logs (NameNode) o Hive query and metastore logs o Hadoop config files • Need better tools to correlate the different layers of the system: hive client, hive metastore, MapReduce job, YARN, HDFS, operating sytem metrics, … By the way… Operating any distributed system would be totally insane without NTP and a standard time zone (UTC).
CASE STUDY – ANALYZE QUERY • Customer provided Hive query + data sets (100GBs to ~5 TBs) • Needed help optimizing the query • Didn’t rewrite query immediately • Wanted to characterize query performance and isolate bottlenecks first
ANALYZE AND TUNE EXECUTION • Ran original query on the datasets in our environment: o Two M/R Stages: Stage-1, Stage-2 • Long running reducers run out of memory o set mapreduce.reduce.memory.mb=5120" o Reduces slots and extends reduce time • Query fails to launch Stage-2 with out of memory o set HADOOP_HEAPSIZE=1024 on client machine • Query has 250,000 Mappers in Stage-2 which causes failure o set mapred.max.split.size=5368709120 to reduce Mappers
ANALYSIS: HOW TO VISUALIZE? • Next challenge - how to visualize job execution? • Existing hadoop/hive logs not sufficient for this task • Wrote internal tools o parse job history files o plot mapper and reducer execution
ANALYSIS: MAP STAGE-1
ANALYSIS: REDUCE STAGE-1 Single!reduce!task!
ANALYSIS: MAP STAGE-2
ANALYSIS: REDUCE STAGE-2
ANALYZE EXECUTION: FINDINGS • Lone, long running reducer in first stage of query • Analyzed input data: o Query split input data by userId o Bucketizing input data by userId o One very large bucket: “invalid” userId o Discussed “invalid” userid with customer • An error value is a common pattern! o Need to differentiate between “Don’t know and don’t care” or “don’t know and do care.”
INTERACTIVE (DRAM CENTRIC) PROCESSING SYSTEMS • Loading data into DRAM makes processing fast! • Examples: Spark, Impala, 0xdata, …, [SAP HANA], … • Streaming systems (Storm, DataTorrent) may be similar • Need to increase YARN container memory size
HIVE + INTERACTIVE: WATCH OUT FOR CONTAINER SIZE • Caution: larger YARN container settings for interactive jobs may not be right for batch systems like Hive • Container size: needs to combine vcores and memory: yarn.scheduler.maximum-allocation-vcores yarn.nodemanager.resource.cpu-vcores ..."
HIVE + INTERACTIVE: WATCH OUT FOR FRAGMENTATION • Attempting to schedule interactive systems and batch systems like Hive may result in fragmentation • Interactive systems may require all-or-nothing scheduling • Batch jobs with little tasks may starve interactive jobs
HIVE + INTERACTIVE: WATCH OUT FOR FRAGMENTATION Solutions for fragmentation… • Reserve interactive nodes before starting batch jobs • Reduce interactive container size (if the algorithm permits) • Node labels (YARN-726) and gang scheduling (YARN-624)
CONCLUSIONS • Hive + Hadoop debugging can get very complex o Sifting through many logs and screens o Automatic transmission versus manual transmission • Static partitioning induced by Java Virtual Machine has benefits but also induces challenges. • Where there are difficulties, there’s opportunity: o Better tooling, instrumentation, integration of logs/metrics • YARN still evolving into an operating system • Hadoop as a Service: aggregate and share expertise • Need to learn from the traditional database community!
QUESTIONS? COMMENTS?

More Related Content

PDF
Debugging Hive with Hadoop-in-the-Cloud by David Chaiken of Altiscale
Data Con LA
 
PDF
Cloudera Impala
Alex Moundalexis
 
PPTX
De-Bugging Hive with Hadoop-in-the-Cloud
DataWorks Summit
 
PDF
Scaling Spark Workloads on YARN - Boulder/Denver July 2015
Mac Moore
 
PDF
Data Engineering with Spring, Hadoop and Hive
Alex Silva
 
PPTX
Spring for Apache Hadoop
zenyk
 
PPTX
How to develop Big Data Pipelines for Hadoop, by Costin Leau
Codemotion
 
PPTX
HiveServer2
Schubert Zhang
 
Debugging Hive with Hadoop-in-the-Cloud by David Chaiken of Altiscale
Data Con LA
 
Cloudera Impala
Alex Moundalexis
 
De-Bugging Hive with Hadoop-in-the-Cloud
DataWorks Summit
 
Scaling Spark Workloads on YARN - Boulder/Denver July 2015
Mac Moore
 
Data Engineering with Spring, Hadoop and Hive
Alex Silva
 
Spring for Apache Hadoop
zenyk
 
How to develop Big Data Pipelines for Hadoop, by Costin Leau
Codemotion
 
HiveServer2
Schubert Zhang
 

What's hot (20)

PPTX
Rapid Development of Big Data applications using Spring for Apache Hadoop
zenyk
 
PPTX
Hadoop World 2011: Hadoop Troubleshooting 101 - Kate Ting - Cloudera
Cloudera, Inc.
 
PPTX
Oozie or Easy: Managing Hadoop Workloads the EASY Way
DataWorks Summit
 
PPTX
Hive and Apache Tez: Benchmarked at Yahoo! Scale
DataWorks Summit
 
PPT
Apache Hadoop India Summit 2011 talk "Oozie - Workflow for Hadoop" by Andreas N
Yahoo Developer Network
 
PDF
NYC HUG - Application Architectures with Apache Hadoop
markgrover
 
PPT
Hadoop @ Yahoo! - Internet Scale Data Processing
Yahoo Developer Network
 
PDF
Owning time series with team apache Strata San Jose 2015
Patrick McFadin
 
PPTX
Why your Spark Job is Failing
DataWorks Summit
 
PDF
a Secure Public Cache for YARN Application Resources
DataWorks Summit
 
PPTX
Introduction to real time big data with Apache Spark
Taras Matyashovsky
 
PPTX
Hive analytic workloads hadoop summit san jose 2014
alanfgates
 
PDF
Using R with Hadoop
Revolution Analytics
 
PDF
Johnny Miller – Cassandra + Spark = Awesome- NoSQL matters Barcelona 2014
NoSQLmatters
 
PDF
Chef & OpenStack: OSCON 2014
Matt Ray
 
PPTX
Pivotal hawq internals
Alexey Grishchenko
 
PDF
Big data analytics with Spark & Cassandra
Matthias Niehoff
 
PDF
Spark Cassandra Connector: Past, Present, and Future
Russell Spitzer
 
PDF
Spark cassandra connector.API, Best Practices and Use-Cases
Duyhai Doan
 
PDF
Analytics with Cassandra & Spark
Matthias Niehoff
 
Rapid Development of Big Data applications using Spring for Apache Hadoop
zenyk
 
Hadoop World 2011: Hadoop Troubleshooting 101 - Kate Ting - Cloudera
Cloudera, Inc.
 
Oozie or Easy: Managing Hadoop Workloads the EASY Way
DataWorks Summit
 
Hive and Apache Tez: Benchmarked at Yahoo! Scale
DataWorks Summit
 
Apache Hadoop India Summit 2011 talk "Oozie - Workflow for Hadoop" by Andreas N
Yahoo Developer Network
 
NYC HUG - Application Architectures with Apache Hadoop
markgrover
 
Hadoop @ Yahoo! - Internet Scale Data Processing
Yahoo Developer Network
 
Owning time series with team apache Strata San Jose 2015
Patrick McFadin
 
Why your Spark Job is Failing
DataWorks Summit
 
a Secure Public Cache for YARN Application Resources
DataWorks Summit
 
Introduction to real time big data with Apache Spark
Taras Matyashovsky
 
Hive analytic workloads hadoop summit san jose 2014
alanfgates
 
Using R with Hadoop
Revolution Analytics
 
Johnny Miller – Cassandra + Spark = Awesome- NoSQL matters Barcelona 2014
NoSQLmatters
 
Chef & OpenStack: OSCON 2014
Matt Ray
 
Pivotal hawq internals
Alexey Grishchenko
 
Big data analytics with Spark & Cassandra
Matthias Niehoff
 
Spark Cassandra Connector: Past, Present, and Future
Russell Spitzer
 
Spark cassandra connector.API, Best Practices and Use-Cases
Duyhai Doan
 
Analytics with Cassandra & Spark
Matthias Niehoff
 
Ad

Viewers also liked (8)

PDF
www.ijerd.com
IJERD Editor
 
PPT
Intro to Accounts
Shashank Gupta
 
PDF
www.ijerd.com
IJERD Editor
 
PDF
www.ijerd.com
IJERD Editor
 
PPTX
Rock Kickoff 2013
HBSROCK
 
PDF
www.ijerd.com
IJERD Editor
 
PDF
www.ijerd.com
IJERD Editor
 
PDF
Documento guia de_wifi eset
Erick Sandoval
 
www.ijerd.com
IJERD Editor
 
Intro to Accounts
Shashank Gupta
 
www.ijerd.com
IJERD Editor
 
www.ijerd.com
IJERD Editor
 
Rock Kickoff 2013
HBSROCK
 
www.ijerd.com
IJERD Editor
 
www.ijerd.com
IJERD Editor
 
Documento guia de_wifi eset
Erick Sandoval
 
Ad

Similar to OC Big Data Monthly Meetup #5 - Session 1 - Altiscale (20)

PPTX
Debugging Hive with Hadoop-in-the-Cloud
Soam Acharya
 
PPTX
Managing growth in Production Hadoop Deployments
DataWorks Summit
 
PPT
ha_module5
Gurmukh Singh
 
PDF
Chapter 10
pavan penugonda
 
PPTX
Hadoop ppt on the basics and architecture
saipriyacoool
 
PPTX
Getting started big data
Kibrom Gebrehiwot
 
PPTX
Big data, just an introduction to Hadoop and Scripting Languages
Corley S.r.l.
 
PDF
Big Data Developers Moscow Meetup 1 - sql on hadoop
bddmoscow
 
PPTX
BIG DATA ANALYSIS
Nitesh Singh
 
PPTX
First Hive Meetup London 2012-07-10 - Tomas Cervenka - VisualDNA
Tomas Cervenka
 
PPTX
Data Analytics and IoT, how to analyze data from IoT
AmmarHassan80
 
PPTX
xPatterns on Spark, Shark, Mesos, Tachyon
Claudiu Barbura
 
PPTX
Cloudera Impala - Las Vegas Big Data Meetup Nov 5th 2014
cdmaxime
 
PDF
myHadoop 0.30
Glenn K. Lockwood
 
PPT
2.2 Hadoop ecosystem -Distribution.ppt
sec22ci043
 
PDF
Review of Calculation Paradigm and its Components
Namuk Park
 
PPTX
Facing enterprise specific challenges – utility programming in hadoop
fann wu
 
PDF
Cassandra Day SV 2014: Spark, Shark, and Apache Cassandra
DataStax Academy
 
PPTX
Intro to Apache Spark by CTO of Twingo
MapR Technologies
 
PPTX
xPatterns ... beyond Hadoop (Spark, Shark, Mesos, Tachyon)
Claudiu Barbura
 
Debugging Hive with Hadoop-in-the-Cloud
Soam Acharya
 
Managing growth in Production Hadoop Deployments
DataWorks Summit
 
ha_module5
Gurmukh Singh
 
Chapter 10
pavan penugonda
 
Hadoop ppt on the basics and architecture
saipriyacoool
 
Getting started big data
Kibrom Gebrehiwot
 
Big data, just an introduction to Hadoop and Scripting Languages
Corley S.r.l.
 
Big Data Developers Moscow Meetup 1 - sql on hadoop
bddmoscow
 
BIG DATA ANALYSIS
Nitesh Singh
 
First Hive Meetup London 2012-07-10 - Tomas Cervenka - VisualDNA
Tomas Cervenka
 
Data Analytics and IoT, how to analyze data from IoT
AmmarHassan80
 
xPatterns on Spark, Shark, Mesos, Tachyon
Claudiu Barbura
 
Cloudera Impala - Las Vegas Big Data Meetup Nov 5th 2014
cdmaxime
 
myHadoop 0.30
Glenn K. Lockwood
 
2.2 Hadoop ecosystem -Distribution.ppt
sec22ci043
 
Review of Calculation Paradigm and its Components
Namuk Park
 
Facing enterprise specific challenges – utility programming in hadoop
fann wu
 
Cassandra Day SV 2014: Spark, Shark, and Apache Cassandra
DataStax Academy
 
Intro to Apache Spark by CTO of Twingo
MapR Technologies
 
xPatterns ... beyond Hadoop (Spark, Shark, Mesos, Tachyon)
Claudiu Barbura
 

More from Big Data Joe™ Rossi (11)

PDF
Hadoop: Past, Present and Future - v2.2 - SQLSaturday #326 - Tampa BA Edition
Big Data Joe™ Rossi
 
PPTX
OC Big Data Monthly Meetup #6 - Session 2 - Basho/Riak
Big Data Joe™ Rossi
 
PDF
OC Big Data Monthly Meetup #6 - Session 1 - IBM
Big Data Joe™ Rossi
 
PDF
SD Big Data Monthly Meetup #4 - Session 2 - WANDisco
Big Data Joe™ Rossi
 
PDF
SD Big Data Monthly Meetup #4 - Session 1 - IBM
Big Data Joe™ Rossi
 
PDF
Hadoop: Past, Present and Future - v2.1 - SQLSaturday #340
Big Data Joe™ Rossi
 
PDF
OC Big Data Monthly Meetup #5 - Session 2 - Sumo Logic
Big Data Joe™ Rossi
 
PDF
Hadoop - Past, Present and Future - v2.0
Big Data Joe™ Rossi
 
PDF
Hadoop - Past, Present and Future - v1.2
Big Data Joe™ Rossi
 
PPTX
Hadoop - Past, Present and Future - v1.1
Big Data Joe™ Rossi
 
PPTX
Huhadoop - v1.1
Big Data Joe™ Rossi
 
Hadoop: Past, Present and Future - v2.2 - SQLSaturday #326 - Tampa BA Edition
Big Data Joe™ Rossi
 
OC Big Data Monthly Meetup #6 - Session 2 - Basho/Riak
Big Data Joe™ Rossi
 
OC Big Data Monthly Meetup #6 - Session 1 - IBM
Big Data Joe™ Rossi
 
SD Big Data Monthly Meetup #4 - Session 2 - WANDisco
Big Data Joe™ Rossi
 
SD Big Data Monthly Meetup #4 - Session 1 - IBM
Big Data Joe™ Rossi
 
Hadoop: Past, Present and Future - v2.1 - SQLSaturday #340
Big Data Joe™ Rossi
 
OC Big Data Monthly Meetup #5 - Session 2 - Sumo Logic
Big Data Joe™ Rossi
 
Hadoop - Past, Present and Future - v2.0
Big Data Joe™ Rossi
 
Hadoop - Past, Present and Future - v1.2
Big Data Joe™ Rossi
 
Hadoop - Past, Present and Future - v1.1
Big Data Joe™ Rossi
 
Huhadoop - v1.1
Big Data Joe™ Rossi
 

Recently uploaded (20)

PPT
Galois Field Theory of Risk: A Perspective, Protocol, and Mathematical Backgr...
Melanie Swan
 
PDF
A review of recent deep learning applications in wood surface defect identifi...
IAESIJAI
 
PDF
STKI Israel Market Study 2025 version august
Dr. Jimmy Schwarzkopf
 
PPTX
Module 1 Introduction to Web Programming .pptx
kamleshkc191
 
DOCX
search engine optimization ppt fir known well about this
StandardCodeLearning
 
PDF
giants, standing on the shoulders of - by Daniel Stenberg
Daniel Stenberg
 
PDF
UiPath Agentic Automation session 1: RPA to Agents
suhanisingh58689
 
DOCX
Basics of Cloud Computing - Cloud Ecosystem
suthak11
 
PDF
Enhancing plagiarism detection using data pre-processing and machine learning...
IAESIJAI
 
PDF
Convolutional neural network based encoder-decoder for efficient real-time ob...
IAESIJAI
 
PDF
Dell Pro Micro: Speed customer interactions, patient processing, and learning...
Principled Technologies
 
PPTX
TEXTILE technology diploma scope and career opportunities
kriti38
 
PDF
How ambidextrous entrepreneurial leaders react to the artificial intelligence...
IAESIJAI
 
PPTX
GROUP4NURSINGINFORMATICSREPORT-2 PRESENTATION
ClarisejaneSartillo1
 
PDF
Transform-Your-Streaming-Platform-with-AI-Driven-Quality-Engineering.pdf
Kalilur Rahman
 
PDF
Early detection and classification of bone marrow changes in lumbar vertebrae...
IAESIJAI
 
PPTX
Training Program for knowledge in solar cell and solar industry
RabindranathKaibarty
 
PDF
4 layer Arch & Reference Arch of IoT.pdf
sendilkumar66
 
PPTX
MicrosoftCybserSecurityReferenceArchitecture-April-2025.pptx
SilvioHayashi
 
PDF
Transform-Your-Factory-with-AI-Driven-Quality-Engineering.pdf
Kalilur Rahman
 
Galois Field Theory of Risk: A Perspective, Protocol, and Mathematical Backgr...
Melanie Swan
 
A review of recent deep learning applications in wood surface defect identifi...
IAESIJAI
 
STKI Israel Market Study 2025 version august
Dr. Jimmy Schwarzkopf
 
Module 1 Introduction to Web Programming .pptx
kamleshkc191
 
search engine optimization ppt fir known well about this
StandardCodeLearning
 
giants, standing on the shoulders of - by Daniel Stenberg
Daniel Stenberg
 
UiPath Agentic Automation session 1: RPA to Agents
suhanisingh58689
 
Basics of Cloud Computing - Cloud Ecosystem
suthak11
 
Enhancing plagiarism detection using data pre-processing and machine learning...
IAESIJAI
 
Convolutional neural network based encoder-decoder for efficient real-time ob...
IAESIJAI
 
Dell Pro Micro: Speed customer interactions, patient processing, and learning...
Principled Technologies
 
TEXTILE technology diploma scope and career opportunities
kriti38
 
How ambidextrous entrepreneurial leaders react to the artificial intelligence...
IAESIJAI
 
GROUP4NURSINGINFORMATICSREPORT-2 PRESENTATION
ClarisejaneSartillo1
 
Transform-Your-Streaming-Platform-with-AI-Driven-Quality-Engineering.pdf
Kalilur Rahman
 
Early detection and classification of bone marrow changes in lumbar vertebrae...
IAESIJAI
 
Training Program for knowledge in solar cell and solar industry
RabindranathKaibarty
 
4 layer Arch & Reference Arch of IoT.pdf
sendilkumar66
 
MicrosoftCybserSecurityReferenceArchitecture-April-2025.pptx
SilvioHayashi
 
Transform-Your-Factory-with-AI-Driven-Quality-Engineering.pdf
Kalilur Rahman
 

OC Big Data Monthly Meetup #5 - Session 1 - Altiscale

  • 1. DEBUGGING HIVE WITH HADOOP IN THE CLOUD Soam Acharya, David Chaiken, Denis Sheahan, Charles Wimmer Altiscale, Inc. #OCBigData @ 20140917T1845-0700
  • 2. WHO ARE WE? • Altiscale: Infrastructure Nerds! • Hadoop As A Service • Rack and build our own Hadoop clusters • Provide a suite of Hadoop tools o Hive, Pig, Oozie o Others as needed: R, Python, Spark, Mahout, Impala, etc. • Monthly billing plan: compute (YARN), storage (HDFS) • https://www.altiscale.com • @Altiscale #HadoopSherpa
  • 3. TALK ROADMAP • Our Platform and Perspective • Hadoop 2 Primer • Hadoop Debugging Tools • Accessing Logs in Hadoop 2 • Hive + Hadoop Architecture • Hive Logs • Hive Issues + Case Studies o Hive + Interactive (DRAM Centric) Processing Engines • Conclusion: Making Hive Easier to Use
  • 4. OUR DYNAMIC PLATFORM • Hadoop 2.0.5 => Hadoop 2.2.0 => Hadoop 2.4.1 • Hive 0.10 => Hive 0.12 => Stinger (Hive 0.13 + Tez) • Hive, Pig and Oozie most commonly used tools • Working with customers on: Spark, Impala, 0xdata, Flume, Camus/Kafka, …
  • 5. ALTISCALE PERSPECTIVE • What we do as a service provider… o Performance + Reliability: Jobs finish faster, fewer failures o Instant Access: Always-on access to HDFS and YARN o Hadoop Helpdesk: Tools + experts ensure customer success o Secure: Networking, SOC 2 Audit, Kerberos o Results: Faster Time-to-Value (TTV), Lower TCO • Operational approach in this presentation… o How to use Hadoop 2 cluster tools and logs to debug and to tune Hive o This talk will not focus on query optimization
  • 6. QUICK PRIMER – HADOOP 2 !!!Hadoop!2!Cluster! Name!Node! ! Node!Managers! +!! Data!Nodes! Hadoop!Slave! Hadoop!Slave! Hadoop!Slave! Resource!Manager! ! Secondary!NameNode! ! Hadoop!Slave!
  • 7. QUICK PRIMER – HADOOP 2 YARN • Resource Manager (per cluster) o Manages job scheduling and execution o Global resource allocation • Application Master (per job) o Manages task scheduling and execution o Local resource allocation • Node Manager (per-machine agent) o Manages the lifecycle of task containers o Reports to RM on health and resource usage
  • 8. HADOOP 1 VS HADOOP 2 • No more JobTrackers, TaskTrackers • YARN ~ Operating System for Clusters o MapReduce is implemented as a YARN application o Bring on the applications! (Spark is just the start…) • Should be Transparent to Hive users
  • 9. HADOOP 2 DEBUGGING TOOLS • Monitoring o System state of cluster: ! CPU, Memory, Network, Disk ! Nagios, Ganglia, Sensu! ! Collectd, statd, Graphite o Hadoop level ! HDFS usage ! Resource usage: • Container memory allocated vs used • # of jobs running at the same time • Long running tasks
  • 10. HADOOP 2 DEBUGGING TOOLS • Hadoop logs o Daemon logs: Resource Manager, NameNode, DataNode o Application logs: Application Master, MapReduce tasks o Job history file: resources allocated during job lifetime o Application configuration files: store all Hadoop application parameters • Source code instrumentation
  • 12. ACCESSING LOGS IN HADOOP 2 • To view the logs for a job, click on the link under the ID column in Resource Manager UI.
  • 13. ACCESSING LOGS IN HADOOP 2 • To view application top level logs, click on logs. • To view individual logs for the mappers and reducers, click on History.
  • 14. ACCESSING LOGS IN HADOOP 2 • Log output for the entire application.
  • 15. ACCESSING LOGS IN HADOOP 2 • Click on the Map link for mapper logs and the Reduce link for reducer logs.
  • 16. ACCESSING LOGS IN HADOOP 2 • Clicking on a single link under Name provides an overview for that particular map job.
  • 17. ACCESSING LOGS IN HADOOP 2 • Finally, clicking on the logs link will take you to the log output for that map job.
  • 18. ACCESSING LOGS IN HADOOP 2 • Fun, fun, donuts, and more fun…
  • 19. HIVE + HADOOP 2 ARCHITECTURE • Hive 0.10+ !!!Hadoop!2!Cluster! Hive!CLI! Hive! Metastore! JDBC/ODBC! Hiveserver! AlaCon,! KeFle,!…!
  • 20. HIVE LOGS • Query Log location • From /etc/hive/hive-site.xml: <property>" <name>hive.querylog.location</name>" <value>/home/hive/log/${user.name}</value>" </property>" " SessionStart SESSION_ID="soam_201402032341" TIME="1391470900594"" "
  • 21. HIVE CLIENT LOGS • /etc/hive/hive-log4j.properties: o hive.log.dir=/var/log/hive/${user.name} 2014-05-29 19:51:09,830 INFO parse.ParseDriver (ParseDriver.java:parse(179)) - Parsing command: select count(*) from dogfood_job_data" 2014-05-29 19:51:09,852 INFO parse.ParseDriver (ParseDriver.java:parse(197)) - Parse Completed" 2014-05-29 19:51:09,852 INFO ql.Driver (PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=parse start=1401393069830 end=1401393069852 duration=22>" 2014-05-29 19:51:09,853 INFO ql.Driver (PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=semanticAnalyze>" 2014-05-29 19:51:09,890 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(8305)) - Starting Semantic Analysis" 2014-05-29 19:51:09,892 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(8340)) - Completed phase 1 of Semantic Analysis" 2014-05-29 19:51:09,892 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1060)) - Get metadata for source tables" 2014-05-29 19:51:09,906 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1167)) - Get metadata for subqueries" 2014-05-29 19:51:09,909 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1187)) - Get metadata for destination tables" "
  • 22. HIVE METASTORE LOGS • /etc/hive-metastore/hive-log4j.properties: o hive.log.dir=/service/log/hive-metastore/${user.name} 2014-05-29 19:50:50,179 INFO metastore.HiveMetaStore (HiveMetaStore.java:logInfo(454)) - 200: source:/10.252.18.94 get_table : db=default tbl=dogfood_job_data" 2014-05-29 19:50:50,180 INFO HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(239)) - ugi=chaiken ip=/10.252.18.94 cmd=source:/10.252.18.94 get_table : db=default tbl=dogfood_job_data " 2014-05-29 19:50:50,236 INFO metastore.HiveMetaStore (HiveMetaStore.java:logInfo(454)) - 200: source:/10.252.18.94 get_table : db=default tbl=dogfood_job_data" 2014-05-29 19:50:50,236 INFO HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(239)) - ugi=chaiken ip=/10.252.18.94 cmd=source:/10.252.18.94 get_table : db=default tbl=dogfood_job_data " 2014-05-29 19:50:50,261 INFO metastore.HiveMetaStore (HiveMetaStore.java:logInfo(454)) - 200: source:/10.252.18.94 get_table : db=default tbl=dogfood_job_data"
  • 23. HIVE ISSUES + CASE STUDIES • Hive Issues o Hive client out of memory o Hive map/reduce task out of memory o Hive metastore out of memory o Hive launches too many tasks • Case Studies: o Hive “stuck” job o Hive “missing directories” o Analyze Hive Query Execution o Hive + Interactive (DRAM Centric) Processing Engines
  • 24. HIVE CLIENT OUT OF MEMORY • Memory intensive client side hive query (map-side join) Number of reduce tasks not specified. Estimated from input data size: 999" In order to change the average load for a reducer (in bytes):" set hive.exec.reducers.bytes.per.reducer=<number>" In order to limit the maximum number of reducers:" set hive.exec.reducers.max=<number>" In order to set a constant number of reducers:" set mapred.reduce.tasks=<number>" java.lang.OutOfMemoryError: Java heap space! at java.nio.CharBuffer.wrap(CharBuffer.java:350)" at java.nio.CharBuffer.wrap(CharBuffer.java:373)" at java.lang.StringCoding$StringDecoder.decode(StringCoding.java: 138)"
  • 25. HIVE CLIENT OUT OF MEMORY • Use HADOOP_HEAPSIZE prior to launching Hive client • HADOOP_HEAPSIZE=<new heapsize> hive <fileName>" • Watch out for HADOOP_CLIENT_OPTS issue in hive-env.sh! • Important to know the amount of memory available on machine running client… Do not exceed or use disproportionate amount. $ free -m" total used free shared buffers cached" Mem: 1695 1388 306 0 60 424" -/+ buffers/cache: 903 791" Swap: 895 101 794" !!
  • 26. HIVE TASK OUT OF MEMORY • Query spawns MapReduce jobs that run out of memory • How to find this issue? o Hive diagnostic message o Hadoop MapReduce logs
  • 27. HIVE TASK OUT OF MEMORY • Fix is to increase task RAM allocation… set mapreduce.map.memory.mb=<new RAM allocation>; " set mapreduce.reduce.memory.mb=<new RAM allocation>;" • Also watch out for… set mapreduce.map.java.opts=-Xmx<heap size>m; " set mapreduce.reduce.java.opts=-Xmx<heap size>m; " • Not a magic bullet – requires manual tuning • Increase in individual container memory size: o Decrease in overall containers that can be run o Decrease in overall parallelism
  • 28. HIVE METASTORE OUT OF MEMORY • Out of memory issues not necessarily dumped to logs • Metastore can become unresponsive • Can’t submit queries • Restart with a higher heap size: export HADOOP_HEAPSIZE in hcat_server.sh • After notifying hive users about downtime: service hcat restart"
  • 29. HIVE LAUNCHES TOO MANY TASKS • Typically a function of the input data set • Lots of little files
  • 30. HIVE LAUNCHES TOO MANY TASKS • Set mapred.max.split.size to appropriate fraction of data size • Also verify that hive.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat"
  • 31. CASE STUDY: HIVE STUCK JOB From an Altiscale customer: “This job [jobid] has been running now for 41 hours. Is it still progressing or has something hung up the map/reduce so it’s just spinning? Do you have any insight?”
  • 32. HIVE STUCK JOB 1. Received jobId, application_1382973574141_4536, from client 2. Logged into client cluster. 3. Pulled up Resource Manager 4. Entered part of jobId (4536) in the search box. 5. Clicked on the link that says: application_1382973574141_4536" 6. On resulting Application Overview page, clicked on link next to “Tracking URL” that said Application Master
  • 33. HIVE STUCK JOB 7. On resulting MapReduce Application page, we clicked on the Job Id (job_1382973574141_4536). 8. The resulting MapReduce Job page displayed detailed status of the mappers, including 4 failed mappers 9. We then clicked on the 4 link on the Maps row in the Failed column. 10. Title of the next page was “FAILED Map attempts in job_1382973574141_4536.” 11. Each failed mapper generated an error message. 12. Buried in the 16th line: Caused by: java.io.FileNotFoundException: File does not exist: hdfs://opaque_hostname:8020/ HiveTableDir/FileName.log.date.seq !
  • 34. HIVE STUCK JOB • Job was stuck for a day or so, retrying a mapper that would never finish successfully. • During the job, our customers’ colleague realized input file was corrupted and deleted it. • Colleague did not anticipate the affect of removing corrupted data on a running job • Hadoop didn’t make it easy to find out: o RM => search => application link => AM overview page => MR Application Page => MR Job Page => Failed jobs page => parse long logs o Task retry without hope of success
  • 35. HIVE “MISSING DIRECTORIES” From an Altiscale customer: “One problem we are seeing after the [Hive Metastore] restart is that we lost quite a few directories in [HDFS]. Is there a way to recover these?”
  • 36. HIVE “MISSING DIRECTORIES” • Obtained list of “missing” directories from customer: o /hive/biz/prod/* • Confirmed they were missing from HDFS • Searched through NameNode audit log to get block IDs that belonged to missing directories. 13/07/24 21:10:08 INFO hdfs.StateChange: BLOCK* NameSystem.allocateBlock: /hive/biz/prod/ incremental/carryoverstore/postdepuis/ lmt_unmapped_pggroup_schema._COPYING_. BP-798113632-10.251.255.251-1370812162472 blk_3560522076897293424_2448396{blockUCState=UNDER_C ONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[10.251.255.177:50 010|RBW], ReplicaUnderConstruction[10.251.255.174:50010|RBW], ReplicaUnderConstruction[10.251.255.169:50010|RBW]]}"
  • 37. HIVE “MISSING DIRECTORIES” • Used blockID to locate exact time of file deletion from Namenode logs: 13/07/31 08:10:33 INFO hdfs.StateChange: BLOCK* addToInvalidates: blk_3560522076897293424_2448396 to 10.251.255.177:50010 10.251.255.169:50010 10.251.255.174:50010 " • Used time of deletion to inspect hive logs
  • 38. HIVE “MISSING DIRECTORIES” QueryStart QUERY_STRING="create database biz_weekly location '/hive/biz/ prod'" QUERY_ID=“usrprod_20130731043232_0a40fd32-8c8a-479c-ba7d- 3bd8a2698f4b" TIME="1375245164667" : QueryEnd QUERY_STRING="create database biz_weekly location '/hive/biz/ prod'" QUERY_ID=”usrprod_20130731043232_0a40fd32-8c8a-479c-ba7d- 3bd8a2698f4b" QUERY_RET_CODE="0" QUERY_NUM_TASKS="0" TIME="1375245166203" : QueryStart QUERY_STRING="drop database biz_weekly" QUERY_ID=”usrprod_20130731073333_e9acf35c-4f07-4f12-bd9d-bae137ae0733" TIME="1375256014799" : QueryEnd QUERY_STRING="drop database biz_weekly" QUERY_ID=”usrprod_20130731073333_e9acf35c-4f07-4f12-bd9d-bae137ae0733" QUERY_NUM_TASKS="0" TIME="1375256014838"
  • 39. HIVE “MISSING DIRECTORIES” • In effect, user “usrprod” issued: At 2013-07-31 04:32:44: create database biz_weekly location '/hive/biz/prod' At 2013-07-31 07:33:24: drop database biz_weekly • This is functionally equivalent to: hdfs dfs -rm -r /hive/biz/prod"
  • 40. HIVE “MISSING DIRECTORIES” • Customer manually placed their own data in /hive – the warehouse directory managed and controlled by hive • Customer used CREATE and DROP db commands in their code o Hive deletes database and table locations in /hive with impunity • Why didn’t deleted data end up in .Trash? o Trash collection not turned on in configuration settings o It is now, but need a –skipTrash option (HIVE-6469)
  • 41. HIVE “MISSING DIRECTORIES” • Hadoop forensics: piece together disparate sources… o Hadoop daemon logs (NameNode) o Hive query and metastore logs o Hadoop config files • Need better tools to correlate the different layers of the system: hive client, hive metastore, MapReduce job, YARN, HDFS, operating sytem metrics, … By the way… Operating any distributed system would be totally insane without NTP and a standard time zone (UTC).
  • 42. CASE STUDY – ANALYZE QUERY • Customer provided Hive query + data sets (100GBs to ~5 TBs) • Needed help optimizing the query • Didn’t rewrite query immediately • Wanted to characterize query performance and isolate bottlenecks first
  • 43. ANALYZE AND TUNE EXECUTION • Ran original query on the datasets in our environment: o Two M/R Stages: Stage-1, Stage-2 • Long running reducers run out of memory o set mapreduce.reduce.memory.mb=5120" o Reduces slots and extends reduce time • Query fails to launch Stage-2 with out of memory o set HADOOP_HEAPSIZE=1024 on client machine • Query has 250,000 Mappers in Stage-2 which causes failure o set mapred.max.split.size=5368709120 to reduce Mappers
  • 44. ANALYSIS: HOW TO VISUALIZE? • Next challenge - how to visualize job execution? • Existing hadoop/hive logs not sufficient for this task • Wrote internal tools o parse job history files o plot mapper and reducer execution
  • 46. ANALYSIS: REDUCE STAGE-1 Single!reduce!task!
  • 49. ANALYZE EXECUTION: FINDINGS • Lone, long running reducer in first stage of query • Analyzed input data: o Query split input data by userId o Bucketizing input data by userId o One very large bucket: “invalid” userId o Discussed “invalid” userid with customer • An error value is a common pattern! o Need to differentiate between “Don’t know and don’t care” or “don’t know and do care.”
  • 50. INTERACTIVE (DRAM CENTRIC) PROCESSING SYSTEMS • Loading data into DRAM makes processing fast! • Examples: Spark, Impala, 0xdata, …, [SAP HANA], … • Streaming systems (Storm, DataTorrent) may be similar • Need to increase YARN container memory size
  • 51. HIVE + INTERACTIVE: WATCH OUT FOR CONTAINER SIZE • Caution: larger YARN container settings for interactive jobs may not be right for batch systems like Hive • Container size: needs to combine vcores and memory: yarn.scheduler.maximum-allocation-vcores yarn.nodemanager.resource.cpu-vcores ..."
  • 52. HIVE + INTERACTIVE: WATCH OUT FOR FRAGMENTATION • Attempting to schedule interactive systems and batch systems like Hive may result in fragmentation • Interactive systems may require all-or-nothing scheduling • Batch jobs with little tasks may starve interactive jobs
  • 53. HIVE + INTERACTIVE: WATCH OUT FOR FRAGMENTATION Solutions for fragmentation… • Reserve interactive nodes before starting batch jobs • Reduce interactive container size (if the algorithm permits) • Node labels (YARN-726) and gang scheduling (YARN-624)
  • 54. CONCLUSIONS • Hive + Hadoop debugging can get very complex o Sifting through many logs and screens o Automatic transmission versus manual transmission • Static partitioning induced by Java Virtual Machine has benefits but also induces challenges. • Where there are difficulties, there’s opportunity: o Better tooling, instrumentation, integration of logs/metrics • YARN still evolving into an operating system • Hadoop as a Service: aggregate and share expertise • Need to learn from the traditional database community!