Abstract image of a woman sitting on books and studying on a laptop

Debugging Hive with Hadoop-in-the-Cloud

Download as PPTX, PDF
Soam Acharya, profile picture
Soam Acharya

The document discusses debugging Hive within Hadoop in a cloud environment, provided by Altiscale, which includes various debugging tools, access to logs, and operational techniques for enhancing performance. It covers common issues such as out-of-memory errors, excessive task launches, and case studies involving stuck jobs and missing directories. The presentation also emphasizes the importance of effective monitoring and logging to resolve Hive-related challenges efficiently.

Data & Analytics
1 of 51
Download to read offline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
DEBUGGING HIVE WITH HADOOP IN THE CLOUD Soam Acharya, David Chaiken, Denis Sheahan, Charles Wimmer Altiscale, Inc. June 4, 2014
WHO ARE WE? • Altiscale: Infrastructure Nerds! • Hadoop As A Service • Rack and build our own Hadoop clusters • Provide a suite of Hadoop tools o Hive, Pig, Oozie o Others as needed: R, Python, Spark, Mahout, Impala, etc. • Monthly billing plan: compute, storage • https://www.altiscale.com • @Altiscale #HadoopSherpa
TALK ROADMAP • Our Platform and Perspective • Hadoop 2 Primer • Hadoop Debugging Tools • Accessing Logs in Hadoop 2 • Hive + Hadoop Architecture • Hive Logs • Hive Issues + Case Studies • Conclusion: Making Hive Easier to Use
OUR DYNAMIC PLATFORM • Hadoop 2.0.5 => Hadoop 2.2.0 • Hive 0.10 => Hive 0.12 • Hive, Pig and Oozie most commonly used tools • Working with customers on: Spark, Stinger (Hive 0.13 + Tez), Impala, …
ALTISCALE PERSPECTIVE • Service provider o Hadoop Dialtone! o Keep Hadoop/Hive + other tools running o Service Level Agreements target application-level metrics o Multiple clusters/customers o Operational scalability o Multi-tenancy • Operational approach o How to use Hadoop 2 cluster tools and logs to debug and to tune o This talk will not focus on query optimization
Hadoop 2 Cluster Name Node Hadoop Slave Hadoop Slave Hadoop Slave Resource Manager Secondary NameNode Hadoop Slave Node Managers + Data Nodes QUICK PRIMER – HADOOP 2
QUICK PRIMER – HADOOP 2 YARN • Resource Manager (per cluster) o Manages job scheduling and execution o Global resource allocation • Application Master (per job) o Manages task scheduling and execution o Local resource allocation • Node Manager (per-machine agent) o Manages the lifecycle of task containers o Reports to RM on health and resource usage
HADOOP 1 VS HADOOP 2 • No more JobTrackers, TaskTrackers • YARN ~ Operating System for Clusters o MapReduce is implemented as a YARN application o Bring on the applications! (Spark is just the start…) • Should be Transparent to Hive users
HADOOP 2 DEBUGGING TOOLS • Monitoring o System state of cluster:  CPU, Memory, Network, Disk  Nagios, Ganglia, Sensu!  Collectd, statd, Graphite o Hadoop level  HDFS usage  Resource usage: • Container memory allocated vs used • # of jobs running at the same time • Long running tasks
HADOOP 2 DEBUGGING TOOLS • Hadoop logs o Daemon logs: Resource Manager, NameNode, DataNode o Application logs: Application Master, MapReduce tasks o Job history file: resources allocated during job lifetime o Application configuration files: store all Hadoop application parameters • Source code instrumentation
Debugging Hive with Hadoop-in-the-Cloud
ACCESSING LOGS IN HADOOP 2 • To view the logs for a job, click on the link under the ID column in Resource Manager UI.
ACCESSING LOGS IN HADOOP 2 • To view application top level logs, click on logs. • To view individual logs for the mappers and reducers, click on History.
ACCESSING LOGS IN HADOOP 2 • Log output for the entire application.
ACCESSING LOGS IN HADOOP 2 • Click on the Map link for mapper logs and the Reduce link for reducer logs.
ACCESSING LOGS IN HADOOP 2 • Clicking on a single link under Name provides an overview for that particular map job.
ACCESSING LOGS IN HADOOP 2 • Finally, clicking on the logs link will take you to the log output for that map job.
ACCESSING LOGS IN HADOOP 2 • Fun, fun, donuts, and more fun…
HIVE + HADOOP 2 ARCHITECTURE • Hive 0.10+ Hadoop 2 Cluster Hive CLI Hive Metastore HiveserverJDBC/ODBC Kettle, Hue, …
HIVE LOGS • Query Log location • From /etc/hive/hive-site.xml: <property> <name>hive.querylog.location</name> <value>/home/hive/log/${user.name}</value> </property> SessionStart SESSION_ID="soam_201402032341" TIME="1391470900594"
HIVE CLIENT LOGS • /etc/hive/hive-log4j.properties: o hive.log.dir=/var/log/hive/${user.name} 2014-05-29 19:51:09,830 INFO parse.ParseDriver (ParseDriver.java:parse(179)) - Parsing command: select count(*) from dogfood_job_data 2014-05-29 19:51:09,852 INFO parse.ParseDriver (ParseDriver.java:parse(197)) - Parse Completed 2014-05-29 19:51:09,852 INFO ql.Driver (PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=parse start=1401393069830 end=1401393069852 duration=22> 2014-05-29 19:51:09,853 INFO ql.Driver (PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=semanticAnalyze> 2014-05-29 19:51:09,890 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(8305)) - Starting Semantic Analysis 2014-05-29 19:51:09,892 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(8340)) - Completed phase 1 of Semantic Analysis 2014-05-29 19:51:09,892 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1060)) - Get metadata for source tables 2014-05-29 19:51:09,906 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1167)) - Get metadata for subqueries 2014-05-29 19:51:09,909 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1187)) - Get metadata for destination tables
HIVE METASTORE LOGS • /etc/hive-metastore/hive-log4j.properties: o hive.log.dir=/service/log/hive-metastore/${user.name} 2014-05-29 19:50:50,179 INFO metastore.HiveMetaStore (HiveMetaStore.java:logInfo(454)) - 200: source:/10.252.18.94 get_table : db=default tbl=dogfood_job_data 2014-05-29 19:50:50,180 INFO HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(239)) - ugi=chaiken ip=/10.252.18.94 cmd=source:/10.252.18.94 get_table : db=default tbl=dogfood_job_data 2014-05-29 19:50:50,236 INFO metastore.HiveMetaStore (HiveMetaStore.java:logInfo(454)) - 200: source:/10.252.18.94 get_table : db=default tbl=dogfood_job_data 2014-05-29 19:50:50,236 INFO HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(239)) - ugi=chaiken ip=/10.252.18.94 cmd=source:/10.252.18.94 get_table : db=default tbl=dogfood_job_data 2014-05-29 19:50:50,261 INFO metastore.HiveMetaStore (HiveMetaStore.java:logInfo(454)) - 200: source:/10.252.18.94 get_table : db=default tbl=dogfood_job_data
HIVE ISSUES + CASE STUDIES • Hive Issues o Hive client out of memory o Hive map/reduce task out of memory o Hive metastore out of memory o Hive launches too many tasks • Case Studies: o Hive “stuck” job o Hive “missing directories” o Analyze Hive Query Execution
HIVE CLIENT OUT OF MEMORY • Memory intensive client side hive query (map-side join) Number of reduce tasks not specified. Estimated from input data size: 999 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapred.reduce.tasks=<number> java.lang.OutOfMemoryError: Java heap space at java.nio.CharBuffer.wrap(CharBuffer.java:350) at java.nio.CharBuffer.wrap(CharBuffer.java:373) at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:138)
HIVE CLIENT OUT OF MEMORY • Use HADOOP_HEAPSIZE prior to launching Hive client • HADOOP_HEAPSIZE=<new heapsize> hive <fileName> • Watch out for HADOOP_CLIENT_OPTS issue in hive-env.sh! • Important to know the amount of memory available on machine running client… Do not exceed or use disproportionate amount. $ free -m total used free shared buffers cached Mem: 1695 1388 306 0 60 424 -/+ buffers/cache: 903 791 Swap: 895 101 794
HIVE TASK OUT OF MEMORY • Query spawns MapReduce jobs that run out of memory • How to find this issue? o Hive diagnostic message o Hadoop MapReduce logs
HIVE TASK OUT OF MEMORY • Fix is to increase task RAM allocation… set mapreduce.map.memory.mb=<new RAM allocation>; set mapreduce.reduce.memory.mb=<new RAM allocation>; • Also watch out for… set mapreduce.map.java.opts=-Xmx<heap size>m; set mapreduce.reduce.java.opts=-Xmx<heap size>m; • Not a magic bullet – requires manual tuning • Increase in individual container memory size: o Decrease in overall containers that can be run o Decrease in overall parallelism
HIVE METASTORE OUT OF MEMORY • Out of memory issues not necessarily dumped to logs • Metastore can become unresponsive • Can’t submit queries • Restart with a higher heap size: export HADOOP_HEAPSIZE in hcat_server.sh • After notifying hive users about downtime: service hcat restart
HIVE LAUNCHES TOO MANY TASKS • Typically a function of the input data set • Lots of little files
HIVE LAUNCHES TOO MANY TASKS • Set mapred.max.split.size to appropriate fraction of data size • Also verify that hive.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat
CASE STUDY: HIVE STUCK JOB From an Altiscale customer: “This job [jobid] has been running now for 41 hours. Is it still progressing or has something hung up the map/reduce so it’s just spinning? Do you have any insight?”
HIVE STUCK JOB 1. Received jobId, application_1382973574141_4536, from client 2. Logged into client cluster. 3. Pulled up Resource Manager 4. Entered part of jobId (4536) in the search box. 5. Clicked on the link that says: application_1382973574141_4536 6. On resulting Application Overview page, clicked on link next to “Tracking URL” that said Application Master
HIVE STUCK JOB 7. On resulting MapReduce Application page, we clicked on the Job Id (job_1382973574141_4536). 8. The resulting MapReduce Job page displayed detailed status of the mappers, including 4 failed mappers 9. We then clicked on the 4 link on the Maps row in the Failed column. 10. Title of the next page was “FAILED Map attempts in job_1382973574141_4536.” 11. Each failed mapper generated an error message. 12. Buried in the 16th line: Caused by: java.io.FileNotFoundException: File does not exist: hdfs://opaque_hostname:8020/HiveTableDir/FileNa me.log.date.seq
HIVE STUCK JOB • Job was stuck for a day or so, retrying a mapper that would never finish successfully. • During the job, our customers’ colleague realized input file was corrupted and deleted it. • Colleague did not anticipate the affect of removing corrupted data on a running job • Hadoop didn’t make it easy to find out: o RM => search => application link => AM overview page => MR Application Page => MR Job Page => Failed jobs page => parse long logs o Task retry without hope of success
HIVE “MISSING DIRECTORIES” From an Altiscale customer: “One problem we are seeing after the [Hive Metastore] restart is that we lost quite a few directories in [HDFS]. Is there a way to recover these?”
HIVE “MISSING DIRECTORIES” • Obtained list of “missing” directories from customer: o /hive/biz/prod/* • Confirmed they were missing from HDFS • Searched through NameNode audit log to get block IDs that belonged to missing directories. 13/07/24 21:10:08 INFO hdfs.StateChange: BLOCK* NameSystem.allocateBlock: /hive/biz/prod/incremental/carryoverstore/postdepuis /lmt_unmapped_pggroup_schema._COPYING_. BP- 798113632-10.251.255.251-1370812162472 blk_3560522076897293424_2448396{blockUCState=UNDER_C ONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[10.251.255.177:50 010|RBW], ReplicaUnderConstruction[10.251.255.174:50010|RBW], ReplicaUnderConstruction[10.251.255.169:50010|RBW]]}
HIVE “MISSING DIRECTORIES” • Used blockID to locate exact time of file deletion from Namenode logs: 13/07/31 08:10:33 INFO hdfs.StateChange: BLOCK* addToInvalidates: blk_3560522076897293424_2448396 to 10.251.255.177:50010 10.251.255.169:50010 10.251.255.174:50010 • Used time of deletion to inspect hive logs
HIVE “MISSING DIRECTORIES” QueryStart QUERY_STRING="create database biz_weekly location '/hive/biz/prod'" QUERY_ID=“usrprod_20130731043232_0a40fd32-8c8a-479c- ba7d-3bd8a2698f4b" TIME="1375245164667" : QueryEnd QUERY_STRING="create database biz_weekly location '/hive/biz/prod'" QUERY_ID=”usrprod_20130731043232_0a40fd32-8c8a-479c- ba7d-3bd8a2698f4b" QUERY_RET_CODE="0" QUERY_NUM_TASKS="0" TIME="1375245166203" : QueryStart QUERY_STRING="drop database biz_weekly" QUERY_ID=”usrprod_20130731073333_e9acf35c-4f07-4f12-bd9d-bae137ae0733" TIME="1375256014799" : QueryEnd QUERY_STRING="drop database biz_weekly" QUERY_ID=”usrprod_20130731073333_e9acf35c-4f07-4f12-bd9d-bae137ae0733" QUERY_NUM_TASKS="0" TIME="1375256014838"
HIVE “MISSING DIRECTORIES” • In effect, user “usrprod” issued: At 2013-07-31 04:32:44: create database biz_weekly location '/hive/biz/prod' At 2013-07-31 07:33:24: drop database biz_weekly • This is functionally equivalent to: hdfs dfs -rm -r /hive/biz/prod
HIVE “MISSING DIRECTORIES” • Customer manually placed their own data in /hive – the warehouse directory managed and controlled by hive • Customer used CREATE and DROP db commands in their code o Hive deletes database and table locations in /hive with impunity • Why didn’t deleted data end up in .Trash? o Trash collection not turned on in configuration settings o It is now, but need a –skipTrash option (HIVE-6469)
HIVE “MISSING DIRECTORIES” • Hadoop forensics: piece together disparate sources… o Hadoop daemon logs (NameNode) o Hive query and metastore logs o Hadoop config files • Need better tools to correlate the different layers of the system: hive client, hive metastore, MapReduce job, YARN, HDFS, operating sytem metrics, … By the way… Operating any distributed system would be totally insane without NTP and a standard time zone (UTC).
CASE STUDY – ANALYZE QUERY • Customer provided Hive query + data sets (100GBs to ~5 TBs) • Needed help optimizing the query • Didn’t rewrite query immediately • Wanted to characterize query performance and isolate bottlenecks first
ANALYZE AND TUNE EXECUTION • Ran original query on the datasets in our environment: o Two M/R Stages: Stage-1, Stage-2 • Long running reducers run out of memory o set mapreduce.reduce.memory.mb=5120 o Reduces slots and extends reduce time • Query fails to launch Stage-2 with out of memory o set HADOOP_HEAPSIZE=1024 on client machine • Query has 250,000 Mappers in Stage-2 which causes failure o set mapred.max.split.size=5368709120 to reduce Mappers
ANALYSIS: HOW TO VISUALIZE? • Next challenge - how to visualize job execution? • Existing hadoop/hive logs not sufficient for this task • Wrote internal tools o parse job history files o plot mapper and reducer execution
ANALYSIS: MAP STAGE-1
Single reduce task ANALYSIS: REDUCE STAGE-1
ANALYSIS: MAP STAGE-2
ANALYSIS: REDUCE STAGE-2
ANALYZE EXECUTION: FINDINGS • Lone, long running reducer in first stage of query • Analyzed input data: o Query split input data by userId o Bucketizing input data by userId o One very large bucket: “invalid” userId o Discussed “invalid” userid with customer • An error value is a common pattern! o Need to differentiate between “Don’t know and don’t care” or “don’t know and do care.”
CONCLUSIONS • Hive + Hadoop debugging can get very complex o Sifting through many logs and screens o Automatic transmission versus manual transmission • Static partitioning induced by Java Virtual Machine has benefits but also induces challenges. • Where there are difficulties, there’s opportunity o Better tooling o Better instrumentation o Better integration of disparate logs and metrics • Hadoop as a Service: aggregate and share expertise • Need to learn from the traditional database community!
QUESTIONS? COMMENTS?

More Related Content

PPTX
Hadoop or Spark: is it an either-or proposition? By Slim Baltagi
Slim Baltagi
 
PDF
20131205 hadoop-hdfs-map reduce-introduction
Xuan-Chao Huang
 
PPTX
Introduction To Big Data with Hadoop and Spark - For Batch and Real Time Proc...
Agile Testing Alliance
 
PDF
Building a Big Data platform with the Hadoop ecosystem
Gregg Barrett
 
PDF
Introduction to Hadoop and MapReduce
eakasit_dpu
 
PPTX
Hands On: Introduction to the Hadoop Ecosystem
Adaryl "Bob" Wakefield, MBA
 
PPTX
Top Hadoop Big Data Interview Questions and Answers for Fresher
JanBask Training
 
PDF
Big data Hadoop Analytic and Data warehouse comparison guide
Danairat Thanabodithammachari
 
Hadoop or Spark: is it an either-or proposition? By Slim Baltagi
Slim Baltagi
 
20131205 hadoop-hdfs-map reduce-introduction
Xuan-Chao Huang
 
Introduction To Big Data with Hadoop and Spark - For Batch and Real Time Proc...
Agile Testing Alliance
 
Building a Big Data platform with the Hadoop ecosystem
Gregg Barrett
 
Introduction to Hadoop and MapReduce
eakasit_dpu
 
Hands On: Introduction to the Hadoop Ecosystem
Adaryl "Bob" Wakefield, MBA
 
Top Hadoop Big Data Interview Questions and Answers for Fresher
JanBask Training
 
Big data Hadoop Analytic and Data warehouse comparison guide
Danairat Thanabodithammachari
 

What's hot (20)

PPTX
Hadoop from Hive with Stinger to Tez
Jan Pieter Posthuma
 
PPTX
Hadoop Summit 2015: Hive at Yahoo: Letters from the Trenches
Mithun Radhakrishnan
 
PPTX
Introduction to the Hadoop EcoSystem
Shivaji Dutta
 
PPTX
Big data Hadoop
Ayyappan Paramesh
 
PDF
Scaling up with hadoop and banyan at ITRIX-2015, College of Engineering, Guindy
Rohit Kulkarni
 
PPTX
Demystify Big Data Breakfast Briefing: Herb Cunitz, Hortonworks
Hortonworks
 
PPTX
Faster Faster Faster! Datamarts with Hive at Yahoo
Mithun Radhakrishnan
 
PDF
Common and unique use cases for Apache Hadoop
Brock Noland
 
PPTX
Big Data and Hadoop Introduction
Dzung Nguyen
 
PDF
Introduction To Hadoop Ecosystem
InSemble
 
PDF
Introduction to the Hadoop Ecosystem (FrOSCon Edition)
Uwe Printz
 
PDF
DBA to Data Scientist
pasalapudi
 
PPTX
Hive at Yahoo: Letters from the trenches
DataWorks Summit
 
PDF
Dba to data scientist -Satyendra
pasalapudi123
 
PPTX
20140228 - Singapore - BDAS - Ensuring Hadoop Production Success
Allen Day, PhD
 
PPTX
Data warehousing with Hadoop
hadooparchbook
 
PPTX
August 2016 HUG: Recent development in Apache Oozie
Yahoo Developer Network
 
PPTX
Hadoop in the Cloud: Real World Lessons from Enterprise Customers
DataWorks Summit/Hadoop Summit
 
PPTX
Data Pipelines in Hadoop - SAP Meetup in Tel Aviv
larsgeorge
 
PPTX
Apache Hadoop at 10
Cloudera, Inc.
 
Hadoop from Hive with Stinger to Tez
Jan Pieter Posthuma
 
Hadoop Summit 2015: Hive at Yahoo: Letters from the Trenches
Mithun Radhakrishnan
 
Introduction to the Hadoop EcoSystem
Shivaji Dutta
 
Big data Hadoop
Ayyappan Paramesh
 
Scaling up with hadoop and banyan at ITRIX-2015, College of Engineering, Guindy
Rohit Kulkarni
 
Demystify Big Data Breakfast Briefing: Herb Cunitz, Hortonworks
Hortonworks
 
Faster Faster Faster! Datamarts with Hive at Yahoo
Mithun Radhakrishnan
 
Common and unique use cases for Apache Hadoop
Brock Noland
 
Big Data and Hadoop Introduction
Dzung Nguyen
 
Introduction To Hadoop Ecosystem
InSemble
 
Introduction to the Hadoop Ecosystem (FrOSCon Edition)
Uwe Printz
 
DBA to Data Scientist
pasalapudi
 
Hive at Yahoo: Letters from the trenches
DataWorks Summit
 
Dba to data scientist -Satyendra
pasalapudi123
 
20140228 - Singapore - BDAS - Ensuring Hadoop Production Success
Allen Day, PhD
 
Data warehousing with Hadoop
hadooparchbook
 
August 2016 HUG: Recent development in Apache Oozie
Yahoo Developer Network
 
Hadoop in the Cloud: Real World Lessons from Enterprise Customers
DataWorks Summit/Hadoop Summit
 
Data Pipelines in Hadoop - SAP Meetup in Tel Aviv
larsgeorge
 
Apache Hadoop at 10
Cloudera, Inc.
 
Ad

Similar to Debugging Hive with Hadoop-in-the-Cloud (20)

PDF
OC Big Data Monthly Meetup #5 - Session 1 - Altiscale
Big Data Joe™ Rossi
 
PDF
Debugging Hive with Hadoop-in-the-Cloud by David Chaiken of Altiscale
Data Con LA
 
PDF
Running Hadoop as Service in AltiScale Platform
InMobi Technology
 
PPTX
Getting started big data
Kibrom Gebrehiwot
 
PDF
Chapter 10
pavan penugonda
 
PPTX
Managing growth in Production Hadoop Deployments
DataWorks Summit
 
PPTX
Data Analytics and IoT, how to analyze data from IoT
AmmarHassan80
 
PDF
Hd insight essentials quick view
Rajesh Nadipalli
 
PDF
Hd insight essentials quick view
Rajesh Nadipalli
 
PDF
HdInsight essentials Hadoop on Microsoft Platform
nvvrajesh
 
PPTX
First Hive Meetup London 2012-07-10 - Tomas Cervenka - VisualDNA
Tomas Cervenka
 
PPTX
BIG DATA ANALYSIS
Nitesh Singh
 
PDF
Big Data Developers Moscow Meetup 1 - sql on hadoop
bddmoscow
 
PPT
2.2 Hadoop ecosystem -Distribution.ppt
sec22ci043
 
PDF
Data Engineering with Spring, Hadoop and Hive
Alex Silva
 
PPTX
Rapid Development of Big Data applications using Spring for Apache Hadoop
zenyk
 
PPTX
Hackathon bonn
Emil Andreas Siemes
 
PPT
ha_module5
Gurmukh Singh
 
PDF
Hadoop content
Hadoop online training
 
PPTX
Containerdays Intro to Habitat
Mandi Walls
 
OC Big Data Monthly Meetup #5 - Session 1 - Altiscale
Big Data Joe™ Rossi
 
Debugging Hive with Hadoop-in-the-Cloud by David Chaiken of Altiscale
Data Con LA
 
Running Hadoop as Service in AltiScale Platform
InMobi Technology
 
Getting started big data
Kibrom Gebrehiwot
 
Chapter 10
pavan penugonda
 
Managing growth in Production Hadoop Deployments
DataWorks Summit
 
Data Analytics and IoT, how to analyze data from IoT
AmmarHassan80
 
Hd insight essentials quick view
Rajesh Nadipalli
 
Hd insight essentials quick view
Rajesh Nadipalli
 
HdInsight essentials Hadoop on Microsoft Platform
nvvrajesh
 
First Hive Meetup London 2012-07-10 - Tomas Cervenka - VisualDNA
Tomas Cervenka
 
BIG DATA ANALYSIS
Nitesh Singh
 
Big Data Developers Moscow Meetup 1 - sql on hadoop
bddmoscow
 
2.2 Hadoop ecosystem -Distribution.ppt
sec22ci043
 
Data Engineering with Spring, Hadoop and Hive
Alex Silva
 
Rapid Development of Big Data applications using Spring for Apache Hadoop
zenyk
 
Hackathon bonn
Emil Andreas Siemes
 
ha_module5
Gurmukh Singh
 
Hadoop content
Hadoop online training
 
Containerdays Intro to Habitat
Mandi Walls
 
Ad

Recently uploaded (20)

PPT
PROJECT CYCLE MANAGEMENT FRAMEWORK (PCM).ppt
GebrelibanosYowhans
 
PPTX
GPS sensor used agriculture land for automation
masanimasani2020
 
PPTX
865628565-Pertemuan-2-chapter-03-NUMERICAL-MEASURES.pptx
DibyenduRoy49
 
PPTX
ai agent creaction with langgraph_presentation_
RithikLogu1
 
PPTX
DATA MODELING, data model concepts, types of data concepts
Shiwangi sharma
 
PPTX
PPT for Diseases.pptx, there are 3 types of diseases
salilbansal7
 
PPTX
transformers as a tool for understanding advance algorithms in deep learning
ajay77470
 
PPT
dsa Lec-1 Introduction FOR THE STUDENTS OF bscs
MuddassarAli6
 
PDF
CS3352FOUNDATION OF DATA SCIENCE _1_MAterial.pdf
mohanapriya462996
 
PPTX
indiraparyavaranbhavan-240418134200-31d840b3.pptx
janbenijami
 
PPTX
recommendation Project PPT with details attached
AlanChen266674
 
PPTX
AI_Agriculture_Presentation_Enhanced.pptx
ssb1984
 
PPT
statistic analysis for study - data collection
NguynNgcBchTrm8
 
PPTX
inbound6529290805104538764.pptxmmmmmmmmm
hacheldelrosario
 
PPTX
machinelearningoverview-250809184828-927201d2.pptx
AnujChetry
 
PPTX
1 hour to get there before the game is done so you don’t need a car seat for ...
atharv94071
 
PDF
Session 11 - Data Visualization Storytelling (2).pdf
VNgcUynPhng
 
PPTX
ifsm.pptx, institutional food service management
salilbansal7
 
PPTX
cp-and-safeguarding-training-2018-2019-mmfv2-230818062456-767bc1a7.pptx
rachellegalit2
 
PDF
Grey Minimalist Professional Project Presentation (1).pdf
abhilashhari77
 
PROJECT CYCLE MANAGEMENT FRAMEWORK (PCM).ppt
GebrelibanosYowhans
 
GPS sensor used agriculture land for automation
masanimasani2020
 
865628565-Pertemuan-2-chapter-03-NUMERICAL-MEASURES.pptx
DibyenduRoy49
 
ai agent creaction with langgraph_presentation_
RithikLogu1
 
DATA MODELING, data model concepts, types of data concepts
Shiwangi sharma
 
PPT for Diseases.pptx, there are 3 types of diseases
salilbansal7
 
transformers as a tool for understanding advance algorithms in deep learning
ajay77470
 
dsa Lec-1 Introduction FOR THE STUDENTS OF bscs
MuddassarAli6
 
CS3352FOUNDATION OF DATA SCIENCE _1_MAterial.pdf
mohanapriya462996
 
indiraparyavaranbhavan-240418134200-31d840b3.pptx
janbenijami
 
recommendation Project PPT with details attached
AlanChen266674
 
AI_Agriculture_Presentation_Enhanced.pptx
ssb1984
 
statistic analysis for study - data collection
NguynNgcBchTrm8
 
inbound6529290805104538764.pptxmmmmmmmmm
hacheldelrosario
 
machinelearningoverview-250809184828-927201d2.pptx
AnujChetry
 
1 hour to get there before the game is done so you don’t need a car seat for ...
atharv94071
 
Session 11 - Data Visualization Storytelling (2).pdf
VNgcUynPhng
 
ifsm.pptx, institutional food service management
salilbansal7
 
cp-and-safeguarding-training-2018-2019-mmfv2-230818062456-767bc1a7.pptx
rachellegalit2
 
Grey Minimalist Professional Project Presentation (1).pdf
abhilashhari77
 

Debugging Hive with Hadoop-in-the-Cloud

  • 1. DEBUGGING HIVE WITH HADOOP IN THE CLOUD Soam Acharya, David Chaiken, Denis Sheahan, Charles Wimmer Altiscale, Inc. June 4, 2014
  • 2. WHO ARE WE? • Altiscale: Infrastructure Nerds! • Hadoop As A Service • Rack and build our own Hadoop clusters • Provide a suite of Hadoop tools o Hive, Pig, Oozie o Others as needed: R, Python, Spark, Mahout, Impala, etc. • Monthly billing plan: compute, storage • https://www.altiscale.com • @Altiscale #HadoopSherpa
  • 3. TALK ROADMAP • Our Platform and Perspective • Hadoop 2 Primer • Hadoop Debugging Tools • Accessing Logs in Hadoop 2 • Hive + Hadoop Architecture • Hive Logs • Hive Issues + Case Studies • Conclusion: Making Hive Easier to Use
  • 4. OUR DYNAMIC PLATFORM • Hadoop 2.0.5 => Hadoop 2.2.0 • Hive 0.10 => Hive 0.12 • Hive, Pig and Oozie most commonly used tools • Working with customers on: Spark, Stinger (Hive 0.13 + Tez), Impala, …
  • 5. ALTISCALE PERSPECTIVE • Service provider o Hadoop Dialtone! o Keep Hadoop/Hive + other tools running o Service Level Agreements target application-level metrics o Multiple clusters/customers o Operational scalability o Multi-tenancy • Operational approach o How to use Hadoop 2 cluster tools and logs to debug and to tune o This talk will not focus on query optimization
  • 6. Hadoop 2 Cluster Name Node Hadoop Slave Hadoop Slave Hadoop Slave Resource Manager Secondary NameNode Hadoop Slave Node Managers + Data Nodes QUICK PRIMER – HADOOP 2
  • 7. QUICK PRIMER – HADOOP 2 YARN • Resource Manager (per cluster) o Manages job scheduling and execution o Global resource allocation • Application Master (per job) o Manages task scheduling and execution o Local resource allocation • Node Manager (per-machine agent) o Manages the lifecycle of task containers o Reports to RM on health and resource usage
  • 8. HADOOP 1 VS HADOOP 2 • No more JobTrackers, TaskTrackers • YARN ~ Operating System for Clusters o MapReduce is implemented as a YARN application o Bring on the applications! (Spark is just the start…) • Should be Transparent to Hive users
  • 9. HADOOP 2 DEBUGGING TOOLS • Monitoring o System state of cluster:  CPU, Memory, Network, Disk  Nagios, Ganglia, Sensu!  Collectd, statd, Graphite o Hadoop level  HDFS usage  Resource usage: • Container memory allocated vs used • # of jobs running at the same time • Long running tasks
  • 10. HADOOP 2 DEBUGGING TOOLS • Hadoop logs o Daemon logs: Resource Manager, NameNode, DataNode o Application logs: Application Master, MapReduce tasks o Job history file: resources allocated during job lifetime o Application configuration files: store all Hadoop application parameters • Source code instrumentation
  • 12. ACCESSING LOGS IN HADOOP 2 • To view the logs for a job, click on the link under the ID column in Resource Manager UI.
  • 13. ACCESSING LOGS IN HADOOP 2 • To view application top level logs, click on logs. • To view individual logs for the mappers and reducers, click on History.
  • 14. ACCESSING LOGS IN HADOOP 2 • Log output for the entire application.
  • 15. ACCESSING LOGS IN HADOOP 2 • Click on the Map link for mapper logs and the Reduce link for reducer logs.
  • 16. ACCESSING LOGS IN HADOOP 2 • Clicking on a single link under Name provides an overview for that particular map job.
  • 17. ACCESSING LOGS IN HADOOP 2 • Finally, clicking on the logs link will take you to the log output for that map job.
  • 18. ACCESSING LOGS IN HADOOP 2 • Fun, fun, donuts, and more fun…
  • 19. HIVE + HADOOP 2 ARCHITECTURE • Hive 0.10+ Hadoop 2 Cluster Hive CLI Hive Metastore HiveserverJDBC/ODBC Kettle, Hue, …
  • 20. HIVE LOGS • Query Log location • From /etc/hive/hive-site.xml: <property> <name>hive.querylog.location</name> <value>/home/hive/log/${user.name}</value> </property> SessionStart SESSION_ID="soam_201402032341" TIME="1391470900594"
  • 21. HIVE CLIENT LOGS • /etc/hive/hive-log4j.properties: o hive.log.dir=/var/log/hive/${user.name} 2014-05-29 19:51:09,830 INFO parse.ParseDriver (ParseDriver.java:parse(179)) - Parsing command: select count(*) from dogfood_job_data 2014-05-29 19:51:09,852 INFO parse.ParseDriver (ParseDriver.java:parse(197)) - Parse Completed 2014-05-29 19:51:09,852 INFO ql.Driver (PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=parse start=1401393069830 end=1401393069852 duration=22> 2014-05-29 19:51:09,853 INFO ql.Driver (PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=semanticAnalyze> 2014-05-29 19:51:09,890 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(8305)) - Starting Semantic Analysis 2014-05-29 19:51:09,892 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(8340)) - Completed phase 1 of Semantic Analysis 2014-05-29 19:51:09,892 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1060)) - Get metadata for source tables 2014-05-29 19:51:09,906 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1167)) - Get metadata for subqueries 2014-05-29 19:51:09,909 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1187)) - Get metadata for destination tables
  • 22. HIVE METASTORE LOGS • /etc/hive-metastore/hive-log4j.properties: o hive.log.dir=/service/log/hive-metastore/${user.name} 2014-05-29 19:50:50,179 INFO metastore.HiveMetaStore (HiveMetaStore.java:logInfo(454)) - 200: source:/10.252.18.94 get_table : db=default tbl=dogfood_job_data 2014-05-29 19:50:50,180 INFO HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(239)) - ugi=chaiken ip=/10.252.18.94 cmd=source:/10.252.18.94 get_table : db=default tbl=dogfood_job_data 2014-05-29 19:50:50,236 INFO metastore.HiveMetaStore (HiveMetaStore.java:logInfo(454)) - 200: source:/10.252.18.94 get_table : db=default tbl=dogfood_job_data 2014-05-29 19:50:50,236 INFO HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(239)) - ugi=chaiken ip=/10.252.18.94 cmd=source:/10.252.18.94 get_table : db=default tbl=dogfood_job_data 2014-05-29 19:50:50,261 INFO metastore.HiveMetaStore (HiveMetaStore.java:logInfo(454)) - 200: source:/10.252.18.94 get_table : db=default tbl=dogfood_job_data
  • 23. HIVE ISSUES + CASE STUDIES • Hive Issues o Hive client out of memory o Hive map/reduce task out of memory o Hive metastore out of memory o Hive launches too many tasks • Case Studies: o Hive “stuck” job o Hive “missing directories” o Analyze Hive Query Execution
  • 24. HIVE CLIENT OUT OF MEMORY • Memory intensive client side hive query (map-side join) Number of reduce tasks not specified. Estimated from input data size: 999 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapred.reduce.tasks=<number> java.lang.OutOfMemoryError: Java heap space at java.nio.CharBuffer.wrap(CharBuffer.java:350) at java.nio.CharBuffer.wrap(CharBuffer.java:373) at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:138)
  • 25. HIVE CLIENT OUT OF MEMORY • Use HADOOP_HEAPSIZE prior to launching Hive client • HADOOP_HEAPSIZE=<new heapsize> hive <fileName> • Watch out for HADOOP_CLIENT_OPTS issue in hive-env.sh! • Important to know the amount of memory available on machine running client… Do not exceed or use disproportionate amount. $ free -m total used free shared buffers cached Mem: 1695 1388 306 0 60 424 -/+ buffers/cache: 903 791 Swap: 895 101 794
  • 26. HIVE TASK OUT OF MEMORY • Query spawns MapReduce jobs that run out of memory • How to find this issue? o Hive diagnostic message o Hadoop MapReduce logs
  • 27. HIVE TASK OUT OF MEMORY • Fix is to increase task RAM allocation… set mapreduce.map.memory.mb=<new RAM allocation>; set mapreduce.reduce.memory.mb=<new RAM allocation>; • Also watch out for… set mapreduce.map.java.opts=-Xmx<heap size>m; set mapreduce.reduce.java.opts=-Xmx<heap size>m; • Not a magic bullet – requires manual tuning • Increase in individual container memory size: o Decrease in overall containers that can be run o Decrease in overall parallelism
  • 28. HIVE METASTORE OUT OF MEMORY • Out of memory issues not necessarily dumped to logs • Metastore can become unresponsive • Can’t submit queries • Restart with a higher heap size: export HADOOP_HEAPSIZE in hcat_server.sh • After notifying hive users about downtime: service hcat restart
  • 29. HIVE LAUNCHES TOO MANY TASKS • Typically a function of the input data set • Lots of little files
  • 30. HIVE LAUNCHES TOO MANY TASKS • Set mapred.max.split.size to appropriate fraction of data size • Also verify that hive.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat
  • 31. CASE STUDY: HIVE STUCK JOB From an Altiscale customer: “This job [jobid] has been running now for 41 hours. Is it still progressing or has something hung up the map/reduce so it’s just spinning? Do you have any insight?”
  • 32. HIVE STUCK JOB 1. Received jobId, application_1382973574141_4536, from client 2. Logged into client cluster. 3. Pulled up Resource Manager 4. Entered part of jobId (4536) in the search box. 5. Clicked on the link that says: application_1382973574141_4536 6. On resulting Application Overview page, clicked on link next to “Tracking URL” that said Application Master
  • 33. HIVE STUCK JOB 7. On resulting MapReduce Application page, we clicked on the Job Id (job_1382973574141_4536). 8. The resulting MapReduce Job page displayed detailed status of the mappers, including 4 failed mappers 9. We then clicked on the 4 link on the Maps row in the Failed column. 10. Title of the next page was “FAILED Map attempts in job_1382973574141_4536.” 11. Each failed mapper generated an error message. 12. Buried in the 16th line: Caused by: java.io.FileNotFoundException: File does not exist: hdfs://opaque_hostname:8020/HiveTableDir/FileNa me.log.date.seq
  • 34. HIVE STUCK JOB • Job was stuck for a day or so, retrying a mapper that would never finish successfully. • During the job, our customers’ colleague realized input file was corrupted and deleted it. • Colleague did not anticipate the affect of removing corrupted data on a running job • Hadoop didn’t make it easy to find out: o RM => search => application link => AM overview page => MR Application Page => MR Job Page => Failed jobs page => parse long logs o Task retry without hope of success
  • 35. HIVE “MISSING DIRECTORIES” From an Altiscale customer: “One problem we are seeing after the [Hive Metastore] restart is that we lost quite a few directories in [HDFS]. Is there a way to recover these?”
  • 36. HIVE “MISSING DIRECTORIES” • Obtained list of “missing” directories from customer: o /hive/biz/prod/* • Confirmed they were missing from HDFS • Searched through NameNode audit log to get block IDs that belonged to missing directories. 13/07/24 21:10:08 INFO hdfs.StateChange: BLOCK* NameSystem.allocateBlock: /hive/biz/prod/incremental/carryoverstore/postdepuis /lmt_unmapped_pggroup_schema._COPYING_. BP- 798113632-10.251.255.251-1370812162472 blk_3560522076897293424_2448396{blockUCState=UNDER_C ONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[10.251.255.177:50 010|RBW], ReplicaUnderConstruction[10.251.255.174:50010|RBW], ReplicaUnderConstruction[10.251.255.169:50010|RBW]]}
  • 37. HIVE “MISSING DIRECTORIES” • Used blockID to locate exact time of file deletion from Namenode logs: 13/07/31 08:10:33 INFO hdfs.StateChange: BLOCK* addToInvalidates: blk_3560522076897293424_2448396 to 10.251.255.177:50010 10.251.255.169:50010 10.251.255.174:50010 • Used time of deletion to inspect hive logs
  • 38. HIVE “MISSING DIRECTORIES” QueryStart QUERY_STRING="create database biz_weekly location '/hive/biz/prod'" QUERY_ID=“usrprod_20130731043232_0a40fd32-8c8a-479c- ba7d-3bd8a2698f4b" TIME="1375245164667" : QueryEnd QUERY_STRING="create database biz_weekly location '/hive/biz/prod'" QUERY_ID=”usrprod_20130731043232_0a40fd32-8c8a-479c- ba7d-3bd8a2698f4b" QUERY_RET_CODE="0" QUERY_NUM_TASKS="0" TIME="1375245166203" : QueryStart QUERY_STRING="drop database biz_weekly" QUERY_ID=”usrprod_20130731073333_e9acf35c-4f07-4f12-bd9d-bae137ae0733" TIME="1375256014799" : QueryEnd QUERY_STRING="drop database biz_weekly" QUERY_ID=”usrprod_20130731073333_e9acf35c-4f07-4f12-bd9d-bae137ae0733" QUERY_NUM_TASKS="0" TIME="1375256014838"
  • 39. HIVE “MISSING DIRECTORIES” • In effect, user “usrprod” issued: At 2013-07-31 04:32:44: create database biz_weekly location '/hive/biz/prod' At 2013-07-31 07:33:24: drop database biz_weekly • This is functionally equivalent to: hdfs dfs -rm -r /hive/biz/prod
  • 40. HIVE “MISSING DIRECTORIES” • Customer manually placed their own data in /hive – the warehouse directory managed and controlled by hive • Customer used CREATE and DROP db commands in their code o Hive deletes database and table locations in /hive with impunity • Why didn’t deleted data end up in .Trash? o Trash collection not turned on in configuration settings o It is now, but need a –skipTrash option (HIVE-6469)
  • 41. HIVE “MISSING DIRECTORIES” • Hadoop forensics: piece together disparate sources… o Hadoop daemon logs (NameNode) o Hive query and metastore logs o Hadoop config files • Need better tools to correlate the different layers of the system: hive client, hive metastore, MapReduce job, YARN, HDFS, operating sytem metrics, … By the way… Operating any distributed system would be totally insane without NTP and a standard time zone (UTC).
  • 42. CASE STUDY – ANALYZE QUERY • Customer provided Hive query + data sets (100GBs to ~5 TBs) • Needed help optimizing the query • Didn’t rewrite query immediately • Wanted to characterize query performance and isolate bottlenecks first
  • 43. ANALYZE AND TUNE EXECUTION • Ran original query on the datasets in our environment: o Two M/R Stages: Stage-1, Stage-2 • Long running reducers run out of memory o set mapreduce.reduce.memory.mb=5120 o Reduces slots and extends reduce time • Query fails to launch Stage-2 with out of memory o set HADOOP_HEAPSIZE=1024 on client machine • Query has 250,000 Mappers in Stage-2 which causes failure o set mapred.max.split.size=5368709120 to reduce Mappers
  • 44. ANALYSIS: HOW TO VISUALIZE? • Next challenge - how to visualize job execution? • Existing hadoop/hive logs not sufficient for this task • Wrote internal tools o parse job history files o plot mapper and reducer execution
  • 49. ANALYZE EXECUTION: FINDINGS • Lone, long running reducer in first stage of query • Analyzed input data: o Query split input data by userId o Bucketizing input data by userId o One very large bucket: “invalid” userId o Discussed “invalid” userid with customer • An error value is a common pattern! o Need to differentiate between “Don’t know and don’t care” or “don’t know and do care.”
  • 50. CONCLUSIONS • Hive + Hadoop debugging can get very complex o Sifting through many logs and screens o Automatic transmission versus manual transmission • Static partitioning induced by Java Virtual Machine has benefits but also induces challenges. • Where there are difficulties, there’s opportunity o Better tooling o Better instrumentation o Better integration of disparate logs and metrics • Hadoop as a Service: aggregate and share expertise • Need to learn from the traditional database community!