HBase Internals And Operations

© 2019 Bloomberg Finance L.P. All rights reserved.
HBase Internals and
Operations
SRECon19 Asia/Pacific
June 13, 2019
Biju Nair
Software Engineer
bnair10@bloomberg.net

Agenda
• Introduction to HBase
• Operating HBase
• Questions

Bloomberg in a nutshell
The Bloomberg Terminal delivers a diverse
array of information on a single platform to
facilitate financial decision-making.

Bloomberg technology by the numbers
• 5,000+ software engineers
• 150+ technologists and data scientists devoted to machine learning
• One of the largest private networks in the world
• 120 billion pieces of data from the financial markets each day, with a peak of more than
10 million messages/second
• 2 million news stories ingested / published each day (500+ news stories ingested/second)
• News content from 125K+ sources
• Over 1 billion messages and Instant Bloomberg (IB) chats handled daily

HBase at Bloomberg
• Started with v0.94.6
• >2 billion reads per day
• >1 billion writes per day
• 51+ TB of compressed data stored in HBase

HBase Principles
• Ordered Key Value Store
• Distributed shared nothing

Key Value
Key-9999 Value-a
Key-9998 Value-b
Key-9997 Value-c
Key-9996 Value-d
…
…
Key-9995 Value-e
Key-9994 Value-f

Ordered Key Value
Key-9999 Value-a
Key-9998 Value-b
Key-9997 Value-c
Key-9996 Value-d
…
…
Key-9995 Value-e
Key-9994 Value-a
Lexicographicorder
Key-9993 Value-g

Ordered Key Value
Key-9999 Value-a
Key-9998 Value-b
Key-9997 Value-c
Key-9996 Value-d
…
…
Key-9995 Value-e
Key-9994 Value-a
Lexicographicorder
Key-9993 Value-g
Region

Distributed Ordered Key Value
Key-199 Value
Key-198 Value
Key-197 Value
Key-499 Value
…
Key-498 Value
Key-497 Value
ordered
…
Key-299 Value
…
Key-298 Value
Key-297 Value
ordered
…
…
…
Key-599 Value
…
Key-598 Value
Key-597 Value
…
Key-399 Value
…
Key-398 Value
Key-397 Value
…
Key-999 Value
…
Key-998 Value
Key-997 Value
…
Region
Region

Distributed Ordered Key Value
RegionServer
Key-199 Value
Key-198 Value
Key-197 Value
Key-499 Value
…
Key-498 Value
Key-497 Value
…
Key-299 Value
…
Key-298 Value
Key-297 Value
…
…
…
Key-599 Value
…
Key-598 Value
Key-597 Value
…
Key-399 Value
…
Key-398 Value
Key-397 Value
…
Key-999 Value
…
Key-998 Value
Key-997 Value
…
Region
Region
RegionServer RegionServer
RegionServer RegionServer RegionServer

Table Row View
Key Value
Column Id ValueRow Id Timestamp

Table Row View
R11|col1|1234567 Value-A
R11|col2|1234567 Value-B
R11|col3|1234567 Value-C
R11|col4|1234567 Value-D
R11
Value-A Value-B Value-C
Col1 Col2 Col3
Value-D
Col4

Versioning
R11|col1|1234567 Value-A1
R11|col3|1234567 Value-CC
R11|col4|1234567 Value-DD
R11|col4|1234560 Value-D1
R11|col4|1234557 Value-D
Descendingorder

cf2cf1
Column Family
R11
Value-A Value-B Value-C
cf1:col1 cf1:col2 cf1:col3
Value-1
cf2:col1
R11|col1|1234567 Value-1
R11|col2|1234567 Value-2
R11|col3|1234567 Value-3
Value-2
cf2:col2
Value-3
cf2:col3

ACIDity
• Atomic at row level
• Consistent to a point in time before the request
• Isolation through MVCC (reads) and row locks (mutations)
• Durability is guaranteed for all successful mutations

Namespace
Namespace: Blue Namespace: Yellow
ACL ACL
ACL ACL
ACL

HBase Write
HBase Region Server
File System
WAL
Write
1

HBase Write
HBase RS
Memstore
File System
WAL
Write
1
2

HBase Write
HBase RS
Memstore
File System
WAL t1 t1 t1
Store files
Write
1
2
3
Block Block Block
Idx BlmData

HBase Write
HBase RS
Memstore
DFSClient
WAL t1 t1 t1
Store files
Write
1
2
3

HBase Write
HBase RS
Memstore
DFSClient
WAL t1 t1 t1
Store files
Write
1
2
3
HDFS DataNode
WAL t1 t1 t1

Compaction
File System
K-x K-x D-1
Store files
K-xK-xK-x

Compaction
File System
Store files
K-x
Compaction
File System
K-x K-x D-1
Store files
K-xK-xK-x

HBase Read
HBase RS Memstore
File System
WAL
Read
1
Block Cache Block
t1 t1 t1
Store files

HBase Read
HBase RS Memstore
File System
WAL
Read
1
Block Cache
2
Block
t1 t1 t1
Store files

HDFS Read
HBase/
DFSClient
HDFS
File System
Data
TCP

HDFS Short-Circuit Read
HBase/
DFSClient
HDFS
File System
Data
TCP
Open
File
Pass
FD

Large Read Cache
HBase Memstore
File System
WAL t1 t1 t1
Store files
Read
1
Block Cache
2
Block

Large Read Cache
HBase
Memstore
File System
WAL t1 t1 t1
Store files
Read
1
Block Cache
3
Block
Off-heap Cache Block
2

HBase Complete
HMaster
RS1 RS2 RS3 RS4 RS5 RS6
system

HMaster
HBase Complete
HMaster
system

HMaster
HBase Complete
HMaster
system
ZK

HMaster
HBase Complete
HMaster
system
ZK
HBase
Client

Region Server Failure
HMaster
system
ZK
T1 r1
HBase
Client

Region Replication
HMaster
system
ZK
T1 r1
HBase
Client
T1 r1

Region Replication
HMaster
system
ZK
T1 r1T1 r1
HBase
Client

Region Replication
HMaster
system
ZK
T1 r1T1 r1
HBase
Client
https://www.youtube.com/watch?v=l6S-Vbs9WsU

Load Balancing
HMaster
system
ZK
HBase
Client

Load Balancing
HMaster
Balancer/AM
system
ZK
HBase
Client

Balancer
• Region Count Cost
• Primary Region Count Cost
• Table Skew Cost
• Locality Cost
• Rack Locality Cost
• Region Replica Host Cost
• Region Replica Rack Cost
• Read Request Cost
• Write Request Cost
• Memstore Size Cost
• Storefile Size Cost
• Move Cost

Other Features
• HBase Replication
• HBase multi-tenancy support
— https://www.youtube.com/watch?v=bZjz2G38Ju0
• HBase Co-processors and Filters
— https://www.slideshare.net/Hadoop_Summit/hbase-coprocessors-uses-abuses-solutions

NameNode
RS4
HMaster
ZQuorum
ZQuorum
Operator View
HMaster
RS1 RS2 RS4 RS5 RS6
HBase
ClientZK
Quorum
DN1 DN2 DN3 DN4 DN5 DN6
NameNode
JVM
OS
JVM
OS
JVM
OS
JVM
OS
JVM
OS
JVM
OS
HW HW HW HW HW HW

ZooKeeper Availability
• ZK Quorum
• One leader and remaining followers
— stat
— ruok
— mntr
• Test for availability
— e.g., List children of a znode

HBase Availability
• Master Availability - http://hmaster-node:16010/jmx
— "name" : "Hadoop:service=HBase,name=Master,sub=Server","tag.isActiveMaster" : "true"
• Dead RegionServers
— "name" : "Hadoop:service=HBase,name=Master,sub=Server", "numDeadRegionServers" : 0
• Region In Transition
— "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManger","ritCount" : 0
— e.g., Query system table by listing tables

HDFS Availability
• Namenode Availability - http://namenode-host:50070/jmx
—"name" : "Hadoop:service=NameNode,name=FSNamesystem", "tag.HAState" : "active"
• Dead Datanodes
— "name" : "Hadoop:service=NameNode,name=NameNodeInfo", "DeadNodes" : "{}“
• Missing Blocks
— "name" : "Hadoop:service=NameNode,name=NameNodeInfo", "NumberOfMissingBlocks" : 0
• Percentage Used
— "name" : "Hadoop:service=NameNode,name=NameNodeInfo", "PercentUsed" : 59
• Under replicated blocks
— "name" : "Hadoop:service=NameNode,name=FSNamesystemState","UnderReplicatedBlocks":0
— e.g., Append data to a test file

HBase Performance
• RegionServer JMX metrics - http://rs-node:60300/jmx
— "name“:"Hadoop:service=HBase,name=RegionServer,sub=Server“
• Blockcache hit ratio
• Request counts
• Request response time
• Compaction related metrics
• Region count
• Flush related metrics
• Percentage of files local
• Split related metrics
— "name" : "Hadoop:service=HBase,name=RegionServer,sub=Tables",
• Table level metrics
https://www.slideshare.net/MichaelStack4/hbaseconasia2018-track31-serving-billions-of-queries-in-millisecond-latencies

JVM
• GC – JMX Metrics
— "name" : "java.lang:type=GarbageCollector,name=ParNew",
— "name" : "java.lang:type=GarbageCollector,name=ConcurrentMarkSweep",
• GC Logging
— -verbose:gc
— -XX:+PrintHeapAtGC
— -XX:+PrintGCDetails
— -XX:+PrintGCTimeStamps
— -XX:+PrintGCDateStamps
— -XX:+PrintGCApplicationStoppedTime
— -XX:+PrintClassHistogram
— -XX:+PrintGCApplicationConcurrentTime
— -XX:+PrintTenuringDistribution
— -Xloggc:

OS/HW
• Memory
• CPU
• Disk
• Networking

Logs
• ZooKeeper Log
• HDFS
— Namenode log
— Datanode log
• HBase
— Master log
— RegionServer log
• OS
— Syslog

Interacting with HBase
• HBase shell
— DDL: create namespace/table, alter
— Security: grant, revoke
— DML: get, put, scan
— Tools: assign, compact, balance
— General: status
• HBase admin API
• HBase client API

Data Backup / Restore
• Snapshot
— hbase shell > snapshot 'table', 'table_mmddyy’
• Restore from snapshot
— hbase shell > restore_snapshot 'table_mmddyy'
• Export Snapshot
— $ hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot
• CopyTable
— hbase org.apache.hadoop.hbase.mapreduce.CopyTable

Thank You!
Acknowledgement: Apache HBase Community
Reference: http://hbase.apache.org
Connect with Hadoop Team: hadoop@bloomberg.net

We are hiring!
Questions?
https://www.bloomberg.com/careers

HBase Internals And Operations

More Related Content

Similar to HBase Internals And Operations (20)

More from Biju Nair (13)

Recently uploaded (20)

HBase Internals And Operations