Maximum Number of Versions

本文探讨了HBase中列族最大版本数的配置参数,解释了该参数的重要性及如何根据应用需求调整。HBase不覆盖行值,而是通过时间戳存储不同版本,过多的版本将显著增加StoreFile大小。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

The maximum number of row versions to store is configured per column family via HColumnDescriptor. The default for max versions is 1. This is an important parameter because as described in Data Model section HBase does not overwrite row values, but rather stores different values per row by time (and qualifier). Excess versions are removed during major compactions. The number of max versions may need to be increased or decreased depending on application needs.

It is not recommended setting the number of max versions to an exceedingly high level (e.g., hundreds or more) unless those old values are very dear to you because this will greatly increase StoreFile size.

hive> select '2025-07-04' dt, count(*) user_churn_count > from dws_user_user_login_td > where dt = '2025-07-04' > and login_last_date <= date_sub('2025-07-04', 6); 40542 [0d7b2f63-06b7-4d2b-835b-fc4e5bf848ea main] WARN org.apache.hadoop.hive.ql.session.SessionState - METASTORE_FILTER_HOOK will be ignored, since hive.security.authorization.manager is set to instance of HiveAuthorizerFactory. %40544 [0d7b2f63-06b7-4d2b-835b-fc4e5bf848ea main] WARN org.apache.hadoop.hive.metastore.ObjectStore - datanucleus.autoStartMechanismMode is set to unsupported value null . Setting it to value: ignored %40556 [0d7b2f63-06b7-4d2b-835b-fc4e5bf848ea main] WARN org.apache.hadoop.hive.metastore.ObjectStore - datanucleus.autoStartMechanismMode is set to unsupported value null . Setting it to value: ignored %42406 [0d7b2f63-06b7-4d2b-835b-fc4e5bf848ea main] WARN org.apache.hadoop.hive.ql.Driver - Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. %Query ID = root_20250709153825_051fb62b-b7e5-4c17-a94f-874bdf465b83 Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number>运行不结束
07-10
0: jdbc:hive2://localhost:10000/default> select count(*) from party; INFO : Compiling command(queryId=stu_20250614034450_e4e8b591-67e8-49a9-a0a1-a1ed84cdab83): select count(*) from party INFO : Semantic Analysis Completed (retrial = false) INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null) INFO : Completed compiling command(queryId=stu_20250614034450_e4e8b591-67e8-49a9-a0a1-a1ed84cdab83); Time taken: 0.126 seconds INFO : Concurrency mode is disabled, not creating a lock manager INFO : Executing command(queryId=stu_20250614034450_e4e8b591-67e8-49a9-a0a1-a1ed84cdab83): select count(*) from party WARN : Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. tez) or using Hive 1.X releases. INFO : Query ID = stu_20250614034450_e4e8b591-67e8-49a9-a0a1-a1ed84cdab83 INFO : Total jobs = 1 INFO : Launching Job 1 out of 1 INFO : Starting task [Stage-1:MAPRED] in serial mode INFO : Number of reduce tasks determined at compile time: 1 INFO : In order to change the average load for a reducer (in bytes): INFO : set hive.exec.reducers.bytes.per.reducer=<number> INFO : In order to limit the maximum number of reducers: INFO : set hive.exec.reducers.max=<number> INFO : In order to set a constant number of reducers: INFO : set mapreduce.job.reduces=<number> INFO : number of splits:1 INFO : Submitting tokens for job: job_local849148416_0004 INFO : Executing with tokens: [] INFO : The url to track the job: http://localhost:8080/ INFO : Job running in-process (local Hadoop) INFO : 2025-06-14 03:44:51,641 Stage-1 map = 0%, reduce = 0% ERROR : Ended Job = job_local849148416_0004 with errors ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask INFO : MapReduce Jobs Launched: INFO : Stage-Stage-1: HDFS Read: 0 HDFS Write: 0 FAIL INFO : Total MapReduce CPU Time Spent: 0 msec INFO : Completed executing command(queryId=stu_20250614034450_e4e8b591-67e8-49a9-a0a1-a1ed84cdab83); Time taken: 1.295 seconds Error: Error while compiling statement: FAILED: Execution Error, return cod
06-15
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值