活动介绍

hive建ods_tmp表

时间: 2025-05-09 14:22:33 浏览: 23
<think>好的,用户想在Hive中创建名为ODS_TMP的表。我需要先确定用户的需求,是创建内部表还是外部表?根据用户之前的问题,可能他们需要参考之前提到的外部表创建方法。引用[2]中的例子用的是外部表,所以可能用户更倾向于外部表,尤其是ODS层通常用于原始数据存储,外部表更适合,因为数据生命周期不受Hive管理。 用户提供的示例中,引用[1]提到了内部表和外部表的区别,以及一些查看表的命令。所以需要提醒用户外部表和内部表的区别,确保他们了解数据存储的位置和管理方式。 接下来,创建表的语法需要包括EXTERNAL关键字,指定存储位置,可能还需要分区和文件格式。例如,用户可能需要指定文本文件格式或者ORC等。假设用户的数据存放在HDFS的特定路径,比如/user/hive/warehouse/ods.db/ods_tmp,需要确认路径是否正确。 另外,用户可能需要注意IF NOT EXISTS来避免表已存在的情况,以及需要添加注释和分区字段。例如,常见的ODS层表可能会有日期分区,比如dt字段。同时,字段的数据类型需要根据实际数据来定义,比如userid string, time string等。 还要考虑是否需要使用ROW FORMAT DELIMITED来指定分隔符,比如FIELDS TERMINATED BY ','。如果数据是其他格式,比如Parquet,可能需要不同的设置。 最后,生成相关问题时要围绕Hive表创建的其他方面,比如分区、查看表结构、数据加载等,帮助用户进一步操作。</think>在Hive中创建ODS_TMP表需要明确表的类型(内部表/外部表)、存储格式和表结构。以下是创建外部表的标准方式(推荐ODS层使用外部表): ```sql CREATE EXTERNAL TABLE IF NOT EXISTS ODS_TMP( userid STRING COMMENT '用户ID', kcid STRING COMMENT '课程ID', time STRING COMMENT '时间戳' ) COMMENT 'ODS层临时表' PARTITIONED BY (dt STRING COMMENT '日期分区') ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' STORED AS TEXTFILE LOCATION '/user/hive/warehouse/ods.db/ods_tmp'; ``` **关键参数说明**: 1. `EXTERNAL`:创建外部表,删除表时仅删除元数据[^1] 2. `IF NOT EXISTS`:防止表已存在时报错[^2] 3. `PARTITIONED BY`:按日期分区管理数据 4. `ROW FORMAT`:指定字段分隔符为制表符 5. `STORED AS`:存储格式(TEXTFILE/ORC/PARQUET等) 6. `LOCATION`:指定HDFS存储路径
阅读全文

相关推荐

set hive.exec.dynamic.partition.mode=nonstrict; set hive.exec.parallel=true; set hive.exec.parallel.thread.number=30; --drop table if exists dm_finrisk.dm_event_walletSettleSuccessEvent_di; create table if not exists dm_finrisk.dm_event_cashLoanEvent_di( event_uuid string ,event_code string ,dt string ,event_time string ,rawSnapshot string ) COMMENT 'cashLoanEvent' PARTITIONED BY ( day bigint)stored as parquet; drop table if exists tmp.tmp_dm_event_cashLoanEvent_di_01_${zdt.addDay(-1).format("yyyyMMdd")}; create table if not exists tmp.tmp_dm_event_cashLoanEvent_di_01_${zdt.addDay(-1).format("yyyyMMdd")} as select uuid as event_uuid ,event as event_code ,from_unixtime(unix_timestamp(cast(day as string),'yyyymmdd'),'yyyy-mm-dd') as dt ,from_unixtime(cast(cast(time AS BIGINT)/1000 AS BIGINT),'yyyy-MM-dd HH:mm:ss') as event_time ,get_json_object(data,'$.hotPublish.rawSnapshot') as rawSnapshot from dm_finrisk.ods_event_hdfs_snapshot_di where code = 'shield@cashLoanEvent' and day=${zdt.addDay(-1).format("yyyyMMdd")} drop table if exists tmp.tmp_dm_event_cashLoanEvent_di_02_${zdt.addDay(-1).format("yyyyMMdd")}; create table if not exists tmp.tmp_dm_event_cashLoanEvent_di_02_${zdt.addDay(-1).format("yyyyMMdd")} as select event_uuid ,event_code ,dt ,event_time ,rawSnapshot from tmp.tmp_dm_event_cashLoanEvent_di_01_${zdt.addDay(-1).format("yyyyMMdd")}; alter table dm_finrisk.dm_event_cashLoanEvent_di drop if exists partition (day = ${zdt.addDay(-1).format("yyyyMMdd")}); insert overwrite table dm_finrisk.dm_event_cashLoanEvent_di partition (day = ${zdt.addDay(-1).format("yyyyMMdd")}) select event_uuid ,event_code ,dt ,event_time ,rawSnapshot from tmp.tmp_dm_event_cashLoanEvent_di_02_${zdt.addDay(-1).format("yyyyMMdd")}分析下上面代码

[root@hadoop01 apache-hive-3.1.3-bin]# hive which: no hbase in (/export/servers/hadoop-3.3.5/bin::/export/servers/apache-hive-3.1.3-bin/bin:/export/servers/flume-1.9.0/bin::/export/servers/apache-hive-3.1.3-bin/bin:/export/servers/flume-1.9.0/bin:/export/servers/flume-1.9.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/export/servers/jdk1.8.0_161/bin:/export/servers/hadoop-3.3.5/bin:/export/servers/hadoop-3.3.5/sbin:/export/servers/scala-2.12.10/bin:/root/bin:/export/servers/jdk1.8.0_161/bin:/export/servers/hadoop-3.3.5/bin:/export/servers/hadoop-3.3.5/sbin:/export/servers/scala-2.12.10/bin:/export/servers/jdk1.8.0_161/bin:/export/servers/hadoop-3.3.5/bin:/export/servers/hadoop-3.3.5/sbin:/export/servers/scala-2.12.10/bin:/root/bin) 2025-06-17 20:45:39,133 INFO conf.HiveConf: Found configuration file file:/export/servers/apache-hive-3.1.3-bin/conf/hive-site.xml Hive Session ID = 7a465677-eec4-40ee-b6a2-5c7b638725a7 2025-06-17 20:45:43,011 INFO SessionState: Hive Session ID = 7a465677-eec4-40ee-b6a2-5c7b638725a7 Logging initialized using configuration in jar:file:/export/servers/apache-hive-3.1.3-bin/lib/hive-common-3.1.3.jar!/hive-log4j2.properties Async: true 2025-06-17 20:45:43,144 INFO SessionState: Logging initialized using configuration in jar:file:/export/servers/apache-hive-3.1.3-bin/lib/hive-common-3.1.3.jar!/hive-log4j2.properties Async: true 2025-06-17 20:45:45,757 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/7a465677-eec4-40ee-b6a2-5c7b638725a7 2025-06-17 20:45:45,806 INFO session.SessionState: Created local directory: /tmp/root/7a465677-eec4-40ee-b6a2-5c7b638725a7 2025-06-17 20:45:45,820 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/7a465677-eec4-40ee-b6a2-5c7b638725a7/_tmp_space.db 2025-06-17 20:45:45,850 INFO conf.HiveConf: Using the default value passed in for log id: 7a465677-eec4-40ee-b6a2-5c7b638725a7 2025-06-17 20:45:45,850 INFO session.SessionState: Updating thread name to 7a465677-eec4-40ee-b6a2-5c7b638725a7 main 2025-06-17 20:45:47,956 INFO metastore.HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore 2025-06-17 20:45:48,005 WARN metastore.ObjectStore: datanucleus.autoStartMechanismMode is set to unsupported value null . Setting it to value: ignored 2025-06-17 20:45:48,017 INFO metastore.ObjectStore: ObjectStore, initialize called 2025-06-17 20:45:48,020 INFO conf.MetastoreConf: Found configuration file file:/export/servers/apache-hive-3.1.3-bin/conf/hive-site.xml 2025-06-17 20:45:48,023 INFO conf.MetastoreConf: Unable to find config file hivemetastore-site.xml 2025-06-17 20:45:48,023 INFO conf.MetastoreConf: Found configuration file null 2025-06-17 20:45:48,024 INFO conf.MetastoreConf: Unable to find config file metastore-site.xml 2025-06-17 20:45:48,024 INFO conf.MetastoreConf: Found configuration file null 2025-06-17 20:45:48,400 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored 2025-06-17 20:45:48,953 INFO hikari.HikariDataSource: HikariPool-1 - Starting... 2025-06-17 20:45:49,485 INFO hikari.HikariDataSource: HikariPool-1 - Start completed. 2025-06-17 20:45:49,573 INFO hikari.HikariDataSource: HikariPool-2 - Starting... 2025-06-17 20:45:49,644 INFO hikari.HikariDataSource: HikariPool-2 - Start completed. 2025-06-17 20:45:50,533 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" 2025-06-17 20:45:50,824 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is MYSQL 2025-06-17 20:45:50,827 INFO metastore.ObjectStore: Initialized ObjectStore 2025-06-17 20:45:51,223 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 20:45:51,224 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 20:45:51,225 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 20:45:51,225 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 20:45:51,225 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 20:45:51,225 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 20:45:54,606 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 20:45:54,607 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 20:45:54,608 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 20:45:54,608 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 20:45:54,609 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 20:45:54,609 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 20:45:59,277 WARN metastore.ObjectStore: Version information not found in metastore. metastore.schema.verification is not enabled so recording the schema version 3.1.0 2025-06-17 20:45:59,278 WARN metastore.ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 3.1.0, comment = Set by MetaStore [email protected] 2025-06-17 20:45:59,555 INFO metastore.HiveMetaStore: Added admin role in metastore 2025-06-17 20:45:59,559 INFO metastore.HiveMetaStore: Added public role in metastore 2025-06-17 20:45:59,663 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty 2025-06-17 20:46:00,035 INFO metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=root (auth:SIMPLE) retries=1 delay=1 lifetime=0 2025-06-17 20:46:00,086 INFO metastore.HiveMetaStore: 0: get_all_functions 2025-06-17 20:46:00,097 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_all_functions Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.Hive Session ID = b7eae93e-640d-4628-b883-4e088aafa6e6 2025-06-17 20:46:00,287 INFO CliDriver: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. 2025-06-17 20:46:00,287 INFO SessionState: Hive Session ID = b7eae93e-640d-4628-b883-4e088aafa6e6 2025-06-17 20:46:00,331 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/b7eae93e-640d-4628-b883-4e088aafa6e6 2025-06-17 20:46:00,336 INFO session.SessionState: Created local directory: /tmp/root/b7eae93e-640d-4628-b883-4e088aafa6e6 2025-06-17 20:46:00,346 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/b7eae93e-640d-4628-b883-4e088aafa6e6/_tmp_space.db 2025-06-17 20:46:00,354 INFO metastore.HiveMetaStore: 1: get_databases: @hive# 2025-06-17 20:46:00,355 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_databases: @hive# 2025-06-17 20:46:00,360 INFO metastore.HiveMetaStore: 1: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore 2025-06-17 20:46:00,364 INFO metastore.ObjectStore: ObjectStore, initialize called 2025-06-17 20:46:00,404 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is MYSQL 2025-06-17 20:46:00,406 INFO metastore.ObjectStore: Initialized ObjectStore 2025-06-17 20:46:00,446 INFO metastore.HiveMetaStore: 1: get_tables_by_type: db=@hive#db_hive1 pat=.*,type=MATERIALIZED_VIEW 2025-06-17 20:46:00,447 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_tables_by_type: db=@hive#db_hive1 pat=.*,type=MATERIALIZED_VIEW 2025-06-17 20:46:00,470 INFO metastore.HiveMetaStore: 1: get_multi_table : db=db_hive1 tbls= 2025-06-17 20:46:00,471 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_multi_table : db=db_hive1 tbls= 2025-06-17 20:46:00,486 INFO metastore.HiveMetaStore: 1: get_tables_by_type: db=@hive#default pat=.*,type=MATERIALIZED_VIEW 2025-06-17 20:46:00,486 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_tables_by_type: db=@hive#default pat=.*,type=MATERIALIZED_VIEW 2025-06-17 20:46:00,495 INFO metastore.HiveMetaStore: 1: get_multi_table : db=default tbls= 2025-06-17 20:46:00,495 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_multi_table : db=default tbls= 2025-06-17 20:46:00,495 INFO metastore.HiveMetaStore: 1: get_tables_by_type: db=@hive#itcast_ods pat=.*,type=MATERIALIZED_VIEW 2025-06-17 20:46:00,496 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_tables_by_type: db=@hive#itcast_ods pat=.*,type=MATERIALIZED_VIEW 2025-06-17 20:46:00,503 INFO metastore.HiveMetaStore: 1: get_multi_table : db=itcast_ods tbls= 2025-06-17 20:46:00,503 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_multi_table : db=itcast_ods tbls= 2025-06-17 20:46:00,503 INFO metadata.HiveMaterializedViewsRegistry: Materialized views registry has been initialized hive>

[root@hadoop01 apache-hive-3.1.3-bin]# bin/hive which: no hbase in (/export/servers/hadoop-3.3.5/bin::/export/servers/apache-hive-3.1.3-bin/bin:/export/servers/flume-1.9.0/bin::/export/servers/apache-hive-3.1.3-bin/bin:/export/servers/flume-1.9.0/bin:/export/servers/flume-1.9.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/export/servers/jdk1.8.0_161/bin:/export/servers/hadoop-3.3.5/bin:/export/servers/hadoop-3.3.5/sbin:/export/servers/scala-2.12.10/bin:/root/bin:/export/servers/jdk1.8.0_161/bin:/export/servers/hadoop-3.3.5/bin:/export/servers/hadoop-3.3.5/sbin:/export/servers/scala-2.12.10/bin:/export/servers/jdk1.8.0_161/bin:/export/servers/hadoop-3.3.5/bin:/export/servers/hadoop-3.3.5/sbin:/export/servers/scala-2.12.10/bin:/root/bin) 2025-06-17 19:17:36,429 INFO conf.HiveConf: Found configuration file file:/export/servers/apache-hive-3.1.3-bin/conf/hive-site.xml Hive Session ID = 174edc05-5da2-4b28-a455-d29ac7bdd8fc 2025-06-17 19:17:39,510 INFO SessionState: Hive Session ID = 174edc05-5da2-4b28-a455-d29ac7bdd8fc Logging initialized using configuration in jar:file:/export/servers/apache-hive-3.1.3-bin/lib/hive-common-3.1.3.jar!/hive-log4j2.properties Async: true 2025-06-17 19:17:39,639 INFO SessionState: Logging initialized using configuration in jar:file:/export/servers/apache-hive-3.1.3-bin/lib/hive-common-3.1.3.jar!/hive-log4j2.properties Async: true 2025-06-17 19:17:42,218 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/174edc05-5da2-4b28-a455-d29ac7bdd8fc 2025-06-17 19:17:42,262 INFO session.SessionState: Created local directory: /tmp/root/174edc05-5da2-4b28-a455-d29ac7bdd8fc 2025-06-17 19:17:42,274 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/174edc05-5da2-4b28-a455-d29ac7bdd8fc/_tmp_space.db 2025-06-17 19:17:42,304 INFO conf.HiveConf: Using the default value passed in for log id: 174edc05-5da2-4b28-a455-d29ac7bdd8fc 2025-06-17 19:17:42,304 INFO session.SessionState: Updating thread name to 174edc05-5da2-4b28-a455-d29ac7bdd8fc main 2025-06-17 19:17:44,440 INFO metastore.HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore 2025-06-17 19:17:44,512 WARN metastore.ObjectStore: datanucleus.autoStartMechanismMode is set to unsupported value null . Setting it to value: ignored 2025-06-17 19:17:44,530 INFO metastore.ObjectStore: ObjectStore, initialize called 2025-06-17 19:17:44,532 INFO conf.MetastoreConf: Found configuration file file:/export/servers/apache-hive-3.1.3-bin/conf/hive-site.xml 2025-06-17 19:17:44,534 INFO conf.MetastoreConf: Unable to find config file hivemetastore-site.xml 2025-06-17 19:17:44,534 INFO conf.MetastoreConf: Found configuration file null 2025-06-17 19:17:44,535 INFO conf.MetastoreConf: Unable to find config file metastore-site.xml 2025-06-17 19:17:44,535 INFO conf.MetastoreConf: Found configuration file null 2025-06-17 19:17:44,958 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored 2025-06-17 19:17:45,507 INFO hikari.HikariDataSource: HikariPool-1 - Starting... 2025-06-17 19:17:46,120 INFO hikari.HikariDataSource: HikariPool-1 - Start completed. 2025-06-17 19:17:46,244 INFO hikari.HikariDataSource: HikariPool-2 - Starting... 2025-06-17 19:17:46,262 INFO hikari.HikariDataSource: HikariPool-2 - Start completed. 2025-06-17 19:17:46,944 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" 2025-06-17 19:17:47,177 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is MYSQL 2025-06-17 19:17:47,179 INFO metastore.ObjectStore: Initialized ObjectStore 2025-06-17 19:17:47,661 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 19:17:47,662 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 19:17:47,664 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 19:17:47,664 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 19:17:47,665 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 19:17:47,665 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 19:17:51,340 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 19:17:51,341 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 19:17:51,342 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 19:17:51,342 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 19:17:51,342 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 19:17:51,342 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 19:17:56,088 WARN metastore.ObjectStore: Version information not found in metastore. metastore.schema.verification is not enabled so recording the schema version 3.1.0 2025-06-17 19:17:56,088 WARN metastore.ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 3.1.0, comment = Set by MetaStore [email protected] 2025-06-17 19:17:56,489 INFO metastore.HiveMetaStore: Added admin role in metastore 2025-06-17 19:17:56,497 INFO metastore.HiveMetaStore: Added public role in metastore 2025-06-17 19:17:56,607 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty 2025-06-17 19:17:56,969 INFO metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=root (auth:SIMPLE) retries=1 delay=1 lifetime=0 2025-06-17 19:17:57,003 INFO metastore.HiveMetaStore: 0: get_all_functions 2025-06-17 19:17:57,011 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_all_functions Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. 2025-06-17 19:17:57,170 INFO CliDriver: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. Hive Session ID = 2804a115-a38a-441b-9d42-6abad28c99f8 2025-06-17 19:17:57,170 INFO SessionState: Hive Session ID = 2804a115-a38a-441b-9d42-6abad28c99f8 2025-06-17 19:17:57,216 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/2804a115-a38a-441b-9d42-6abad28c99f8 2025-06-17 19:17:57,222 INFO session.SessionState: Created local directory: /tmp/root/2804a115-a38a-441b-9d42-6abad28c99f8 2025-06-17 19:17:57,228 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/2804a115-a38a-441b-9d42-6abad28c99f8/_tmp_space.db 2025-06-17 19:17:57,231 INFO metastore.HiveMetaStore: 1: get_databases: @hive# 2025-06-17 19:17:57,231 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_databases: @hive# 2025-06-17 19:17:57,233 INFO metastore.HiveMetaStore: 1: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore 2025-06-17 19:17:57,239 INFO metastore.ObjectStore: ObjectStore, initialize called 2025-06-17 19:17:57,272 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is MYSQL 2025-06-17 19:17:57,272 INFO metastore.ObjectStore: Initialized ObjectStore 2025-06-17 19:17:57,288 INFO metastore.HiveMetaStore: 1: get_tables_by_type: db=@hive#db_hive1 pat=.*,type=MATERIALIZED_VIEW 2025-06-17 19:17:57,288 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_tables_by_type: db=@hive#db_hive1 pat=.*,type=MATERIALIZED_VIEW 2025-06-17 19:17:57,314 INFO metastore.HiveMetaStore: 1: get_multi_table : db=db_hive1 tbls= 2025-06-17 19:17:57,314 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_multi_table : db=db_hive1 tbls= 2025-06-17 19:17:57,316 INFO metastore.HiveMetaStore: 1: get_tables_by_type: db=@hive#default pat=.*,type=MATERIALIZED_VIEW 2025-06-17 19:17:57,316 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_tables_by_type: db=@hive#default pat=.*,type=MATERIALIZED_VIEW 2025-06-17 19:17:57,321 INFO metastore.HiveMetaStore: 1: get_multi_table : db=default tbls= 2025-06-17 19:17:57,321 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_multi_table : db=default tbls= 2025-06-17 19:17:57,322 INFO metastore.HiveMetaStore: 1: get_tables_by_type: db=@hive#itcast_ods pat=.*,type=MATERIALIZED_VIEW 2025-06-17 19:17:57,322 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_tables_by_type: db=@hive#itcast_ods pat=.*,type=MATERIALIZED_VIEW 2025-06-17 19:17:57,326 INFO metastore.HiveMetaStore: 1: get_multi_table : db=itcast_ods tbls= 2025-06-17 19:17:57,326 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_multi_table : db=itcast_ods tbls= 2025-06-17 19:17:57,326 INFO metadata.HiveMaterializedViewsRegistry: Materialized views registry has been initialized hive>

[root@hadoop01 apache-hive-3.1.3-bin]# $HIVE_HOME/bin/hive which: no hbase in (/export/servers/hadoop-3.3.5/bin::/export/servers/apache-hive-3.1.3-bin/bin:/export/servers/flume-1.9.0/bin::/export/servers/apache-hive-3.1.3-bin/bin:/export/servers/flume-1.9.0/bin:/export/servers/flume-1.9.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/export/servers/jdk1.8.0_161/bin:/export/servers/hadoop-3.3.5/bin:/export/servers/hadoop-3.3.5/sbin:/export/servers/scala-2.12.10/bin:/root/bin:/export/servers/jdk1.8.0_161/bin:/export/servers/hadoop-3.3.5/bin:/export/servers/hadoop-3.3.5/sbin:/export/servers/scala-2.12.10/bin:/export/servers/jdk1.8.0_161/bin:/export/servers/hadoop-3.3.5/bin:/export/servers/hadoop-3.3.5/sbin:/export/servers/scala-2.12.10/bin:/root/bin) 2025-06-17 19:30:31,773 INFO conf.HiveConf: Found configuration file file:/export/servers/apache-hive-3.1.3-bin/conf/hive-site.xml Hive Session ID = b6ee71f0-5d43-4149-99c7-808d6c553bb8 2025-06-17 19:30:36,006 INFO SessionState: Hive Session ID = b6ee71f0-5d43-4149-99c7-808d6c553bb8 Logging initialized using configuration in jar:file:/export/servers/apache-hive-3.1.3-bin/lib/hive-common-3.1.3.jar!/hive-log4j2.properties Async: true 2025-06-17 19:30:36,195 INFO SessionState: Logging initialized using configuration in jar:file:/export/servers/apache-hive-3.1.3-bin/lib/hive-common-3.1.3.jar!/hive-log4j2.properties Async: true 2025-06-17 19:30:40,759 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/b6ee71f0-5d43-4149-99c7-808d6c553bb8 2025-06-17 19:30:40,863 INFO session.SessionState: Created local directory: /tmp/root/b6ee71f0-5d43-4149-99c7-808d6c553bb8 2025-06-17 19:30:40,874 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/b6ee71f0-5d43-4149-99c7-808d6c553bb8/_tmp_space.db 2025-06-17 19:30:40,916 INFO conf.HiveConf: Using the default value passed in for log id: b6ee71f0-5d43-4149-99c7-808d6c553bb8 2025-06-17 19:30:40,916 INFO session.SessionState: Updating thread name to b6ee71f0-5d43-4149-99c7-808d6c553bb8 main 2025-06-17 19:30:43,385 INFO metastore.HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore 2025-06-17 19:30:43,505 WARN metastore.ObjectStore: datanucleus.autoStartMechanismMode is set to unsupported value null . Setting it to value: ignored 2025-06-17 19:30:43,521 INFO metastore.ObjectStore: ObjectStore, initialize called 2025-06-17 19:30:43,523 INFO conf.MetastoreConf: Found configuration file file:/export/servers/apache-hive-3.1.3-bin/conf/hive-site.xml 2025-06-17 19:30:43,526 INFO conf.MetastoreConf: Unable to find config file hivemetastore-site.xml 2025-06-17 19:30:43,526 INFO conf.MetastoreConf: Found configuration file null 2025-06-17 19:30:43,528 INFO conf.MetastoreConf: Unable to find config file metastore-site.xml 2025-06-17 19:30:43,528 INFO conf.MetastoreConf: Found configuration file null 2025-06-17 19:30:44,060 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored 2025-06-17 19:30:44,701 INFO hikari.HikariDataSource: HikariPool-1 - Starting... 2025-06-17 19:30:45,564 INFO hikari.HikariDataSource: HikariPool-1 - Start completed. 2025-06-17 19:30:45,707 INFO hikari.HikariDataSource: HikariPool-2 - Starting... 2025-06-17 19:30:45,741 INFO hikari.HikariDataSource: HikariPool-2 - Start completed. 2025-06-17 19:30:46,209 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" 2025-06-17 19:30:46,656 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is MYSQL 2025-06-17 19:30:46,662 INFO metastore.ObjectStore: Initialized ObjectStore 2025-06-17 19:30:47,806 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 19:30:47,807 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 19:30:47,808 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 19:30:47,809 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 19:30:47,809 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 19:30:47,810 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 19:30:51,421 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 19:30:51,422 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 19:30:51,422 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 19:30:51,422 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 19:30:51,423 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 19:30:51,423 WARN DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2025-06-17 19:30:56,388 WARN metastore.ObjectStore: Version information not found in metastore. metastore.schema.verification is not enabled so recording the schema version 3.1.0 2025-06-17 19:30:56,388 WARN metastore.ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 3.1.0, comment = Set by MetaStore [email protected] 2025-06-17 19:30:57,110 INFO metastore.HiveMetaStore: Added admin role in metastore 2025-06-17 19:30:57,116 INFO metastore.HiveMetaStore: Added public role in metastore 2025-06-17 19:30:57,279 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty 2025-06-17 19:30:57,779 INFO metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=root (auth:SIMPLE) retries=1 delay=1 lifetime=0 2025-06-17 19:30:57,891 INFO metastore.HiveMetaStore: 0: get_all_functions 2025-06-17 19:30:57,923 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_all_functions Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. 2025-06-17 19:30:58,126 INFO CliDriver: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. Hive Session ID = 06c85488-8547-4e38-a0d1-5e386cd373f1 2025-06-17 19:30:58,131 INFO SessionState: Hive Session ID = 06c85488-8547-4e38-a0d1-5e386cd373f1 2025-06-17 19:30:58,173 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/06c85488-8547-4e38-a0d1-5e386cd373f1 2025-06-17 19:30:58,195 INFO session.SessionState: Created local directory: /tmp/root/06c85488-8547-4e38-a0d1-5e386cd373f1 2025-06-17 19:30:58,203 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/06c85488-8547-4e38-a0d1-5e386cd373f1/_tmp_space.db 2025-06-17 19:30:58,210 INFO metastore.HiveMetaStore: 1: get_databases: @hive# 2025-06-17 19:30:58,211 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_databases: @hive# 2025-06-17 19:30:58,213 INFO metastore.HiveMetaStore: 1: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore 2025-06-17 19:30:58,217 INFO metastore.ObjectStore: ObjectStore, initialize called 2025-06-17 19:30:58,253 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is MYSQL 2025-06-17 19:30:58,255 INFO metastore.ObjectStore: Initialized ObjectStore 2025-06-17 19:30:58,272 INFO metastore.HiveMetaStore: 1: get_tables_by_type: db=@hive#db_hive1 pat=.*,type=MATERIALIZED_VIEW 2025-06-17 19:30:58,272 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_tables_by_type: db=@hive#db_hive1 pat=.*,type=MATERIALIZED_VIEW 2025-06-17 19:30:58,305 INFO metastore.HiveMetaStore: 1: get_multi_table : db=db_hive1 tbls= 2025-06-17 19:30:58,305 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_multi_table : db=db_hive1 tbls= 2025-06-17 19:30:58,310 INFO metastore.HiveMetaStore: 1: get_tables_by_type: db=@hive#default pat=.*,type=MATERIALIZED_VIEW 2025-06-17 19:30:58,310 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_tables_by_type: db=@hive#default pat=.*,type=MATERIALIZED_VIEW 2025-06-17 19:30:58,327 INFO metastore.HiveMetaStore: 1: get_multi_table : db=default tbls= 2025-06-17 19:30:58,327 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_multi_table : db=default tbls= 2025-06-17 19:30:58,327 INFO metastore.HiveMetaStore: 1: get_tables_by_type: db=@hive#itcast_ods pat=.*,type=MATERIALIZED_VIEW 2025-06-17 19:30:58,327 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_tables_by_type: db=@hive#itcast_ods pat=.*,type=MATERIALIZED_VIEW 2025-06-17 19:30:58,332 INFO metastore.HiveMetaStore: 1: get_multi_table : db=itcast_ods tbls= 2025-06-17 19:30:58,332 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_multi_table : db=itcast_ods tbls= 2025-06-17 19:30:58,332 INFO metadata.HiveMaterializedViewsRegistry: Materialized views registry has been initialized hive> SHOW DATABASES; 2025-06-17 19:31:11,344 INFO conf.HiveConf: Using the default value passed in for log id: b6ee71f0-5d43-4149-99c7-808d6c553bb8 2025-06-17 19:31:11,624 INFO ql.Driver: Compiling command(queryId=root_20250617193111_702231fb-8543-45f5-b003-86a49e9c4298): SHOW DATABASES 2025-06-17 19:31:12,626 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager 2025-06-17 19:31:12,673 INFO ql.Driver: Semantic Analysis Completed (retrial = false) 2025-06-17 19:31:12,819 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:database_name, type:string, comment:from deserializer)], properties:null) 2025-06-17 19:31:12,982 INFO exec.ListSinkOperator: Initializing operator LIST_SINK[0] 2025-06-17 19:31:13,009 INFO ql.Driver: Completed compiling command(queryId=root_20250617193111_702231fb-8543-45f5-b003-86a49e9c4298); Time taken: 1.484 seconds 2025-06-17 19:31:13,011 INFO reexec.ReExecDriver: Execution #1 of query 2025-06-17 19:31:13,011 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager 2025-06-17 19:31:13,012 INFO ql.Driver: Executing command(queryId=root_20250617193111_702231fb-8543-45f5-b003-86a49e9c4298): SHOW DATABASES 2025-06-17 19:31:13,034 INFO ql.Driver: Starting task [Stage-0:DDL] in serial mode 2025-06-17 19:31:13,041 INFO metastore.HiveMetaStore: 0: get_databases: @hive# 2025-06-17 19:31:13,041 INFO HiveMetaStore.audit: ugi=root ip=unknown-ip-addr cmd=get_databases: @hive# 2025-06-17 19:31:13,057 INFO exec.DDLTask: results : 3 2025-06-17 19:31:13,244 INFO ql.Driver: Completed executing command(queryId=root_20250617193111_702231fb-8543-45f5-b003-86a49e9c4298); Time taken: 0.233 seconds OK 2025-06-17 19:31:13,245 INFO ql.Driver: OK 2025-06-17 19:31:13,245 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager 2025-06-17 19:31:13,276 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 2025-06-17 19:31:13,381 INFO mapred.FileInputFormat: Total input files to process : 1 2025-06-17 19:31:13,458 INFO exec.ListSinkOperator: RECORDS_OUT_INTERMEDIATE:0, RECORDS_OUT_OPERATOR_LIST_SINK_0:3, db_hive1 default itcast_ods Time taken: 1.725 seconds, Fetched: 3 row(s) 2025-06-17 19:31:13,479 INFO CliDriver: Time taken: 1.725 seconds, Fetched: 3 row(s) 2025-06-17 19:31:13,480 INFO conf.HiveConf: Using the default value passed in for log id: b6ee71f0-5d43-4149-99c7-808d6c553bb8 2025-06-17 19:31:13,480 INFO session.SessionState: Resetting thread name to main hive>

Connected to: Apache Hive (version 3.1.2) Driver: Hive JDBC (version 3.1.2) Transaction isolation: TRANSACTION_REPEATABLE_READ Beeline version 3.1.2 by Apache Hive 0: jdbc:hive2://bigdata1:10000> use ods; No rows affected (0.145 seconds) 0: jdbc:hive2://bigdata1:10000> SHOW TABLES LIKE 'customer_inf'; +---------------+ | tab_name | +---------------+ | customer_inf | +---------------+ 1 row selected (0.452 seconds) 0: jdbc:hive2://bigdata1:10000> select * from ods.crstomer_inf;Closing: 0: jdbc:hive2://bigdata1:10000 ^C^C[root@bigdata1 ~]# ^C [root@bigdata1 ~]# beeline -u "jdbc:hive2://bigdata1:10000" -p root SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/module/hadoop-3.1.3/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/module/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Connecting to jdbc:hive2://bigdata1:10000 Connected to: Apache Hive (version 3.1.2) Driver: Hive JDBC (version 3.1.2) Transaction isolation: TRANSACTION_REPEATABLE_READ Beeline version 3.1.2 by Apache Hive 0: jdbc:hive2://bigdata1:10000> use ods; No rows affected (0.142 seconds) 0: jdbc:hive2://bigdata1:10000> select * from customer_inf; Error: Error while compiling statement: FAILED: SemanticException Unable to determine if hdfs://bigdata1:9000/user/hive/warehouse/ods.db/customer_inf is encrypted: org.apache.hadoop.security.AccessControlException: Permission denied: user=anonymous, access=EXECUTE, inode="/user":root:supergroup:drwx------ at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:399) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:315) at org.apache.hadoop.

bin/sqoop import \ > --connect jdbc:mysql://192.168.249.161:3306/nev \ > --username hive \ > --password hive \ > --query "select > id,create_date_time,session_id,sid,create_time,seo_source,seo_keywords,ip,area, > country,province,city,origin_channel,user as > user_match,manual_time,begin_time,end_time,last_customer_msg_time_stamp,last_agent_msg_time_stamp, reply_msg_count,msg_count,browser_name,os_info from > web_chat_ems_2019_07 where 1=1 and \$CONDITIONS" \ > --hcatalog-database hnsoft_ods \ > --hcatalog-table web_chat_ems_ods \ > -m 2 \ > --split-by id Warning: /home/hadoop/app/sqoop/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /home/hadoop/app/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/hadoop/app/hadoop-2.9.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/hadoop/app/hbase-1.2.0/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 25/05/28 14:51:28 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6 25/05/28 14:51:28 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 25/05/28 14:51:28 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 25/05/28 14:51:28 INFO tool.CodeGenTool: Beginning code generation Loading class com.mysql.jdbc.Driver'. This is deprecated. The new driver class is com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary. 25/05/28 14:51:29 INFO manager.SqlManager: Executing SQL statement: select id,create_date_time,session_id,sid,create_time,seo_source,seo_keywords,ip,area, country,province,city,origin_channel,user as user_match,manual_time,begin_time,end_time,last_customer_msg_time_stamp,last_agent_msg_time_stamp, reply_msg_count,msg_count,browser_name,os_info from web_chat_ems_2019_07 where 1=1 and (1 = 0) 25/05/28 14:51:30 INFO manager.SqlManager: Executing SQL statement: select id,create_date_time,session_id,sid,create_time,seo_source,seo_keywords,ip,area, country,province,city,origin_channel,user as user_match,manual_time,begin_time,end_time,last_customer_msg_time_stamp,last_agent_msg_time_stamp, reply_msg_count,msg_count,browser_name,os_info from web_chat_ems_2019_07 where 1=1 and (1 = 0) 25/05/28 14:51:30 INFO manager.SqlManager: Executing SQL statement: select id,create_date_time,session_id,sid,create_time,seo_source,seo_keywords,ip,area, country,province,city,origin_channel,user as user_match,manual_time,begin_time,end_time,last_customer_msg_time_stamp,last_agent_msg_time_stamp, reply_msg_count,msg_count,browser_name,os_info from web_chat_ems_2019_07 where 1=1 and (1 = 0) 25/05/28 14:51:30 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/hadoop/app/hadoop 注: /tmp/sqoop-root/compile/413c3b2426f9b2007f8209241e804082/QueryResult.java使用或覆盖了已过时的 API。 注: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。 25/05/28 14:51:41 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/413c3b2426f9b2007f8209241e804082/QueryResult.jar 25/05/28 14:51:41 INFO mapreduce.ImportJobBase: Beginning query import. 25/05/28 14:51:41 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatOutputFormat at org.apache.sqoop.mapreduce.DataDrivenImportJob.getOutputFormatClass(DataDrivenImportJob.java:178) at org.apache.sqoop.mapreduce.ImportJobBase.configureOutputFormat(ImportJobBase.java:94) at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:259) at org.apache.sqoop.manager.SqlManager.importQuery(SqlManager.java:729) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:499) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605) at org.apache.sqoop.Sqoop.run(Sqoop.java:143) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227) at org.apache.sqoop.Sqoop.main(Sqoop.java:236) Caused by: java.lang.ClassNotFoundException: org.apache.hive.hcatalog.mapreduce.HCatOutputFormat at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 12 more

0: jdbc:hive2://bigdata1:10000> select * from ods.crstomer_inf; Error: Error while compiling statement: FAILED: SemanticException [Error 10001]: Line 1:14 Table not found 'crstomer_inf' (state=42S02,code=10001) 0: jdbc:hive2://bigdata1:10000> select * from ods.customer_inf; Error: Error while compiling statement: FAILED: SemanticException Unable to determine if hdfs://bigdata1:9000/user/hive/warehouse/ods.db/customer_inf is encrypted: org.apache.hadoop.security.AccessControlException: Permission denied: user=anonymous, access=EXECUTE, inode="/user":root:supergroup:drwx------ at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:399) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:315) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:242) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:193) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:606) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkTraverse(FSDirectory.java:1804) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkTraverse(FSDirectory.java:1822) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.resolvePath(FSDirectory.java:677) at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:114) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3131) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1173) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos

大家在看

recommend-type

Xilinx ISE rs_decoder_ipcore and encoder License

Xilinx ISE RS编码解码IP核的License
recommend-type

毕业设计&课设-一个基于Matlab的PET仿真和重建框架,具有系统矩阵的分析建模,能够结合各种数据….zip

matlab算法,工具源码,适合毕业设计、课程设计作业,所有源码均经过严格测试,可以直接运行,可以放心下载使用。有任何使用问题欢迎随时与博主沟通,第一时间进行解答! matlab算法,工具源码,适合毕业设计、课程设计作业,所有源码均经过严格测试,可以直接运行,可以放心下载使用。有任何使用问题欢迎随时与博主沟通,第一时间进行解答! matlab算法,工具源码,适合毕业设计、课程设计作业,所有源码均经过严格测试,可以直接运行,可以放心下载使用。有任何使用问题欢迎随时与博主沟通,第一时间进行解答! matlab算法,工具源码,适合毕业设计、课程设计作业,所有源码均经过严格测试,可以直接运行,可以放心下载使用。有任何使用问题欢迎随时与博主沟通,第一时间进行解答! matlab算法,工具源码,适合毕业设计、课程设计作业,所有源码均经过严格测试,可以直接运行,可以放心下载使用。有任何使用问题欢迎随时与博主沟通,第一时间进行解答! matlab算法,工具源码,适合毕业设计、课程设计作业,所有源码均经过严格测试,可以直接运行,可以放心下载使用。有任何使用问题欢迎随
recommend-type

MATLAB机械臂简单控制仿真(Simulink篇-总).zip

MATLAB下机器人可视化与控制---simulink篇中的简单例子,在Simulink中做了预定义轨迹的运动和Slider Gain控制的运动,用GUI控制的关节代码在MATLAB下机器人可视化与控制
recommend-type

使用 GCC 构建 STM23F0 ARM 项目的模板源码

使用 GCC 构建 STM23F0 ARM 项目的模板源码,具体请看 README
recommend-type

详细说明 VC++的MFC开发串口调试助手源代码,包括数据发送,接收,显示制式等29782183com

详细说明 VC++的MFC开发串口调试助手源代码,包括数据发送,接收,显示制式等29782183com

最新推荐

recommend-type

计算机网络学习中学员常见问题与改进方法

计算机网络学习中学员常见问题与改进方法+
recommend-type

美国国际航空交通数据分析报告(1990-2020)

根据给定的信息,我们可以从中提取和分析以下知识点: 1. 数据集概述: 该数据集名为“U.S. International Air Traffic data(1990-2020)”,记录了美国与国际间航空客运和货运的详细统计信息。数据集涵盖的时间范围从1990年至2020年,这说明它包含了长达30年的时间序列数据,对于进行长期趋势分析非常有价值。 2. 数据来源及意义: 此数据来源于《美国国际航空客运和货运统计报告》,该报告是美国运输部(USDOT)所管理的T-100计划的一部分。T-100计划旨在收集和发布美国和国际航空公司在美国机场的出入境交通报告,这表明数据的权威性和可靠性较高,适用于政府、企业和学术研究等领域。 3. 数据内容及应用: 数据集包含两个主要的CSV文件,分别是“International_Report_Departures.csv”和“International_Report_Passengers.csv”。 a. International_Report_Departures.csv文件可能包含了以下内容: - 离港航班信息:记录了各航空公司的航班号、起飞和到达时间、起飞和到达机场的代码以及国际地区等信息。 - 航空公司信息:可能包括航空公司代码、名称以及所属国家等。 - 飞机机型信息:如飞机类型、座位容量等,这有助于分析不同机型的使用频率和趋势。 - 航线信息:包括航线的起始和目的国家及城市,对于研究航线网络和优化航班计划具有参考价值。 这些数据可以用于航空交通流量分析、机场运营效率评估、航空市场分析等。 b. International_Report_Passengers.csv文件可能包含了以下内容: - 航班乘客信息:可能包括乘客的国籍、年龄、性别等信息。 - 航班类型:如全客机、全货机或混合型航班,可以分析乘客运输和货物运输的比例。 - 乘客数量:记录了各航班或航线的乘客数量,对于分析航空市场容量和增长趋势很有帮助。 - 飞行里程信息:有助于了解国际间不同航线的长度和飞行距离,为票价设置和燃油成本分析提供数据支持。 这些数据可以用于航空客运市场分析、需求预测、收益管理等方面。 4. 数据分析和应用实例: - 航空流量分析:通过分析离港航班数据,可以观察到哪些航线最为繁忙,哪些机场的国际航空流量最大,这有助于航空公司调整航班时刻表和运力分配。 - 市场研究:乘客数据可以揭示不同国家和地区之间的人口流动趋势,帮助航空公司和政府机构了解国际旅行市场的需求变化。 - 飞机利用率:结合飞机机型和飞行频率信息,可以对特定机型的使用率进行分析,评估飞机维护需求和燃油效率。 - 安全监管:通过对比不同航空公司和航班的安全记录,监管机构可以更有效地评估航空公司的安全性能,并采取必要的监管措施。 5. 技术和方法论: 分析此类数据通常涉及数据清洗、数据整合、统计分析、时间序列分析、预测建模等数据科学方法。使用Excel、SQL、R、Python等工具进行数据处理和分析是常见的做法。例如,可以使用Python的Pandas库来清洗和准备数据,使用Matplotlib和Seaborn库来可视化数据,然后利用Scikit-learn或Statsmodels库来构建预测模型。 通过以上知识点的提取和分析,我们可以理解到“U.S. International Air Traffic data(1990-2020)-数据集”的重要性,它不仅记录了跨越30年的航空交通数据,还为各种分析和应用提供了详实的基础信息。对于航空业从业者、政策制定者、研究人员以及数据分析师来说,这是一个极具价值的数据资源。
recommend-type

统计学视角:深入理解最小二乘法的概率论基础

# 1. 最小二乘法的基本概念 最小二乘法(Least Squares Method, LSM)是统计学和数据分析中广泛使用的一种数学优化技术。其主要目的是通过最小化误差的平方和来寻找数据的最佳函数匹配。这种方法最早由高斯提出,但在工程、物理、经济和其他领域中有着广泛的应用。 在本章中,我们将首先了
recommend-type

vscode中使用Codeium

<think>好的,我需要回答用户如何在VSCode中使用Codeium插件的问题。首先,我需要回顾用户提供的引用内容,看看是否有相关信息。用户提供的引用[1]提到了Codeium Chat可以解释代码中的问题,引用[2]提到了在IDEA中的一些问题,可能对VSCode的安装有帮助。用户还提到了安装步骤、基本功能、常见问题等。 首先,我应该分步骤说明安装过程,包括在VSCode扩展商店搜索Codeium并安装。然后,登录部分可能需要用户访问仪表板获取API密钥,引用[2]中提到登录问题,可能需要提醒用户注意网络或权限设置。 接下来是基本功能,比如代码自动补全和Chat功能。引用[1]提到C
recommend-type

UniMoCo:统一框架下的多监督视觉学习方法

在详细解析“unimoco”这个概念之前,我们需要明确几个关键点。首先,“unimoco”代表的是一种视觉表示学习方法,它在机器学习尤其是深度学习领域中扮演着重要角色。其次,文章作者通过这篇论文介绍了UniMoCo的全称,即“Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning”,其背后的含义是在于UniMoCo框架整合了无监督学习、半监督学习和全监督学习三种不同的学习策略。最后,该框架被官方用PyTorch库实现,并被提供给了研究者和开发者社区。 ### 1. 对比学习(Contrastive Learning) UniMoCo的概念根植于对比学习的思想,这是一种无监督学习的范式。对比学习的核心在于让模型学会区分不同的样本,通过将相似的样本拉近,将不相似的样本推远,从而学习到有效的数据表示。对比学习与传统的分类任务最大的不同在于不需要手动标注的标签来指导学习过程,取而代之的是从数据自身结构中挖掘信息。 ### 2. MoCo(Momentum Contrast) UniMoCo的实现基于MoCo框架,MoCo是一种基于队列(queue)的对比学习方法,它在训练过程中维持一个动态的队列,其中包含了成对的负样本。MoCo通过 Momentum Encoder(动量编码器)和一个队列来保持稳定和历史性的负样本信息,使得模型能够持续地进行对比学习,即使是在没有足够负样本的情况下。 ### 3. 无监督学习(Unsupervised Learning) 在无监督学习场景中,数据样本没有被标记任何类别或标签,算法需自行发现数据中的模式和结构。UniMoCo框架中,无监督学习的关键在于使用没有标签的数据进行训练,其目的是让模型学习到数据的基础特征表示,这对于那些标注资源稀缺的领域具有重要意义。 ### 4. 半监督学习(Semi-Supervised Learning) 半监督学习结合了无监督和有监督学习的优势,它使用少量的标注数据与大量的未标注数据进行训练。UniMoCo中实现半监督学习的方式,可能是通过将已标注的数据作为对比学习的一部分,以此来指导模型学习到更精准的特征表示。这对于那些拥有少量标注数据的场景尤为有用。 ### 5. 全监督学习(Full-Supervised Learning) 在全监督学习中,所有的训练样本都有相应的标签,这种学习方式的目的是让模型学习到映射关系,从输入到输出。在UniMoCo中,全监督学习用于训练阶段,让模型在有明确指示的学习目标下进行优化,学习到的任务相关的特征表示。这通常用于有充足标注数据的场景,比如图像分类任务。 ### 6. PyTorch PyTorch是一个开源机器学习库,由Facebook的人工智能研究团队开发,主要用于计算机视觉和自然语言处理等任务。它被广泛用于研究和生产环境,并且因其易用性、灵活性和动态计算图等特性受到研究人员的青睐。UniMoCo官方实现选择PyTorch作为开发平台,说明了其对科研社区的支持和对易于实现的重视。 ### 7. 可视化表示学习(Visual Representation Learning) 可视化表示学习的目的是从原始视觉数据中提取特征,并将它们转换为能够反映重要信息且更易于处理的形式。在UniMoCo中,无论是无监督、半监督还是全监督学习,最终的目标都是让模型学习到有效的视觉表示,这些表示可以用于下游任务,如图像分类、目标检测、图像分割等。 ### 8. 标签队列(Label Queue) UniMoCo通过标签队列维护受监管的标签,这可能意味着对于那些半监督或全监督学习的任务,模型在进行对比学习时,会参考这些来自标签队列的数据。标签队列机制能帮助模型更好地利用有限的标注数据,增强模型的泛化能力。 ### 结论 UniMoCo的提出,以及其官方PyTorch实现的发布,将对计算机视觉领域产生深远影响。它不仅提供了一个统一的对比学习框架,使得从无监督到全监督的学习过程更加灵活和高效,而且为研究者们提供了一个强力的工具,以便更好地探索和实现各种视觉任务。UniMoCo的研究和应用前景,为机器学习尤其是深度学习在视觉领域的研究和实践提供了新的视角和可能。
recommend-type

【MATLAB算法精讲】:最小二乘法的实现与案例深度分析

# 1. 最小二乘法的基本原理 最小二乘法是一种数学优化技术,它通过最小化误差的平方和来寻找数据的最佳函数匹配。其核心思想是选择一条曲线,使得所有观察点到这条曲线的距离之和最小。这种方法广泛应用于统计学、信号处理、工程学和经济学等领域,尤其适用于需要通过一组数据点来确定函数参数的情况。 ## 1.1 统计学视角下的最小二乘法 在统计学中,最小二乘法经常用于
recommend-type

Idea使用教程+jdk配置

<think>好的,我现在需要回答用户关于IntelliJ IDEA使用教程和JDK配置方法的问题。首先,用户明确提到了参考站内引用中的内容,引用[1]和引用[2]都涉及JDK的配置步骤,但看起来这些引用内容可能来自其他问题或文档,可能与当前问题不完全相关,但用户希望我利用这些信息来组织回答。 首先,我需要确认IntelliJ IDEA配置JDK的基本步骤,并整合用户提供的引用内容。引用[1]提到选择JDK安装根目录,例如D:\develop\Java\jdk-17,这说明配置时需要定位到JDK的主目录。引用[2]则提到了通过New按钮选择JDK版本,并完成项目创建,这部分可能涉及到项目设置
recommend-type

GitHub入门实践:审查拉取请求指南

从提供的文件信息中,我们可以抽取以下知识点: **GitHub入门与Pull Request(PR)的审查** **知识点1:GitHub简介** GitHub是一个基于Git的在线代码托管和版本控制平台,它允许开发者在互联网上进行代码的托管和协作。通过GitHub,用户可以跟踪和管理代码变更,参与开源项目,或者创建自己的私有仓库进行项目协作。GitHub为每个项目提供了问题跟踪和任务管理功能,支持Pull Request机制,以便用户之间可以进行代码的审查和讨论。 **知识点2:Pull Request的作用与审查** Pull Request(PR)是协作开发中的一个重要机制,它允许开发者向代码库贡献代码。当开发者在自己的分支上完成开发后,他们可以向主分支(或其他分支)提交一个PR,请求合入他们的更改。此时,其他开发者,包括项目的维护者,可以审查PR中的代码变更,进行讨论,并最终决定是否合并这些变更到目标分支。 **知识点3:审查Pull Request的步骤** 1. 访问GitHub仓库,并查看“Pull requests”标签下的PR列表。 2. 选择一个PR进行审查,点击进入查看详细内容。 3. 查看PR的标题、描述以及涉及的文件变更。 4. 浏览代码的具体差异,可以逐行审查,也可以查看代码变更的概览。 5. 在PR页面添加评论,可以针对整个PR,也可以针对特定的代码行或文件。 6. 当审查完成后,可以提交评论,或者批准、请求修改或关闭PR。 **知识点4:代码审查的最佳实践** 1. 确保PR的目标清晰且具有针对性,避免过于宽泛。 2. 在审查代码时,注意代码的质量、结构以及是否符合项目的编码规范。 3. 提供建设性的反馈,指出代码的优点和需要改进的地方。 4. 使用清晰、具体的语言,避免模糊和主观的评论。 5. 鼓励开发者间的协作,而不是单向的批评。 6. 经常审查PR,以避免延迟和工作积压。 **知识点5:HTML基础** HTML(HyperText Markup Language)是用于创建网页的标准标记语言。它通过各种标签(如`<p>`用于段落,`<img>`用于图片,`<a>`用于链接等)来定义网页的结构和内容。HTML文档由元素组成,这些元素通过开始标签和结束标签来标识。例如,`<p>This is a paragraph.</p>`。HTML的最新版本是HTML5,它引入了许多新的元素和API,增强了对多媒体、图形和本地存储的支持。 **知识点6:GitHub Pages功能介绍** GitHub Pages是一个静态站点托管服务,允许用户直接从GitHub仓库中发布个人、组织或项目的网站。你可以通过设置一个专门的分支来存放你的网站源代码,然后利用GitHub Pages的设置选项,选择分支并发布你的网站。发布的网站将可以通过一个自定义的URL访问,这个URL通常是`username.github.io/repo-name`的格式。这为开发者提供了一个简单而快速的方法来搭建个人或项目的展示页面。 **知识点7:简单的游戏开发实践** 文件描述中提到了一个基于项目的学习活动,通过游戏的形式,让入门人员体验操作和理解基本的游戏开发概念。在这个活动中,参与者通过键盘操作控制形状的旋转和移动,目标是创建无空隙的完整行,这涉及到游戏逻辑、用户输入处理和图形界面显示等基础知识。该活动可能使用了HTML、JavaScript和CSS等前端技术实现,参与者通过实践操作来学习如何编程,并理解基本的游戏设计原理。
recommend-type

【R语言高级教程】:最小二乘法从入门到精通

# 1. 最小二乘法的理论基础 最小二乘法是数学优化技术之一,广泛应用于统计学、数据分析和工程学等领域。其核心思想在于找到一条直线(或曲线),使得所有观测点到该直线的垂直距离之和最小,即误差平方和最小。这一方法不仅简单易行,而且可以有效减少异常值对模型的影响,使得参数估计更加稳定。 ## 1.1 最小二乘法的数学表述 在最基本的线性回归模型中,最小二乘法通过最小化误差的平方和来寻找最
recommend-type

cadence画PCB时改变线长

<think>好的,我现在需要帮助用户解决在Cadence PCB设计中调整走线长度的问题。首先,我得回想一下用户提到的引用内容,尤其是引用1、2和3,里面提到了实时显示走线长度、约束导向设计以及过孔对长度的影响。用户可能已经知道如何显示长度,现在想知道如何调整它们。 用户的问题是如何调整走线长度以满足特定需求,比如等长布线或时序要求。我应该先分步骤说明不同的调整方法,比如手动调整、使用自动优化工具、蛇形走线,以及设置约束管理器中的规则。 记得引用中的信息,比如引用2提到的约束导向环境和实时长度显示,所以需要提到约束管理器的使用。引用3讨论了过孔对长度的影响,调整过孔数量可能也是一种方法。