eclipse写程序从hdfs上下载文件到本地报错：at org.apache.hadoop.util.Shell.runCommand

最新推荐文章于 2022-05-25 17:18:48 发布
原创 最新推荐文章于 2022-05-25 17:18:48 发布 · 2.1k 阅读 · 2 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
文章标签：

#d

本文介绍了解决Hadoop环境下winutils.exe缺失或无法正常工作的问题步骤，包括检查环境变量配置、验证winutils.exe的存在及可用性以及通过重启计算机或在代码中设置属性来解决问题。

摘要生成于 C知道 ，由 DeepSeek-R1 满血版支持， 前往体验 >

1.看看是否有配置HADOOP_HOME这个环境变量，并且在PATH里引用（比如HADOOP_HOME配置C:\hadoop-2.6.4，PATH里加%HADOOP_HOME%\bin）

2.HADOOP_HOME中bin目录下是否有winutils.exe这（对应版本的，我的是window7）exe，并保证winutils.exe双击运行没有报***.dll 缺失的错误。

3.如果还是不行，就试着重启电脑（我的没有配置HADOOP_HOME这个变量，配置完后还是出现这个问题，但是重启电脑就好了），或者在代码中加入System.setProperty("hadoop.home.dir", "C:/hadoop-2.6.4")，第二个参数为你的HADOOP_HOME目录


lshhhf
关注
Eclipse新建Maven项目并运行（打包）java程序实现hdfs与本地之间的上传或下载文件
Wxp_csdn的博客
05-22 1349
Eclipse新建Maven项目并运行（打包）java程序实现hdfs与本地之间的上传或下载文件一、在Eclipse里新建Maven Project二、输入com.bla，项目名称HDFSTest，如果项目要运行到web的话在package处选择war，我们这里选择jar的形式。3、自动生成Maven项目，新建包hdfs.files，包里新建class：4个java文件。四、可直接运行四个java...
Hive: java.lang.ClassNotFoundException Class org.apache.hive.hcatalog.data.JsonSerDe not found
砍柴人的博客
02-25 988
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. Query ID = root_20210225123435_1324c293-cd8b-42be-802d-339619ddd885 Tot..
Windows环境下eclipse提交到远程wordcount程序报错 at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
www_jun的专栏
10-23 5939
远程hadoop2.7，本地的windows7 程序报错如下： 2015-09-28 22:04:21,423 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:(62)) - Unable to load native-hadoop library for your platform... using builtin-java
atorg.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:557)
10-25 296
错误原因： 你当前开发环境中{Hadoop_HOME}\bin\hadoop.dll 文件和你当前的hadoop版本不匹配。 解决方案： 网络下载相应版本的hadoop.dll，并将该文件放入c:\windows\system32下，同时替换hadoop目录bin下的hadoop.dll 例如： 下载https://codeload.github.com/...
at org.apache.hadoop.hdfs.server.namenode.ha.standbystate.checkoperation
qq_39128254的博客
03-29 860
解决办法，将hdfs正在使用的active namenode重启解决。
eclipse连接hadoop下载文件时出错org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Ljava/
u014774853的专栏
11-21 411
eclipse连接hadoop下载文件时出错org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Ljava/lang/String;I)V 解决办法：找到hadoop对应版本的hadoop.dll，然后将hadoop.dll拷贝到C:\Windows\System32 hadoop.dll3.1.1版本，链接: https://pan.baidu.com/s/1X5quZkuQUP7xZzVIR37.
Java对hdfs操作报如下错误，请问怎么解决？错误如下：Exception in thread "main" java.io.IOException: (null) entry in command string: null chmod 0700 I:\tmp\hadoop-22215\mapred\staging\222151620622033\.staging at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:770) at org.apache.hadoop.util.Shell.execCommand(Shell.java:866) at org.apache.hadoop.util.Shell.execCommand(Shell.java:849) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733) at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:491) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:532) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:509) at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:305) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:133) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:144) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308) at com.sl.maxTemperature.main(maxTemperature.java:41)
04-23
chmod -R 777 I:\tmp\hadoop-22215\mapred\staging\222151620622033\.staging 如果这个错误依然存在，你也可以尝试使用其他账户，或者联系系统管理员解决权限问题。如果还有其他问题，您可以在具体的平台上寻求...
[root@hadoop04 ~]# hadoop jar film1.jar CleanDriver -D mapreduce.job.summary.log.level=DEBUG /film/input /film/outputs/cleandata 25/03/15 14:57:08 INFO client.RMProxy: Connecting to ResourceManager at hadoop04/192.168.100.104:8032 25/03/15 14:57:08 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 25/03/15 14:57:10 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/root/.staging/job_1742021469568_0001 Exception in thread "main" org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://hadoop04:9000/user/root/CleanDriver at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:323) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:265) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:387) at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:301) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:318) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:196) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308) at CleanDriver.main(CleanDriver.java:35) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:226) at org.apache.hadoop.util.RunJar.main(RunJar.java:141) [root@hadoop04 ~]#
03-16
嗯，用户在执行Hadoop作业的时候遇到了问题，报错是Input path does not exist。我需要仔细看看这个错误信息。首先，用户运行的命令是： hadoop jar film1.jar CleanDriver -D mapreduce.job.summary.log.level=...
hive> create databases bclcredits; 2025-06-18 16:00:45,495 INFO conf.HiveConf: Using the default value passed in for log id: 949847db-5ab3-4891-b83b-d3071c4b727b 2025-06-18 16:00:45,495 INFO session.SessionState: Updating thread name to 949847db-5ab3-4891-b83b-d3071c4b727b main 2025-06-18 16:00:45,496 INFO ql.Driver: Compiling command(queryId=aaa_20250618160045_00ce6840-bf74-4e81-a3c7-f7ede97059fb): create databases bclcredits NoViableAltException(85@[917:1: ddlStatement : ( createDatabaseStatement | switchDatabaseStatement | dropDatabaseStatement | createTableStatement | dropTableStatement | truncateTableStatement | alterStatement | descStatement | showStatement | metastoreCheck | createViewStatement | createMaterializedViewStatement | dropViewStatement | dropMaterializedViewStatement | createFunctionStatement | createMacroStatement | dropFunctionStatement | reloadFunctionStatement | dropMacroStatement | analyzeStatement | lockStatement | unlockStatement | lockDatabase | unlockDatabase | createRoleStatement | dropRoleStatement | ( grantPrivileges )=> grantPrivileges | ( revokePrivileges )=> revokePrivileges | showGrants | showRoleGrants | showRolePrincipals | showRoles | grantRole | revokeRole | setRole | showCurrentRole | abortTransactionStatement | killQueryStatement | resourcePlanDdlStatements );]) at org.antlr.runtime.DFA.noViableAlt(DFA.java:158) at org.antlr.runtime.DFA.predict(DFA.java:116) at org.apache.hadoop.hive.ql.parse.HiveParser.ddlStatement(HiveParser.java:4244) at org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:2494) at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1420) at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:220) at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:74) at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:67) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:616) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1826) at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1773) at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1768) at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126) at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:323) at org.apache.hadoop.util.RunJar.main(RunJar.java:236) FAILED: ParseException line 1:7 cannot recognize input near 'create' 'databases' 'bclcredits' in ddl statement 2025-06-18 16:00:45,520 ERROR ql.Driver: FAILED: ParseException line 1:7 cannot recognize input near 'create' 'databases' 'bclcredits' in ddl statement org.apache.hadoop.hive.ql.parse.ParseException: line 1:7 cannot recognize input near 'create' 'databases' 'bclcredits' in ddl statement at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:223) at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:74) at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:67) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:616) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1826) at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1773) at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1768) at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126) at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:323) at org.apache.hadoop.util.RunJar.main(RunJar.java:236) 2025-06-18 16:00:45,521 INFO ql.Driver: Completed compiling command(queryId=aaa_20250618160045_00ce6840-bf74-4e81-a3c7-f7ede97059fb); Time taken: 0.025 seconds 2025-06-18 16:00:45,521 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager 2025-06-18 16:00:45,521 INFO conf.HiveConf: Using the default value passed in for log id: 949847db-5ab3-4891-b83b-d3071c4b727b 2025-06-18 16:00:45,521 INFO session.SessionState: Resetting thread name to main
最新发布
06-19
但是，从错误信息来看，当前错误是由于使用了`databases`（复数）导致的，所以先修正为单数。因此，正确的创建数据库的语句应该是：CREATEDATABASEbclcredits;或者，如果数据库名是关键字或特殊字符，则使用反引号：...
hdfs和本地进行文件互传报错
qq_45771067的博客
09-02 280
在maven项目中进行hdfs和local文件传输时出现异常： org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Ljava/lang/String;JJJI)Ljava/io/FileDescriptor; org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(II[BI[BIILjava/lang/String;JZ)V 解...
Shell run Java
01-04
在Unix Shell脚本里运行Java application
[HDFS]FileSystem.copyToLocalFile下载文件报错Unable to load native-hadoop library for your platform...
05-25 1297
应用在服务器上启动后，从HDFS下载文件失败，报错如下： Unable to load native-hadoop library for your platform... using builtin-java classes where applicable The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 第一句提示不能加载Hadoop的库，检查库，提示hadoop命令
hadoop错误java.io.IOException at org.apache.hadoop.mapred.pipes.OutputHandler。。。
oizz01的专栏
11-23 1133
错误日志 12/09/30 18:35:26 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String). 12/09/30 18:35:26 INFO util.NativeCodeLoader: Loaded
Hadoop远程调试删除文件报错：org.apache.hadoop.security.AccessControlException: Permission denied: user=
小蚯蚓的博客
02-29 1629
一、问题描述 使用Idea操作hdfs上文件，进行文件删除，结果报错因为用户名没有权限，而无法操作对应文件 [ERROR] 09:13:32.239 [main] ERROR com.hadoop.hdfs.HdfsApi - HdfsApi delete IOException org.apache.hadoop.security.AccessControlException: Permis...
Hadoop2.x.x用IDEA直接调用
ZZHMMD957
10-31 304
不知道描述清楚没有，就是在IEDA中直接运行main方法然后本地调用服务器上的mapreduce！！！ 你需要做以下的工作： 1.在windows下解压一个hadoop2.x.x包然后把winutils.exe文件放到bin目录下 2.添加环境变量: HADOOP_HOME=E:\hadoop-2.7.4把%HADOOP_HOME%\bin;加到Path里边 3.把你服务器上的hadoo
hadoop3.1.3项目中get HDFS文件到本地出错
qq_40659767的博客
03-21 787
java idea hadoop get hdfs Could not locate Hadoop executable: ---\bin\winutils.exe HADOOP_HOME and hadoop.home.dir
CDH 5.16.2 异常踩坑
学无止境，永不停歇
05-07 1610
20/05/05 23:52:24 WARN CorruptStatistics: Ignoring statistics because created_by could not be parsed (see PARQUET-251): parquet-mr org.apache.parquet.VersionParser$VersionParseException: Could not par...
hadoop异常之输入路径不存在
alanlonglong的博客
01-23 5582
org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://mycluster/pythontout         at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedList
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值