spark on yarn 报 org.apache.hadoop.util.Shell$ExitCodeException: 问题

最新推荐文章于 2023-04-15 02:58:06 发布

原创最新推荐文章于 2023-04-15 02:58:06 发布 · 3.6k 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#yarn #spark #jdk #hadoop

spark 专栏收录该内容

6 篇文章

订阅专栏

在将基于Spark 1.6、JDK 1.7和Hadoop 2.7的Scala程序移植到使用JDK 1.6、Hadoop 2.2的环境中进行Spark on YARN测试时，遇到`org.apache.hadoop.util.Shell$ExitCodeException`错误。尝试更新jar、driver-class-path等方式未解决问题。通过分析日志发现，问题可能与JDK版本有关。实验表明，使用JDK 1.7的NodeManager可以成功运行，而JDK 1.6的则失败。最终，通过降级到Spark 1.4.0成功解决了问题，证实问题确实由版本不兼容引起。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

今天把新数据中心基于 spark 1.6 jdk 1.7 hadoop2.7的scala程序拿到老数据中心jdk1.6/jdk.17 hadoop 2.2 环境进行 spark on yarn测试
使用spark-shell 或者 spark-submit 提交时报如下错误

org.apache.hadoop.util.Shell$ExitCodeException: 
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
        at org.apache.hadoop.util.Shell.run(Shell.java:379)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)


.Failing this attempt.. Failing the application.
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: 0
         queue: default
         start time: 1481532476948
         final status: FAILED
         tracking URL: xxx:8088/cluster/app/application_1480324693568_0053

Exception in thread "main" org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:124)
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:64)
        at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:530)
        at Regex$.main(Regex.scala:14)
        at Regex.main(Regex.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

查看container日志报如下错误

xxx(&container_1480324693568_0052_01_000001έstderr661Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/deploy/yarn/ExecutorLauncher
Caused by: java.lang.ClassNotFoundException: org.apache.spark.deploy.yarn.ExecutorLauncher
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

一开始以为是 spark基于hadoop的包spark_home/lib/spark-assembly-1.6.1-hadoop2.3.0.jar 没有加载还试着用–jar 或者 –diver-class-path 加载但是后来发现了启动客户端日志里面有这句话

16/12/12 16:47:54 INFO yarn.Client: Uploading resource file:/home/xxx/spark-1.6.1/lib/spark-assembly-1.6.1-hadoop2.3.0.jar -> hdfs://ns1/user/xxx/.sparkStaging/application_1480324693568_0053/spark-assembly-1.6.1-hadoop2.3.0.jar

说明该工具包已经上送的yarn环境还怀疑这个类不在这个包里但是通过eclipse 反查类这个类是在里面的
比较巧合的是程序在有两个节点运行成功了后来发现这两个节点的nodemanager是jdk1.7运行的而其他的失败的节点是 jdk1.6运行的所以怀疑是jdk版本早造成的所以换这个环境试试spark-1.4.0-bin-hadoop2.3
最后成功
看来还是版本的问题