Hadoop源码详解之Job 类

最新推荐文章于 2024-07-29 03:29:36 发布

说文科技

最新推荐文章于 2024-07-29 03:29:36 发布

阅读量2.4k

点赞数

CC 4.0 BY-SA版权

分类专栏： # Hadoop

喜欢文章？请私信联系作者。

本文链接：https://blog.csdn.net/liu16659/article/details/85196352

Hadoop 专栏收录该内容

52 篇文章

订阅专栏

本文深入剖析了Hadoop中Job类的功能与用法，包括其构造器、waitForCompletion方法等关键部分，展示了如何通过Job类配置并提交MapReduce任务。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Hadoop源码详解之`Job`类

1. 源码

包：org.apache.hadoop.mapreduce
继承的接口有：AutoCloseable，JobContext，org.apache.hadoop.mapreduce.MRJobConfig

The job submitter’s view of the Job.
It allows the user to configure the job, submit it, control its execution, and query the state. The set methods only work until the job is submitted, afterwards they will throw an IllegalStateException.
Normally the user creates the application, describes various facets of the job via Job and then submits the job and monitor its progress.

作业提交者层次上的作业视图。
它允许用户配置job，提交它，并且控制它的运行，然后查询状态。set 方法仅仅工作直到job被提交，否则会抛出IllegalStateException。
正常情况下，用户创建一个应用，描述工作的各个方面，通过Job 类，并且提交job，然后监测它的进度。
下面给出一个示例关于如何使用 Job 类去提交一个job。

	  // Create a new Job
     Job job = Job.getInstance();
     job.setJarByClass(MyJob.class);
     
     // Specify various job-specific parameters     
     job.setJobName("myjob");
     
     job.setInputPath(new Path("in"));
     job.setOutputPath(new Path("out"));
     
     job.setMapperClass(MyJob.MyMapper.class);
     job.setReducerClass(MyJob.MyReducer.class);

     // Submit the job, then poll for progress until the job is complete
     job.waitForCompletion(true);

2. 方法详解

2.1 构造器

在这里插入图片描述
以前的构造器全部建议不再使用，转而使用getInistance(...)这个方法。

2.2 `waitForCompletion`

方法释义

Submit the job to the cluster and wait for it to finish.
提交job到集群，并且等待它完成。

方法源码

 /**
   * @param verbose print the progress to the user
   * @return true if the job succeeded
   * @throws IOException thrown if the communication with the JobTracker is lost
   */
  public boolean waitForCompletion(boolean verbose
                                   ) throws IOException, InterruptedException,
                                            ClassNotFoundException {
    if (state == JobState.DEFINE) {
      submit();
    }
    if (verbose) {
      monitorAndPrintJob();
    } else {
      // get the completion poll interval from the client.
      int completionPollIntervalMillis = 
        Job.getCompletionPollInterval(cluster.getConf());
      while (!isComplete()) {
        try {
          Thread.sleep(completionPollIntervalMillis);
        } catch (InterruptedException ie) {
        }
      }
    }
    return isSuccessful();
  }