Hadoop实战（一）之统计平均分

Hadoop实战（一）之统计平均分

前言

Hadoop中分为hdfs、mapreduce等等，本文以mapreduce的一个代码实例来加深对mapreduce的理解

一、mapreduce是什么？

MapReduce 是一个分布式运算程序的编程框架，是用户开发基于 Hadoop 的数据分析应用的核心框架，MapReduce分为map阶段和reduce阶段，在此不再赘述

二、项目分析

1.项目要求

要求如下：生成数据为常用APP的调查问卷数据，一共有n个用户的评分数据，是每个用户对这30个APP的评分
需求：统计出每个APP的每个指标的平均分数
数据格式如下，中间使用的是制表符分隔：
用户ID APP名称 UI设计 响应速度 使用率 喜爱程度
测试数据样式：

生成的结果样式：

2.开始编写代码

2.1.编写mapper类

代码如下（示例）：

class CountAverageMapper extends Mapper<LongWritable, Text, Text, FlowBean> {
    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        //将输入的纯文本文件的数据转化为String
        String line = value.toString();
        //将输入的数据按行进行分割
        String[] strings = StringUtils.split(line, "\t");
        //分别对每一行进行处理
        //APP名称部分
        String stringName = strings[1];
        //APP评分部分
        double number1 = Double.parseDouble(strings[2]);
        double number2 = Double.parseDouble(strings[3]);
        double number3 = Double.parseDouble(strings[4]);
        double number4 = Double.parseDouble(strings[5]);
        double sum = number1 + number2 + number3 + number4;
        context.write(new Text(stringName), new FlowBean(stringName, number1, number2, number3, number4, sum));
    }
}

先对输入数据进行处理，但很明显，需要自定义数据格式，所以在此又写了一个类

2.2.自定义数据格式，新建一个类

代码如下（示例）：

public class FlowBean implements Writable {

    public double getNumber1() {
        return number1;
    }

    public void setNumber1(double number1) {
        this.number1 = number1;
    }

    private double number1;

    public double getNumber2() {
        return number2;
    }

    public void setNumber2(double number2) {
        this.number2 = number2;
    }

    private double number2;

    public double getNumber3() {
        return number3;
    }

    public void setNumber3(double number3) {
        this.number3 = number3;
    }

    private double number3;

    public double getNumber4() {
        return number4;
    }

    public void setNumber4(double number4) {
        this.number4 = number4;
    }

    private double number4;

    public double getSum() {
        return sum;
    }

    public void setSum(double sum) {
        this.sum = sum;
    }

    private double sum;

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    private String name;

    public FlowBean() {

    }

    public FlowBean(String name, double number1, double number2, double number3, double number4, double sum) {
        super();
        this.name = name;
        this.number1 = number1;
        this.number2 = number2;
        this.number3 = number3;
        this.number4 = number4;
        this.sum =  Double.parseDouble(String.format("%.2f",(number1 + number2 + number3 + number4)/4));
    }
    @Override
    public void readFields(DataInput dataInput) throws IOException {
        name = dataInput.readUTF();
        number1 = dataInput.readDouble();
        number2 = dataInput.readDouble();
        number3 = dataInput.readDouble();
        number4 = dataInput.readDouble();
        sum = dataInput.readDouble();
    }
    @Override
    public void write(DataOutput dataOutput) throws IOException {
        dataOutput.writeUTF(name);
        dataOutput.writeDouble(number1);
        dataOutput.writeDouble(number2);
        dataOutput.writeDouble(number3);
        dataOutput.writeDouble(number4);
        dataOutput.writeDouble(sum);
    }

    @Override
    public String toString() {
        return this.number1 + "\t" + this.number2 + "\t" + this.number3 + "\t" + this.number4 + "\t" + this.sum;
    }
}

2.3.重写reducer类

class CountAverageReducer extends Reducer<Text, FlowBean, Text, FlowBean> {
    @Override
    protected void reduce(Text key, Iterable<FlowBean> values, Context context) throws IOException, InterruptedException {
        double number1 = 0;
        double number2 = 0;
        double number3 = 0;
        double number4 = 0;
        double sum = 0;
        int count = 0;
       for (FlowBean iterator:values){
            //计算分数
            number1 += iterator.getNumber1();
            number2 += iterator.getNumber2();
            number3 += iterator.getNumber3();
            number4 += iterator.getNumber4();
            sum += iterator.getSum();
            //统计出现的次数
            count++;
        }
        //计算平均分
        double average = number1 / count;
        double average2 = number2 / count;
        double average3 = number3 / count;
        double average4 = number4 / count;
        double avg = sum / count;
        context.write(key, new FlowBean(key.toString(),Double.parseDouble(String.format("%.2f",average)),
                Double.parseDouble(String.format("%.2f",average2)), Double.parseDouble(String.format("%.2f",average3)),
                Double.parseDouble(String.format("%.2f",average4)), Double.parseDouble(String.format("%.2f",avg))));

    }
}

2.4.main方法

public class CountAverage {
    public static void main(String[] args) throws Exception {
        //创建配置对象
        Configuration configuration = new Configuration();

        //创建操作对象
        Job job = Job.getInstance(configuration);
        job.setJarByClass(CountAverage.class);
        //设置Map，Combine，Reduce处理类
        job.setMapperClass(CountAverageMapper.class);
        job.setReducerClass(CountAverageReducer.class);

        //必须手动设置map的输出类型
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(FlowBean.class);

        //将输入的数据集分割成小数据块splites，提供一个RecordReder
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(FlowBean.class);

        //提供一个RecordWriter的实现，负责数据输出
        job.setOutputFormatClass(TextOutputFormat.class);
        //设置输入和输出目录
        FileInputFormat.addInputPath(job, new Path("F:\\wc\\homeworkin\\2.log"));
        FileOutputFormat.setOutputPath(job, new Path("C:\\Users\\不归客\\Desktop\\output1"));

        job.waitForCompletion(true);
    }
}

3.运行代码，测试

生成结果如下，可以发现是比较让人满意的

总结

这次讲解的实例主要是为了锻炼mapreduce的自定义数据格式，来让读者更容易掌握mapreduce