spark读取当前处理数据的文件名

最新推荐文章于 2024-06-10 00:43:41 发布

转载最新推荐文章于 2024-06-10 00:43:41 发布 · 4.9k 阅读

spark 专栏收录该内容

3 篇文章

订阅专栏

本文介绍如何使用Spark在Hadoop环境中读取文件名，并提供了两种不同版本的实现方式：一种基于mapreduce，另一种基于mapred。通过具体代码示例展示了如何配置SparkConf，创建SparkContext，以及如何利用NewHadoopRDD和HadoopRDD处理文件。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

参考

http://hanyingjun318.iteye.com/blog/2277512

环境

idea；sbt；

hadoop

在hadoop中读取文件名

InputSplit inputSplit=(InputSplit)context.getInputSplit();  
String filename=((FileSplit)inputSplit).getPath().getName();

spark

spark代码，在本地测试有效，mr有mapred和mapreduce两种
mapreduce

import org.apache.hadoop.io.{LongWritable, Text}
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.hadoop.mapreduce.InputSplit
import org.apache.hadoop.mapreduce.lib.input.{FileSplit, TextInputFormat}
import org.apache.spark.rdd.NewHadoopRDD

object testPath {
  def main(args: Array[String]): Unit = {

    val conf = new SparkConf()
    conf.setAppName("testtoParquet")
    conf.setMaster("local")
    val sc = new SparkContext(conf)
    var input = "/home/dwj/data/testSpark/20180409"
    var output = ""

    val fileRDD = sc.newAPIHadoopFile[LongWritable, Text, TextInputFormat](input)
    val hadoopRDD = fileRDD.asInstanceOf[NewHadoopRDD[LongWritable, Text]]
    val fileAdnLine = hadoopRDD.mapPartitionsWithInputSplit((inputSplit:InputSplit, iterator:Iterator[(LongWritable, Text)]) => {
      val file = inputSplit.asInstanceOf[FileSplit]
      iterator.map(x => {file.getPath.toString()+"\t" + x._2})
    })
    fileAdnLine.foreach(println)
  }
}

mapred

import org.apache.hadoop.io.{LongWritable, Text}
import org.apache.hadoop.mapred.{FileSplit, InputSplit, TextInputFormat}
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.rdd.HadoopRDD

object testPathOld {
  def main(args: Array[String]): Unit = {

    val conf = new SparkConf()
    conf.setAppName("testtoParquet")
    conf.setMaster("local")
    val sc = new SparkContext(conf)
    var input = "/home/dwj/data/testSpark/20180409"

    val fileRDD = sc.hadoopFile[LongWritable, Text, TextInputFormat](input)
    val hadoopRDD = fileRDD.asInstanceOf[HadoopRDD[LongWritable, Text]]
    val fileAdnLine = hadoopRDD.mapPartitionsWithInputSplit((inputSplit:InputSplit,iterator:Iterator[(LongWritable, Text)]) => {
      val file = inputSplit.asInstanceOf[FileSplit]
      iterator.map(x => {file.getPath.toString()+"\t" + x._2})
    })
    fileAdnLine.foreach(println)
  }
}