value toDF is not a member of org.apache.spark.rdd.RDD[People]-CSDN博客

本文介绍了一个常见的Spark SQL错误“valuetoDFisnotamemberoforg.apache.Spark.rdd.RDD[People]”的原因及解决方案。指出仅需将case class定义移出作用域即可解决问题，并提供了一段完整的Spark SQL应用示例。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

编译如下代码时，出现value toDF is not a member of org.apache.Spark.rdd.RDD[People] 错误

val rdd : RDD[People]= sparkSession.sparkContext.textFile(hdfsFile,2).map(line => line.split(",")).map(arr => People(arr(0),arr(1).trim.toInt))

rdd.toDF

参考http://stackoverflow.com/questions/33704831/value-todf-is-not-a-member-of-org-apache-spark-rdd-rdd，针对此错误有人提出需要做到以下两点

1. import sqlContext.implicits._ 语句需要放在获取sqlContext对象的语句之后

2. case class People(name : String, age : Int) 的定义需要放在方法的作用域之外（即Java的成员变量位置）

实际上只需要做到第二点即可解决错误，如下：

import org.apache.spark.{SparkContext, SparkConf}

/**
 * Created by ESRI on 2017/6/14.
 */

object sqltest2 {
  case class Person(name: String, age: Int)
  def main(args: Array[String]) {
    println("I Love You Scala")

    System.setProperty("hadoop.home.dir", "E:\\bigdataTools\\hadoop\\hadoop-2.6.0\\hadoop-2.6.0")
    val conf = new SparkConf().setMaster("local").setAppName("wordCount")
    val sc = new SparkContext(conf)
    val sqlContext = new org.apache.spark.sql.SQLContext(sc)

    import sqlContext.implicits._

    // Define the schema using a case class.
    // Note: Case classes in Scala 2.10 can support only up to 22 fields. To work around this limit,
    // you can use custom classes that implement the Product interface.


    // Create an RDD of Person objects and register it as a table.
    val people = sc.textFile("E:\\testData\\spark\\spark1.6\\people.txt").map(_.split(",")).map(p => Person(p(0).trim.toString, p(1).trim.toInt)).toDF()
    people.registerTempTable("people")

    // SQL statements can be run by using the sql methods provided by sqlContext.
    val teenagers = sqlContext.sql("SELECT name, age FROM people WHERE age >= 13 AND age <= 19")

    // The results of SQL queries are DataFrames and support all the normal RDD operations.
    // The columns of a row in the result can be accessed by field index:
    teenagers.map(t => "Name: " + t(0)).collect().foreach(println)

    // or by field name:
    teenagers.map(t => "Name: " + t.getAs[String]("name")).collect().foreach(println)

    // row.getValuesMap[T] retrieves multiple columns at once into a Map[String, T]
    //teenagers.map(_.getValuesMap[Any](List("name", "age"))).collect().foreach(println)

  }
}