SparkStreaming - 写入到mysql ForeachRdd

最新推荐文章于 2025-05-21 15:52:24 发布

大米饭精灵

最新推荐文章于 2025-05-21 15:52:24 发布

阅读量490

点赞数

CC 4.0 BY-SA版权

分类专栏： SparkStreaming 文章标签： SparkStreaming

本文链接：https://blog.csdn.net/qq_15300683/article/details/80689998

SparkStreaming 专栏收录该内容

7 篇文章

订阅专栏

本文介绍如何使用Apache Spark Streaming处理实时数据流，并通过最佳实践将处理后的数据批量写入MySQL数据库。文章展示了如何设置Spark环境、创建数据流、处理数据以及实现与MySQL的集成。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

import java.sql.DriverManager import org.apache.spark.SparkConf import org.apache.spark.streaming.{Seconds, StreamingContext} object ForeachRDDApp { def main(args: Array[String]) { val sparkConf = new SparkConf() .setAppName("ForeachRDDApp") .setMaster("local[2]") val ssc = new StreamingContext(sparkConf, Seconds(10)) val lines = ssc.socketTextStream("hadoop000",9997) val results = lines.flatMap(_.split(",")).map((_,1)).reduceByKey(_+_) // TODO... 将results写入到MySQL中 // results.foreachRDD(rdd => { // rdd.foreach(x => { // val connection = createConnection() // val word = x._1 // val count = x._2.toInt // val sql = s"insert into wc(word, c) values ('$word', $count)" // connection.createStatement().execute(sql) // }) // }) // 最佳实践 results.foreachRDD(rdd => { rdd.foreachPartition(partition => { val connection = createConnection() partition.foreach(x => { val word = x._1 val count = x._2.toInt val sql = s"insert into wc(word, c) values ('$word', $count)" connection.createStatement().execute(sql) }) connection.close() }) rdd.foreach(x => { val connection = createConnection() val word = x._1 val count = x._2.toInt val sql = s"insert into wc(word, c) values ('$word', $count)" connection.createStatement().execute(sql) }) }) ssc.start() // 一定要写 // lines.print() ssc.awaitTermination() } def createConnection() = { Class.forName("com.mysql.jdbc.Driver") DriverManager.getConnection("jdbc:mysql://hadoop000:3306/ss2","root","root") } }