【Spark】Spark常用方法总结3-DataFrame常用方法（Python版本）

最新推荐文章于 2024-04-27 03:21:45 发布

「已注销」

最新推荐文章于 2024-04-27 03:21:45 发布

阅读量584

点赞数

CC 4.0 BY-SA版权

分类专栏： Spark 文章标签：大数据 spark hadoop

本文链接：https://blog.csdn.net/refbit/article/details/104109126

Spark 专栏收录该内容

5 篇文章

订阅专栏

本文总结了Spark DataFrame在Python中的常用操作，包括创建、显示、创建虚拟表、过滤、连接和选择等关键方法。通过实例展示了如何使用这些方法进行数据处理，如`show()`、`createGlobalTempView()`、`filter()`/`where()`、`join()`及`distinct()`和`select()`。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

创建

df1 = spark.read.load(r'E:\常用基础数据\po.csv', 'csv')

schema = StructType(
    fields=list(map(lambda x: StructField(x, StringType(), True), [chr(ord('A') + x) for x in range(10)])))
df1 = spark.read.load(r'E:\常用基础数据\po.csv', 'csv', schema=schema)

show

#默认是(n=20, truncate=True, vertical=False)
n显示多少条记录, truncate是否省略长度大于20的字符串, vertical竖向显示

df1.show(n=20, truncate=True, vertical=False)

创建虚拟表

创建

df1.createOrReplaceTempView('purchase_order1')
df1.createGlobalTempView('purchase_order2')
df1.createTempView('purchase_order3')

查询

df2 = spark.sql('select * from purchase_order1 limit 10')

createGlobalTempView需要加global_temp

df3 = spark.sql('select * from global_temp.purchase_order2 limit 10')
df4 = spark.sql('select * from purchase_order3 limit 10')

filter/where

df1.filter(df1.A == 1)
df1.filter('A = 1')

join

other – Right side of the join
on – a string for the join column name, a list of column names, a join expression (Column), or a list of Columns. If on is a string or a list of strings indicating the name of the join column(s), the column(s) must exist on both sides, and this performs an equi-join.
how – str, default inner. Must be one of: inner, cross, outer, full, full_outer, left, left_outer, right, right_outer, left_semi, and left_anti.

join_df = df3.join(other=df4, on=(df3.H == df4.B), how='left_outer')
join_df = join_df.filter('fc.A is null')

distinct

buyer_names = df3.select(df3.D)
34693
print(buyer_names.count())
173
print(buyer_names.distinct().count())

select

a_col = df3.select(df3['A'])
a_col = df3.select((df3['A'] * 100).alias('test'))