Spark Programming
map(): 将源RDD的每一个元素通过一个函数处理并返回新的RDD
def add(x):
return x+1
rdd_data = spark.parallelize([1,2,3])
results = rdd_data.map(add)
results.collect() #[2,3,4]
filter:选中源数据中使得filter条件返回true的元素,组成新的数据集
rdd_data2 = spark.parallelize([3,1,2,5,5])
result3 = rdd_data2.filter(lambda x:x>2)
result3.collect #[3,5,5]
rdd_data2 = spark.parallelize([3,1,2,5,5,5])
result4 = rdd_data2.distinct()
reulst4.collect()
randomSplit(): 按给定的权重将RDD切分成多个子段
rdd_data3 = spark.parallelize(range(10),1) #创建1-10的list
result = rdd_data3.randomSplit([0.6,0.4],seed=42) #加入seed使得切分模式可复现
print('first:%s'%res[0].collect()) #权重0.6的segment
print('second:%s'%res[1].collect()) #权重0.4的segment
groupBy():按照定义的条件返回分组后的RDD
rdd_data4 = spark.parallelize([1,1,2,3,5,8])
result = rdd_data4.groupBy(lambda x: x%2) #能被2整除的数字
res = result.collect()
faltMap(): 对每个元素进行map处理并将结果摊平
rdd_data7 = spark.parallelize(['hello world', 'hi', 'dog'])
result7 = rdd_data7.flatMap(lambda x: x.split(' ')) # split each phrase by space
result7.collect()
#flatMap() output:['hello', 'world', 'hi', 'dog', 'fish', 'orange', 'apple']
result8 = rdd_data7.map(lambda x: x.split(' '))
result8.collect()
#map() output: [['hello', 'world'], ['hi'], ['dog'], ['fish'], ['orange'], ['apple']]
union(): 返回所有输入RDD的拼接结果(首尾相接)
rddData1 = spark.parallelize([1, 1, 2, 3])
rddData2 = spark.parallelize([4, 5, 6, 7])
rddData3 = spark.parallelize([21, 22, 23, 25])
rddData1.union(rddData2).union(rddData3).collect()
#output:[1,1,2,3,4,5,6,7,21,22,23,25]
intersection(): 返回所有输入RDD的重叠部分
rddData1 = spark.parallelize([1, 1, 2, 3])
rddData2 = spark.parallelize([3, 4, 5, 7])
rddData1.intersection(rddData2).collect()
#output:[3]
subtract(): 返回self中不被other包含的部分
rddData1 = spark.parallelize([1, 1, 2, 3])
rddData2 = spark.parallelize([3, 4, 5, 7])
sorted(rddData1.subtract(rddData2).collect())
#output:[1,1,2]
cartesian(): 返回多个RDD的笛卡尔积(把每一个元素都相互组合)
rddData1 = spark.parallelize([1, 1, 2, 3])
rddData2 = spark.parallelize([3, 4, 5, 7])
rddData1.cartesian(rddData2).collect()
#output:[(1, 3)...(3, 7)]
RDD Action
rddData1 = spark.parallelize([21,1,2,3])
rddData1.first()
#output:21
take(): 取RDD中的前N个元素 (返回的是集合)
rddData1 = spark.parallelize([21, 1, 2, 3])
rddData1.take(2)
#output:[21,1]
min()/max()/stdev(): 分别取最小值/最大值/标准差
spark.parallelize([10, 1, 2, 9, 3, 4, 5, 6, 7]).min() #1
spark.parallelize([10, 1, 2, 9, 3, 4, 5, 6, 7]).max() #10
spark.parallelize([10, 1, 2, 9, 3, 4, 5, 6, 7]).stdev () #10
spark.parallelize([1, 2, 3, 4]).mean() #2.5
spark.parallelize([2, 3, 4]).count() #3
spark.parallelize([1, 2, 3, 4]).sum() #10
- key-value pair RDD: 是一种特殊的RDD, 所有数据都以键值对形式存放,key充当唯一的ID
filter()
kvRDD = spark.parallelize([(3,4), (3,6), (5,6), (1,2)])
kvRDD.filter(lambda kv: kv[0] < 5).collect() #kv[0] = key,此处保留key<5的元素
kvRDD = spark.parallelize([(3,4), (3,6), (5,6), (1,2)])
kvRDD.filter(lambda kv: kv[1] < 5).collect() #kv[1] = value,此处保留value<5的元素
mapValue():对每一个 value 元素施加函数
kvRDD = spark.parallelize([(3,4),(3,1),(5,6),(1,2)])
kvRDD.mapValues(lambda kv: kv**2).collect()
reduceByKey(): 把key归类并对同类的value应用聚合函数
from operator import add
kvRDD = spark.parallelize([(3,4), (3,1), (5,6), (1,2)])
kvRDD.reduceByKey(add).collect() #此处应用相加总和
链接