Spark Programming

RDD Transformation(single)

  • map(): 将源RDD的每一个元素通过一个函数处理并返回新的RDD
def add(x):
	return x+1
	
rdd_data = spark.parallelize([1,2,3])
results = rdd_data.map(add)
results.collect()    #[2,3,4]
  • filter:选中源数据中使得filter条件返回true的元素,组成新的数据集
rdd_data2 = spark.parallelize([3,1,2,5,5])
result3 = rdd_data2.filter(lambda x:x>2)
result3.collect    #[3,5,5]
  • distinct():剔除重复项
rdd_data2 = spark.parallelize([3,1,2,5,5,5])
result4 = rdd_data2.distinct()
reulst4.collect()
  • randomSplit(): 按给定的权重将RDD切分成多个子段
rdd_data3 = spark.parallelize(range(10),1)    #创建1-10的list
result = rdd_data3.randomSplit([0.6,0.4],seed=42)    #加入seed使得切分模式可复现
print('first:%s'%res[0].collect())    #权重0.6的segment
print('second:%s'%res[1].collect())    #权重0.4的segment
  • groupBy():按照定义的条件返回分组后的RDD
rdd_data4 = spark.parallelize([1,1,2,3,5,8])
result = rdd_data4.groupBy(lambda x: x%2)    #能被2整除的数字
res = result.collect()
  • faltMap(): 对每个元素进行map处理并将结果摊平
rdd_data7 = spark.parallelize(['hello world', 'hi', 'dog'])
result7 = rdd_data7.flatMap(lambda x: x.split(' ')) # split each phrase by space
result7.collect()
#flatMap() output:['hello', 'world', 'hi', 'dog', 'fish', 'orange', 'apple']
result8 = rdd_data7.map(lambda x: x.split(' '))
result8.collect()
#map() output: [['hello', 'world'], ['hi'], ['dog'], ['fish'], ['orange'], ['apple']]

RDD Transformation(multiple)

  • union(): 返回所有输入RDD的拼接结果(首尾相接)
rddData1 = spark.parallelize([1, 1, 2, 3])
rddData2 = spark.parallelize([4, 5, 6, 7])
rddData3 = spark.parallelize([21, 22, 23, 25])
rddData1.union(rddData2).union(rddData3).collect()
#output:[1,1,2,3,4,5,6,7,21,22,23,25]
  • intersection(): 返回所有输入RDD的重叠部分
rddData1 = spark.parallelize([1, 1, 2, 3])
rddData2 = spark.parallelize([3, 4, 5, 7])
rddData1.intersection(rddData2).collect()
#output:[3]
  • subtract(): 返回self中不被other包含的部分
rddData1 = spark.parallelize([1, 1, 2, 3])
rddData2 = spark.parallelize([3, 4, 5, 7])
sorted(rddData1.subtract(rddData2).collect())
#output:[1,1,2]
  • cartesian(): 返回多个RDD的笛卡尔积(把每一个元素都相互组合)
rddData1 = spark.parallelize([1, 1, 2, 3])
rddData2 = spark.parallelize([3, 4, 5, 7])
rddData1.cartesian(rddData2).collect()
#output:[(1, 3)...(3, 7)]

RDD Action

  • fisrt(): 返回RDD的第一个元素
rddData1 = spark.parallelize([21,1,2,3])
rddData1.first()
#output:21
  • take(): 取RDD中的前N个元素 (返回的是集合)
rddData1 = spark.parallelize([21, 1, 2, 3])
rddData1.take(2) 
#output:[21,1]
  • min()/max()/stdev(): 分别取最小值/最大值/标准差
spark.parallelize([10, 1, 2, 9, 3, 4, 5, 6, 7]).min()    #1
spark.parallelize([10, 1, 2, 9, 3, 4, 5, 6, 7]).max()    #10
spark.parallelize([10, 1, 2, 9, 3, 4, 5, 6, 7]).stdev ()    #10
  • mean()/count()/sum()
spark.parallelize([1, 2, 3, 4]).mean()    #2.5
spark.parallelize([2, 3, 4]).count()    #3
spark.parallelize([1, 2, 3, 4]).sum()    #10

Single Key-Value Pair RDD Transformation

  • key-value pair RDD: 是一种特殊的RDD, 所有数据都以键值对形式存放,key充当唯一的ID
  • filter()
kvRDD = spark.parallelize([(3,4), (3,6), (5,6), (1,2)])
kvRDD.filter(lambda kv: kv[0] < 5).collect()   #kv[0] = key,此处保留key<5的元素
kvRDD = spark.parallelize([(3,4), (3,6), (5,6), (1,2)])
kvRDD.filter(lambda kv: kv[1] < 5).collect()   #kv[1] = value,此处保留value<5的元素
  • mapValue():对每一个 value 元素施加函数
kvRDD = spark.parallelize([(3,4),(3,1),(5,6),(1,2)])
kvRDD.mapValues(lambda kv: kv**2).collect()
  • reduceByKey(): 把key归类并对同类的value应用聚合函数
from operator import add
kvRDD = spark.parallelize([(3,4), (3,1), (5,6), (1,2)])
kvRDD.reduceByKey(add).collect()    #此处应用相加总和

Multiple Key-Value Pair RDDs Transformations


链接