仅使用DataFrame API规范Spark DataFrame中多个列的值
问题描述：
我试图通过减去均值并除以每列的stddev来标准化spark数据帧中多列的值.这是我到目前为止的代码:
I am trying to normalize the values of multiple columns in a spark dataframe, by subtracting the mean and dividing by the stddev of each column. Here's the code I have so far:
from pyspark.sql import Row
from pyspark.sql.functions import stddev_pop, avg
df = spark.createDataFrame([Row(A=1, B=6), Row(A=2, B=7), Row(A=3, B=8),
Row(A=4, B=9), Row(A=5, B=10)])
exprs = [x - (avg(x)) / stddev_pop(x) for x in df.columns]
df.select(exprs).show()
哪个给了我结果?
+------------------------------+------------------------------+
|(A - (avg(A) / stddev_pop(A)))|(B - (avg(B) / stddev_pop(B)))|
+------------------------------+------------------------------+
| null| null|
+------------------------------+------------------------------+
我希望的地方:
+------------------------------+------------------------------+
|(A - (avg(A) / stddev_pop(A)))|(B - (avg(B) / stddev_pop(B)))|
+------------------------------+------------------------------+
| -1.414213562| -1.414213562|
| -0.707106781| -0.707106781|
| 0| 0|
| 0.707106781| 0.707106781|
| 1.414213562| 1.414213562|
+------------------------------+------------------------------+
我相信我可以使用 StandardScaler mllib中的类，但我更愿意在可能的情况下仅使用dataframe API进行此操作-如果只是作为学习练习.
I believe I can do this with the StandardScaler class from mllib, but I'd prefer to do this using only the dataframe API if possible - if only as a learning exercise.
答
With thanks to the answer here, I came up with this:
from pyspark.sql.functions import stddev_pop, avg, broadcast
cols = df.columns
stats = (df.groupBy().agg(
*([stddev_pop(x).alias(x + '_stddev') for x in cols] +
[avg(x).alias(x + '_avg') for x in cols])))
df = df.join(broadcast(stats))
exprs = [(df[x] - df[x + '_avg']) / df[x + '_stddev'] for x in cols]
df.select(exprs).show()
+------------------------+------------------------+
|((A - A_avg) / A_stddev)|((B - B_avg) / B_stddev)|
+------------------------+------------------------+
| -1.414213562373095| -1.414213562373095|
| -0.7071067811865475| -0.7071067811865475|
| 0.0| 0.0|
| 0.7071067811865475| 0.7071067811865475|
| 1.414213562373095| 1.414213562373095|
+------------------------+------------------------+