如何将嵌套列添加到DataFrame
问题描述:
我有一个具有以下架构的数据框df
:
I have a dataframe df
with the following schema:
root
|-- city_name: string (nullable = true)
|-- person: struct (nullable = true)
| |-- age: long (nullable = true)
| |-- name: string (nullable = true)
我想做的是在我的person
结构中添加一个嵌套列,例如car_brand
.我该怎么办?
What I want to do is add a nested column, say car_brand
to my person
structure. How would I do it?
预期的最终模式如下:
root
|-- city_name: string (nullable = true)
|-- person: struct (nullable = true)
| |-- age: long (nullable = true)
| |-- name: string (nullable = true)
| |-- car_brand: string (nullable = true)
答
您可以解压缩该结构并将其添加到新的结构中,包括同时添加新的列.例如,将"bmw"添加到数据框中的所有用户都是这样的:
You can unpack the struct and add it to a new one, including the new column at the same time. For example, adding "bmw" to all persons in the dataframe be done like this:
df.withColumn("person", struct($"person.*", lit("bmw").as("car_brand")))