How to Change a Column Type of a DataFrame in PySpark

Published Jan 3, 2022  ∙  Updated May 2, 2022

How can we change the column type of a DataFrame in PySpark?

Suppose we have a DataFrame df with column num of type string.

Let’s say we want to cast this column into type double.

Luckily, Column provides a cast() method to convert columns into a specified data type.

Cast using cast() and the singleton DataType

We can use the PySpark DataTypes to cast a column type.

from pyspark.sql.types import DoubleType
df = df.withColumn("num", df["num"].cast(DoubleType()))
# OR
df = df.withColumn("num", df.num.cast(DoubleType()))

We can also use the col() function to perform the cast.

from pyspark.sql.functions import col
from pyspark.sql.types import DoubleType
df = df.withColumn("num", col("num").cast(DoubleType())) 

Cast using cast() and simple strings

We can also use simple strings.

from pyspark.sql.types import DoubleType
df = df.withColumn("num", df["num"].cast("double"))
# OR
df = df.withColumn("num", df.num.cast("double"))

Get simple string from DataType

Here is a list of DataTypes to simple strings.

BinaryType: binary
BooleanType: boolean
ByteType: tinyint
DateType: date
DecimalType: decimal(10,0)
DoubleType: double
FloatType: float
IntegerType: int
LongType: bigint
ShortType: smallint
StringType: string
TimestampType: timestamp

Simple strings for any DataType can be obtained using getattr() and simpleString().

We can get the simple string for any DataType like so:

from pyspark.sql import types
simpleString = getattr(types, 'BinaryType')().simpleString()
from pyspark.sql.types import BinaryType
simpleString = BinaryType().simpleString()

We can also write out simple strings for arrays and maps: array<int> and map<string,int>.

Read more about cast() in the PySpark documentation.