Ravi Writes

5. Columns and Expressions

3 min read

Columns and Expressions

Column = a named, typed expression over a DataFrame — a reference to a field, or a computation built on one, not data itself. You get one with col("name"), df["name"], df.name, or by combining expressions (col("amount") * 1.1), and pass it into transformations like select, filter, withColumn.

Expression here means a recipe for computing a value, not the value itself — like a spreadsheet formula (=A1*1.1), not the number it evaluates to. It’s typed because once resolved against a DataFrame’s schema it has a concrete Spark SQL type (DoubleType, BooleanType, …). It’s named because every Column has an output name — inherited from the source field, auto-generated from the expression, or set explicitly with .alias() — that becomes the column header when it’s selected.

Example

from pyspark.sql.functions import col, expr
 
df.select(col("amount") * 1.1)
df.filter(col("status") == "ACTIVE")
df.withColumn("amount_rounded", col("amount").cast("int"))
 
# expr() writes the same kind of Column as a SQL-like string instead
df.select(expr("amount * 1.1"))

Column the Type vs col() the Function

In Spark’s supported languages, Column is a type — an object with public methods. col() is just a standard built-in function that constructs and returns a Column object; it isn’t the type itself, and it’s not the only way to get one.

from pyspark.sql import Column
from pyspark.sql.functions import col
 
c = col("amount")
type(c)                # <class 'pyspark.sql.column.Column'>
isinstance(c, Column)  # True
 
# other ways to end up with a Column object — none of these call col()
df["amount"]           # indexing the DataFrame
df.amount               # attribute access
col("amount") * 1.1    # an expression built on a Column is itself a Column
 
# because it's an object, it has public methods you can call directly
c.alias("amt")
c.cast("double")
c.desc()
c.isNull()

Key Properties

  • Not data, an expression — a Column doesn’t hold values; it holds an expression tree describing how to compute or reference one. Nothing runs until it’s used inside a transformation on an actual DataFrame.
  • Composable — Columns combine with operators (+, ==, …) and functions (F.trim, F.upper, …) to build up more complex expressions from simpler ones.
  • Resolved late — a Column referencing a field that doesn’t exist only fails once it’s analyzed against a real DataFrame’s schema, not when the Column expression itself is written.

Ab Initio Equivalent

A Column maps to a field reference inside a Record format or DML expression. In a Reformat’s out.customer_id :: string_lrtrim(in.id);, in.id is the field reference — the Column-equivalent, qualified by its input port — and string_lrtrim(in.id) is that reference combined with a DML function, the same way F.trim(col("id")) combines a Column with a PySpark function. The DML block plays the role the surrounding Python code plays; a single qualified field name inside it plays the role a Column plays.