DataFrame
DataFrame = a distributed collection of rows, organized into named columns with a defined schema — Spark’s core structured data abstraction. Conceptually like a table (or a pandas DataFrame), but physically split across partitions, spread across executors, and evaluated lazily.
Example
data = [(1, "Alice", 250.0), (2, "Bob", 400.0)]
schema = ["id", "name", "amount"]
df = spark.createDataFrame(data, schema)
df.show()
# +---+-----+------+
# | id| name|amount|
# +---+-----+------+
# | 1|Alice| 250.0|
# | 2| Bob| 400.0|
# +---+-----+------+
df.printSchema()
# root
# |-- id: long (nullable = true)
# |-- name: string (nullable = true)
# |-- amount: double (nullable = true)Key Properties
- Distributed — rows are spread across partitions, not held in one place; operations run in parallel across executors.
- Schema — every DataFrame has a schema (column names + types), whether inferred or explicitly declared.
- Immutable — transformations never modify a DataFrame in place; they return a new DataFrame.
- Lazy — transformations (
select,filter,withColumn, …) just build up a logical plan; nothing actually runs until an action (show,write,collect, …) triggers it.
Ab Initio Equivalent
A DataFrame isn’t a component — it’s data, not a processing step. Think of it as the record stream flowing along a flow line between components, shaped by a DML record format. A DataFrame’s schema is the DML record format. A DataFrame’s partitions are the parallel lanes of that flow line under a given layout.
graph LR A["Input File"] -->|"out → in (record stream)"| B["Filter By Expression"] B -->|"out → in (record stream)"| C["Reformat"] C -->|"out → in (record stream)"| D["Output File"]
The components (Input File, Filter By Expression, Reformat, Output File) aren’t the DataFrame — they’re the transformations. The flow line connecting an out port to the next component’s in port is what’s carrying the record stream, and that is the DataFrame equivalent: not the ports themselves, but the stream of records passing through them.
Dataset vs DataFrame
Spark actually has two structured APIs: Dataset and DataFrame. A DataFrame is, under the hood, just a Dataset[Row] — an untyped Dataset, checked at runtime rather than compile time. The typed version, Dataset[T], only exists in Scala and Java, since it relies on compile-time type checking that a dynamically-typed language like Python doesn’t have. So in PySpark, DataFrame isn’t a simplified version of something else — it’s the only structured API there is.