Ravi Writes

6. Rows

4 min read

Rows

Row = a single record within a DataFrame — an ordered collection of fields, accessible by name or position. It’s what you get back when you pull data out of the distributed, lazy world into concrete Python objects, e.g. via .collect(), .first(), or .take().

Because a Row is an object — just an ordered collection of fields — you can instantiate one directly in each of Spark’s supported languages (Scala, Java, Python, R), standalone, without needing an existing DataFrame first. That’s different from Dataset[T] (see DataFrame), which exists only in Scala and Java because it depends on compile-time typing. A Row has no such restriction: it’s a generic, language-level container for field values, so every language gets the same Row(...) constructor, not a scaled-down version of something bigger.

Example

from pyspark.sql import Row
 
r = Row(id=1, name="Alice", amount=250.0)
type(r)       # <class 'pyspark.sql.types.Row'>
r.name        # "Alice"
r["name"]     # "Alice"
r[1]          # "Alice"
 
rows = df.collect()
# [Row(id=1, name='Alice', amount=250.0), Row(id=2, name='Bob', amount=400.0)]
 
rows[0][0]    # 1        — id of the first row, by position
rows[0][1]    # "Alice"  — name of the first row, by position
 
# a DataFrame can be built directly from a list of Rows
data = [Row(id=1, name="Alice", amount=250.0), Row(id=2, name="Bob", amount=400.0)]
df2 = spark.createDataFrame(data)
df2.show()
# +---+-----+------+
# | id| name|amount|
# +---+-----+------+
# |  1|Alice| 250.0|
# |  2|  Bob| 400.0|
# +---+-----+------+

Where Rows Actually Show Up

Day-to-day PySpark work stays column-oriented — DataFrame + Column expressions — because that’s what lets Spark optimize and parallelize. Row shows up specifically at the boundaries where you step outside that world:

  • Pulling results to the driver — .collect(), .first(), .take() for small results (a computed max date, a row count, a config value) that drive program logic, not big data itself.
  • Unit tests / sample data — constructing expected or fixture data by hand, as in the example above.
  • RDD-style, row-at-a-time processing — df.rdd.map(lambda row: ...), foreach(), custom per-record logic (e.g. calling an external API per row) where you need the whole record together, not a vectorized column operation.
  • Assertions in tests — assert result.first().amount == 100.0.

It’s a real, purposeful escape hatch — for when you need one concrete record instead of a distributed column expression — not the default way of working with data.

Key Properties

  • One record, not one field — where a Column describes something vertically, across every row (one field, all records), a Row is horizontal: every field, for one record.
  • Immutable — like a DataFrame, a Row can’t be modified in place.
  • Local, not distributed — a Row only exists once data has actually been pulled out of Spark’s distributed, lazy world onto the driver (.collect(), .first(), .take(), …). While the DataFrame is still a logical plan, there are no Row objects yet.
  • Matches a schema — a Row’s fields correspond to whatever DataFrame it came from; you can also build one manually to help define a DataFrame’s schema (spark.createDataFrame([Row(...), ...])).

Ab Initio Equivalent

If a DataFrame maps to the record stream flowing along a flow line, a Row maps to a single record instance passing through that stream — one instance of the DML record format, at one moment, rather than the format itself. The DML record format is the schema (~ a DataFrame’s schema); an actual record flowing through, with real field values, is the Row.

Of the three (DataFrame, Column, Row), the Row is the one that maps most naturally onto how Ab Initio actually executes. Considering the execution model of an Ab Initio graph, the concept of a record is far more directly observable: the Co>Operating System typically processes records one by one, in sequence, passing each one from component to component along the flow. Spark’s DataFrame and Column, by contrast, describe a logical, lazy, columnar plan that doesn’t correspond to any single moment of execution — a Row is the closest thing Spark has to what Ab Initio is doing concretely, record by record, the whole time.