Data Types and Schema

Schema = the structure of a DataFrame: an ordered list of columns, each with a name, a data type, and a nullable flag. Spark represents it as a StructType (the record) built from StructFields (the columns), where each field’s type comes from the pyspark.sql.types catalog (StringType, IntegerType, DecimalType, …). Every DataFrame has one, whether you declared it or Spark inferred it.

Example

from decimal import Decimal
from pyspark.sql.types import (
    StructType, StructField, IntegerType, StringType, DecimalType,
)
 
schema = StructType([
    StructField("id",     IntegerType(),      nullable=False),
    StructField("name",   StringType(),       nullable=True),
    StructField("amount", DecimalType(10, 2), nullable=True),
])
 
df = spark.createDataFrame(
    [(1, "Alice", Decimal("250.00")), (2, "Bob", Decimal("400.00"))],
    schema,
)
 
df.printSchema()
# root
#  |-- id: integer (nullable = false)
#  |-- name: string (nullable = true)
#  |-- amount: decimal(10,2) (nullable = true)

Two ways to write a schema

The StructType above is the programmatic form — explicit, but verbose. Spark accepts the exact same schema as a compact DDL string — SQL-like name type column definitions — anywhere a schema is expected:

schema = "id INT NOT NULL, name STRING, amount DECIMAL(10,2)"
 
df = spark.createDataFrame(data, schema)          # building a DataFrame
df = spark.read.schema(schema).csv("/path.csv")   # reading with a schema

The two forms are fully interchangeable — StructType.fromDDL(schema) parses the string into the object form, and df.schema.toDDL() goes back the other way. Use the DDL string for brevity and the StructType when you need to build a schema in code (loop over fields, reuse types, generate it dynamically).

Of the two, the DDL string is the closest thing PySpark has to an Ab Initio DML record format: a single compact block of typed field declarations, read top to bottom, rather than a list of constructor calls.

Basic Data Types

Every type lives in pyspark.sql.types and is written with a trailing () — it’s a class you instantiate (StringType(), not StringType).

Spark type	Python value	Ab Initio DML
`StringType`	`str`	`string(n)`
`IntegerType`	`int` (32-bit)	`integer(4)`
`LongType`	`int` (64-bit)	`integer(8)`
`DoubleType`	`float`	`real(8)`
`FloatType`	`float`	`real(4)`
`DecimalType(p, s)`	`decimal.Decimal`	`decimal(p.s)`
`BooleanType`	`bool`	(none — often `integer(1)`)
`DateType`	`datetime.date`	`date("YYYY-MM-DD")`
`TimestampType`	`datetime.datetime`	`datetime("...")`
`ArrayType(t)`	`list`	`t[n]` (vector)
`MapType(k, v)`	`dict`	(no direct type)
`StructType([...])`	`Row` / nested	nested `record ... end`

For anyone coming from Ab Initio, DecimalType(precision, scale) is the important one: financial pipelines live on decimal(18.2)-style fields, and DecimalType is the exact, non-lossy match — reach for it instead of DoubleType wherever the DML used decimal.

Explicit vs. inferred schema

Spark can figure out a schema for you, or you can hand it one:

# Inferred — Spark samples the data to guess types (a scan, and guesses can be wrong)
df = spark.read.option("header", True).option("inferSchema", True).csv("/path/data.csv")
 
# Explicit — you supply the StructType; no guessing, no extra scan
df = spark.read.option("header", True).schema(schema).csv("/path/data.csv")

Inference is convenient for exploration but has real costs: an extra pass over the data, and types that can drift between runs (an all-integer column one day, a stray decimal the next). For anything production, declare the schema.

Ab Initio Equivalent

A Spark schema is an Ab Initio DML record format — the same idea, different syntax. The parallel is tightest with the DDL string: a record ... end block of typed fields on one side, a compact name type string on the other, read the same way top to bottom.

# Schema of record (record stream)
record
  integer(4)    id;
  string(20)    name = NULL("");
  decimal(10.2) amount = NULL;
end;

# PySpark DDL string
"id INT NOT NULL, name STRING, amount DECIMAL(10,2)"

The deeper difference is when the schema is enforced, and it echoes Transformations vs Actions:

Ab Initio is schema-first and strict. Every flow carries an explicit DML; there’s no “infer.” The Co>Operating System validates records against that DML, and mismatches surface as reject/error records at run time — but the format itself is fixed and checked at compile.
Spark is schema-on-read. The schema is attached when data is read (inferred or declared), not baked into the file. A type only truly bites when an action forces the data through it — a value that doesn’t fit a declared type becomes null (or errors, depending on mode) at that point, not before.

So the mental shift isn’t “learn a new concept” — you already think in typed record formats. It’s “the DML now lives in your code as a StructType, and Spark will happily infer one if you don’t provide it — which you usually should.”

Ravi Writes

Explorer

4. Data Types and Schema