Core Operations
The operation-by-operation reference — the practical half that picks up where the core-concept pages leave off. Each note covers a single PySpark operation (reading, transforming, or writing data), mapped to its Ab Initio component, with syntax and a worked example. If the concepts aren’t solid yet, start with Transformations (what/why), then Transformations vs Actions and Narrow vs Wide Transformations (how they run).
Most of these operations center on the DataFrame — reads produce one, transformations reshape it, writes consume one.
Operations: Spark ↔ Ab Initio
| Spark | Narrow / Wide | Ab Initio Equivalent |
|---|---|---|
spark.read | — | Input File / Input Table |
select | Narrow | Reformat (select/project fields) |
filter / where | Narrow | Filter By Expression |
withColumn | Narrow | Reformat (derive/add a field) |
withColumnRenamed | Narrow | Reformat (rename a field) |
drop | Narrow | Reformat (drop a field) |
union | Narrow | Concatenate |
sample | Narrow | Sample |
groupBy / agg | Wide | Partition By Key + Rollup |
join | Wide | Partition By Key + Join |
distinct | Wide | Partition By Key + Sort + Dedup Sorted |
repartition | Wide | Partition By Key / Partition By Round-Robin |
orderBy / sort | Wide | Partition By Key + Sort |
df.write | — | Output File / Output Table |
(spark.read and df.write are a read and a write, not transformations, so they have no narrow/wide class — hence the —.)