Transformations
ETL or ELT — the debate over ordering misses the point. E and L are plumbing: where the data comes from, when it moves. Wherever you draw that line, “T” doesn’t shrink or disappear — it’s the elephant in the room, the one step that decides whether data is actually ready to be used.
- E/L is source-aware and time-aware, T is consumer-aware. E/L’s job is fidelity — capture the source faithfully, preserve it so state can be reconstructed later. T’s job is interpretation — shape that faithfully captured data for a specific purpose, whatever that purpose needs.
- T is where raw becomes true. Dedup, conform keys, resolve conflicting sources, handle late arrivals. Before T: what the source happened to record. After T: what the business believes actually happened.
- T is where trust is built or broken. E/L tend to fail loudly — a dropped connection, a missing file. T fails quietly — a silent null, a wrong join, a bad grain.
PySpark DataFrame vs. Ab Initio Record Stream
In Ab Initio Graph, transformations act on a record stream — records shaped by a DML record format, flowing through a component (Reformat, Join, Rollup) one at a time, and that stream can itself be partitioned across multiple lanes for parallel processing. In PySpark, transformations act on a DataFrame — the whole distributed collection at once, described declaratively (select, withColumn, groupBy) and only executed when an action triggers it, with partitioning handled underneath across the cluster.
Two things worth understanding about how transformations actually run in PySpark:
With those concepts in place, the next step is working through the operations one by one — see the Core Operations reference.