Ab Initio to Spark
I’ve spent years building pipelines visually — dragging components onto a canvas, wiring them together, tuning partitioning and parallelism as I went. That habit of thinking doesn’t disappear just because the interface does. Moving from Ab Initio to PySpark isn’t starting from zero — it’s the same engineering instincts, just expressed in code instead of connected components.
These notes are for Ab Initio Graph Developers transitioning to PySpark. Each topic pairs a familiar Ab Initio graph pattern with its PySpark equivalent, side by side, so you can map concepts you already know (components, DML expressions/functions, record formats) onto their PySpark counterparts rather than learning Spark from a blank page.
A living series
This is a living series — I keep adding new parts and revising existing ones as I go, so pages can change over time. If a topic you’re after isn’t here yet, check back.
Educational purpose only: Ab Initio is proprietary, closed-source software. Any Ab Initio-specific syntax, DML, or component behaviour shown here is included purely for educational/comparison purposes, to aid the transition to Apache Spark (via PySpark). All comparisons reflect nothing more than my own experience working professionally with Ab Initio and experimenting with PySpark — they are not official Ab Initio documentation, not affiliated with or endorsed by Ab Initio Software Corporation, and shouldn’t be treated as a reference for the product itself. There is no intention of any monetary benefit from these notes.