Ab Initio to PySpark
Before Spark, the most popular way to implement production-grade data-processing workflows was a GUI-based platform — Informatica, Ab Initio, IBM DataStage and the like.
That visual-first, low-code approach was popular, and genuinely successful. And among those platforms, Ab Initio was a different beast — well ahead of its time, but very secretive, and with a licensing cost to match. Ab Initio + Teradata was a niche stack, and the preferred one at big banks through the on-prem years.
Then Spark became generally available, and things changed. Though I’d argue it wasn’t Spark alone — it was Cloud + Spark that changed the game, and the rise of serverless data-processing platforms like Databricks and AWS Glue.
Ab Initio did evolve, and made the product cloud-compatible. But cloud-compatible is not cloud-native.
before Spark
Visual-first, low-code
Informatica · Ab Initio · DataStage. Licensed, on-prem, GUI-built.
then
Spark goes GA
Open source, code-first. The pipeline becomes a script.
and now
Cloud + Spark
Databricks · AWS Glue. Serverless, and the reason it really stuck.
Anyway — I’m not writing this series to debate which tooling is better. I’m writing it to help Ab Initio Graph developers move into the PySpark world, or build a hybrid Ab Initio + PySpark skillset. One thing is certain: there are more and more opportunities in Spark, or Ab Initio + Spark, than in Ab Initio alone.
The good news
Your experience developing Ab Initio Graphs makes this transition really smooth. You already think in data flows, partitioning, joins and performance — what changes is the way you express it, from visual to code.
We’ll explore it step by step, right from the beginning.
A living series
This is a living series — I keep adding new parts and revising existing ones as I go, so pages can change over time. If a topic you’re after isn’t here yet, check back.
Contents
- Ab Initio to PySpark — the vocabulary: every idea you use daily, and its PySpark counterpart.
- Ab Initio Graph to PySpark Script — one small pipeline, drawn as a graph and written as a script, side by side.
Educational purpose only: Ab Initio is proprietary, closed-source software. Any Ab Initio-specific syntax, DML, or component behaviour shown here is included purely for educational/comparison purposes, to aid the transition to Apache Spark (via PySpark). All comparisons reflect nothing more than my own experience working professionally with Ab Initio and experimenting with PySpark — they are not official Ab Initio documentation, not affiliated with or endorsed by Ab Initio Software Corporation, and shouldn’t be treated as a reference for the product itself. There is no intention of any monetary benefit from these notes.