1. Ab Initio Graph vs PySpark Script

Ab Initio Graph vs PySpark Script

An Ab Initio Graph is a visual program built using Ab Initio’s own tool, GDE (Graphical Development Environment), and saved as an .mp file (a proprietary text format). It runs on the Ab Initio Co>Operating System, which can be installed on a single machine or across a cluster of machines. A graph can be executed via the air command-line tool, or run interactively straight from GDE.

A PySpark Script is a plain-text program — regular Python code, written in any editor or IDE — saved as a .py file. It runs on Apache Spark, which can be installed on a single machine (local mode) or across a distributed cluster (standalone, YARN, or Kubernetes). A script can be executed via the spark-submit command-line tool, or run interactively from a notebook (e.g. Jupyter).

Ab Initio Graph Skeleton

Ab Initio Graph = Graph Components connected logically + each Component configured using logic written in DML. ** DML (Data Manipulation Language)** is Ab Initio’s proprietary language to describe, parse, and manipulate data.

graph LR
    A[Input File] --> B[Filter By Expression]
    B --> C[Reformat]
    C --> D[Output File]

Note: this is a simplified representation of the actual Ab Initio graph, not the graph itself — the real graph is the .mp file opened in GDE.

# Input File Component is configured for
# 1. Location of dataset
# 2. Schema of dataset
 
# Input File (Record Format - DML)
record
utf8 string("|") id = NULL("");
utf8 string("|") name = NULL("");
utf8 string("|") amount = NULL("");
utf8 string("\n") status = NULL("");
end;

# Filter By Expression Component is configured for
# Filter Logic (DML Logic)
straing_upcase(string_lrtrim(status)) == "ACTIVE"
 
# Input File Record format = FBE Input Record format
# FBE Input Record format = FBE Output Record format

# Reformat Component is configured for 
# Renaming fields, cleaning data and changing data-type (DML Logic)
out :: reformat(in)=
begin
out.customer_id :: string_lrtrim(in.id);
out.customer_name :: string_lrtrim(in.name);
out.amount :: (decimal("|")) string_lrtrim(in.amount);
end;
 
# FBE Output Record format = Reformat Input Record format
# Reformat Output Record format = Output File Record format

# Output File Component is configured for
# 1. Location of dataset
# 2. Schema of dataset
 
# Ouput File (Record Format - DML)
record
utf8 string("|") customer_id = NULL("");
utf8 string("|") customer_name = NULL("");
decimal("|") amount = NULL("");
end;

Equivalent PySpark Script

PySpark Script = DataFrame transformations chained together logically + each transformation configured using logic written in PySpark’s DataFrame API. Unlike Ab Initio, there’s no separate DML — the same Python/PySpark code both wires the pipeline together and expresses the per-column logic (via pyspark.sql.functions).

from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql.functions import col
from pyspark.sql.types import StructType, StructField, StringType, DecimalType
 
# Spark Entry Point
spark = SparkSession.builder.appName("AbInitioEquivalent").getOrCreate()

# Input File (Record Format - DML)
# record
# utf8 string("|") id = NULL("");
# utf8 string("|") name = NULL("");
# utf8 string("|") amount = NULL("");
# utf8 string("\n") status = NULL("");
# end;
input_schema = StructType([
    StructField("id", StringType(), True),
    StructField("name", StringType(), True),
    StructField("amount", StringType(), True),
    StructField("status", StringType(), True),
])
 
df = spark.read.csv("/volumes/data/input.csv", sep="|", schema=input_schema, header=False)

# Filter By Expression
# string_upcase(string_lrtrim(status)) == "ACTIVE"
df_filtered = df.filter(F.upper(F.trim(col("status"))) == "ACTIVE")

# Reformat (Function - DML)
# out.customer_id :: string_lrtrim(id);
# out.customer_name :: string_lrtrim(name);
# out.amount :: (decimal("|")) string_lrtrim(amount);
df_reformatted = df_filtered.select(
    F.trim(col("id")).alias("customer_id"),
    F.trim(col("name")).alias("customer_name"),
    F.trim(col("amount")).cast(DecimalType(18, 2)).alias("amount"),
)

# Output File (Record Format - DML)
# record
# utf8 string("|") customer_id = NULL("");
# utf8 string("|") customer_name = NULL("");
# decimal("|") amount = NULL("");
# end;
output_schema = StructType([
    StructField("customer_id", StringType(), True),
    StructField("customer_name", StringType(), True),
    StructField("amount", DecimalType(18, 2), True),
])
assert df_reformatted.schema == output_schema
 
df_reformatted.write.csv("/volumes/data/output.csv", sep="|", header=False, mode="overwrite")
# /volumes/data/output.csv is a directory (logical dataset)
 
spark.stop()

Ab Initio (or other visual ETL tools like Informatica) and Apache Spark are still genuinely different — in execution model, tooling, and philosophy. But the comparison above shows the core ideas carry over more directly than they might first appear.