8. Transformations vs Actions

Transformations vs Actions

Every operation on a PySpark DataFrame is one of two kinds. A transformation (select, filter, withColumn, join, groupBy, …) never touches data — it just adds a step to a logical plan and returns a new DataFrame describing that plan. An action (show, count, collect, write, take, …) is what actually triggers execution: Spark hands the accumulated logical plan to Catalyst, which optimizes it, compiles it into a physical plan, and only then runs it across the cluster. Chain a hundred transformations and nothing happens until an action is called — that’s what “lazy” means.

df2 = df.filter(F.col("status") == "ACTIVE")      # transformation — no data read yet
df3 = df2.withColumn("amount", F.col("amount") * 2)  # transformation — still nothing runs
df3.write.csv("/volumes/data/output.csv")            # action — this is what actually runs the job

The diagram below traces what those three lines do. The driver does read your script in sequence — line by line, top to bottom — but each transformation only adds an entry to the logical plan. What’s deferred isn’t the reading; it’s the actual data work, which waits until the action runs.

graph LR
    subgraph Submit["spark-submit"]
        SS["Script submitted<br/>to the cluster"]
    end

    subgraph Build[" "]
        T1["df.filter(...)<br/>transformation"] -->|"adds entry"| ULP["Logical Plan<br/>(Unresolved at first)"]
        T2["df.withColumn(...)<br/>transformation"] -->|"adds entry"| ULP
    end

    SS --> T1
    T1 --> T2
    ULP -.->|"df.write(...) — action"| EX(["Spark analyzes + optimizes,<br/>compiles a physical plan,<br/>then executes it across the cluster"])

    NOTE["Note: simplified view"]
    style NOTE fill:none,stroke:none

Ab Initio Equivalent

Ab Initio has no lazy/eager split — wiring the graph in GDE is building the plan, so the graph itself is the plan (a specification of what should happen — closest in spirit to Spark’s logical plan). It does have a compiler/optimizer that can apply some optimizations under the hood, but Co>Op keeps that hidden — you never get resolved/optimized/physical plans as separate, inspectable artifacts the way Spark does.

When you Run the graph, Co>Op first does prep work — environment setup, parameter resolution, and translating the graph into low-level commands — and only then does execution start, finally launching the component processes that actually push records through.

graph LR
    subgraph Submit["Run Graph"]
        G["Graph submitted<br/>to Co>Op cluster<br/>(graph ≈ logical plan)"]
    end

    G --> P["Prep work:<br/>Env setup ·<br/>Parameter resolution ·<br/>Graph → low-level commands"]
    P -.->|"execution starts"| EX(["Component processes run<br/>across the cluster —<br/>records flow through the graph"])

    NOTE["Note: simplified view"]
    style NOTE fill:none,stroke:none