Beyond “Just a File”: A Data Engineer’s Perspective

As a data engineer, the smallest “unit of data” you work with is generally either a file or a database table. Even in the age of real-time streaming, files remain one of the most convenient and reliable ways to exchange data between systems. Whether in traditional ETL or modern data engineering, they show up as:

• Input datasets
• Intermediate artifacts
• Final outputs

But beyond being “just a file,” the way data is structured and stored — its storage format — directly affects how efficiently it can be read, processed, and moved through a pipeline.

A map of file types

Zoom out and the formats fall into a fairly clean taxonomy — split first by whether a file is plain text, then by how the data inside is structured:

graph TD
    F["File"]
    F --> PT["Plain-Text File"]
    F --> NPT["not Plain-Text File"]

    PT --> PTQ["How is the data<br/>structured within the file?"]
    PTQ --> SS["Semi-Structured-Data File"]
    PTQ --> ST["Structured-Data /<br/>Tabular-Data File"]
    PTQ --> UT["Unstructured-Data File"]

    SS --> JSON["JSON"]
    SS --> XML["XML"]

    ST --> FL["Fixed-Length /<br/>Fixed-Width File"]
    ST --> VL["Variable-Length /<br/>Variable-Width File"]
    VL --> DEL["Delimited File"]
    VL --> BOTH["File with both fixed-length<br/>and delimited fields"]
    DEL --> CSV["CSV (Comma-Separated Values)"]
    DEL --> TSV["TSV (Tab-Separated Values)"]
    DEL --> PSV["PSV (Pipe-Separated Values)"]
    DEL --> SOH["0x01 (SOH) delimited"]
    DEL --> NUL["0x00 (NUL) delimited"]

    UT --> LOG["free-form text file<br/>e.g. log file"]

    NPT --> NPQ["How is the data<br/>structured within the file?"]
    NPQ --> ST2["Structured-Data File"]
    NPQ --> UT2["Unstructured-Data File"]

    ST2 --> LP["Length-Prefixed File"]
    ST2 --> COB["COBOL Data File"]
    ST2 --> XLS["Excel File (.xls / .xlsx)"]
    ST2 --> PAR["Apache Parquet"]
    ST2 --> AVRO["Apache Avro"]
    ST2 --> ORC["Apache ORC"]

    UT2 --> RICH["Rich-Formatted text file<br/>(.docx / .rtf / .pdf)"]

    classDef fmt fill:#5f4b9b,stroke:#4a3a7a,color:#ffffff;
    class JSON,XML,FL,BOTH,CSV,TSV,PSV,SOH,NUL,LOG,LP,COB,XLS,PAR,AVRO,ORC,RICH fmt;

The purple nodes are concrete formats; the rest are the categories they fall under.

Traditional ETL file types

In legacy settings, you’ll frequently encounter:

• Delimited text files — CSV, PSV, and even \x01 (SOH) delimiters.
• Fixed-length / fixed-width files — remember the old debate: fixed-length vs. delimited?
• Length-prefixed files.
• COBOL data files.
• Mixed-field files — a complex blend of fixed-length, variable-length, and binary numeric fields.

Modern data engineering standards

In modern stacks, the focus shifts toward optimisation and scale:

• CSV — the universal “language” of data exchange.
• Apache Parquet — the backbone of modern architectures (data lakes / lakehouses) and a Spark favourite, thanks to its columnar storage efficiency.

Why it matters

Understanding these formats is the difference between a pipeline that just “runs” and one that is truly optimised for performance.

---

Originally published on LinkedIn — feel free to read and react to it there too.