Pipelines: ETL, ELT, batch and streaming

A pipeline is a series of steps that takes data from a source, changes it, and puts it somewhere useful. Sources are app databases, files, APIs and event logs. Destinations are usually a warehouse or a lake.

Good pipelines are repeatable. If you run the same day twice, you get the same result, not double the rows. This property is called idempotency, and it is one of the most useful words in the job.

ETL means extract, transform, load: you clean data before it reaches the warehouse. ELT means extract, load, transform: you land raw data first, then transform it inside the warehouse with SQL.

ELT became the default for analytics because cloud warehouses are fast and storage is cheap. Keeping the raw data means you can fix a transformation later and rebuild. Tools like dbt are built around this approach. ETL still makes sense when data must be filtered or masked before it is stored, for example personal data.

Batch pipelines process data in chunks on a schedule: every hour, every night. They are simpler, cheaper and easier to rerun. Most company reporting is batch, and that is fine.

Streaming pipelines process events as they arrive, within seconds. You need them for fraud checks, live tracking or alerts. Common tools are Apache Kafka for moving events and Apache Flink or Spark Structured Streaming for processing. Streaming adds real complexity, such as late and out-of-order events.

Before you build anything real-time, ask the people who will use it how fresh the data must be. A dashboard read once each morning does not need streaming. Many teams start with batch and add streaming only for the one use case that needs it.