The Log: What Every Software Engineer Should Know About Real-Time Data’s Unifying Abstraction

A concise, visual retelling of Jay Kreps’ classic essay. All diagrams are from Kreps’ own Stanford EE380 lecture (slides).

The one-sentence version: the humble append-only log is the single abstraction underneath databases, distributed systems, data integration, and stream processing — and once you see it, you can’t unsee it.

---

Table of Contents

1. Part 1 — What is a log?
2. Part 2 — Logs and distributed systems
3. Part 3 — Logs and data integration
4. Part 4 — Logs and stream processing
5. Part 5 — Building systems with the log
6. The whole idea in 30 seconds

---

Part 1 — What is a log?

Forget application logs (the human-readable ERROR: ... lines from log4j or syslog). This is the other kind of log — the one databases and distributed systems are built on.

A log is an append-only, totally-ordered sequence of records, ordered by time.

• Records are appended to the end; reads go left → right.
• Each record gets a monotonically increasing number — an offset — that acts as its logical timestamp.
• That ordering is decoupled from any physical clock. Order is the point.

You already use this shape all the time — even a web server’s access_log is one:

The key insight: tables and logs are dual

A log is just a record of changes. A table is the current state after applying those changes. You can always convert between them:

• Log → Table: replay every change in order and you rebuild the table.
• Table → Log: capture each change as it happens and you produce the log.

It’s exactly like version control: the log is the sequence of commits/patches; the table is the checked-out working copy.

This table ⇄ log duality is the thread running through everything that follows. Databases, replication, caches, indexes, and stream processors are all just different materializations of the same log.

---

Part 2 — Logs and distributed systems

Databases have quietly relied on logs for decades:

• Crash recovery / durability — write the change to the log first (write-ahead log), then apply it. On crash, replay the log.
• Replication — ship the log to replicas (Oracle, MySQL, Postgres all do this).

Why does replaying a log work? The State Machine Replication Principle:

If two identical, deterministic processes start in the same state and receive the same inputs in the same order, they produce the same output and end in the same state.

The log is “the same inputs in the same order.” That’s why consensus algorithms — Paxos, Raft, ZAB, Viewstamped Replication — are, at their core, protocols for agreeing on the contents of a log.

Two flavors of using the log:

Style	What’s in the log	How replicas work
State-machine replication (active-active)	the requests / operations	every replica runs every operation itself
Primary-backup	the resulting state changes	leader runs the operation, followers just apply the diff

Either way, the log is the source of truth, and a replica’s position is just an offset into it. A replica that falls behind or restarts simply resumes reading from where it left off.

---

Part 3 — Logs and data integration

Data integration = making all of an organization’s data available in all of its systems and services.

It’s unglamorous plumbing, but it’s the base of the pyramid. You can’t do reliable analytics, ML, or “AI” if the data can’t even flow reliably first.

Two trends made this hard:

1. Event data exploded — clicks, impressions, pageviews, metrics, logs. Orders of magnitude more than traditional database rows.
2. Specialized systems multiplied — OLAP, search, key-value stores, graph indexes, batch (Hadoop), caches. Each one needs the data.

Connect N sources to M systems with point-to-point pipelines and you get an O(N×M) mess — fragile, duplicated, impossible to operate:

The fix: put a log in the middle

Every source writes to a central log. Every destination subscribes and reads at its own pace. O(N×M) → O(N+M).

This is exactly what Kafka was built for at LinkedIn. The log becomes “the pipe” between all systems, and it buys you:

• A logical clock for the whole org — offsets let any two systems talk about “where they are” consistently.
• Producer/consumer decoupling — consumers read at their own speed; a slow or down consumer just catches up later.
• Zero-touch onboarding — a new destination system subscribes without touching any producer.

Rethinking ETL. Traditional ETL tangles two separate jobs together. Split them:

• Producers publish clean, canonical data to the log (this is the org’s contract).
• Destinations do only their own system-specific transforms.

Real-time transforms in the middle (sessionize, enrich, join) just produce new derived logs — which look identical to source logs to anyone downstream.

Making the log scale

A central log has to eat the entire firehose. Kafka handles it with:

• Partitioning — split the log into independent partitions. Order is guaranteed within a partition; partitions scale horizontally with no coordination.
• Batching — group reads, writes, disk flushes, and network transfers.
• A single binary format — the same bytes in memory, on disk, and on the wire, enabling zero-copy transfer.

At LinkedIn this was already running at serious scale:

---

Part 4 — Logs and stream processing

Stream processing isn’t “SQL on fast data.” It’s simply continuous data processing — the natural way to handle data that is, in reality, produced continuously.

Batch processing was always an artifact of batch collection (think: the census, collected once every 10 years). Once data arrives continuously, processing it in daily chunks is the artificial choice.

A stream processor reads logs and writes logs. Chain them and your whole organization becomes a graph of logs and the jobs that transform them:

Stream processing = logs + jobs.

The log plays two indispensable roles in this graph:

1. Ordering + multi-subscriber — every consumer sees the same records in the same order, so no reordering bugs.
2. Buffering / isolation — jobs fail, restart, and run at different speeds independently, without one stalling another.

Stateful processing and the changelog

Real work — aggregations, joins, enrichments — needs state. The best pattern reuses the duality from Part 1:

• Keep a local table (LevelDB, RocksDB, Lucene, …) right next to the processor for fast access.
• Journal every change to that table as its own log (a “changelog”).
• Crash? Rebuild the local table by replaying its changelog.

So the processor’s state is itself a log — which means other processors can subscribe to it too.

Log compaction — keeping it from growing forever

• Event streams: keep a time/size window (e.g. last N days), drop the rest.
• Keyed/table-like streams: keep only the latest value per key. That’s enough to fully reconstruct the table, while bounding storage.

---

Part 5 — Building systems with the log

Zoom out and an entire company looks like one big distributed database: the log is the commit log, and each specialized system (search, graph, OLAP, cache) is just a differently-shaped index over it.

Concrete example — “user views a job.” The frontend doesn’t call five services. It just emits one “job viewed” event to the log. Everyone interested — Hadoop, security, analytics, the recommendation engine, monitoring — subscribes independently. Adding a sixth consumer touches nothing upstream.

The log-centric architecture: split “log” from “serving”

Factor every data system into two layers:

• Log layer — captures state changes in order. Handles consistency, replication, durability, recovery, rebalancing.
• Serving layer — subscribes to the log and builds whatever query-optimized index it needs.

Writes go to the log; serving nodes apply changes to their local index. Clients get read-your-writes consistency by passing the write’s offset with their query. One log can feed many serving systems, amortizing its cost — and each index picks the storage that suits it (logs love cheap sequential disk; serving layers can use SSD/RAM).

This layering is what makes the log’s plumbing worth centralizing: it handles the genuinely hard distributed-systems problems once, so each serving system doesn’t have to re-solve them.

Where does this go? Three futures

1. Status quo — many specialized systems persist → an external log is essential glue.
2. Re-consolidation — one mega-system swallows everything (hard in practice).
3. Unbundling — open-source building blocks (Kafka, ZooKeeper, Mesos, Lucene, RocksDB, protobuf…) snap together into custom data systems. The log is the backbone that connects them.

---

The whole idea in 30 seconds

• A log is an append-only, totally-ordered record of changes.
• Tables and logs are dual — replay a log to get a table; capture a table’s changes to get a log.
• That single idea powers database recovery, replication, and consensus.
• Put one log in the middle of your architecture and O(N×M) integration spaghetti becomes O(N+M).
• Stream processing = logs + jobs; stateful jobs keep local tables backed by changelogs.
• A whole company is one big distributed database: the log is the commit log, every system is an index over it.

“You can’t fully understand databases, NoSQL stores, key-value stores, replication, Paxos, Hadoop, version control, or almost any software system without understanding logs.” — Jay Kreps

---

Learn more

• 📄 Original essay — The Log (LinkedIn Engineering)
• 🎥 Talk — I ♥ Logs, Stanford EE380
• 🖼️ Lecture slides (PDF)
• 🔧 Apache Kafka · Apache Samza

_{This is a condensed, illustrated summary for study purposes. All concepts and diagrams are the work of Jay Kreps; credit goes entirely to the original author.}