I recently found myself doing a deep dive into how Apache Flink® SQL can be used for deduplication. What I discovered is that a thorough understanding of deduplication requires quite a lot of knowledge about the Flink runtime, including event time and watermarks, state management, and changelog processing. I was surprised that exploring deduplication took me so far into the weeds, and I hope it will be instructive to share what I learned.

When working with Apache Flink®’s SQL API, it’s not unusual for several possible solutions to suggest themselves. Some solutions may have unexpected consequences, affecting flexibility, performance, cost, and maintainability.

In this post I’ll share what I’m looking for and thinking about when I review a Flink SQL statement: state, append-only vs. updating streams, late events, latency, and determinism.

The details about the Flink SQL runtime presented here are based on Flink 2.3.0.