I recently found myself doing a deep dive into how Apache Flink® SQL can be used for deduplication. What I discovered is that a thorough understanding of deduplication requires quite a lot of knowledge about the Flink runtime, including event time and watermarks, state management, and changelog processing. I was surprised that exploring deduplication took me so far into the weeds, and I hope it will be instructive to share what I learned.

In this post I’ll share what I’m looking for when I review an Apache Flink® SQL statement.

If you are approaching Flink SQL by thinking about how you would solve the same problem in a SQL database (e.g., Postgres), that’s an excellent start, but without some knowledge of how the Flink runtime handles various SQL operations, you could run into trouble. Some solutions can have unexpected consequences, affecting flexibility, performance, cost, and maintainability.