Back
November 14, 2023
5
min read

Checkpoint Chronicle - November 2023

By
Robin Moffatt
Share this post

Welcome to the Checkpoint Chronicle, a monthly roundup of interesting stuff in the data and streaming space. Your hosts and esteemed curators of said content are Gunnar Morling and Robin Moffatt - feel free to send our way any choice nuggets that you think we should feature in future editions. 

  • Apache Iceberg has some notable big adopters, such as Netflix (where it was created), and Apple. In this fascinating talk from QCon New York 2023, Stephen Wu from Apple talks about using Iceberg as a streaming source into Apache Flink, including the reasons why they included Iceberg in the pipeline and not just Apache Kafka throughout.

  • Several long-time developers contributing to the Kafka Streams project within Apache Kafka have launched a startup called responsive—and with it an absolute barn-stormer of a blog post: A Size for Every Stream: The Expert's Guide to Sizing Kafka Streams

  • In this excellent talk from QCon London 2023 and just published on the QCon website (alongside the previous blog on the same subject), Matt Boyle and Andrea Medda from Cloudflare detail their journey to 1 trillion messages in Kafka and the issues and learnings they had along the way. Some of the points included dealing with message compatibility whilst retaining loose-coupling as they moved away from a monolithic implementation, internal tooling and libraries to improve developer velocity, and accurate monitoring and instrumentation.

  • From the Decodable stables this month we have a useful set of blogs: 
    • If you’re new to stream processing then this post from Robert is just what you need to give you an overview of the space. 
    • Gunnar then goes on to cover what change data capture (CDC)  is and the wide range of use cases for it. 
    • To finish off the introductory blog posts, check out Robin’s view of what is Apache Flink as he sets out on his learning journey.

  • A sign of the increasing maturity of the stream processing space is that we are moving beyond solely “Hey I implemented Hello World with Flink!” blog posts—although there’s nothing wrong with these too, and I always applaud learning in public—and onto the gnarlier topics that come with running stream processing applications for real. This post from Yaroslav Tkachenko discusses a zero-downtime technique for deploying new versions of Flink applications based on the blue-green deployment pattern.

  • Arroyo are one of the many new companies in the increasingly-crowded streaming space, and earlier this year published this excellent blog that explains streaming SQL clearly along with a discussion of dealing with updates, comparing two different approaches that it calls Dataflow Semantics and Update Semantics.

  • Another new kid on the block in streaming is Epsio, who published this useful explainer of their implementation of a streaming SQL engine.
  • A year after their first post, the Netflix team returned with an update on the Data Mesh platform, detailing their move from individual Flink operators to adoption of Flink SQL. It’s interesting to see how Flink is used, and also detail around exactly how companies at this scale implement streaming technologies.
    It’s worth noting that Netflix’s Data Mesh != Zhamak Dehghani’s Data Mesh - it’s just a naming collision, and one which Netflix discusses here.

  • The Open Table Format (OTF) squabbles continue, with this blog from Iceberg co-creator and Tabular co-founder Ryan Blue discussing flaws in Apache Hudi’s Atomicity, Consistency, and Isolation guarantees, which one of Hudi’s co-creators (and Onehouse founder), Vinoth Chander, disagrees with in this post. Databricks’ Ali Ghodsi even threw his tuppence worth in too.
    It seems there are at least two angles to this argument: 
    • Arguments over the technical correctness of claimed features (ACID, etc)
    • Strategic positioning; Both Databricks and OneHouse see the OTF future as being multiple formats with some form of interoperability between them (Uniform and OneTable, respectively). The impression that I’ve got is that Tabular see Iceberg as the format around which others will and should converge.
  • Replication slots not advancing in certain circumstances used to be a notorious source of headaches for users of change data capture with Postgres. The engineering team at Zalando—who are heavy users of Debezium—took a stab at addressing this, using keep-alive messages in order to address this issue. They discuss how they patched the Postgres JDBC driver in this in-depth blog post. No more unexpected WAL growth!

Paper of the Month

Who doesn’t love to read them a good paper? In this spirit, we’re going to reference one research or industry paper (either classic, or just hot off the press) in the field of data processing and streaming each month, starting with one of our all-time favourites:

📄 One SQL to Rule Them All: An Efficient and Syntactically Idiomatic Approach to Management of Streams and Tables (arXiv:1905.12133)

In this paper from 2019, Edmon Begoli et al. discuss how the “pervasive use of time-varying relations, robust event-time semantics support, and materialization control can substantially improve the ease-of-use of streaming SQL”. Definitely an inspiring read!

New Releases

Events

  • Data Council is going to take place in Austin again (26-28 March 2024). The CfP for it is open for just a few more days (closing on 17th November), so hurry to get your proposals in.

  • Current 2023 was held in San Jose this year with attendees from across the data streaming space. If you weren’t able to make it but don’t want to feel left out, you can read a variety of write-ups, including from Confluent themselves, the trade press, vendors including Cloudera and Tabular, and independents such as Yaroslav Tkachenko and Bitrock. 

‍

That’s all for this month! We hope you’ve enjoyed the newsletter, and are all-ears for any feedback or suggestions you’ve got. 

Gunnar (LinkedIn / X / Mastodon / Email)

Robin (LinkedIn / X / Mastodon / Email)

📫 Email signup 👇

Did you enjoy this issue of Checkpoint Chronicle? Would you like the next edition delivered directly to your email to read from the comfort of your own home?

Simply enter your email address here and we'll send you the next issue as soon as it's published—and nothing else, we promise!

👍 Got it!
Oops! Something went wrong while submitting the form.
Robin Moffatt

Robin is a Principal DevEx Engineer at Decodable. He has been speaking at conferences since 2009 including QCon, Devoxx, Strata, Kafka Summit, and Øredev. You can find many of his talks online and his articles on the Decodable blog as well as his own blog.

Outside of work, Robin enjoys running, drinking good beer, and eating fried breakfasts—although generally not at the same time.

Table of contents

Heading 2
Let's Get Decoding
Start freeTalk To An Expert

Welcome to the Checkpoint Chronicle, a monthly roundup of interesting stuff in the data and streaming space. Your hosts and esteemed curators of said content are Gunnar Morling and Robin Moffatt - feel free to send our way any choice nuggets that you think we should feature in future editions. 

  • Apache Iceberg has some notable big adopters, such as Netflix (where it was created), and Apple. In this fascinating talk from QCon New York 2023, Stephen Wu from Apple talks about using Iceberg as a streaming source into Apache Flink, including the reasons why they included Iceberg in the pipeline and not just Apache Kafka throughout.

  • Several long-time developers contributing to the Kafka Streams project within Apache Kafka have launched a startup called responsive—and with it an absolute barn-stormer of a blog post: A Size for Every Stream: The Expert's Guide to Sizing Kafka Streams

  • In this excellent talk from QCon London 2023 and just published on the QCon website (alongside the previous blog on the same subject), Matt Boyle and Andrea Medda from Cloudflare detail their journey to 1 trillion messages in Kafka and the issues and learnings they had along the way. Some of the points included dealing with message compatibility whilst retaining loose-coupling as they moved away from a monolithic implementation, internal tooling and libraries to improve developer velocity, and accurate monitoring and instrumentation.

  • From the Decodable stables this month we have a useful set of blogs: 
    • If you’re new to stream processing then this post from Robert is just what you need to give you an overview of the space. 
    • Gunnar then goes on to cover what change data capture (CDC)  is and the wide range of use cases for it. 
    • To finish off the introductory blog posts, check out Robin’s view of what is Apache Flink as he sets out on his learning journey.

  • A sign of the increasing maturity of the stream processing space is that we are moving beyond solely “Hey I implemented Hello World with Flink!” blog posts—although there’s nothing wrong with these too, and I always applaud learning in public—and onto the gnarlier topics that come with running stream processing applications for real. This post from Yaroslav Tkachenko discusses a zero-downtime technique for deploying new versions of Flink applications based on the blue-green deployment pattern.

  • Arroyo are one of the many new companies in the increasingly-crowded streaming space, and earlier this year published this excellent blog that explains streaming SQL clearly along with a discussion of dealing with updates, comparing two different approaches that it calls Dataflow Semantics and Update Semantics.

  • Another new kid on the block in streaming is Epsio, who published this useful explainer of their implementation of a streaming SQL engine.
  • A year after their first post, the Netflix team returned with an update on the Data Mesh platform, detailing their move from individual Flink operators to adoption of Flink SQL. It’s interesting to see how Flink is used, and also detail around exactly how companies at this scale implement streaming technologies.
    It’s worth noting that Netflix’s Data Mesh != Zhamak Dehghani’s Data Mesh - it’s just a naming collision, and one which Netflix discusses here.

  • The Open Table Format (OTF) squabbles continue, with this blog from Iceberg co-creator and Tabular co-founder Ryan Blue discussing flaws in Apache Hudi’s Atomicity, Consistency, and Isolation guarantees, which one of Hudi’s co-creators (and Onehouse founder), Vinoth Chander, disagrees with in this post. Databricks’ Ali Ghodsi even threw his tuppence worth in too.
    It seems there are at least two angles to this argument: 
    • Arguments over the technical correctness of claimed features (ACID, etc)
    • Strategic positioning; Both Databricks and OneHouse see the OTF future as being multiple formats with some form of interoperability between them (Uniform and OneTable, respectively). The impression that I’ve got is that Tabular see Iceberg as the format around which others will and should converge.
  • Replication slots not advancing in certain circumstances used to be a notorious source of headaches for users of change data capture with Postgres. The engineering team at Zalando—who are heavy users of Debezium—took a stab at addressing this, using keep-alive messages in order to address this issue. They discuss how they patched the Postgres JDBC driver in this in-depth blog post. No more unexpected WAL growth!

Paper of the Month

Who doesn’t love to read them a good paper? In this spirit, we’re going to reference one research or industry paper (either classic, or just hot off the press) in the field of data processing and streaming each month, starting with one of our all-time favourites:

📄 One SQL to Rule Them All: An Efficient and Syntactically Idiomatic Approach to Management of Streams and Tables (arXiv:1905.12133)

In this paper from 2019, Edmon Begoli et al. discuss how the “pervasive use of time-varying relations, robust event-time semantics support, and materialization control can substantially improve the ease-of-use of streaming SQL”. Definitely an inspiring read!

New Releases

Events

  • Data Council is going to take place in Austin again (26-28 March 2024). The CfP for it is open for just a few more days (closing on 17th November), so hurry to get your proposals in.

  • Current 2023 was held in San Jose this year with attendees from across the data streaming space. If you weren’t able to make it but don’t want to feel left out, you can read a variety of write-ups, including from Confluent themselves, the trade press, vendors including Cloudera and Tabular, and independents such as Yaroslav Tkachenko and Bitrock. 

‍

That’s all for this month! We hope you’ve enjoyed the newsletter, and are all-ears for any feedback or suggestions you’ve got. 

Gunnar (LinkedIn / X / Mastodon / Email)

Robin (LinkedIn / X / Mastodon / Email)

📫 Email signup 👇

Did you enjoy this issue of Checkpoint Chronicle? Would you like the next edition delivered directly to your email to read from the comfort of your own home?

Simply enter your email address here and we'll send you the next issue as soon as it's published—and nothing else, we promise!

Robin Moffatt

Robin is a Principal DevEx Engineer at Decodable. He has been speaking at conferences since 2009 including QCon, Devoxx, Strata, Kafka Summit, and Øredev. You can find many of his talks online and his articles on the Decodable blog as well as his own blog.

Outside of work, Robin enjoys running, drinking good beer, and eating fried breakfasts—although generally not at the same time.

Related Posts

The Wonders of Postgres Logical Decoding Messages for CDC
March 6, 2023
12
min read
Blog

The Wonders of Postgres Logical Decoding Messages for CDC

By
Gunnar Morling

Let's Get Decoding

Decodable is free to try. Register for access and see how easy it is.

Start Free