Blog /

New CDC connectors for MySQL and PostgreSQL: Don’t Let Your Data be a Day Late and a Dollar Short

Charles Harding

Decodable’s support for CDC (Change Data Capture) advanced further with the release of source CDC connectors for MySQL and PostgreSQL. Source CDC connectors convert traditional database tables into sources of streaming data for processing and delivery to sink systems via Decodable. To dive right in, check out our short video demos, broken into CDC sourcing and change stream processing episodes.

Need a bit more context on CDC first?

Simply put, CDC is the ability to capture just the changed records of a datastore for the benefit of downstream systems. An event will be generated for operations like update, insert, and delete that can be incorporated at the downstream system to give the most current version of the data.  In many cases CDC is useful when the user makes an action and we want to respond to it.

With many recommender systems a cached feature store is the data source for the machine learning algorithm.  Unless this cache is updated with new customer behavior information it will continue to make recommendations based on past customer behavior.

Let's look at the case of a couple visiting a popular travel site to plan their next vacation.  The travel site stores data on their travel habits and has no trouble making travel recommendations on the type of hotel room they prefer or if they like window or aisle seats on the airplane because of the information that is stored in the cached feature store.  They start looking at trip information for Belize, but the ad deals they are sent are for Egypt because it is the last place they visited.  A little frustrated, they decide to have dinner and talk about where they want to go.  During dinner they decide they would rather go to Fiji so when they are done they start looking up trips to Fiji and now they receive special offers for Belize.  They decide to bag it for the night and try again in the morning on a different site.

This “day late and a dollar short” behavior is typical of a cached feature store that is refreshed by a periodic batch update. A better solution is to use CDC because it can update the feature store with the user’s current search behavior that will result in the recommender coming up with more accurate results. If the travel site from our previous scenario were enhanced with CDC, when the couple starts searching for trip information for Belize they will receive ad deals for Belize instead of Egypt.

In the past many system designers shied away from CDC because of the complexity of the systems that needed to be built and maintained.

Watch The Demo

Now we've cleared that up, check out our demo of CDC with Decodable in action:

You can get started with Decodable for free - our developer account includes enough for you to build a useful pipeline and - unlike a trial - it never expires.

Learn more:

Join the community Slack

The Top 5 Streaming ETL Patterns

ETL and ELT are traditionally scheduled batch operations, but as the need for always-on, always-current data services becomes the norm, realtime ELT operating on streams of data is the goal of many organizations - if not the reality, yet.In real world usage, the ‘T’ in ETL represents a wide range of patterns assembled from primitive operations. In this blog we’ll explore these operations and see examples of how they’re implemented as SQL statements.

Learn more

Using Rest APIs to Build Pipelines & Stream Data with Postman

In this blog, we build a streaming data pipeline with Decodable’s REST APIs using Postman to invoke the APIs from a GUI, and then use Postman to send streaming data into the new pipeline using Decodable's HTTP connector.

Learn more


Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique. Duis cursus, mi quis viverra ornare, eros dolor interdum nulla, ut commodo diam libero vitae erat. Aenean faucibus nibh et justo cursus id rutrum lorem imperdiet. Nunc ut sem vitae risus tristique posuere.

Learn more
Pintrest icon in black

Start using Decodable today.