New CDC connectors for MySQL and PostgreSQL: Don’t Let Your Data be a Day Late and a Dollar Short

Decodable’s support for CDC (Change Data Capture) advanced further with the release of source CDC connectors for MySQL and PostgreSQL. Source CDC connectors convert traditional database tables into sources of streaming data for processing and delivery to sink systems via Decodable. To dive right in, check out our short video demos, broken into CDC sourcing and change stream processing episodes.

Need a bit more context on CDC first?

Simply put, CDC is the ability to capture just the changed records of a datastore for the benefit of downstream systems. An event will be generated for operations like update, insert, and delete that can be incorporated at the downstream system to give the most current version of the data.  In many cases CDC is useful when the user makes an action and we want to respond to it.

With many recommender systems a cached feature store is the data source for the machine learning algorithm.  Unless this cache is updated with new customer behavior information it will continue to make recommendations based on past customer behavior.

Let's look at the case of a couple visiting a popular travel site to plan their next vacation.  The travel site stores data on their travel habits and has no trouble making travel recommendations on the type of hotel room they prefer or if they like window or aisle seats on the airplane because of the information that is stored in the cached feature store.  They start looking at trip information for Belize, but the ad deals they are sent are for Egypt because it is the last place they visited.  A little frustrated, they decide to have dinner and talk about where they want to go.  During dinner they decide they would rather go to Fiji so when they are done they start looking up trips to Fiji and now they receive special offers for Belize.  They decide to bag it for the night and try again in the morning on a different site.

This “day late and a dollar short” behavior is typical of a cached feature store that is refreshed by a periodic batch update. A better solution is to use CDC because it can update the feature store with the user’s current search behavior that will result in the recommender coming up with more accurate results. If the travel site from our previous scenario were enhanced with CDC, when the couple starts searching for trip information for Belize they will receive ad deals for Belize instead of Egypt.

In the past many system designers shied away from CDC because of the complexity of the systems that needed to be built and maintained.

Watch The Demo

Now we've cleared that up, check out our demo of CDC with Decodable in action:

You can get started with Decodable for free - our developer account includes enough for you to build a useful pipeline and - unlike a trial - it never expires.

Learn more:

• Read the docs
• Check out our other blogs
• Subscribe to our YouTube Channel

Join the community Slack