Blog /

The Future of Real Time Data Engineering

Eric Sammer

Historically, real-time data engineering has been the land of low-level Java and C++ developers agonizing over frameworks, data serialization, messaging guarantees, distributed state checkpointing algorithms, and pipeline recovery semantics. Concerns over these low-level details aren't limited to people building these systems. They extend to those building pipelines to run on top. Even the most modern systems require that pipeline developers hold a PhD in distributed systems - or to have at least dropped out of a good PhD program - to build fast, correct, and manageable pipelines.

The truth is, while it's fun to get into the details (we love it here at Decodable), most people need to build pipelines quickly that "just work". When a node fails, when a partition goes offline, when a poison pill event shows up, when there's back-pressure in the system, when the power suddenly goes out, or when the data volume scales up, "just working," it turns out, is a tall order.

Assuming that infrastructure challenges are a solved problem for teams (they're typically not), they're still left with the real-world complexities of real-time data engineering. Whether it's coordinating a schema change, establishing the practices to make those changes safely, testing pipelines against real data, performing safe deployments, creating a safe way to share data across teams, or just knowing their pipelines are alive and healthy, teams are inevitably left to build the necessary tooling around the development and management of real-time pipelines. In most cases, teams only need to filter, restructure, parse, aggregate, or enrich data to drive a microservice, ML pipeline, or populate a table in a database or dataset in an object store.

Decodable grew out of this need to up-level, simplify, and accelerate the development and management of real-time pipelines. Application developers, data engineers, and data scientists should be able to build and deploy a production pipeline in minutes, just as they do with batch data pipelines. That means making the infrastructure effectively disappear, leaving what feels like a serverless data engineering process.

Rather than worrying about the low-level details, we think it should be possible to think in terms of connections to data sources and sinks, streams that connect to them, and pipelines that process those streams.

The developer experience building and deploying those pipelines should work with existing tools and processes rather than forcing you to learn an entirely new way of working. Finally, you shouldn't need to re-platform your existing microservices or applications. We get that we need to be a good citizen within your existing data platform, so we have no proprietary formats, work with de facto open source systems and standards.

Building pipelines should be fast, safe, and even fun. If you're a data engineer working in the batch world and need to handle real-time data, we've got you covered. If you're an application engineer building microservices or other data-powered infrastructure, Decodable is for you.

Create an account today to get rolling, and join our Slack community to talk directly to the team and folks just like you.

A Practical Introduction to the Data Mesh

There’s been quite a bit of talk about data meshes recently, both in terms of philosophy and technology. Unfortunately, most of the writing on the subject is thick with buzzwords, targeted toward VP and C-level executives, unparsable to engineers. The motivation behind the data mesh, however, is not only sound but practical and intuitive.

Learn more

Announcing General Availability of the Decodable Real-Time Data Platform

Now GA, Decodable puts the power of real-time data engineering in the hands of every developer, so you can use your existing SQL skills to connect sources and sinks, to build and deploy real-time pipelines – that just work. Getting started with Decodable is free, easy and only takes a few minutes.

Learn more

Press Release: Decodable raises $20m Series A Funding

Decodable, the real-time data engineering company, today announced that it has raised more than $20 million in Series A funding led by Venrock and Bain Capital Ventures, with participation from additional investors

Learn more


Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique. Duis cursus, mi quis viverra ornare, eros dolor interdum nulla, ut commodo diam libero vitae erat. Aenean faucibus nibh et justo cursus id rutrum lorem imperdiet. Nunc ut sem vitae risus tristique posuere.

Learn more
Pintrest icon in black

Start using Decodable today.