.svg)
.svg)
Apache Kafka Overview
Decodable supports both sourcing data from, and sinking data to, Apache Kafka, which is an open-source distributed event streaming platform optimized for ingesting and processing streaming data in real-time. Kafka provides three main functions to its users:
- Publish and subscribe to streams of records
- Effectively store streams of records in the order in which records were generated
- Process streams of records in real time
Kafka is primarily used to build real-time streaming data pipelines and applications that adapt to the data streams. It combines messaging, storage, and stream processing to allow storage and analysis of both historical and real-time data.
Decodable + Apache Kafka
Decodable regards Apache Kafka as both a source which can send a stream of data, and a sink which can receive one or more streams of data from a pipeline. You could use Decodable to take data from Kafka and send to an analytical database or machine learning model, or even send back to another Kafka topic after transformation. With Kafka as a sink, Decodable can take a stream of data from a messaging system such as Kinesis, a REST API or even a standard database like MySQL or Postgres running in CDC (change data capture) mode. Transforming data between Kafka topics can be the basis for building sophisticated event-driven microservice applications without writing a single line of code.
In the following video you'll see how to perform ETL, taking data from one Kafka topic, processing it in Decodable, and consuming it into another Kafka topic.
In the following video you'll see how to use Kafka as a source, streaming data into Decodable which transforms it and outputs to Amazon S3.
