Let’s get decoding.
Register for access and see how easy it is.
No more low level code and stitching together complex systems. Build and deploy pipelines in minutes with SQL.
A data engineering service that makes it easy for developers and data engineers to build and deploy real-time data pipelines for data-driven applications.
decodable stream create \
--name http_events \
--field event_time=timestamp \
--field hostname=string \
--field path=string \
--field response_code=int
Created stream http_events (a1b2c3d4)
# Connect an existing Kafka topic to our stream.
decodable connection create
--name prod_http_events
--connector kafka
--type source
--prop bootstrap.brokers=my-broker:9091
--prop format=json
--prop topic=http_events
--stream-id a2b2c3d4
Created connection prod_http_events (e1f2g3h4)
Pre-built connectors for messaging systems, storage systems, and database engines make it easy to connect and discover available data. For each connection you make, you get a stream to or from the system.
# Preview the data in our stream.
decodable pipeline preview "select * from http_events"
{ "event_time": "2022-01-01T00:00:00Z", "hostname": "www.decodable.co", "path": "/", "response_code": 200 }
{ "event_time": "2022-01-01T00:00:09Z", "hostname": "www.decodable.co", "path": "/platform", "response_code": 200 }
{ "event_time": "2022-01-01T00:01:13Z", "hostname": "www.decodable.co", "path": "/missing-page", "response_code": 404 }
# Create a pipeline to send successful http events to a
# stream connected to S3 for analytics.
decodable pipeline create
--name http_to_s3 "insert into successful_http_events select * from http_events where response_code between 200 and 299"
Created pipeline i1j2k3l4 version 1
# Activate our new pipeline.
decodable pipeline activate i1j2k3l4
Pipeline activated!
With Decodable you can build your pipelines with SQL. Pipelines use streams to send data to, or receive data from, your connections.
You can also use streams to connect pipelines together to handle the most complex processing tasks.
Streams can serve many pipelines, so they make it easy to serve data to other teams.
# Show all pipelines.
decodable pipeline list
id name create time
m1n2o3p4 http_to_s3 2021-11- 01T21:54:52Z
c15448bc rxie-v2-passthrough 2021-10-19T23:19:02Z
fd4f8016 parse_envoy 2021-08-24T18:57:55Z
# Rollback to version 2 of a pipeline rather than the latest
decodable pipeline activate m1n2o3p4 --version 2
# See real-time health metrics of an active pipeline\
decodable pipeline get m1n2o3p4
http_to_s3
id i1j2k3l4
version 2
is latest false
target state RUNNING
actual state RUNNING
description Filter output failed HTTP events.
create time 2021-11-01T21:54:52Z
last runtime error
message Failed to deserialize json
timestamp 2021-11-02T03:10:42Z
input metrics
http_events 37 records / 8.1KB per second | 23,493,201 / 4.90GB total
output metrics
successful_http_events 24 records / 5.2KB per second | 15,238,833 / 3.14GB total
insert into successful_http_events select *
from http_events
where
response_code between 200 and 299
Observe your pipelines to ensure data keeps flowing. Create curated streams for other teams. Define retention policies on streams to avoid data loss during external system failures. Real-time health and performance metrics let you know everything’s working.
Decodable gets you out of the business of managing complex infrastructure,
so you can focus on your business logic and data flow.
Teams can create and share streams that act as a public API to their data.
Everyone else can then customize the data and use the way they want.
No more centralized schema committees or custom code to maintain, just the data you need in the format you want, when you need it.
Decodable connects to any messaging system, database, storage system, or other system you control.