Blog
Blog /

Optimizing Data For InfluxDB Using Decodable

Charles Harding
Decodable

According to gestalt psychology, "the whole is more than the sum of its parts." This is certainly true of Decodable and InfluxDB. Decodable with its effortless streaming, data cleansing, and enrichment paired with InfluxDB’s powerful time series database’s ability to analyze and manage data is a formidable system. Some of the specific ways Decodable works with InfluxDB to improve performance (with some links back to the InfluxDB best practice guidance) are:

Optimize Writes

You can use Decodable to easily capture streaming events and push them to InfluxDB when they reach the optimal 5,000 lines. You can also modify timestamps to follow the Influx best practice of “Use the coarsest time possible” and follow rate-limiting best practices by using Decodable as a buffer, smoothing out ingest. Another typical pattern: Use Decodable to lower the resolution of the dataset by calculating results over an interval and sending that data to InfluxDB (e.g. average the values over the course of 5 seconds to smooth the data, and send the result to InfluxDB). This eliminates unnecessary complexity, speeds up queries, and reduces the cost of storage and computation. Read the InfluxDB best practice.

Reduce Cardinality

Excessively high cardinality can cause memory issues and slow your InfluxDB experience. There are several patterns to mitigate this without losing important data nuance. You can use Decodable to reformat events to either group them or reorder fields to choose a more suitable field for the key. Read the InfluxDB best practice.

Eliminate Duplicate Events 

With Decodable’s built-in exactly-once mandate, you can effortlessly prevent duplicate events simply by passing through Decodable. You can also easily use Decodable to keep a cache of previous events, checking for matches in specific fields and discarding duplicates. Alternatively, you can use Decodable to add a unique field (or increment the timestamp) to an event that appears to be duplicated so that it’s handled correctly by InfluxDB. Read the InfluxDB best practice.

Improve Schemas  

As InfluxDB documentation states, “Complex measurements make some queries impossible”. You frequently can’t control upstream formats, but you can take control once the data comes to you. Using Decodable allows you to follow InfluxDB best practices to remove keywords and special characters, convert to numeric types, and ensure keys don’t repeat. You can also parse out concatenated attributes and avoid regex queries, and take control of tags. Read the InfluxDB best practice.

Join Streaming Tables

Engineers often need to enrich data using an outside database before landing in Influx. This is called a stream-table join, and it’s commonly used to bring more data into Influx and reduce the need for joins/lookups in InfluxDB. There are a few reasons for this:

  • Joins in-database can have performance considerations
  • Database joins can be an expensive operation that requires data to be shuffled around at query time. If you can pre-join your data in the stream processor, then you'll end up with a single, denormalized, pristine table in InfluxDB

Pre-process Data

Decodable can easily pre-process the data to meet some of the most common processing needs so that it arrives ready to use:

  • Flattening data into the relational data model
  • Filter unneeded columns to reduce the data that needs to be processed
  • Create new fields that are produced from existing fields
  • Convert numeric columns to string columns or visa versa

In Decodable, window functions can be used to create subsets of your streams with a start time and end time. A pipeline can then aggregate the windowed data and the aggregate can be written into InfluxDB.

Get Started Today

If you are an InfluxDB user today and you have a streaming platform (Kafka, Kinesis, Pulsar, RedPanda, etc), you can log into Decodable.co and easily connect and transform your data today. If you need help, set up a free session with one of our experts here and we’ll assist in creating your pipeline.


You can get started with Decodable for free - our developer account includes enough for you to build a useful pipeline and - unlike a trial - it never expires.

Learn more:

Join the community Slack

Decodable Platform Overview

I joined Decodable about a year ago as one of the founding engineers. One year on, we’ve built a self-service real-time data platform supporting hundreds of accounts in multiple regions. In this blog post, I’ll share why we built Decodable, the design decisions we made, and the platform architecture.

Learn more

Decodable's Imply Polaris Connector: The Druid Easy Button

Decodable now includes a native sink connector to send data to Imply Polaris. Imply Polaris® is a real-time database for modern analytics applications, built from Apache Druid® and delivered as a fully managed database as a service (DBaaS). It provides a complete, integrated experience that simplifies everything from ingesting data to using visualizations to extracting valuable insights from your data.

Learn more

New CDC connectors for MySQL and PostgreSQL: Don’t Let Your Data be a Day Late and a Dollar Short

Decodable’s support for CDC (Change Data Capture) advanced further with the release of source CDC connectors for MySQL and PostgreSQL. Source CDC connectors convert traditional database tables into sources of streaming data for processing and delivery to sink systems via Decodable. This blog explains more about CDC with demos for the CDC connectors and Decodable's change processing capabilities.

Learn more

Transforming XML on the Stream

Integration combines disparate systems as well as diverse sources and types of data. This diversity is almost certain to include XML formats which are still used heavily today and need to be integrated with more modern data formats. In this blog, we'll show how simple it is to transform XML records into JSON records with Decodable.

Learn more

Heading

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique. Duis cursus, mi quis viverra ornare, eros dolor interdum nulla, ut commodo diam libero vitae erat. Aenean faucibus nibh et justo cursus id rutrum lorem imperdiet. Nunc ut sem vitae risus tristique posuere.

Learn more
Tags
Pintrest icon in black
ETL
InfluxDB

Start using Decodable today.