Kinesis is a fantastic cloud-based queueing service from Amazon. As an enhancement to that service, Amazon recently released the Kinesis Producer Library (KPL), which allows consumers of the service to aggregate messages into a smaller number of records. With that aggregation, total record throughput decreases, which makes pricing for the service competitive, and in some cases cheaper than running your own infrastructure. Even with the KPL however, there are still a substantial number of considerations to make before integrating with Kinesis. You need to consider back-pressure, metrics, monitoring, etc.
To address those considerations, Monetate created and open-sourced Koupler. As a comparison, if you are a home-owner that has wrestled with do-it-yourself plumbing projects, you are probably well acquainted with the shark-bite coupler. It is amazing. It can connect anything to anything, simply and without much effort. The invention has brought complex plumbing projects within the reach of mere mortals. Koupler aims to do the same for Kinesis integrations.
Koupler accounts for the complexities associated with event-queue connections and simplifies the integration, offering familiar interface mechanisms that allow Koupler to be used out-of-the-box with many systems. Specifically, Koupler provides TCP, UDP, HTTP and IPC interfaces to Kinesis. This enables simple tooling and command line mechanisms to integrate with Kinesis with little or no application code. For example, one could wire system logs to Kinesis via simple syslogd configuration.
Koupler also provides insight into client-side throughput metrics. Those metrics are posted back to CloudWatch, which is Amazon’s dashboard that allows administrators to monitor systems.
To get started, and for more information, please visit Monetate’s github repository: https://github.com/monetate/koupler
Monetate encourages contributions and community involvement. Please do not hesitate to reach out directly or via github. (firstname.lastname@example.org)
About Monetate Monetate is the global leader in providing multi-channel personalization to the world’s best brands. Built for speed, the Monetate Platform’s easy-to-use interface allows marketers to create, test and deploy an unlimited number of personalized digital experiences with limited need for IT or consulting resources.
With Monetate, marketing is transformed from a world of discrete campaigns to one of always-on, personalized experiences that maximize the lifetime value of each individual customer.
Monetate is used by the world’s leading retail brands to grow revenue faster by being faster. Founded in 2008, Monetate influences billions of dollars in revenue every year for world-class companies like Macy’s, QVC, Patagonia, Sur la Table, The North Face and hundreds of other market leaders.