How to Build a Pluggable IoT Pipeline in AWS

Abstract

As far as modern architectures go, there are few more complicated than an IoT pipeline. You’ve got to consider an ingestion layer (typically streaming) that may undergo manic load. You’ve got to think of data tagging, storage (probably across multiple engines), archival and access—both internal and external. And all of it has to scale like crazy, be as cost effective as possible, and use automation wherever it can. Oh, and your boss needs the IoT pipeline built by tomorrow. Short timelines? Tight budget? Unrealistic expectations? Unfortunately, these asks are realities for many cloud professionals. AWS knows this and is here to help.

Sample

AWS IoT does a lot of the work for you

In tandem with the launch of Amazon Alexa, AWS released AWS IoT in 2015. AWS IoT is a fully-managed IoT back-end platform that integrates with services, such as Lambda (serverless compute), Greengrass (allows running of Lambda on edge devices) and API Gateway (a secured, scalable API front end). This means that those looking to build the core of the pipeline—the ingestion, storage, and access layers—now have a lot of their work already done for them by AWS. But AWS IoT alone is not (yet) going to be enough to build everything.

In order to build an end-to-end, low-cost, highly-scalable and yet still pluggable (you can swap out various pieces) IoT Pipeline in AWS, you’ll need to follow AWS’ guidance to plan, deploy and optimize:

• Plan – Define KPIs of the project (do you want infinite scale, lower costs, more agility? What, exactly will constitute project “success”?). Identify services and architectures that enable those KPIs. Build architecture diagrams and proof of concept designs.
• Deploy – Build out the production system, tweak any design considerations (security, service choice, etc.), and start driving real traffic to the pipeline.
• Optimize – Should be an important part of your “Plan” phase—how will you maintain the app, and how easily will it be to swap out individual pieces? A huge part of this phase is constant review of new AWS products and services.

“Plan,” the toughest of the three phases, is the focus of this white paper. The other two phases will be referenced tangentially.