https://www.mparticle.com/blog/kinesis-vs-kafka/
Space missions, satellites, stock markets, and sports telecasts have been transmitting data with little or no perceptible lag for several decades now. However, these streaming solutions have historically been prohibitively costly for most organizations to deploy.
Today, streaming platforms like Apache Kafka and Amazon Kinesis make it possible for organizations to send, receive, and analyze data streams in real time – without requiring expensive infrastructure or herculean maintenance efforts.
In this article, we present an unbiased comparison of both these streaming platforms and help you choose the right solution for your needs.
The performance of a streaming platform takes into account how much data (throughput) and how quickly (latency) you can move it through a pipeline. Higher throughput rates and lower latency time translate into a more scalable and realtime streaming platform.
Kafka has a slight performance edge over Kinesis because it can be further fine-tuned for your unique needs. However, there are no noticeable performance differences between the two platforms.
Kinesis:
Kinesis, as a fully managed pay-per-use service, performs well. Its base throughput units are called shards. Each Kinesis shard provides a capacity of 1 MB per second of input data, 2 MB per second of output data, and up to 1,000 PutRecords per second. Depending on your workload, Kinesis scales the number of shards as required.
Kafka:
When tested on three cheap on-premise servers, Kafka delivered 2 million writes per second with a peak throughput of 193 MB per second and an average p99 latency of 3 milliseconds. When tested on managed cloud infrastructure, its peak throughput was 605 MB per second with a p99 latency of 5 milliseconds.
The cumulative effort required to get both these streaming platforms to work as intended should be considered in relation to your existing hardware and DevOps capacity. Otherwise, you risk running into unexpected cost and resource overruns.
Kinesis is a readily deployable solution that better suits teams with little or no DevOps expertise. Kafka is better suited for DevOps teams that can manually set up, configure, and fine-tune the deployment.
Kinesis:
Kinesis comes with automated deployment templates that can set up your production environment in a few hours. Because it is a serverless service, you don’t have to manually set up or configure any servers – it automatically scales resources to meet your workload spikes as needed.
Kafka:
Deploying Kafka involves several manual steps. First, you need to configure Kafka on the on-premise or cloud server of your choice and allocate the required resources. Next, you’ll need to deploy ZooKeeper and a Kafka broker. Last, you’ll need to manually set up nodes, partitions, and replication rules. Testing, optimizing, and fine-tuning the deployment to suit your needs can take many days.