Apache Kafka Reviews

• Distributed
• Persistence
• Offset management by consumer

This is the base streaming component of our IoT platform.

In case of disaster recovery, we mirror the data in the cluster by maintaining the offsets and store the data within Hadoop 2.8 HDFS.

• It needs a separate cluster and a separate administrator to manage the Kafka cluster, adding an extra cost.
• It is challenging when data is moved to a mirror cluster, in the case of disaster recovery. It doesn't keep the offset.

I have used this solution for one year.

The open source community is very strong. Also, distributors like Cloudera and Hortonworks provide paid support.

For big data, we did not have a previous solution. I have used Microsoft MQ for building traditional systems.

The setup was straightforward.

This is open source with the cost of a cluster administrator.

We did not look at anything else. At that time, this was already accepted by the industry for streaming data processing.

If the Hadoop distribution is MapR, then consider MapR Streaming. MapR Streaming has overcome these fundamental issues. It stores data within the MapR-FS itself. So there is extra overhead, but with a licensing cost.

We have used Kafka for streaming customer web clicks from live sessions to understand customer behavioral patterns.

The ability to partition data on Kafka is valuable. But Kafka needs support and management. It is better to have it fully managed on the cloud.

The only reason I give Kafka as product a low rating is because there are far superior and cheaper alternatives in cloud-based solutions, where we save money on manpower, electricity, servers, datacenters, networking, etc.

In fact, this is the view I have for pretty much all open source software compared to cloud based services. They just make things cheaper, faster, scalable and manageable. Kafka is good, but Kafka as a cloud service is awesome!!

This is a relative rating (compared to cloud services), not that something is wrong with Kafka. I hope that is clear.

The product is good, but it needs implementation and on-going support. The whole cloud engagement model has made the adoption of Kafka better due to PaaS (Amazon Kinesis, a fully managed service by AWS).

No issues here with stability.

Ah, scalability!!! We need to set up multiple servers again for handling the load, which makes Kafka not scalable, unless you subscribe to cloud services.

It’s an Apache-community based support, so it is not really prioritized if you have a business issue. This is why most enterprise customers pay for cloud services.

We didn’t have a previous solution. We started with Kafka and then switched to Amazon Kinesis (PaaS for Kafka). I think Microsoft Azure also released a competing service.

The setup was straightforward.

Licensing issues are not applicable. Apache licensing makes it simple with almost zero cost for the software itself.

We unsuccessfully, and kind of foolishly, tried Apache Camel. They were not similar in services, so we moved to Kafka rightfully, and then to AWS cloud ultimately.

If you have a dedicated Kafka resource to implement and manage the services, then go for Apache Kafka. Otherwise, do consider cloud-based services from AWS or Azure.

I am an enterprise architect involved in Big Data and integration projects using Apache Kafa. We use it for integrating our different management systems.

The most important feature for me is the guaranteed delivery of messages from producers to consumers.

More adapters for connecting to different systems need to be available.

I have been using Kafka for about six months.

This is a stable solution and we haven't had any complexities.

This solution is scalable.

I have used IBM MQ and it is better in terms of the adapters that are available. However, the price of IBM MQ is very high.

The initial setup is easy.

Kafka is more reasonably priced than IBM MQ.

Although we are deployed on-premises at the moment, we are looking to have a cloud-based deployment in a year or two.

This is a solution that I can recommend but it will take a lot of time to develop the adapters.

I would rate this solution a seven out of ten.

Excellent speeds for publishing messages faster.

Too much dependency on the zookeeper and leader selection is still the bottleneck for Kafka implementation.

RESTful API implementation actually uses the Kafka Broker to publish the messages but I am not able to find it becoming scalable. Partially, the reason might be there is no load balancer for the RESTful API web server.

Setup is very much straightforward for development, and cluster setup is also easy. I am not aware of the production setup yet.

I implemented it on my own.

We are a solution provider and Apache Kafka is being used in our client's company.

The most valuable feature is the support for a high volume of data.

The initial setup and deployment could be less complex.

Integration is one of the main concerns that we have.

We have been using Apache Kafka for two years.

Kafka is a stable product.

This is a scalable solution.

The technical support is quite good, and we have no problem with it.

We also use IBM MQ. It is also a stable product.

IBM MQ is probably easier to deploy than Kafka.

In addition to these, I have also worked with RabbitMQ.

Deploying Kafka is more complex than IBM MQ.

My customer has asked me to choose between IBM MQ and Apache Kafka. I will be comparing these two solutions in the near future. My impression is that Kafka is going to better suit my customer, but I have to consider their specific needs before I can be sure.

This is a solution that I may recommend, but its suitability depends on the needs and requirements.

I would rate this solution an eight out of ten.

• Replication, partitioning, and reliability are the most valuable features.
• Even if one of my clusters fails, the replication factor of a topic makes sure that I have the data available for processing, so I won't lose any of it.
• Partitioning enables me to process the parallel requests. It helps in reaching the throughput.

One improvement is in regards to the OS memory management. In case there are too many partitions, it runs into memory issues. Although this is a very rare scenario, it can happen.

I have been using this product for a year now.

There were no stability issues.

Kafka is a highly scalable product. We have not faced any scalability issues so far.

Since it's an open source product, no technical support is available. However, the open source community is very active.

The initial setup was straightforward. Just go through the Kafka documentation and it will be up and running in no time.

Since it's an open source product, there is no pricing for it.