Apache Kafka Reviews

Kafka has a guaranteed delivery mechanism that is very easy to set up. When starting out with minimal hardware, it can handle very large data volumes. When prototyping and creating a proof of concept, Kafka has helped to speed up the timeline from the prototype all the way to production volumes.

Apache Kafka is actually a distributed commit log. That is different than most messaging and queuing systems before it. I find the ability to write data at one velocity and have subscribing consumers read at different velocities to be the best feature.

The GUI tools for monitoring and support are still very basic and not very rich. There is no help in determining a shard key for performance.

We did not have any issues with stability.

We did not have any issues with scalability.

• Kafka is open source from LinkedIn and support comes from the community of users.
• You can go with Confluent, the company that was founded by the original engineers from LinkedIn.
• You can go with a cloud hosting service, like AWS EMR or Azure HDInsight.

We used traditional message queues and file semaphores. There was a lot of overhead with asynchronous messages being put into an order and making sure nothing got dropped. It required a lot of code and maintenance.

Since it is open source, you are on your own for setup. However, the tutorials from the Apache foundation and online sources have been an immense help.

Getting started is very easy. The complexity of very large volumes of data and appropriate sharding, however, is difficult. There are fewer resources for tuning and best practices.

When starting to look at a distributed message system, look for a cloud solution first. It is an easier entry point than an on-premises hardware solution. A lot of the complexity has already been taken care of. Both AWS and Azure have supported Kafka clusters that can be provisioned very easily.

We looked at RabbitMQ and Spark Streaming.

Be sure to define the use cases as best as possible at first.

Kafka is very good, but it is complex to support. It can handle any message size, whereas native cloud options have size limitations.

Be sure to understand what messages will be sent and how many discrete topics will be needed.

Be aware that you must code both producers and consumers.

The bulk of the work is with the consumer.

The Apache stack for Kafka is very open source. There are essentially no tools other than command line options to monitor brokers and topic health. So there are 3rd party tools that will help with that, some free, some paid – but it requires that you install agents on the servers hosting Kafka and open up ports for netbeans on the scripts that start up the Kafka services. Additionally, you also have to monitor zookeeper – which is very memory intensive. Cloud offerings that provide the whole modern data architecture stack – like AWS EMR and Azure HDInsight as well as Hortonworks and Cloudera provide a console GUI as part of each of their offerings. Also Confluent, a company founded by the Linked-In engineers that designed Kafka, also have a paid enterprise offering that has much better tools for maintain the kafka cluster. But apache Kafka with the community – you are on your own.

• Replication, partitioning, and reliability are the most valuable features.
• Even if one of my clusters fails, the replication factor of a topic makes sure that I have the data available for processing, so I won't lose any of it.
• Partitioning enables me to process the parallel requests. It helps in reaching the throughput.

One improvement is in regards to the OS memory management. In case there are too many partitions, it runs into memory issues. Although this is a very rare scenario, it can happen.

I have been using this product for a year now.

There were no stability issues.

Kafka is a highly scalable product. We have not faced any scalability issues so far.

Since it's an open source product, no technical support is available. However, the open source community is very active.

The initial setup was straightforward. Just go through the Kafka documentation and it will be up and running in no time.

Since it's an open source product, there is no pricing for it.

Kafka was at the base of our system architecture. The system was designed as an event based architecture. Almost all the interactions among micro-services and the same data are used as input to our analytics infrastructure.

• Scalability
• Reliability
• Ease of use

Stability of the API and the technical support could be improved.

The Kafka API is changing quite radically with the different releases. There are many new improvements and that's good. But the inherent cost of adapting to a new version of the platform was worrying me at the time.

The documentation was sometimes misleading, since it was describing some feature in the new version of the API rather than the one we were using.

We did not encounter any issues with stability.

We did not encounter any issues with scalability.

We were not completely satisfied with the technical support. We subscribed to the Confluent professional platform to receive guidance and support on development and deployment. Whilst the development side is quite well covered by their consultants, the deployment and administration is not at the same level.

The previous solution was not really an equivalent one. I have been using several messaging systems, but Kafka fits us better for a more scalable system.

The initial setup was straightforward.

I would not subscribe to the Confluent platform, but rather stay on the free open source version. The extra cost wasn't justified.

We didn't evaluate other options, as we already had a positive experience across the team with Kafka. Everybody agreed to work with it.

We were considering Kinesis too, since we were running on AWS. We preferred to opt for a tool with which people were more familiar.

The product is easy to use. However, to leverage its power, there is a need for good knowledge of event based processing. I suggest using the massive amount of material shared by the Confluent team, or what is available online.

One of the best features which I have worked with is replay.

Real-time log aggregation which was earlier done with rsync has been moved to Kafka infrastructure along with other real-time streams.

• GUI for Kafka infrastructure monitoring and deployment

I have used it for two years.

Documentation is quite comprehensive.

I found it very stable.

No issues with scalability.

We used the open-source version.

We used the open-source version.

We previously used rsync, which was not real-time.

Initial setup was mostly intuitive (based on rsync).

Implementation was in-house based on the open-source version.

Target was to achieve real-time service.

Before choosing this product, we did not evaluate other options.

Excellent speeds for publishing messages faster.

Too much dependency on the zookeeper and leader selection is still the bottleneck for Kafka implementation.

RESTful API implementation actually uses the Kafka Broker to publish the messages but I am not able to find it becoming scalable. Partially, the reason might be there is no load balancer for the RESTful API web server.

Setup is very much straightforward for development, and cluster setup is also easy. I am not aware of the production setup yet.

I implemented it on my own.

• Supports more than 10,000 events/second.
• Scalability
• Replication

It is a good product for event-driven architecture.

We use Kafka for reactive architecture, track and trace, mail and parcel.

A good free monitor tool would be great for Apache Kafka (from Apache foundation).

We used Kafka 0.8 for 2 years and Kafka 0.10 for 3 months.

We have not encountered any stability issues.

We have not encountered any scalability issues.

We haven’t used technical support.

Apache MQ is different. It is a message bus (log rotate) than can manage more than 10,000 events/sec.

The basic configuration is quite good. We have built a Hadoop cluster and the Kafka service was included.

We use a community version.

Kafka processes asynchronous exchanges, so there are no transactional interactions.

The most valuable features are topic-based eventing, scalability, and retention periods.

My organization is transforming by using the new SOA/eventing-based architecture. The application depends on the employees’ information events. Kafka is very helpful in implementing this. It increases the performance and gives the details to multiple external/internal teams using Kafka topics in an asynchronous manner.

For example, if someone is moving from one office to another one, we have to update the software. While updating it, the system puts that event in a topic so that all other consumers can update that person’s new location. This can include the payroll team, the insurance team, and the hospital network.

The retention period helps us retain the data in the topic for the configured number of days. In this example, if any of the consumers fail to consume the message from the topic, then that message will be there until the retention period ends.

I would like to see a more user-friendly GUI.

We have used this solution since December, 2015.

If you are using the same group ID for multiple topics, it may shut down the application. We have faced this issue before.

We have not had any scalability issues.

I would give technical support a rating of 6 out of 10.

We were using ActiveMQ, which is just a messaging system. We are changing because of Kafka’s added value of scalability, retention, and high payload support.

The installation was somewhat straightforward.

The solution is worth the money.

This is the best tool I have ever used for asynchronous, event-based solutions.

We use the product for high-scale distributed messaging. The processing capability of the product is enormous. Being a distributed platform, multiple consumers can sync with it and fetch messages.

Another great feature is the consumer offset log which tells you where the consumer left and where he needs to start again. Consumers aren’t required to code and put extra effort to maintain the offset.

We were using another commercial messaging engine, which was not scalable unless you paid more. Each hub that we provisioned was expensive. This solution is open source, which is much easier to use and doesn’t cost us anything.

This product guarantees at-least-once delivery. We have asked JIRA to provide features such as at-most-once delivery to remove duplicate message consumption.

We haven’t faced any issues so far. Some of the clusters churn millions of records per seconds with ease.

We have clustered environments and we haven’t seen any scalability issues. We can provision a new node in as little as 45 minutes.

It is open source, so support is in our own hands. The only option is to make a new feature request through JIRA. When multiple people in the community make a request for similar feature, it gets priority.

We switched from a previous solution mainly to reduce costs and to have a more scalable solution.

The initial setup was a bit complex in terms of how to manage it across data centers. But once it was setup, we never faced issues.

We evaluated multiple options, such as ActiveMQ and RabbitMQ. We leaned towards this solution.

I would advise others to start with non-SSL implementations and try to do PoCs. Afterwards, they should move towards more secure features.