Apache Kafka Reviews

We use Kafka for a stage event-driven process from a process perspective. Our platform is an ID platform, so after registration data is received, it has to be stored from various registration locations. The process includes stages like quality checking, consistency, format, biometric data checking, and so on. We are using both Kafka and ActiveMQ for almost three years now.

Using Kafka has saved us costs compared to proprietary solutions. It has allowed for significant customizations internally, which would have been much costlier with other solutions. We have saved a lot by employing Kafka and only needing basic salary agreements for our experts.

The real-time data processing capability of Apache Kafka is a significant pro. We also find Kafka to be relatively stable under large data volumes and its performance to be consistent. The convenience in setting up after major problems like data center blackouts is a notable feature.

Kafka has some limitations in terms of queue management. Specifically, it lacks the capability to handle larger queues for external system interactions. It would be beneficial if Kafka included more robust, high-capacity queue management features for integration with external systems.

We have been using Kafka for almost three years now.

Kafka is stable even under high data sizes. Even if the data size is at the highest possible capability, Kafka executes relatively stable compared to other solutions we have used. I would rate its stability as a nine out of ten.

Scalability is one of the strong suits of Kafka. Initially, we handled up to five thousand registrations of data items per day. Now we manage up to forty thousand. We have scaled Kafka on the same Kubernetes environment, and it has been very convenient without critical problems.

We have not directly communicated with technical support for Kafka. Instead, we have relied on the open community for advice and employed senior experts to work on the platform.

Positive

We did not use any other solutions before Apache Kafka for these use cases.

The initial setup for Kafka was a bit difficult and required additional configurations. It took us about one to two hours to deploy the whole platform, which includes other components like PostgreSQL database, antivirus, and directory management solutions. Kafka itself would take only a fraction of this time.

Our DevOps team currently consists of four people working directly on this platform. For Kafka, we also involve one person from our infrastructure team, making it a total of five people.

We have definitely seen a return on investment in terms of cost savings. Using Kafka as an open-source solution has saved us a lot compared to proprietary solutions. We have been able to make significant customizations internally at a lower cost.

I would rate the overall cost of using Kafka as a three out of ten, indicating that it is rather affordable, considering the benefits and savings it provides.

We did not evaluate other options or vendors before choosing Apache Kafka. We followed recommendations from other experts.

Kafka is one of the most convenient tools with a fair level of performance. Its stability is impressive, allowing us to recover from data center blackouts quickly. I highly recommend it for similar needs.

I'd rate the solution nine out of ten.

I was planning to use the tool for real-time analysis in terms of data processing and real-time analytics workflows. The real-time IoT data comes through with a few challenges, and that is for one time, so it is more like a Kafka topic. I want to actually use multiple Kafka topics where one of them can be directly fed into the data pipeline, another one can be fed into the real-time alert system, and the next one can be fed into machine learning.

The most valuable features of the solution revolve around areas like the latency part, where the tool offers very little latency and the sequencing part. The sequencing part actually helps to aggregate things in a way that I don't need to write another function or kind of sequence it, and I write an aggregate function to figure out the maximum value in the last ten samples.

One complexity that I faced with the tool stems from the fact that since it is not kind of a stand-alone application, it won't integrate with native cloud, like AWS or Azure. Apache Kafka has another mask on it, so if users can have a direct service, like Grafana, that can actually be used as a stand-alone tool with Grafana cloud, or you can use a mix of AWS and Grafana, so there is not much difference with it. I expect Apache Kafka to have Grafana's same nature.

The product's support and the cloud integration capabilities are areas of concern where improvements are required.

I have been using Apache Kafka for a year.

Stability-wise, I rate the solution an eight out of ten.

Scalability-wise, I rate the solution an eight out of ten.

Around four people in my company use the product.

I did not interact much with the product technical support team. I did not have dedicated support that responded to all my queries since I was using the product's free version. I rate the support a seven out of ten.

Neutral

I have worked with Databricks. I use Databricks and Apache Kafka simultaneously.

The product's deployment phase is neither complex nor straightforward. As the software has evolved a lot, users can actually keep it even simpler by opting for a plug-and-play model.

The solution is deployed on an on-premises model.

The solution can be deployed in two or three days.

I was involved with the tool's installation process.

I cannot comment on the tool's ROI since I did not use it for production purposes.

I was using the product's free version.

I did not come across any scenarios involving fault tolerance because when it comes to the issue data consistency issues, like missing or incorrect value of data are actually part of the system where the data is being fed. Nevertheless here, when it comes to the missing values, I never tried the option, especially whenever a value is missing, that can allow one to impute the value with another parameter.

Speaking about if I incorporated any emerging data stream streaming trends in Apache Kafka workflows, for example, utilization of AI, I would say that I use it as a local system, so if I have an EC2 server where I kind of read the sample and then populate the regression and reintegration model on top of it, but that is done locally and not on the cloud.

I recommend the product to those who plan to use it. I like Kafka and Flink, and I want to actually create a system in AWS mainly for real-time streaming so that I don't need to worry about multiple data copies.

Considering the improvements needed in the product's support, and the cloud integration capabilities, while looking at the simplicity during the installation phase, I rate the tool a seven out of ten.

Kafka is used as a streaming platform where multiple producers and consumers exchange high load and high volume of messages asynchronously without affecting each other's performance. 

It serves as an industry-standard platform for such operations. Kafka is also integrated into data system architecture for applications like monitoring events on platforms like LinkedIn to enable further analytical insights.

Kafka makes data streaming asynchronous and decouples the reliance of events on consumers. 

It was the first of its kind to provide a streaming pipeline, setting a new component in the tech architecture and ecosystem. It allows continuous messaging without impacting performance.

Confluent has improved aspects like documentation and cloud support, yet Kafka's reliance on older architectures like ZooKeeper in previous versions is a limitation. 

Its language and architecture could be further improved to solve issues in consensus algorithms, as Red Panda does.

I have been working with Kafka for about a year and feel comfortable using it.

I have not had any issues in terms of performance; however, there may be performance issues due to Java's garbage collector, which can cause memory issues if bloated.

While I have not tried setting Kafka up on Docker containers, it is possible. I have only run a single-node broker for Kafka.

I also use Redpanda, which is similar to Kafka in features, however, they differ in internal workings affecting performance and resource usage.

The setup process is straightforward as per the documentation. It involves unpacking zip files with the necessary packages, ensuring Java and JVM are installed. 

Previously, Kafka relied on ZooKeeper, requiring two configuration files. However, with the newer KRAP version, the setup does not need ZooKeeper, which simplifies the process.

Apache Kafka was part of a college curriculum, and I set IP up myself. I found setting it up manageable.

I definitely recommend Kafka, as it is the industry standard for streaming platforms. While Red Panda is similar, Kafka remains the stronger choice in the market for its established support and usage in big companies.

I'd rate the solution nine out of ten.

My company uses Apache Kafka to keep some intermediate data in the workflow.

Apache Kafka's most valuable features include clustering and sharding. Though we have not started using Apache Kafka Streams in our company, I have heard that it is one of the good features of the product we plan to use. The good features let you replay and reconsume messages in Apache Kafka, allowing you to have multiple consumer groups. The rebalancing feature of the product is also useful since if one consumer dies, then Apache Kafka does a rebalancing. With Apache Kafka, we use clustering, sharding, and partitioning features in our company.

In Apache Kafka, it is currently difficult to create a consumer. The implementation of Apache Kafka's features, like rebalancing, is possible only when you create a consumer, which is a very difficult task since it is overly complicated. To create a consumer in Apache Kafka, a person needs to have a very strong knowledge of the internal functioning of Apache. I feel that Kaka needs to provide a consumer so that its users don't spend time in the creation of consumers. In general, Apache Kafka must provide users with a more user-friendly UI.

I have been using Apache Kafka for five years. I am just a customer of the solution.

It is a pretty stable solution. Stability-wise, I rate the solution a nine out of ten.

Scalability-wise, I rate the solution an eight out of ten.

The users of the solution are not directly involved with it. Those who use Apache Kafka in our company use it to push orders on the frontend, and then the frontend calls for some microservice, after which the microservice pushes data to Apache Kafka. Around 10,000 to 15,000 people in my organization follow the aforementioned procedure.

I won't be able to comment on the technical support team of Apache Kafka since some other team members in my company were involved with them. In general, my company is satisfied with the technical support team of Apache Kafka. I rate Apache Kafka's technical support an eight out of ten.

Positive

My company started off with Apache Kafka, so they did not use any other solutions previously.

If you are using the latest version of Apache Kafka in which the use of Zookeeper is not required, then it uses the KRaft protocol, which is built into Kafka broker. Since the use of Zookeeper is no longer required, I think the setup phase of dissolution is better than its previous versions. I rate the initial setup of the product somewhere between seven to eight out of ten.

Apache Kafka's initial setup is very straightforward.

The solution is deployed on an on-premises model.

Apache Kafka was deployed in our company within a couple of days.

Three people were involved in the deployment process of Apache Kafka.

I rate Apache Kafka's pricing a five on a scale of one to ten, where one is cheap and ten is expensive. There are no additional costs apart from the licensing fees for Apache Kafka.

During my company's evaluation process for other options from Apache Kafka, we came across RabbitMQ. My company chose Apache Kafka over RabbitMQ since it was one of the market's more popular tools. With Apache Kafka, more materials, support, online technical groups, and forums were available for consumers.

Apache Kafka as a broker tool is a very stable and good product. When you need to create a consumer in any programming language, including Java, Golang or any other programming language, the team involved in the process of the creation of a consumer should have very strong knowledge and expertise in the use of Apache Kafka since it is not at all easy to create a consumer for the product. A highly qualified person with a good amount of experience should also know the internals of the solution, which may not seem too straightforward. Anyone cannot use Apache Kafka easily without proper knowledge or experience. When you use Apache Kafka in your actual application, you need to create some producers and some consumers. To create a consumer, you need to have a very strong understanding of the solution since it is not a process that anyone can manage easily. A company needs to have a very strong team with good technical knowledge to be able to use the product.

I rate the overall solution a nine out of ten.

We have multiple use cases for our Kafka system. One is Kafka Connect, which is used to facilitate communication between different regions with Grid Deal. Another is to distribute events and projects to multiple downstream. We publish all the messages to Kafka and other listeners subscribe and write them to different MQs. Lastly, Kafka Connect is used especially for inter-application communication.

We had been using a lot of expensive licenses earlier, such as SOLEIL, as well as some legacy versions, which were not only costly but also caused memory issues and required highly technical personnel to manage. This posed a huge challenge in terms of resourcing and cost, and it simply wasn't worth investing more in. However, Kafka was comparatively free as it was open source, and we were able to build our own monitoring system on top of it. Kafka is an open-source platform that allows us to develop modern solutions with relative ease. Additionally, there are many resources available in the market to quickly train personnel to work with this platform. Kafka is user-friendly and does not require an extensive learning curve, unlike other tools. Furthermore, the configuration is straightforward. All in all, Kafka provides us with a great platform to build upon with minimal effort.

The use of Kafka's logging mechanism has been extremely beneficial for us, as it allows us to sequence messages, track pointers, and manage memory without having to create multiple copies. We are currently on a legacy version and have found that the latest version of Kafka has solved many of the issues we were facing, such as sequencing, memory management, and more. Additionally, the fact that it is open source is a major benefit.

Multiple people have constructed conflict resolution with successful solutions on top of open-source platforms. Unfortunately, open source does not have the monitoring and capabilities these solutions offer, so organizations must create their own. Investing in these solutions may be beneficial for many companies, who prefer to use open-source options. 

There is a lot of information available for the solution and it can be overwhelming to sort through. The solution can improve by including user-friendly documentation.

I have been using the solution for four years.

We have not experienced any issues with the stability of the solution. We had some issues with Grid Gain and Kafka Connect, but we believe it was more of an issue on Grid Gain's side since they informed us of a bug. Our result has been that we have not encountered many issues on the Kafka side.

We use the solution in the distributed mode in multiple regions – the US, London, and Hong Kong. We have increased the number of nodes to ensure it is available to us at all times.

I give the scalability an eight out of ten.

We have around 600 people within my team using the solution.

The initial setup was relatively easy for us since we already had Zookeeper and the necessary setup in place. We also had good knowledge of Kafka. Therefore, it was not a difficult challenge. In general, I believe that it is manageable. There are benefits and the setup is not overly complex.

Our company has implemented Ship, making our lives easier when it comes to changes or version updates. We can package everything in one place and deploy it with Ship, then implement the virtual number with a minimum of 50 changes.

Deployment time depends on our location and the task at hand. Initially, there is a lot of setup and configuration that must be done, but this can become easier with experience. Nowadays, the process is not too difficult, as all the version numbers and conflict files are already in place. However, if this is a new task for us, it may take some time to figure out all the configurations.

One person was dedicated to deploying Kafka. This person got help from our release team, who had already set up Zookeeper and other necessary components.

The implementation was completed in-house.

I give the solution an eight out of ten.

Maintaining Kafka, the open source, can be difficult without the proper version purchased or the right infrastructure in place. However, once the initial setup is complete, it is relatively simple to maintain. The open-source version of Kafka is not a complete package, so additional maintenance may be required.

I strongly recommend reading the documentation for any issues because it is likely to contain the answer we are looking for. There is a lot of information provided that may not be immediately obvious, so take the time to explore thoroughly.

Currently, I work for an observability company. We stream customer data into our cloud, digest the information, enrich it, transform it, save it, and use on-the-fly aggregation with Kafka. Previously, I worked for a security company doing normal detection using streaming with Kafka. 

I also worked for a company with a data platform based on Kafka, where we ingested clickstream data and enriched it before streaming.

The most valuable feature of Kafka is the Kafka Streams client. Unlike other systems like Flink or Spark Streaming, you don't need a separate engine to do real-time transformations and analytics. The amount of data that can be streamed into the platform and the scalability are also significant benefits.

Kafka requires fine-tuning to find the best architecture, number of nodes, and partitions for your use case. It’s a trial-and-error process with no one-size-fits-all solution. Issues may arise until it’s appropriately tuned. 

While it can scale out efficiently, scaling down is more challenging, making deleting data or reducing activity harder.

I have been working with the Kafka product for more than ten years.

Since Kafka is written in Java, it's not as stable as it should be on the JVM. The stability depends on fine-tuning the system to find the best architecture for your use case. However, the replication factor helps avoid data loss despite the stability issues.

Kafka's architecture allows for scalability by adding nodes and partitions to topics. However, it's not as effective in scaling in, making reducing activity and deleting data harder. 

Scalability can be managed both manually and automatically to meet demands.

I used to work with Spark Streaming and Flink, however, not in the past year.

If you are unfamiliar with Kafka, setting up the cluster can be quite difficult. You need to understand the architecture and components and compute the data volume upfront. For experienced individuals, the setup is less difficult yet still requires preparation.

From a time-saving perspective, onboarding new customers is straightforward, requiring them merely to stream their data into our platform.

We use Apache Kafka, which is open-source, so we don't have fees. I can't comment on ownership costs as I am not responsible for that domain.

Apart from Kafka, I have experience working with Spark Streaming and Flink.

When implementing Kafka, it's important to plan the cluster size upfront to ensure easy scalability. Adding or removing nodes can disrupt the clusters, so proper sizing and planning are key. 

I would rate Kafka as a solution as a nine.

Apache Kafka is an open-source solution that can be used for messaging or event processing.

Apache Kafka has performance issues that cause it to lag.

We did a couple of POCs on Apache Kafka for more than two years for messaging and event processing.

I rate Apache Kafka an eight out of ten for stability.

I rate Apache Kafka a seven out of ten for scalability.

Since it's an open-source solution, there is no technical support, and users often rely on the community edition.

I have previously worked with Confluent and Anypoint MQ. Confluent is completely an event-driven architecture. Anypoint MQ is a typical messaging software and cannot be used for an event-driven architecture.

The solution's initial setup is quite straightforward. You just have to upgrade a couple of configuration files.

Apache Kafka is an open-source solution.

A non-enterprise business with a low message load can use an open-source solution like Apache Kafka.

I would recommend the solution to enterprise businesses depending on their use cases. Suppose an enterprise business doesn't have any integration or a middleware platform and wants to do a greenfield implementation. I'll evaluate the use cases and refer Apache Kafka to them if messaging is needed only for exception handling or transferring the messages.

I have recommended Apache Kafka to some customers who wanted asynchronous messaging for logging purposes. Those messages were not business-critical messages as such.

I would recommend Apache Kafka to other users. Apache Kafka is more relevant when we use open-source integrations and when customers want to reduce the TCO. As an architect, I recommend the solution to customers based on their messaging needs. Apache Kafka and Anypoint MQ are the only two messaging products available today. The open-source Apache Kafka is always recommended if the customer really doesn't want to get into any of the license models.

Overall, I rate Apache Kafka an eight out of ten.

We are using Apache Kafka to extract data from a Portuguese data source, utilizing an open-source project for data capture. The connector for this project is linked to both Kafka and Confluence platforms. We then transform the extracted data and store it in Elasticsearch.

We appreciate the ability to persistently and quickly write data, as well as the flexibility to customize it for multiple customers. Additionally, we like the ability to retain data within Apache Kafka and use features, such as time travel to access past customer data. The connection with other systems, such as Apache Kafka and IBM DB2. 

Apache Kafka can improve by adding a feature out of the box which allows it to deliver only one message.

I have used Apache Kafka within the last 12 months.

Apache Kafka is a stable solution.

The scalability of Apache Kafka is good. It can process many requests simultaneously.

We have approximately 600 people using this solution in my organization.

I have not contacted the support from Apache Kafka.

The initial setup is relatively easy as I am using Docker and the files provided by Confluent. However, setting up Apache Kafka in a production environment is not as straightforward. I prefer to use solutions, such as Confluence that already have everything preconfigured. As a developer, creating an environment for it is not a problem for me, but I think it can be challenging for those responsible for the production environment. There have been issues with data loss and other problems in the past. Configuring it for production is not easy.

My deployment was very quick because I am using it locally. We have someone else that does the cloud deployment.

I did our local implementation and we have someone else that does the cloud deployment.

The price of Apache Kafka is good.

I rate the price of Apache Kafka an eight out of ten.

I don't see any major issues with using Apache Kafka. Many companies use it and it's a good solution. My advice would be to use it as a software-as-a-service rather than setting up your own cluster. This way, you can benefit from a preconfigured and maintained platform. It's better to opt for a software-as-a-service solution.

I rate Apache Kafka an eight out of ten.