Apache Kafka on Confluent Cloud Reviews

I use the solution in my company mostly for log-related purposes.

The solution's most valuable feature is that we used it during the building of data pipelines. From that, we read, wrote, and managed all the processes if there was any event happening. So, in real-time, we are just tracking how many users or logs are coming into our company.

We use Apache Kafka on Confluent Cloud for streaming large volumes of data in real-time. It's employed in scenarios such as handling events from various countries and streaming them efficiently for our clients. 

We also utilize it for data analytics and in client versions for topic creation, consumer consumption, and ACL provisioning.

Apache Kafka on Confluent Cloud provides an enhanced level of reliability and resources compared to Apache Kafka alone. It offers more features which are beneficial for our clients, including cluster linking, schema registry, error handling, and dead-letter queues. It significantly improves customer and publisher satisfaction, especially with topic integration and data streaming.

In my company, we are not using the tool for analytics and it is more for CDC processes, so we change the capture processes. It is used to extract data from a database and make it available in other parts of our systems or produce events that inform us of data updates.

There are some premium connectors, for example, available in Confluent, which you cannot access in the marketplace, so there are some limitations. From Confluent's point of view, I understand where they come from, but I believe its deployment model is a little strict. Confluent Platform can be installed in your own infrastructure. Even if you install Confluent Platform on your own platform, you need to use the components that Confluent offers. Otherwise, the support is very limited, and I think this is an idea of improvement for Confluent. Confluent is pretty solid, so I don't have much in terms of improvement.

Whenever you need to handle a huge load of real-time data processing, Kafka is useful. We currently use it for an output management system for insurance, where the system receives data in a fixed amount and has to process it in several steps. 

We manage these steps with Kafka because the load can be quite big, with millions of XMLs coming into the system that need to be processed in near real-time.

The most valuable feature to me is it scales really well when there is a data influx, irrespective of the peak time. It can handle both large and small amounts of data. 

Also, the state-saving feature is very much appreciated. It allows me to rewind a certain process if I see an error and then reprocess it.

We use Apache Kafka with Confluent Cloud for specific real-time transaction use cases, both on-premise and in the cloud. We have been using Confluent Cloud for about five years.

We initially used it for data reputation, then expanded to microservices integration and Kubernetes, focusing on improving data quality and enabling real-time location tracking.

We configure it for data transactions across various topics and partitions, depending on the specific use case and required throughput.

From an IT perspective, I've used this product across all domains: system development, operations, data management, and system quality.

We have experience using Kafka on Confluent Cloud for data pipelines. We've implemented several techniques to optimize topic usage, integrated it with microservices, and even utilized change data capture (CDC) components.

It's basically four bands of use cases, where we publish data on Kafka topics and stream it across microservices.

Some of the retention policies of Kafka topics have been most beneficial for data management specifically.

Based on our experience, there are different use cases where data needs to be handled in different ways. Sometimes we want to get rid of it once it has been consumed, or we have to store it for a longer period. 

Kafka provides handy properties that allow us to directly configure the data, whether to keep it or discard it after use.

Our use case is for real-time data integration. It was a preferred tool for this purpose. Additionally, we employed Azure EventHub, another service, as an indicator for real-time data in a couple of larger programs focused on integrating real-time data and visualization.

Kafka and Confluent Cloud have proven to be cost-effective, especially when compared to other tools. In a recent BI integration program over the past year, we assessed multiple use cases spanning ship-to-shore and various Azure integrations. Our findings revealed that Confluent Kafka performed exceptionally well, standing out alongside Genesys and Azure Event Hubs. While these three are top contenders, the choice among other tools depends on the specific use case and project requirements.

The customer initially used tools like SMQs, FITRA, and Stream for real-time data processing. However, after our recommendation, Confluent Cloud proved to be a superior choice, capable of replacing these three tools and simplifying their data infrastructure. This shift to a single tool, Confluent Cloud, streamlined their operations, making maintenance and management more efficient for their internal projects.