Apache Kafka on Confluent Cloud Reviews

I have used Apache Kafka on Confluent Cloud for one of my projects with regard to log monitoring. My main use case for Apache Kafka on Confluent Cloud in that project was mainly streaming of the logs. I wanted to capture logs coming from various interconnected systems into a unified place, so Confluent helped me to streamline all those logs into one place, and then I was consuming those logs that were produced.

Having all my logs unified helped our team a lot because the main challenge we were trying to solve was that in the current scenario we were working on, there was no place where we could view the logs in one place like Grafana or anything similar. It was not available, and for this scenario, we had to use multiple systems in order to check the logs, which could be databases, different applications, and logs for various other APIs. So it was not unified in one place. Now we unified all those logs by producing it to Apache Kafka on Confluent Cloud and then we were consuming it.

It was very much easier because I know Kafka and using Confluent made it much simpler. It was much more easy to understand, grasp, and very well structured with regard to even the JSON response and everything.

I am using Apache Kafka on Confluent Cloud with the Confluent servers itself; we had taken a subscription to Confluent, so it would be the private cloud.

In terms of the development it took us to set this whole thing up using Confluent, we were able to do it at a quicker rate. If we went with the ideal vanilla Kafka, it would require much more manual effort, but here it was easier because of the user interface and the experience, which was mainly very much drag and drop, so we could easily get it done faster.

We did initially have Grafana, but the only problem with Grafana was it was limited to certain applications, and ours was not among them. Because of that, we thought of shifting to a unified log monitoring system and started off with having vanilla Kafka installed on our servers. But we found Apache Kafka on Confluent Cloud much more convenient regarding how we would set it up, so to get things done faster, we shifted to Confluent.

The best features Apache Kafka on Confluent Cloud offers would be the connection with various external systems through various languages such as Python and C#. That gave us an edge to use this tool, and we were able to connect to it.

The multi-language support helped my workflow because we did use two kinds of languages, mainly .NET or C# and Python. So we were able to get logs because we had these out-of-the-box connectors available in Confluent. We just had to read the documents, understand how to configure it, how to send the logs, and we were through by just adding a few lines of code in our applications in the respective languages.

Apache Kafka on Confluent Cloud positively impacted my organization because this was just one of our projects we had utilized, while there are many more I was not involved in, so I cannot really speak on them. But this helped a lot with regard to log monitoring and many other streaming use cases where we were able to get all the data in one place and stream it at a quicker rate, and it was much more simpler compared to using plain vanilla Kafka. So Confluent made that difference for us.

I think Apache Kafka on Confluent Cloud can be improved by probably working more around Confluent or the tool. In my opinion, it should utilize the response structures in a better way or be able to detect if there is any variable or if there is any data structure that is mismatched, as it would be easier than us manually having to put in the exact name in order for it to match the response.

Regarding additional improvements, I would say probably around error handling, where when we encounter errors specific to our response structures and everything, or the tables or anything of that nature, it would be better if we were prompted with better error handling mechanisms.

I do not think there are any other improvements Apache Kafka on Confluent Cloud needs, aside from error handling and response structures.

I have been working in my current field for four or more years.

Apache Kafka on Confluent Cloud is stable.

Regarding scalability, I think it depends on the subscriptions and the way we set it up on the cloud. According to me, it is quite scalable in terms of all the data it can handle and stream, so I would say it's quite scalable.

I did interact with the customer support team when I was checking on my credits issued and the cost of my credits utilized. During that time, I had an interaction through email, and it went very well. I was getting prompt responses, and it was nicely handled regarding the support.

Positive

We did initially have Grafana, but the only problem with Grafana was it was limited to certain applications, and ours was not among them. Because of that, we thought of shifting to a unified log monitoring system and started off with having vanilla Kafka installed on our servers. But we found Apache Kafka on Confluent Cloud much more convenient regarding how we would set it up, so to get things done faster, we shifted to Confluent.

In terms of the development it took us to set this whole thing up using Confluent, we were able to accomplish it at a quicker rate. If we went with the ideal vanilla Kafka, it would require much more manual effort, but here it was easier because of the user interface and the experience, which was mainly very much drag and drop, so we could easily get it done faster.

We did not really look at any other options. We were, in fact, given Apache Kafka on Confluent Cloud as the top solution when we were doing our solution designing, so that is the reason we directly started our testing and trials on Confluent Cloud.

I do not have metrics to share regarding the return on investment, so I cannot really give an insight on that.

My experience with pricing, setup cost, and licensing went well. When I did my testing around it or research, I used the free credits. However, I did not realize that I was crossing the credits until I received an email regarding billing, so I had to email the team to let them know that I was not aware of that. I thought Confluent would stop me when I crossed the credits, but it did not, and then I got charged. So, because of that, there was back and forth with the team regarding the cost, but overall, I feel it would have been easier to handle if there were a mechanism that let us know when the credits were being fully utilized or when the limits were getting crossed.

We did not really look at any other options. We were, in fact, given Apache Kafka on Confluent Cloud as the top solution when we were doing our solution designing, so that is the reason we directly started our testing and trials on Confluent Cloud.

My advice to others looking into using Apache Kafka on Confluent Cloud is that it is easier and has a low learning curve. If there is any use case regarding streaming, I would suggest starting off or definitely trying out Apache Kafka on Confluent Cloud and getting the application maybe set up on it, if not evaluating how it returns you the returns. This is a good way to start off because of how it is easier to set up; it saves time, allowing us to utilize the time for other business logic or other things we would need to do. I would rate this solution a 9 out of 10.

We need to send a lot of asynchronous messages in this project, and we use the middleware and Apache Kafka on Confluent Cloud to guarantee asynchronous messaging between the services.

We use Apache Kafka on Confluent Cloud in the cloud.

We also use it with AWS.

The valuable features with Apache Kafka on Confluent Cloud are the messaging and the asynchronous messages; it's the basic, not advanced usage. It's only to create clusters to receive and send messages.

The point is the asynchronous messages and the scalability; it is important for us.

To guarantee the compliance of the architecture and the patterns for the company, to provide scalability, and to guarantee the security to send the messages.

The Kafka Streams API helps with real-time data transformations and aggregations.

It's very fast and helps us to create the project, guarantee the message delivery, and the performance. It's a good experience with very impressive processing and a very impressive project and product.

If it were easier to configure clusters and had more straightforward configuration, high-level API abstraction in the APIs could improve it. The clustering is a little hard for juniors and clients. It's suitable for senior engineers, but the configuration and clustering are very hard for juniors.

I don't use Apache Kafka on Confluent Cloud more in this project these days. We are analyzing if we can use Apache Kafka on Confluent Cloud in another project next month, but today, we don't need this software.

I don't remember exactly the problem, but we did have a problem creating the cluster and scaling messages. After the correct configuration, we don't have more problems.

I did have experience with AWS support.

Positive

We did not use different streaming analytics before using Apache Kafka on Confluent Cloud.

We evaluated other options including WebSphere MQ and AMQP, and there was another solution I don't remember.

We didn't use support from the companies. We used AWS support, and for the internal team, the infrastructure team has some knowledge about Apache Kafka on Confluent Cloud, but we never contracted a third-party company to support this project.

Synchronized messages were very important for us.

We track performance and throughput as key metrics to evaluate Apache Kafka on Confluent Cloud's handling of event streams.

The benefits include improving the number of new customers that you can attain, having a more modular system, new functionalities, and high availability.

The main differences between WebSphere and Apache Kafka on Confluent Cloud are performance, throughput, and the knowledge from the developers in Brazil.

I have many people working with Apache Kafka on Confluent Cloud in Brazil, and I can send you their contact information if you send me an email.

I rate Apache Kafka on Confluent Cloud 10 out of 10.

The use cases with this product are events. I use Apache Kafka on Confluent Cloud, and that's what events are.

Some of the best features with Apache Kafka on Confluent Cloud are streaming and event capabilities. Streaming and event capabilities are significant. These features are important due to scalability and resiliency.

In terms of auditing and compliance, Stream Lineage has helped by providing understanding of the entire data domain across the movement and what is happening in the domain and the system.

In terms of improvements, observability and monitoring are areas that could be enhanced. They are lacking in terms of observability and monitoring compared to other products.

I have dealt with Apache Kafka on Confluent Cloud for around nine years.

The initial setup is complex. It takes a month to deploy.

To evaluate Apache Kafka on Confluent Cloud's handling of event streams, I track scalability and resiliency.

I have dealt with the support directly. On a scale of one to 10 with Apache Kafka on Confluent Cloud support, I rate them as good. I would rate them eight if 10 was the best and one was the worst.

A reasonable amount of people deploys it. There is enough documentation available for addressing issues that we come across.

In regards to operational impacts and return on investment, returns depend on the application you deploy and the amount of benefits you are getting, which depends on how many applications you are deploying, what are the sorts of applications, and what are the requirements. If your application has a requirement of eventing, you would have returns to it, but I cannot articulate what sort of dollar value we're talking about.

That's just for all my applications. I rate Apache Kafka on Confluent Cloud a nine out of ten.

Positive

We use Apache Kafka on Confluent Cloud for streaming large volumes of data in real-time. It's employed in scenarios such as handling events from various countries and streaming them efficiently for our clients. 

We also utilize it for data analytics and in client versions for topic creation, consumer consumption, and ACL provisioning.

Apache Kafka on Confluent Cloud provides an enhanced level of reliability and resources compared to Apache Kafka alone. It offers more features which are beneficial for our clients, including cluster linking, schema registry, error handling, and dead-letter queues. It significantly improves customer and publisher satisfaction, especially with topic integration and data streaming.

Apache Kafka on Confluent Cloud is more reliable and frequent to use compared to Apache Kafka. Its features such as schema registry, cluster linking, error handling, and dead-letter queues provide significant benefits. It also offers enhanced visibility and integration for data streaming, helping clients and customers use it efficiently.

Some areas for improvement in Apache Kafka on Confluent Cloud include issues faced during migration with Kubernetes pods. This aspect could be smoother to better support migration processes.

I have been working with Apache Kafka on Confluent Cloud since September 2022 after joining my current company in June 2022.

The solution is stable and monitors activities, ensuring reliable operations. It provides alerts for unusual activities that allow us to take proactive actions.

Confluent Kafka's scalability is rated eight out of ten. It's capable of horizontal scalability by adding more consumers to handle high message throughput.

Technical support for Confluent Kafka is very good. Their efforts to provide timely solutions to bugs and defects have been excellent.

Positive

Previously, solutions such as Red Hat AMQ and Google's PubSub were considered, but Apache Kafka on Confluent Cloud was ultimately chosen.

The initial setup is straightforward with provided resources and documentation. I started working with it in the middle stages, not from the initial deployment.

Deployment can be done by two members based on requirements, and a single DevOps engineer can handle both deployment and maintenance.

I'm not sure about the pricing of Apache Kafka on Confluent Cloud.

I studied the PubSub and AMQ platforms yet did not have hands-on experience with them since Confluent Kafka was already implemented in my company.

I recommend new users start by going through the Confluence page and training to learn about Confluent Kafka's features and differences from Apache Kafka.

We find that the best features include using the CDC functionality with the connector to take the data from our SQL database and publish it to many consumers. Any changes enable us to easily publish changes about their domain business objects without too much code and work from domain teams. In this way, we can more easily provide a very robust layer of API and events.

The second use case is easier projection of data. We found that many teams were struggling to create projections and read stores with regular event buses, and Apache Kafka on Confluent Cloud helped us because of all sorts of features, such as the log architecture they have, and other features. KSQL also helped us there.

When order is more important, we rely on Apache Kafka on Confluent Cloud.

The benefits that I have seen from having a real-time architecture include better velocity for developers. That is the main one. Instead of developing many of those capabilities in each team, we can rely on Apache Kafka on Confluent Cloud to provide those functionalities we want, and the teams can focus on their own business instead of providing all sorts of APIs and dependencies to other domains, allowing everyone to run faster.

We find that the best features include using the CDC functionality with the connector to take the data from our SQL database and publish it to many consumers. Any changes enable us to easily publish changes about their domain business objects without too much code and work from domain teams. In this way, we can more easily provide a very robust layer of API and events.

The second use case is easier projection of data. We found that many teams were struggling to create projections and read stores with regular event buses, and Apache Kafka on Confluent Cloud helped us because of all sorts of features, such as the log architecture they have. KSQL also helped us there.

I think what I would improve about the solution is the cost, mostly. From my standpoint, it's the cost. From an engineering perspective, it works really well.

There's always room for improvement. One more point is sometimes it's more UI-related issues. Some of the more high-end features are more complicated to execute. But overall, it's a good product.

I have been using Apache Kafka on Confluent Cloud for around a year, maybe two.

When it comes to assessing the impact of the automated scaling features, we don't measure it, but it's part of our technology stack selection criteria - it's pretty much a must today.

We don't want to increase the headcount in our DBA team. They are the ones managing all our databases, queues, and data sources. So for us, having a very thin layer of management is critical, and we sit with other compute. That's very important for us because headcount is the most expensive part.

We looked at other products, specifically other Kafka providers. We have Apache Kafka and AWS. We looked at self-hosting it, but we wanted Apache Kafka on Confluent Cloud.

Neutral

We were looking for specific use cases. We compared different Kafka solutions, not necessarily competitors. We have a message bus already. We wanted the log capability, mostly.

The setup was easy enough. We got a lot of support from people at Confluent and AWS as well.

Regarding ROI in any capacity, whether it's savings from employees or cloud, the ROI was very significant. Although, specifically with Apache Kafka on Confluent Cloud, it was a bit more challenging to increase adoption because it's very expensive. So we had to pick and choose where we implemented to make sure that ROI is positive.

I don't remember the exact number because it's been a while since we did a pricing talk, but it was expensive.

They charge per topic and other resources. Because we are very cost sensitive, we want to approve it and make sure people don't just use it unnecessarily.

I would give Apache Kafka on Confluent Cloud a rating of seven out of ten.

For somebody who's shopping around, looking in this space to decide what to purchase, Apache Kafka on Confluent Cloud is a market leader. It's almost the first choice.

Going with AWS Apache was also very compelling to us because it's very quick to enable stuff in AWS and try it. I would start with those, but first understand if this is actually what you need. There are other much cheaper solutions that serve other use cases, and sometimes people can mix those and just pick the wrong product.

Overall, I would rate Apache Kafka on Confluent Cloud a nine out of ten.

In my company, we are not using the tool for analytics and it is more for CDC processes, so we change the capture processes. It is used to extract data from a database and make it available in other parts of our systems or produce events that inform us of data updates.

There are some premium connectors, for example, available in Confluent, which you cannot access in the marketplace, so there are some limitations. From Confluent's point of view, I understand where they come from, but I believe its deployment model is a little strict. Confluent Platform can be installed in your own infrastructure. Even if you install Confluent Platform on your own platform, you need to use the components that Confluent offers. Otherwise, the support is very limited, and I think this is an idea of improvement for Confluent. Confluent is pretty solid, so I don't have much in terms of improvement.

I have experience with Apache Kafka on Confluent Cloud.

Confluent gives you 99.99 percent availability, so I rate the tool's stability a nine out of ten.

If you use Confluent Cloud, the platform's case can go up and down depending on your needs, and it is very easy from the point of view of storage as well because if you are getting more advanced, it basically scales up your storage. If you are given a number of events using your storage device, it is very easy. If you use Confluent Platform, you have a little bit more manual management there, although being a product that assists you with some side components like CFK.

With Confluent, if you have its tools, I rate the support an eight out of ten, but if you have mixed products, then it is a six out of ten.

Neutral

I have experience with Confluent Cloud and Amazon MSK. With Confluent Cloud, we are really happy with the ecosystem that is made available, along with the connectors, SQL, DB, and other such aspects. The tool can be provided in a very easy way, and it was really effective for the type of activities that we do. The tool presents quite a range of possibilities for integration between different sources and things.

While you use Confluent, all of the services that are needed to manage the enterprise-level EDA are available to you, and you have an integrated schema registry, together with the entire schema registry, and you have a portal for publishing your schema. You can do routing and filtering by configuration. You have CFK, which allows management of your cluster, allows monitoring of your cluster, and allows you basically to connect to the managed connectors on your cluster. Confluent is a full-fledged platform for an event-driven architecture that can be deployed at an enterprise scale, while Amazon MSK is just Kafka as a service from AWS.

A part of the delivery team does the setup, but it was pretty easy on both sides, as with AWS and Confluent, the team didn't have much trouble.

The main return on investment was in the maintenance space because going for Confluent Cloud means you remove all the platform management that you have in terms of these resources that can be allocated to other tasks, where the tool takes basically ownership of all of these. We saw, at the end of the year's end, improvements that were substantial, especially when it comes to the need to resolve issues, as we can deploy the minimum team possible for Confluent because the support model allows for the Confluent team to take ownership of the issue. With AWS, the tool's team supports us, but we have to deploy the right people and take them out of all other initiatives. The most important part is the cost related to platform maintenance and issue resolution.

Using Confluent, you have more licensing prices to account for when you calculate. I think the pricing is fair, but Confluent requires a little bit more thinking because the price can go up really quickly when it comes to premium connectors.

Speaking about data security and privacy requirements, I would say that there are some BAA or legal agreements in the tool. We did not have issues in terms of security or breaches, but before any adoption, with PII or PHI type of data and before having this data flowing to other clouds or other platforms, the BAA needs to be signed because of IPAC.

Confluent Cloud handles data volume pretty well.

If you are starting to deploy a fully-fledged ETA platform where you do not just have information streaming and go for CDC, and you have some legacy systems that have to communicate on your systems, then I suggest you go for Confluent Cloud.

I rate the tool an eight out of ten.

I use the solution in my company mostly for log-related purposes.

The solution's most valuable feature is that we used it during the building of data pipelines. From that, we read, wrote, and managed all the processes if there was any event happening. So, in real-time, we are just tracking how many users or logs are coming into our company.

Maybe in terms of Apache Kafka's integration with other Microsoft tools, our company faced some challenges. With other products, specifically non-Apache ones or Microsoft, our company had created some wrapper classes around them.

The integration issues of Apache Kafka revolve around other products like Tomcat, other web servers, or Node.js and .NET Core-related areas. After creating some wrapper classes around them, we were able to get the information we wanted.

My company has not used any AI features in the product, but I would like to see AI features in the product because they currently help generate real-time content. AI can help our company slowly create some LLM. My company is not currently implementing any AI features, but I see that there will be a lot of opportunities to use such functionalities. I have experience with Databricks, for which I have used AI.

I have been using Apache Kafka on Confluent Cloud for more than a year.

The tool's reliability and scalability are good.

We did not have to ask for much help from the product's support team since we are still exploring the solution. My company mostly looks at forums or blogs, and so we are able to solve any issues associated with the tool. I rate the support an eight out of ten.

Positive

My company has previously used Redis, ActiveMQ, and RabbitMQ. Apache Kafka is better than RabbitMQ. The installation, configuration, and real-time streaming are good with Apache Kafka. Usually, the tool is configurable.

The product's installation phase is pretty straightforward for us since we know how to use it.

Initially, the solution was deployed in a few hours, but after that, we were able to install it quickly.

I consider that the product's price falls under the middle range category.

I have not exactly used the tool for data management, but I have used it for log aggregation.

The tool does not need to be maintained currently, but we are still exploring how we can make it perfect. My company's client is asking for a little bit more from the tool, so we will deploy it. Right now, we have our own products, and in those tools, we are just seeing how we can use Apache Kafka.

My company has not used any AI features in the product.

I rate the tool an eight out of ten.

Whenever you need to handle a huge load of real-time data processing, Kafka is useful. We currently use it for an output management system for insurance, where the system receives data in a fixed amount and has to process it in several steps. 

We manage these steps with Kafka because the load can be quite big, with millions of XMLs coming into the system that need to be processed in near real-time.

The most valuable feature to me is it scales really well when there is a data influx, irrespective of the peak time. It can handle both large and small amounts of data. 

Also, the state-saving feature is very much appreciated. It allows me to rewind a certain process if I see an error and then reprocess it.

The administration port could be more extensive. Additionally, managing the states of certain events could be made easier, perhaps with automatic rollback instead of having to program it manually.

I have been using it for a few months. 

For us, the stability has been great. No complaints here.

Scalability is great. We haven't gone into production yet, but so far, it scales very well.

I would rate the scalability a nine out of ten. There are maybe a dozen or so people who really have to interact with Kafka.

The setup was straightforward. Confluent Cloud takes a lot of the work out of the user's hands, and it's easy to set up. We're quite happy with it.

We didn't need any help. We have one developer, and that's it.

The benefit is that we can offer clients cloud streaming for data processing, which we host for them. They benefit from the speed and reliability.

Since we are in development, we have no license. We will need one eventually. The price isn't low, but it's not too high.

I evaluated other options for workflow solutions like Camunda, and other messaging services, like NQ series.

Overall, I would rate the product an eight out of ten. So far, we haven't found any obstacles, and we're happy that it's been straightforward and problem-free.

The product is very good, but if you need to set it up on your own, there's quite a bit of work. That's why we chose Confluent as a partner who has made it tailored for us. That's surely a good way to start. When you have more experience, you can try setting it up on your own.

We use Apache Kafka with Confluent Cloud for specific real-time transaction use cases, both on-premise and in the cloud. We have been using Confluent Cloud for about five years.

We initially used it for data reputation, then expanded to microservices integration and Kubernetes, focusing on improving data quality and enabling real-time location tracking.

We configure it for data transactions across various topics and partitions, depending on the specific use case and required throughput.

From an IT perspective, I've used this product across all domains: system development, operations, data management, and system quality.

We have experience using Kafka on Confluent Cloud for data pipelines. We've implemented several techniques to optimize topic usage, integrated it with microservices, and even utilized change data capture (CDC) components.

We leverage topic configurations and partitions extensively. We simulate various use cases with different partition numbers, like high throughput scenarios with 45 partitions or high transaction environments with other configurations.

In our microservices architecture running on Kubernetes, Confluent Cloud helps us manage transactions effectively. Additionally, it integrates seamlessly with our data analysis tools like DataStage, Big Data, and Teradata, providing a smooth flow for large data volumes.

The overall integration with other tools and efficient transaction management are the key benefits I experience with Confluent Cloud for large-scale data streams.

I saw an interesting improvement related to the analytics environment.

We have been using this solution since 2018.

We have a well-defined process and platform for scaling big data solutions. When multiple providers propose their options, we configure a custom platform based on our current use cases. 

However, we're planning to migrate to a new big data platform within the next fifteen months. This timeframe is due to our internal process for evaluating and deploying new platforms.

In terms of configuring the product, specifically Confluent, understanding the design and configuring values for various parameters is something only I am familiar with. The initial setup, including the initial Non-Disclosure Agreement (NDA) and progress in implementation, is quite difficult.

We primarily use on-premises Kafka for high-transaction scenarios. If something crashes there, we handle data processing manually. It might not be the most efficient, but we haven't considered it a major concern. 

For other use cases, we also prefer on-premises. 

The implementation took us one year. It involved configuring the platform over a year. The time required for configuring or implementing use cases varies; some take longer, while others might also take up to a year.

I attempted the deployment myself. However, there were three of us involved in these tasks within this analytical environment.

My role revolves around deploying use cases in analytics. I also operate within Architect areas, focusing on data architecture.

For maintenance, the same three people take care of it. We might need two more, but for now, three is sufficient.

The platform and container operations themselves provided significant value.