Amazon MSK's features that have proven most effective for handling large volumes of data stem from the fact that it acts as Kafka's underlying capability to support millions of rides per minute, and it is what helps us scale very easily. Amazon MSK's scalability impacts our data processing tasks, and one thing for sure is that it's a highly scalable system. Usually, we don't see any challenges because it supports millions of rides per minute with Kafka underlying. One challenge is if you want to scale it out by adding new nodes or brokers. People should finalize the number of partitions before they start using MSK and create a topic because changing partitions is quite challenging. I rate the tool an eight out of ten.
Senior Software Engineer at a recreational facilities/services company with 1,001-5,000 employees
Real User
Top 20
2024-06-12T19:48:00Z
Jun 12, 2024
I highly recommend AWS to any company or team. It offers fast delivery, easy deployment for new features, and a clear separation of concerns. The interface is user-friendly and requires minimal worry. I would rate it around an eight out of ten due to initial setup challenges that are overcome with familiarity.
Integration Solution Architect at a consultancy with 11-50 employees
Real User
Top 5
2024-05-24T18:34:00Z
May 24, 2024
With Amazon MSK, we don't use or handle large data volumes. From what we have seen in terms of our volumes, both Confluent Cloud and AWS are pretty much in line, and they both handle it pretty well. The AI strategy is finalized, but we have to do some PoCs, although we really don't have any deep AI usage. There are areas of the business that I don't touch, but AI is used for detection. In my area, I am just involved with the PoCs. AI for the business strategy has not yet been finalized. I would recommend Amazon MSK and Confluent Cloud, but I will make sure that I understand the use case. If you are not in need of connectors and are not in need of deploying a fully-fledged EDA platform, I would say go for Amazon MSK, as you can save some money. I rate the tool a seven out of ten.
Senior Data Engineer at a computer software company with 201-500 employees
Real User
Top 5
2024-05-09T20:45:00Z
May 9, 2024
Maintenance is pretty low. There's only one person needed for the initial setup. We wanted to build something on our own instead of using an already existing solution because it's quite expensive. If you want to build something like a custom data platform, MSK is very easy to use. It offers multiple products. For example, you can add multiple features like Glue, streaming, and more. So, it's very easy to plug in and use. I rate the solution an eight out of ten.
Software Team Lead at a tech services company with 201-500 employees
Real User
Top 20
2023-09-04T10:52:00Z
Sep 4, 2023
It is important to understand completely the technology stack for the solution to be utilized properly, but there is a lack of information availability. Overall, I would rate it seven out of ten.
Overall, I would rate Amazon MSK a seven out of ten. The only reason I am giving it seven is because I have not used all the aspects of this solution. Only thing I would suggest is to go through the documentation before starting anything. Amazon has already provided documentation on its site, which is highly informative. If there are still any issues or questions, users can directly raise a support ticket with the support center. In fact, they may not even need to raise a support ticket because the documentation already covers everything, including instructions on how to install agent collectors on Windows or Linux machines.
My advice would be to compare it with other details, too, like Data streaming services, and then see which is cheaper than the other based on the use case. Because if you need to pass maybe a hundred messages through a day, then Kinesis would be much cheaper than setting up a Kafka cluster. And the other thing is with the Kafka cluster; you need a minimum of three nodes for higher availability. So having to set up a cluster for just a hundred messages daily. That's overkill. And, it becomes expensive in that sense. I rate it eight out of ten.
Team Lead at a tech services company with 10,001+ employees
Real User
2021-03-16T03:58:00Z
Mar 16, 2021
If you're interested in using this solution, be sure to do some POC before integrating it. Overall, on a scale from one to ten, I would give this solution a rating of eight. Kafka is open-source software. In order for you to leverage it behind the scenes, you have to use ZooKeeper, which is essentially a type of server — you have to integrate with it. If you don't use MSK, then you'll have to do this manually, by yourself. MSK has already launched ZooKeeper and the cluster. This is included as an out-of-the-box feature, which is very convenient.
Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a fully managed service that enables you to build and run applications that use Apache Kafka to process streaming data. Amazon MSK provides the control-plane operations, such as those for creating, updating, and deleting clusters.
Amazon MSK's features that have proven most effective for handling large volumes of data stem from the fact that it acts as Kafka's underlying capability to support millions of rides per minute, and it is what helps us scale very easily. Amazon MSK's scalability impacts our data processing tasks, and one thing for sure is that it's a highly scalable system. Usually, we don't see any challenges because it supports millions of rides per minute with Kafka underlying. One challenge is if you want to scale it out by adding new nodes or brokers. People should finalize the number of partitions before they start using MSK and create a topic because changing partitions is quite challenging. I rate the tool an eight out of ten.
I highly recommend AWS to any company or team. It offers fast delivery, easy deployment for new features, and a clear separation of concerns. The interface is user-friendly and requires minimal worry. I would rate it around an eight out of ten due to initial setup challenges that are overcome with familiarity.
With Amazon MSK, we don't use or handle large data volumes. From what we have seen in terms of our volumes, both Confluent Cloud and AWS are pretty much in line, and they both handle it pretty well. The AI strategy is finalized, but we have to do some PoCs, although we really don't have any deep AI usage. There are areas of the business that I don't touch, but AI is used for detection. In my area, I am just involved with the PoCs. AI for the business strategy has not yet been finalized. I would recommend Amazon MSK and Confluent Cloud, but I will make sure that I understand the use case. If you are not in need of connectors and are not in need of deploying a fully-fledged EDA platform, I would say go for Amazon MSK, as you can save some money. I rate the tool a seven out of ten.
Maintenance is pretty low. There's only one person needed for the initial setup. We wanted to build something on our own instead of using an already existing solution because it's quite expensive. If you want to build something like a custom data platform, MSK is very easy to use. It offers multiple products. For example, you can add multiple features like Glue, streaming, and more. So, it's very easy to plug in and use. I rate the solution an eight out of ten.
I advise others to understand the exact business requirements before purchasing. I rate Amazon MSK a six out of ten.
It is important to understand completely the technology stack for the solution to be utilized properly, but there is a lack of information availability. Overall, I would rate it seven out of ten.
Overall, I would rate Amazon MSK a seven out of ten. The only reason I am giving it seven is because I have not used all the aspects of this solution. Only thing I would suggest is to go through the documentation before starting anything. Amazon has already provided documentation on its site, which is highly informative. If there are still any issues or questions, users can directly raise a support ticket with the support center. In fact, they may not even need to raise a support ticket because the documentation already covers everything, including instructions on how to install agent collectors on Windows or Linux machines.
My advice would be to compare it with other details, too, like Data streaming services, and then see which is cheaper than the other based on the use case. Because if you need to pass maybe a hundred messages through a day, then Kinesis would be much cheaper than setting up a Kafka cluster. And the other thing is with the Kafka cluster; you need a minimum of three nodes for higher availability. So having to set up a cluster for just a hundred messages daily. That's overkill. And, it becomes expensive in that sense. I rate it eight out of ten.
My advice to others is this solution is straightforward and the integration is flawless. I rate Amazon MSK a seven out of ten.
If you're interested in using this solution, be sure to do some POC before integrating it. Overall, on a scale from one to ten, I would give this solution a rating of eight. Kafka is open-source software. In order for you to leverage it behind the scenes, you have to use ZooKeeper, which is essentially a type of server — you have to integrate with it. If you don't use MSK, then you'll have to do this manually, by yourself. MSK has already launched ZooKeeper and the cluster. This is included as an out-of-the-box feature, which is very convenient.