Horizontal scale-out is actually not easy, making it an area where improvements are required. There is a limitation on the size of the message that you can put and it needs to be increased.
Senior Software Engineer at a recreational facilities/services company with 1,001-5,000 employees
Real User
Top 20
2024-06-12T19:48:00Z
Jun 12, 2024
In my opinion, there are areas in Amazon MSK that could be improved, particularly in terms of configuration. Initially setting it up and getting it connected was quite challenging. The naming conventions for policies were updated by AWS, and some were undocumented, leading to confusion with outdated materials. It took us weeks of trial and error before discovering new methods through hidden tutorials and official documentation.
Integration Solution Architect at a consultancy with 11-50 employees
Real User
Top 5
2024-05-24T18:34:00Z
May 24, 2024
From AWS, I would consider more MSK schema validation is needed. It is easy to integrate if you have an application, but on-topic integration is more complex. You can do it with EvenBridge very easily, but not with MSK. One of the reasons why we prefer Kafka is because the support is a little bit difficult to manage with Amazon MSK. You have Azure and AWS there, but if you use a connector from the market, the support model is marketplace-based, which means you have to go to the developer of this connector.
Senior Data Engineer at a computer software company with 201-500 employees
Real User
Top 5
2024-05-09T20:45:00Z
May 9, 2024
We don't have many use cases involving ingesting large amounts of data and scaling up and down. We have a clear understanding of our data volume, which remains relatively constant throughout the week. While we're aware of other features Amazon MSK offers, we feel confident in our current setup. If our requirements change significantly in the future, we'll reassess our needs and consider adopting Amazon MSK.
The product's schema support needs enhancement. It will help enhance integration with many kinds of languages of programming languages, especially for environments using languages like .NET. They should build schema access as efficiently as developed by Confluent Cloud.
Using Exabeam has provided one valuable aspect, which is the availability of separate agents for Windows and Linux. However, there is a difference when it comes to Linux. On Linux, we need to perform certain command line operations, whereas, for Windows, we have the convenience of an MSI package. One thing I personally feel is that it would be beneficial to have command line installation for Windows as well. Additionally, we have to install the package one by one on each server. It would be really helpful if Amazon MSK could provide a single installation that covers all the servers. If there are multiple servers, I would otherwise need to perform individual installations on each one. All of my servers are either VDI or virtual machines, so I have to log in to each virtual machine, copy the necessary files, and then initiate the installation. In future releases, I would like to see if there a probability of having a one-click installation that can be applied to all servers.
There's a bit of operational overhead, but it's not a transmission. It's something you can live with because the setting out of the Kafka cluster is less managed compared to Kinesis. Once it's set up, everything goes smoothly without going through that operational overhead again. So maybe that could be some disadvantage, but it's not a transmission related to what you're getting from Kafka. Operational overhead for setting up the Kafka cluster compared to the Kinesis data stream is easier. For Kinesis, you have to do a few clicks and your data stream is ready if I need to do some config to set up the transfer. The solution does not auto-scale. The pricing can be improved.
Team Lead at a tech services company with 10,001+ employees
Real User
2021-03-16T03:58:00Z
Mar 16, 2021
MSK should be easier to integrate with other solutions. It should be more flexible, integration-wise. MSK has a private network that's an out-of-the-box feature. For our case, we needed something public. We can't handle private-only. So, in order to make the data public, we had to do some extra work. It became too risky for us. That's the main reason we stopped using it.
Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a fully managed service that enables you to build and run applications that use Apache Kafka to process streaming data. Amazon MSK provides the control-plane operations, such as those for creating, updating, and deleting clusters.
Horizontal scale-out is actually not easy, making it an area where improvements are required. There is a limitation on the size of the message that you can put and it needs to be increased.
In my opinion, there are areas in Amazon MSK that could be improved, particularly in terms of configuration. Initially setting it up and getting it connected was quite challenging. The naming conventions for policies were updated by AWS, and some were undocumented, leading to confusion with outdated materials. It took us weeks of trial and error before discovering new methods through hidden tutorials and official documentation.
From AWS, I would consider more MSK schema validation is needed. It is easy to integrate if you have an application, but on-topic integration is more complex. You can do it with EvenBridge very easily, but not with MSK. One of the reasons why we prefer Kafka is because the support is a little bit difficult to manage with Amazon MSK. You have Azure and AWS there, but if you use a connector from the market, the support model is marketplace-based, which means you have to go to the developer of this connector.
We don't have many use cases involving ingesting large amounts of data and scaling up and down. We have a clear understanding of our data volume, which remains relatively constant throughout the week. While we're aware of other features Amazon MSK offers, we feel confident in our current setup. If our requirements change significantly in the future, we'll reassess our needs and consider adopting Amazon MSK.
The product's schema support needs enhancement. It will help enhance integration with many kinds of languages of programming languages, especially for environments using languages like .NET. They should build schema access as efficiently as developed by Confluent Cloud.
The configuration seems a little complex and the documentation on the product is not available.
Using Exabeam has provided one valuable aspect, which is the availability of separate agents for Windows and Linux. However, there is a difference when it comes to Linux. On Linux, we need to perform certain command line operations, whereas, for Windows, we have the convenience of an MSI package. One thing I personally feel is that it would be beneficial to have command line installation for Windows as well. Additionally, we have to install the package one by one on each server. It would be really helpful if Amazon MSK could provide a single installation that covers all the servers. If there are multiple servers, I would otherwise need to perform individual installations on each one. All of my servers are either VDI or virtual machines, so I have to log in to each virtual machine, copy the necessary files, and then initiate the installation. In future releases, I would like to see if there a probability of having a one-click installation that can be applied to all servers.
There's a bit of operational overhead, but it's not a transmission. It's something you can live with because the setting out of the Kafka cluster is less managed compared to Kinesis. Once it's set up, everything goes smoothly without going through that operational overhead again. So maybe that could be some disadvantage, but it's not a transmission related to what you're getting from Kafka. Operational overhead for setting up the Kafka cluster compared to the Kinesis data stream is easier. For Kinesis, you have to do a few clicks and your data stream is ready if I need to do some config to set up the transfer. The solution does not auto-scale. The pricing can be improved.
Amazon MSK could improve on the features they offer. They are still lagging behind Confluence.
MSK should be easier to integrate with other solutions. It should be more flexible, integration-wise. MSK has a private network that's an out-of-the-box feature. For our case, we needed something public. We can't handle private-only. So, in order to make the data public, we had to do some extra work. It became too risky for us. That's the main reason we stopped using it.