Apache Flink Reviews

Apache Flink's best feature is its data streaming tool.

Apache Flink's documentation should be available in more languages.

I've been using Apache Flink for four to five years.

Apache Flink is quite stable.

Apache Flink is scalable.

The initial setup was easy, and the cloud-based version takes no more than a minute to deploy.

I would recommend Apache Flink to other users and rate it seven out of ten.

We use it for batch processing, specifically for hiding certain data, including location information and other entity-specific attributes.

The significant advantage is the learning curve as it is easy. It's not associated with any specific cloud entity. It provides us the flexibility to deploy it on any cluster without being constrained by cloud-based limitations. It enables freedom for making optimizations, allowing for specific customizations as needed.

There is room for improvement in the initial setup process. I found myself spending a significant amount of time navigating through documentation to configure it. It would be beneficial to have streamlined commands, where the environment could be quickly initialized, including database setup, providing a more efficient and convenient starting point for projects. This would be particularly advantageous for development and experimentation, allowing more focus on feature testing rather than spending time on the setup process.

I have been working with it for one year.

In the tech industry, there's always room for enhancements. It's a dynamic environment where products evolve over time. I would rate it eight out of ten.

I would rate its scalability capabilities eight out of ten. Currently, a team of fifteen individuals is involved in everyday processes.

Regarding tech support, the challenge lies in the open-source realm where there's no inherent accountability. Modules are developed, and users often rely on forums for assistance. However, for more critical or specific problems, especially those related to compliance or complaints, a paid service is available with a dedicated support team. I would rate it seven out of ten.

Neutral

We initially utilized Databricks for batch processing, which also supports real-time capabilities. However, due to limited computing power usage, we transitioned to a more focused approach. We now employ smaller containers for handling about five to six attributes specific to each entity in real time which allows us to experiment with alerts, actions, and responses for various purposes.

The initial setup was complex.

Deployment takes a total of two days: one day for learning and another day for the actual deployment process. We have a small dedicated team that manages updates and handles the networking aspects of the development website. I would rate it eight out of ten.

It's an open source.

I would give it a rating of seven out of ten due to the need for improvement in command automation and perhaps some stability concerns. However, overall, I would still recommend it.

We utilize IoT devices to gather data for our clients. This data is analyzed to produce reports and insights, and we leverage machine learning and artificial intelligence models.

The product helps us to create both simple and complex data processing tasks. Over time, it has facilitated integration and navigation across multiple data sources tailored to each client's needs. We use Apache Flink to control our clients' installations. 

Apache Flink should improve its data capability and data migration. 

I have been using the product for five years. 

The solution is stable. 

Apache Flink is scalable. 

The product's deployment can be completed in minutes, and we have a special team. It is straightforward. We initiate our requirements within our secure software and utilize Jenkins and our pipelines to carry out the deployment process, whether for expanding services on the cloud or on-premise servers.

Apache Flink can be deployed in-house. 

I rate the product an eight out of ten. 

The last POC we did was for map-making. I work for a map-making company. India is one ADR and you have states within, you have districts within, and you have cities within. There are certain hierarchical areas. When you go to Google and when you search for a city within India, you would see the entire hierarchy. It's falls in India. 

We get third party sources, government sources, or we get it from different sources, if we can. We get the data, and this data is geometry. It's not a straightforward index. If we get raw geometry, we will get the entire map and the layout. 

We do geometry processing. Our POC was more of processing geometry in a distributed way. The exploration that I did was more about distributing this geometry and breaking this big geometry.

Flink moved on to becoming a standard technology for location platform. There's only one location platform available right now by an open location platform. That platform leverages Flink. Because Flink is the component for streaming screen property. It's an optic whose data extends within the organization. Anywhere we need low latency applications, we use Flink.

Apache Flink is meant for low latency applications. You can take one event opposite if you want to maintain a certain state. When another event comes and you want to associate those events together, in-memory state management is a key feature for this. 

Checkpointing was important since we have the consumption done by Kafka. There was a continuous pool of data coming in from cars and which was put into Kafka. This particular Apache Flink component came in and it started processing it. When there's a failure or something effective, checkpointing is very important. 

It also helped us in exactly one standard. Another valuable feature is API extraction, which is nicely done. Anyone can understand it. It's not very complex. Anyone can go through all the transformations and everything they have. It's easy to use that. It's a well-balanced extraction.

In terms of improvement, there should be better reporting. You can integrate with reporting solutions but Flink doesn't offer it themselves. 

They're more about the processing side. Low latency processing is out of their scope. As ar as low latency is concerned, you can integrate to other backend solutions as well. They have that flexibility. APIs are good enough. Its in-memory is so fast, you could have faster-developed data and stuff like that.

The stability was good enough. There are a few issues that were application dependent. From the processing standpoint, it did what it was expected to do as such, there were a few issues with Python integration like checkpointing. The checkpointing was not done properly at times, but again that was more about integration going faster and also optimizing our checkpointing intervals and stuff like that. As Flink is concerned, they have good checkpointing and safe points.

There were 50 developers and DevOps working on it.

I was on the DevOps side. Support was all driven from Chicago. There was a different team in Chicago who was driving all this stuff. I was a completely hands-on developer. My interaction was more from using the API and developing applications. I don't need to use support. Flink was straightforward. 

The deployment can be declared on any kind of distributed managers. I haven't used it, but that's a good option that you could even integrate it within APIs. This adds flexibility to it. 

I was not part of the deployment when it was initially done. When I came into the picture, it was more about the API. We had already started using it at the application level at the organization. initially, when it was done in my previous organization, it was an earlier version of Flink. I think they started off in 2016 and there might have been some glitches or some technical issues. When I came in it was pretty smooth. I didn't find any issues and really I hopped into Flink.

We also looked at Spark Streaming versus Apache. Spark Streaming is not real-time. That's where we understood that Flink is good enough when you want to have real-time Processing. That's the only process that we have right now and Spark Streaming is more of a big data set.

If you want real-deal real-time processing, you have to invest in Flink but part of it is more of when you use Flink, you have everything stored. You store the state also in-memory so you add up the cost of using that engine compared to PAC streaming. It's not mandatory it's up to the application but if you want to really have real-time processing if you want to store the state and if you really want to have a low latency application, that's when I would go with Flink. Whereas Spark streaming would be more of whenever it's okay to have a bit of a delay like really low latency applications.

Flink gives you flexibility. The reason we chose Spark was because people in our company were already familiar with it. We haven't started working on it yet because it's a half-done POC.

Flink is really simple and simple to adopt. You can use any backend state management tools, like DB or something of that sort. it has the visibility to integrate with different technologies, that's also very important. It's pretty welded and I believe for low latency. The API is pretty well written that way to support you. 

I would rate Apache Flink an eight out of ten. 

My company is a cab aggregator, similar to Uber in terms of scale as well. Just like Uber, we have two sources of real-time events. One is the customer mobile app, one is the driver mobile app. We get a lot of events from both of these sources, and there are a lot of things which you have to process in real-time and that is our primary use case of Flink. It includes things like surge pricing, where you might have a lot of people wanting to book a cab so the price increases and if there are fewer people, the price drops. All that needs to be done quickly and precisely. We also need to process events from drivers' mobile phones and calculate distances. It all requires a lot of data to be processed very quickly and in real-time.

The end-to-end latency was drastically reduced, and our capability of handling high throughput has increased by using Flink. It provides a lot of functionality with its windows and maps and that gives us a lot of extra features and power that other frameworks don't have. The solution has helped us by enabling a lot of creative features so we are now able to detect if something abnormal is happening, like a driver has deviated from the set route or the car has not moved for a long period of time, all in real time. Being able to check this has led to more secure rides for our customers. 

The most valuable feature of Apache Flink would be that it is truly real-time. Unlike Spark and other technologies, it's not recurring batch processing and it also allows me better control over resources. For example, in Spark, it's very difficult to create multiple parallel streams and it consumes the memory of your entire cluster very greedily. With Flink, I have very good control, can choose the number of task managers with a fixed amount of memory, and configured parallelism. This flexibility is very useful in scaling of Flink.

Flink has become a lot more stable but the machine learning library is still not very flexible. There are some models which are not able to plug and play. In order to use some of the libraries and models, I need to have a Python library because there might be some pre-processing or post-processing requirements, or to even parse and use the models. The lack of Python support is something they can maybe work on in the future. 

I've been using this solution for two years. 

The solution has become a lot more stable over time. We have around 10-12 users and most are software developers. Even if we are running our task managers on cheap servers, we make sure that our job manager definitely runs on a very expensive server, which never goes down. Things remain more stable that way. We're a large company and have teams dedicated to dealing with the infrastructure and taking care of the maintenance and infra, making sure that jobs runs smoothly. A small company could do the maintenance itself. 

It's a good product and allows me to scale very easily. If I'm getting more and more data, I can very easily increase the memory allocated for every task manager, and increase the number of parallelism. We are increasing usage of Flink as much as possible. There are some things that we still run on Spark, but whenever we need to scale, have easy resource management, and a more real-time streaming solution, we usually now go for Flink. We have never faced any scalability issues and we are running at a very high profile. I think we are yet to reach the scaling limits of Flink.

We have never used Apache's tech support. We usually just Google for our questions. If we don't get the answers directly from Google, we go through the documentation, which is comprehensive, and usually find our answers there. 

We previously used Spark for streaming, but not for real time applications. We have moved some of our services from Spark to Flink. We also use Kafka extensively, but that is mostly for asynchronous communication between different services. Kafka is a totally different use case. You cannot substitute it with Flink. Overall in terms of streaming, we have used Spark, Kafka and Flink.

The initial setup was not very straightforward because compared to other frameworks, Flink is quite new. There isn't yet a good online community, online blogs, or guidance. You have to rely more or less on their documentation for everything. Even if you go to Stack Overflow, for Spark, you will get lots of questions and answers which will help you. With Flink, you have to actually read a lot. It's not as straightforward as other frameworks.

We have only used the open-source version of Flink. 

My advice would be to make sure you understand your requirements, flink's architecture, how it works and whether it is the right solution for you. They provide very good documentation which is useful. The solution isn't suitable for every case and it may be that Spark or some other framework is more suitable. If you are a major company that cannot afford any downtime, and given that Flink is a relatively new technology, it might be worthwhile investing in the monitoring. That would include writing scripts for monitoring and making sure that the throughput of the applications is always steady. Make sure your monitoring and your SOPs around monitoring, are in place.

I would rate this solution a seven out of 10. 

Services that need real-time and fast updates as well as lot of data to process, flink is the way to go. Apache Flink with kubernetes is a good combination. Lots of data transformation grouping, keying, state mangements are some of the features of Flink. My use case is to provide faster and latest data as soon as possible in real time.

The main advantage is the turnaround time, which has been reduced drastically because of Apache Flink. Earlier, it used to take a lot of processing time but now things have changed, and everything is in almost real time. We get the latest data in a very less time. There is no waiting or lag of data in the application, time has been one of the important factors. 

The other factor is memory. The utilization of the machine has been more efficient since we started to use this solution. The big data applications definitely use a large group of machines to process the data. These machines are not optimally utilized. Some of the machines might not have been required, but they still take hold of the resources. In Kubernetes, we can provide resources, and in Apache Flink, there is a configuration where you can do the deployment in combination with a single cluster node. Scalability is quite flexible in flink with task managers and resource configuration.

MapFunction, FilterFunction are the most useful or the most used function in Flink. Data transformation becomes easy for example, Applications that store information about people and when they want to retrieve those person's details in some kind of relation, in the map function, we can actually filter all persons based on their age group. That's why the mapping function is very useful. This could be helpful in analytics to target specific news to specific age group.

TimeWindow feature. The timing of the content and the windowing is a bit changed in 1.11. They have introduced watermarks.

Watermark is basically associating data in the stream with a timestamp. Documentation can be referred. They have updated rest of the documentaion but not the testing documentation. Therefore, We have to manually try and understand few concepts.  

Integration of Apache Flink with other metric services or failure handling data tools needs some kind of update or its in-depth knowledge is expected before integrating. Consider a use case where you want to actually analyze or get analytics about how much data you have processed and how many failed? Prometheus is one of the common metric tools out of the box supported by flink, along with other metric services. The documentation is straight forward. There is a learning curve with metric services, which can consume a lot of time, if not well versed with those tools.

Failure handling basic documentation is provided by flink, like restart on task failure, fixed delay restart...etc.

I have been using Apache Flink for almost nine months.

The uptime of our services has increased, resources are better utilized, restarts is automated on failures, alerts are triggered when infrastructure breaches threshold and application failure metrics are logged, due to which the application has become robust, scaling is something which can be tweaked as per usage of application, business rules. Also Flink parameters are configurable, altogether it made our application more stable and maintainable. 

I haven't actually hit a lot of performance or stress test to see the scaling.
There is no detailed documentation for scaling. There is no fixed solution as well, it depends on use cases, there are rest APIs to scale your task dynamically in Flink Documentation, haven't personally tried it yet.

I haven't actually used their technical support yet.

I have tried batch processing, It was not that much effective in my use case. It was time consuming and not real time solution.

The initial setup is straightforward. A new joined person (with some experience in software industry) would not find it that much difficult to understand and then contribute to the application. When you want to start writing the code for it, then things get tricky and more complex because if you are not involved with what Apache Flink is providing and what you need to do, it is very difficult. If followed the documentation, there are various examples provided and different deployment strategies as well.

It is good solution for time and cost saving.

Being open source licensed, cost is not a factor. The community is strong to support.

To get your hands wet on streaming or big data processing applications, to understand the basic concepts of big data processing and how complex analytics or complications can be made simple. For eg: If you want to analyze tweets or patterns, its a simple use case where you just use flink-twitter-connector and provide that as your input source to Flink. The stream of random tweets keeps on coming and then you can apply your own grouping, keying, filtering logic to understand their concepts.

An important thing I learned while using flink, is basic concepts of windowing, transformation, Data Stream API should be clear, or atleast be aware of what is going to be used in your application, or else you might end up increasing the time rather than decreasing. You should also understand your data, process, pipeline, flow, Is flink the right candidate for your architecture or an over kill? 
It is flexible and powerful is all I can say.

I use the solution for detection of streaming data.

I am using the Python API and I have found the solution to be underdeveloped compared to others. There needs to be better integration with notebooks to allow for more practical development. Additionally, there are no managed services. For example, on Azure, you would have to set everything up yourself.

In a future release, they could improve on making the error descriptions more clear.

I have been using the solution for two weeks.

I have used many different competing solutions in the past such as Pyspark, Hadoop and Storm.

The setup was not too difficult. 

To deploy the solution locally it was relatively easy.

The solution is open-source, which is free.

When choosing this solution you have to look at your use case to see if this is the best choice for you. If you need to have super-fast realtime streaming, and you can develop in Scala, then it might make a lot of sense to use it. If you are looking at delays of seconds, and you are working on Python, then Pyspark might be a better solution.

I rate Apache Flink a six out of ten.

We use Apache Flink in-house to develop the Tectonic platform.

It's usable and affordable.

It is user-friendly and the reporting is good.

There is a learning curve. It takes time to learn.

The initial setup is complex, it could be simplified.

I have been using Apache Flink for more than one year.

I am using the latest version.

Apache Flink is a scalable product. We have no issues with the stability.

It's a very scalable solution. We have more than 100 people in our organization who are using it.

We use community resources. There is a lot of documentation available online.

The initial setup is complex.

It's an open-source solution.

We have not evaluated competitors. We followed the trends and based on the experience and opinions of people from all over the world, we selected Apache Flink.

I would recommend Apache Flink to others who are interested in using it.

I would rate this solution an eight out of ten.