Apache Airflow Reviews

We are a technology, media, and entertainment-technology company. We are using Apache Airflow for architecting our media workflows. We are using it for two major workflows.

We have had it set up for some time on our own cloud. Recently, we migrated the setup to AWS.

Airflow is our first choice because we wanted a clear separation of our functional logic from our operational logic. We don't want our microservices to have the cross-cutting responsibilities of our operational logic. Right now, our microservices are the core business' inner functional logic. The majority of our distribution, our decision making, and the majority of our workflow operational responsibilities have been added to Airflow.

The reason we went with Airflow is its DAG presentation, that shows the relationships among everything. It's more of a configuration-driven workflow. 

It's all Python, as well. The majority of the configuration is Python-friendly.

One specific feature that is missing from Airflow is that the steps of your workflow are not pipelined, meaning the stageless steps of any workflow. Not every workflow can be implemented within Airflow. For example, Step 1 of my workflow will have output which I definitely want to automatically be provided as an input to my Step 2. At the workflow level, we want to have common state management where, across steps, we'll be able to reach the state information. Right now, we're using an external state repository to maintain the state.

If Airflow could come up with some kind of implementation, where not every step of the pipeline is an independent step, that would be helpful. I would like it if a part of the output of your previous steps could be Apache input for your next step. That kind of pipeline is missing. When we consider other products like jBPM, Camunda, or Cadence, they have the concept of pipelining.

I would also like to see support for more platforms, in terms of programming BPMs. Cadence supports Golang and Java. Legacy components can be from any platform, so if they could provide more client support for Java client library and Golang, that would be helpful. I want it to program in Java.

I have been using Apache Airflow for more than a year.

It's definitely scalable.

We have been using Airflow for sometime but we are not heavily dependent on it. We only have a couple of use cases being executed by Airflow. 

Because we have some data engineering problems, we have a good amount of analytics systems. We have a high volume of data that comes into our system, along with a lot of email, and we have to have an automated data pipeline. Given that, we have all these computing capabilities that are built of microservices. The beauty of it is its scalability. It has every step of your workflow, and it has scheduler capabilities. Every step of your workflow is delegated to one of your nodes. That is being scaled per your computing needs.

We are still evolving. Our business processes are not completely automatic. We're still in the process of identifying what all the automation cases are that we can bring under Airflow. We would like to leverage one common orchestrator or workflow BPM for our complete ecosystem. So we have some architects in our system who are happy with Airflow and others who would like to migrate to some other BPM like Cadence or Apache NiFi. There are a lot of orchestrators and we're just out of the gate. Airflow is still not being heavily used in our enterprise.

This is the first workflow BPM tool that we are using in our platforms.

There is comprehensive documentation for setting up a simple workflow and you just follow the documentation for setting things up. We're all engineers so we don't mind if the steps are lengthy, in terms of setting up the system. I'm quite okay with the documentation provided for getting your system up and running.

But I would appreciate it if they published a portal where we could see in what way other businesses, or other technology companies are solving their problems, with some case studies, using Airflow. It would help us to review their case studies. My biggest problem at the time when I was deciding whether Airflow fit our needs or not, was that I was looking for some case studies of technology companies that are already using the solution. With Camunda and jBPM, there is a good quantity of case studies available online.

There is no scarcity of BPMs. There are many products online: either open-source or community products or licensed products. There are many good BPMs. The reason that Airflow is in my system is that some of our workflows which we have onboarded are also on Python. Airflow complements that. But the first and foremost ability of any orchestrator should be to integrate with any underlying platform, be it a Java platform or a Python platform. That's the beauty of an orchestrator.

We have a team of people, four to five team members, who initially evaluated Airflow and  wanted to implement it.

We have customers onboarded on our legacy systems. I cannot disrupt the service and bring everything into Airflow. I have to onboard Airflow seamlessly, while I protect my current, ongoing business systems. So I'm trying to balance things here. We have only been able to onboard a couple of workflows. Eventually, we want to do it more fully, but there were a few challenges as I told you: There is no pipeline to take information, which is forcing me to retain my state in a separate state repository. That would be the next big area where I would like to see improvement.

We normally use the solution for creating a specific flow for data transformation. We have several pipelines that we use and due to the fact that they're pretty well-defined, we use it in conjunction with other tools that do the mediation portion. With Airflow, we do the processing of such data.

The product integrates well with other pipelines and solutions.

The ease of building different processes is very valuable to us. The difference between Kafka and Airflow, is that it's better for dealing with the specific flows that we want to do some transformation. It's very easy to create flows. 

The graphics in the past have not been ideal.

We have several areas where we feel they could improve in terms of being a little bit more flexible. One is implementation. Even though we customized it, there were some specific things we had to do with the image by itself.

The management integration was challenging as well. It requires a lot of work on our end. We were creating our own way to integrate things specifically with specific tools. There's not really an ease of management out-of-the-box option for integration. We needed to become a little bit creative to solve that ourselves.

The scalability of the solution itself is not as we expected. Being on the cloud, it should be easy to scale, however, it's not.

There is no SDC versioning. There's no virtual control for pipelines. We have to build several pipelines for several flows, yet there's not a virtual control to generate them.

There's no Python SDK. We need to generate our own scripts and upload them and put them there. However, there's not a realistic case that we can get connected to them. On top of that, the API sets that are provided are very limited. They are not as rich as others. You cannot do much with them.

I've been using the solution for maybe three years at this point. It hasn't been too long.

The solution is largely stable. Obviously when you start creating more use cases, then you realize the limitations, however, it's not really, really bad.

Due to the fact that the solution is on the cloud, we thought it would be fairly easy to scale. This is proving not to be the case and scalability is limited.

The challenging part is to make it really flexible in a cloud-native environment. With other applications, what you have there is the scalability that can be sensitive to your needs, based on the amount of data you are putting into the flow.

Instead of you having to create your own logic to scale it up, it should be a little more efficient on how it gets integrated into the whole environment. You have to get a little bit creative and put some commands and some logic in there and be monitoring everything. You build everything - versus other options that are more out of the box. With other solutions, if you have these bursts of data they ultimately can scale up and they are more native.

Technical support has been pretty good. We don't really have anything to complain about. We're satisfied with the service so far.

For this particular category, due to the fact that we're testing all the other tools and they were too much of what we needed and due to the fact that we have used other products in other projects, and nothing really worked for us. Airflow, being a bit different, we decided that it was a nice player and a good open-source tool. 

We do use other tools. However, this one seems to work quite well for us.

The initial setup isn't as straightforward as we hoped. It's not as flexible as other options. You need to be a bit creative during the process.

This product is open-source.

We're just customers and end-users. We don't have a special business relationship with Apache.

I'm not sure of which version of the solution we're using. It's likely the most up-to-date, or at the very most back two or three versions as we are not using any of the older versions.

I'd advise others considering the solution to first understand what exactly you're trying to achieve. You either select a non-cloud native Apache workflow manager or select something that is way too big for what you are actually trying to achieve. Understand what is exactly what you need and the volumes that you need, and what exactly are the use cases.

After that, in terms of deployment, that depends on what you exactly are trying to do. If all of your solutions are cloud-native, try to do it with a cloud-native tools solution. Specifically, go to the CMCS site and look into the solutions that there. Those have been tested at least for the cloud-native solutions that exist.

Then, just make sure that the components you have will match and will be available to whatever you're trying to build. For example, the user management is something that is important for us and for this specific setup. Probably for some others, it's not going to be. 

Take into consideration, what are the different connection points and make sure that they are either supported or that you can support the integration of such items. You need to have a proper developer that can help you build your connector or your API.

In general, I would rate the solution at a seven out of ten. If they fix the APIs and the price on LTK, I'd rate it closer to a nine.

Our primary use case for the solution is setting up workflows and processes applied everywhere because most industries are based on workflows and processes. We've deployed it for all kinds of workflows within the organization.

The ability to easily set up and deploy workflows with Airflows is valuable. Additionally, designing processes and workflows is easier, and it assists in coordinating all of the different processes.

The solution can be improved by creating a tool that allows us to do these kinds of things graphically instead of just writing scripts. Hence, the graphical user interface can be improved.

We have been using the solution for approximately one year and are currently using the latest version.

The solution is stable.

The solution is scalable. Approximately hundreds of thousands of people are utilizing it.

We have not had any issues that require customer service and support.

The initial setup is intermediate, and two people are required for deployment.

There is a return on investment because it's free, open source and very useful, so there is a significant return on investment.

I rate the solution an eight out of ten.

Our primary use case is to integrate with SLAs.

The most valuable feature is the workflow.

Technical support is an area that needs improvement. The contact numbers should be readily available so that we can call to get support as required.

In the future, I would like to see a single-click installation.

We have been working with Apache Airflow for approximately one month.

In our company, we are doing a POC and there are only three users. We have also implemented it for clients.

We do plan to increase our usage and the POC that we are now working on is something that we will implement for other clients if it works.

We are not satisfied with technical support. We rely on using Google to identify solutions for the problems we have.

We did not use another similar solution prior to Airflow.

The initial setup was straightforward and it does not take long to complete. The deployment took no more than an hour.

We evaluated Control-M and another similar product from IBM.

This is a good product and I definitely recommend it.

I would rate this solution a seven out of ten.

We mainly used the solution in banking, finance, and insurance. We are looking for some opportunities in production companies, but this is only at the very early stages.

I do not have specific feedback because it is quite early in the review stage for comment.

The dashboard is connected into the BPM flow that could be improved.

I have been using the solution for half a year.

We have been quite satisfied with the stability of the solution.

The scalability of the solution is good.

We had no issue with technical support.

The installation is straightforward.

The pricing for the product is reasonable.

We are evaluating Camunda as well as this solution. We are investigating and trying to determine how suitable they are for production facilities. Additionally, we are seeing where the solutions are actually suitable in what type of processes.

We are unsure of which solution we will end up with, we are testing them currently. We are trying to get into new business types and new industries. We are looking into how well the solutions can be used in production facilities.

I rate Apache Airflow an eight out of ten.