Apache Airflow Reviews

I'm a data engineer. In the past, I used Airflow for building data pipelines and to populate data warehouses. With my current company, it's a data product or datasets that we sell to biopharma companies.

We are using those pipelines to generate those datasets.

I like the UI rework, it's much easier.

I use XCom for derived variables that need to pass between tasks. I don't really tend to use it for passing data, but only for a derived variable. For example, I don't have to re-query something every time, with one-task uses. I use the JSON comp for overwriting certain parameters.

In our use cases, some of the inputs of the dataset are files that we pulled out of S3. Sometimes they need to re-do those files, but we don't need to change any logic, we just need to redo the bills. Rather than redeploying the code to point to a new S3 bucket, we overwrite it to point to a different S3 key.

I have read that there are many different workflow pipelining tools in the biotech space, such as Snakemake and Nextflow.

There is also a CWL plugin that we may look into at some point. 

Eventually, we might have a use case where a researcher has a pipeline they run locally, and then we want to convert that to a DAG. 

The CWL-Airflow plugin would be useful for that. This might be something to look into later. But that would be like months, or maybe a year from now.

I am using a Celery Executor and I find that it crashes and I can't see any logs. I can only assume that it's a memory issue and have to blindly restart until eventually, it starts up again.

One of the use cases is triggered by input rather than a batch process. For example, we receive a batch of data, it goes through tasks one, two, and three, and a new batch comes in, each subsequent task should be operating on just that data from the prior task.

I am used to working on it as the output gets written to a table and then the next task selects all from that upstream table. It could be coded where you are only writing the data for that portion of the task. It could handle state machines and state changes as opposed to the batch proxy.

I would like to see it more friendly for other use cases.

In my current company, I just introduced it within the last couple of months. But I've used it at my prior two jobs as well.

We are using Version 2.0.1.

We are using the open-source version of Apache Airflow.

I usually create my own custom operators every time. We upgraded to 2.0, but I am not using any of the new features. 

I haven't yet used DAG of DAGs or the new way of using Python functions in the Python operator yet. But we might use DAG of DAGs eventually.

I Love this solution and I would rate it a nine out of ten.

Private Cloud

Amazon Web Services (AWS)

We use Apache Airflow to send our data to a third-party system.

We are already on Python. Since Apache works very well on Python, we can manage everything and create pipelines there.

Adding more automated components in Apache Airflow for basic things like exporting the data would be helpful. Apache Airflow is not that easy to use, but we have gotten used to it.

I have been using Apache Airflow for three years.

Apache Airflow is a stable solution.

Apache Airflow is not a scalable solution for our use cases. We have a very huge list of use cases. Over 10 developers use Apache Airflow in our organization.

Apache Airflow's technical support team is good and provides assistance almost 90% of the time.

Apache Airflow's initial setup is easy. It's not that difficult, but it has a learning curve.

Apache Airflow is a cheap solution.

Depending on your use case, if you are looking for a quick solution to work on and know Python, you should go ahead with Apache Airflow.

Apache Airflow is a good enough tool for managing data pipelines. However, the solution is not up to the mark as you scale up and go at the higher performance. Apache Airflow has introduced the DAG connector for managing data pipelines.

Overall, I rate Apache Airflow an eight out of ten.

Public Cloud

Amazon Web Services (AWS)

Apache Airflow's best feature is its flexibility.

Apache Airflow could be improved with the addition of more frameworks.

I've been using Apache Airflow for four years.

Apache Airflow is stable.

Apache Airflow is scalable.

The initial setup was very easy.

We used an in-house team.

Apache Airflow is open-source and free of charge.

I would rate Apache Airflow eight out of ten.

Public Cloud

Amazon Web Services (AWS)

The primary use case for this solution is to automate ETL process for datawarehouse.

The most valuable feature is the UI, for automation.One can monitor all ETL processes in single screen. Complex workflows are shown as DAGs SVG images.

This is a simple tool to automate using Python.

There are some drawbacks to this solution. The code does not cover all tasks in the data warehouse automation process.  Currently , in production, we have a large installation with a complex workflow that includes hundreds of tasks. Most of them are dispatched by existing engine, but not all.
For example, sometimes we need to create cycles in our workflow but we are not able to, because Airflow supports only Direct Acyclic Graphs ( DAGs )

We need to develop our workflow description and notations because out of the box, Apache Airflow does not provide some features that are needed. It is our understanding that it is limited by design.

We will wait for the latest 2.0 version, as it is awaited to be much more mature than the 1.8-1.10 version. We believe that it will be better.

There should be some improvement made to the Doc Management features from within the UI. They should think about Outlook integration, which should be out of the box, and the object model should be expanded to support cyclic graphs inside the workflow.

We have been using this solution for eighteen months.

This solution is not very stable. There are a number of configurations issues.

This solution is scalable. We use this solution in a single node, but it is possible to have a  cluster of workers.

It can be used for one or two thousand related tasks and should be done in a cluster configuration. 

We don't use a cluster, rather we only use single nodes. It is sufficient for our tasks. Tasks are long and the parallelism is limited by the database engine, and not by the workflow engine. 

We would like to evaluate clusters in the future.

We are using the Cron Task scheduling feature for Apache Airflow. Users can configure the Apache Airflow themselves. There are up to ten users that can configure Apache Airflow.

This is a part of the wage solution, and it is the initial point of the wage slot process. The wage solution has hundreds of users.

We don't use any paid technical support, as it is an open-source solution. We have used Stack Overflow and other open information sources, but we know that some companies provide technical support. 

As we have studied their solutions that are available on the internet, it is my understanding, that, we are on a pretty high level and could provide commercial support ourselves. 

We don't use any support from commercial companies, but some very useful recent solutions we could extract from Apache Airflow GitHub, as an example.  

Previously, we used Control-M for a short period. It was a solution used by our customers, and we needed to understand their difficulties and the results. 

For low to middle scaled tasks, Apache Airflow could be a substitute for Control-M. 

The deployment model we used was through a private cloud. It was a private installation on Google Cloud.

In-house team.

It 's measured jut now. Precise data is awaited in 3..4 months. First conclusion - positive ROI

There are no costs associated with this solution. Apache Airflow is a free solution that can be downloaded and ready for use at any moment.

Our tasks can be automated by simple Jenkins, but our customer wanted to implement it on Apache Airflow. This was a solution used by our customer.

Apache Airflow is mainstream and everyone wants to use it. Google provides Apache Airflow as part of the Cloud services.

My advice would be to use this solution for simple tasks. 

They should have a Python expert for features that are not available out of the box, as it is not enough. 

It could be a good solution for enterprise workflow automation and solutions like Control-M within the next two to three years.

We are happy and satisfied with this solution, but not fully satisfied, as this solution has some positive and negative aspects.

I would rate this solution a seven out of ten.

We use Apache Airflow for data orchestration.

Apache Airflow is a pretty useful tool for collecting information. Apache Airflow is a pretty easy solution that can be used with Python. The solution's UI allows me to collect all the information and see the code lines.

I have some issues with the solution's communication. The solution uses the same database or data set. Sometimes, we consume the same data and send it to a different place when doing a different DAG. When using the UI, I want to see that we use the same data set more than once.

I have been using Apache Airflow for five years.

I rate Apache Airflow a seven out of ten for stability.

I rate Apache Airflow an eight out of ten for scalability. Around 400 users are using the solution in our organization.

I previously used Control-M and some AWS and Google Cloud Platform tools.

Apache Airflow's initial setup is pretty straightforward. Apache Airflow is quite intuitive to set up and create DAGs.

It takes around two days to deploy Apache Airflow. A DAG can be created in just a few hours.

Apache Airflow is deployed on-cloud in our organization.

Overall, I rate Apache Airflow a nine out of ten.

We use this solution to monitor BD tasks.

The solution is quite configurable so it is easy to code within a configuration kind of environment. 

The ease of learning and using the solution is quite good. The learning curve is low so new users can learn in a short period of time in comparison to other products. 

The following should be improved:

• Dashboards

• Security

• Airflow web UI

• Telemetry for logging, monitoring, and alerting purposes

• Documentation 
  

I have used the solution for six months. 

The solution is 99% stable. We have a few glitches here and there but have been able to fix them. 

The solution is quite scalable. You can grow in terms of users and environment. You can grow to multi-server applications. You can use the solution on desktops, mobile, or other devices. 

We have an internal tech support team so have not needed support from the vendor. 

The setup is straightforward. The time for deployment depends on the environment and user base.

We implement the solution in-house. We have one implementation with 60 users and another with 75 users. 

We have a tech support team that consists of ten engineers who support implementations. They follow up on issues that might arise during the process automation or implementation of the workflow itself. 

For example, our tech support team will resolve a workflow that gets stuck during the MDM workflow engine. The tech team has the knowledge base to resolve any of these issues. 

The solution is open source.

I do not have exposure to use cases for large organizations with a huge user environment, so I cannot speak to the solution's effectiveness in these scenarios. 

I rate the solution an eight out of ten. 

I use this solution for scheduling purposes. We have our own Python framework to run jobs, do the extractions, and for transformation loading.

We have 20 people who are using Airflow. It's being used on a daily basis. We don't have any plans to increase usage because we have low data sets.

The solution is deployed on cloud. The cloud provider is Azure.

Everything is in the Python framework now. I would like to see some no-code capabilities and drag and drop abilities in Airflow.

We're expecting a few more improvements in the log generator. Currently, it's very clumsy.

I have used Apache Airflow for three years.

It's stable.

It's scalable. So far, we haven't needed more scalability because it's totally controlled by administrators.

The only difference between Apache Airflow and BPM software is the pricing.

Setup is about medium difficulty. You need to have some prior knowledge and experience with docker containers and AKS.

It's open-source.

I would rate this solution as seven out of ten.

Public Cloud

Microsoft Azure

Our primary use case for this solution is scheduling task rates. We capture the data from the SQL Server location and migrate it to the central data warehouse.

The best feature is the customization that can be done using Python. For example, there are use cases where we have to tweak the algorithm and with Apache Script Rate, we have extra functionality that helps to change the underlying process. We can define our algorithms and processes using Python.

The solution could be improved by simplifying the integration process and providing access to its support team to guide integration.

We have been using this solution for two months and it is deployed on-premises.

The solution is stable but primarily depends on the support team and how they manage it.

Apache Airflow is scalable. Approximately 20 people use this solution on my team.

We haven't had any experience with customer service and support.

Previously, we were using SQL server integration tools and integration service SSIS packages. We had project orders and wanted to migrate everything as it was an open source rate and no license was required. We switched to Apache Flow because we are trying to migrate all the projects developed in SSIS using Python.

The initial setup was straightforward. However, if a script is written, it takes four to five minutes to set up.

Apache Airflow is open source, so I cannot comment on licensing costs.

We chose this solution because it was suitable for our business needs.

I rate this solution a seven out of ten. My advice to new users is to have good proficiency with Python language. The solution is good but can be improved by simplifying its integration process.

On-premises