Apache Airflow Reviews

Apache Airflow is utilized for automating data engineering tasks. When creating a sequence of tasks, Airflow can assist in automating them.

The most valuable feature of Apache Airflow is creating and scheduling jobs. Additionally, the reattempt at failed jobs is useful.

We have faced scenarios where Apache Airflow becomes non-responsive, leading to job failures. To resolve such situations, we had to manually reboot Apache Airflow since it doesn't provide an option to restart within the application. This necessitated modifying some configurations to initiate a restart of all Apache Airflow components. Although Apache Airflow is generally dependable, it may occasionally encounter glitches that can disrupt production flows and batches.

I have been using Apache Airflow for approximately three years.

We experienced some glitches using the solution with some errors.

I rate the stability of Apache Airflow a five out of ten.

Apache Airflow is scalable because it is within Amazon AWS.

I rate the scalability of Apache Airflow an eight out of ten.

The technical support is good, they are able to debug issues.

The initial setup of Apache Airflow was simple because it was all managed by Amazon AWS. The process took a few minutes.

The solution is free if you use Amazon AWS.

I would recommend this solution for projects even though there have been glitches. Once the solution has become stable it would be ideal for critical projects.

I rate Apache Airflow a seven out of ten.

This is great software to build data pipelines. However, we had many glitches that were causing some problems in production.

I have used Apache Airflow for various purposes, such as orchestrating Spark jobs, EMR clusters, Glue jobs, and submitting jobs within the DCP data flow on Azure Databricks including ad hoc queries. For instance, if there's a need to execute queries on Redshift or other databases.

If you are working with APIs or databases, you must write SQL queries and formulate the right statements to retrieve everything. But with the UI, it's more like plug-and-play. You go there, select the task you want to see, like logs, and click on it. It will promptly display the details of the logs, automatically showing the returned logs. However, if you're accessing logs manually from the web server, you must write commands and perform additional tasks. These overheads can be efficiently managed using the UI.

Apache Airflow is easy to use and can monitor task execution easily. For instance, when performing setup tasks, you can conveniently view the logs without delving into the job details. All logs are readily accessible within the interface itself. Examining the logs lets you discern which steps and processes are being executed.

You don't have to configure SMTP for everything. You need to configure email settings, such as email on error, failure, or alert access. With Apache Airflow, you can send emails with just a few lines of code. You don't have to write extensive code to configure SMTP; all those configurations can be accomplished within a few lines of code.

I managed a complex workflow for a finance application project. They use Apache Airflow to orchestrate processes, such as retrieving data from SFTP and landing it into S3. From S3, they trigger Glue jobs based on certain conditions. Additionally, they use the Glue catalog in Glusoft for data management, all orchestrated using Airflow. Furthermore, various logics are written in Airflow DAGs to handle scenarios like security mismatches. For instance, files are sent accordingly if there's a missing security.

Apache Airflow triggers a set of tasks based on DAGs. If you have multiple tags, such as raw, transform, and ready layers, instead of manually triggering each DAGs. In that case, you can integrate them to trigger one, automatically triggering the others. Also, you can put conditions.

Airflow should support the dynamic drag creation.

I have been using Apache Airflow for over 8 years.

The solution is stable. 

I rate the solution's stability a nine-point five out of ten.

We were using Apache Airflow on Kubernetes. As more requests came in, it scaled dynamically based on the available ports. There are almost 15 data engineers who are using Apache Airflow.

I rate the solution's scalability a nine out of ten.

The initial setup is straightforward. It will be tricky if you go with an executor or Kubernetes operator.

If you're into plug-and-play convenience, Apache Airflow supports various deployment methods like Docker, Helm, or Kubernetes. If you want to spin up Airflow, it will take more than 10-15 minutes. However, if you're making customizations or prefer not to use existing databases, the setup time could be extended due to customization requests.

You use Apache Airflow to automate your data pipelines. When you have a data pipeline, such as a Spark job or any other job, and want to automate it, triggering the job manually is not always necessary. You need to configure these DAGs accordingly. For instance, Airflow can initiate the job when the data becomes available. We don't need to keep the cluster running all the time, 24/7. We start the cluster using Airflow when we need to submit the job. Once the job is completed, we terminate the cluster.

I recommend the solution.

Overall, I rate the solution a nine out of ten.

Since the solution is programmatic, it allows users to define pipelines in code rather than drag and drop.

There is an area for improvement in onboarding new people. They should make it simple for newcomers. Else, we have to put a senior engineer to operate it.

I have been using Apache Airflow for five years. We are using the latest version of the solution.

I rate the solution’s stability a seven out of ten.

The scalability is good. We have five people working on it for five different projects.

I rate the solution’s scalability a ten out of ten.

We have used open-source Apache NiFi for data flow, Talend, and secret server integration services.

We chose Apache Airflow because it is quite popular, adopted by many people, and has an open-source community and engineers. We moved with the crowd and chose it based on popularity.

I rate the initial setup five on a scale of one to ten, one being difficult and ten being easy.

The deployment required a senior engineer and took a week to complete.

The solution is cheap.

A new user has to be prepared to adopt a new paradigm and treat the data baseline as code rather than drag and drop. An organization should have a dedicated or experienced person looking into this.

Overall, I rate the solution an eight out of ten.

We use the product to orchestrate data engines and process new data files.

The product's most valuable feature is scalability. It helps us run hundreds of data jobs every day.

The platform's stability needs improvement, particularly regarding occasional interruptions due to networking issues. It requires manual intervention to resume jobs. Additionally, while extending the code is possible, it sometimes necessitates creating custom plugins.

We have been using Apache Airflow for four years.

We have more than 100 Apache Airflow users in our organization.

The initial setup on Google Cloud using Cloud Composer is straightforward and simplified. However, deploying it on-premises can be complex and challenging.

The product is worth the investment.

It is an open-source solution, so there are no hidden fees or licensing costs associated with the software. However, users need to cover the operational costs for the actual infrastructure, such as the virtual machines (VMs).

The directed acyclic graph (DAG) functionality in Apache Airflow has significantly enhanced our workflow management. It provides a visual representation of data processing tasks.

The user interface for monitoring and managing workflows has been excellent, particularly in the latest version. It is difficult for beginners to use the platform, and some training is required.

I recommend the product to others, and it is much better than our competitors. It is an open source. I rate it a seven out of ten.

We are using Apache Airflow, which has a modular architecture, integrated with our pipeline and with the code generation using Python and the code pipeline. We are running it in Airflow. It is easy for us to use to build the machine learning models, transport data, and manage our infrastructure.

Apache Airflow is easy for us to use to build machine learning models, transport data, and manage our infrastructure. It generates data using the pipeline, and we use Terraform code to run the airflow engine. The solution is easy for us to install and maintain and is very scalable. Apache Airflow is an open-source platform that allows easy integration with AWS, Azure, and Google Cloud Platform.

There is a minor issue with the manual work in Airflow, as everyday activities are managed manually. There is no dashboard for us to check all the Directed Acyclic Graphs (DAGs); a dashboard would help us analyze the work better.

I have been working with Apache Airflow for six months.

I would rate the stability of the solution as ten out of ten. It is very stable.

I rate the scalability at nine out of ten. The solution is very scalable.

Apache Airflow is an open-source platform, which means we solve issues by ourselves. However, there is enough documentation available, and the community support is good.

Neutral

Apache Airflow is easy for us to install. Our reports allow us to spin up the Airflow container and all the servers automatically within five to ten minutes.

Apache Airflow is a community-based platform and is not a licensed product.

I would rate Apache Airflow from one to ten points overall as ten points, which means it is a very good solution.

We use the product for scheduling and defining workflows. It helps us extensively to manage complex workflows within Cloudera's ecosystem, particularly for handling and processing data.

The solution has been beneficial in automating and managing our data workflows efficiently. It has integrated well with our Cloudera environment, enabling us to handle complex workflows with greater ease and reliability.

The solution's most valuable feature is its ability to run workflows without saving changes. It allows us to execute tasks without permanently altering our configurations, which is useful for temporary adjustments and testing.

One area for improvement would be to address specific functionalities removed in recent updates that were previously useful for our operations.

Additional features that could enhance the product include more flexibility in parameterization and improved tools for managing and debugging workflows.

I have been working with Airflow for approximately a year and a half, focusing on the current version for the past eight months.

The product has been stable in our environment.

The product is scalable. 

The technical support team has been responsive and helpful. They addressed issues related to removed functionalities and ensured critical features were restored in subsequent updates.

Positive

We previously used Hortonworks but switched to Cloudera CDP. We also used other Cloudera tools but found Airflow to be a better fit for our current needs due to its capabilities in workflow management.

The initial setup was complex due to the integration with various data sources and configuration requirements, but once properly set up, it has proven effective.

The implementation was carried out with guidance from Cloudera's support team, who provided valuable assistance in configuring the solution to meet our requirements.

We evaluated other data workflow solutions but found Airflow the most suitable due to its integration with Cloudera and comprehensive feature set for managing complex workflows.

Airflow integrates well with Cloudera and effectively supports complex operations. However, users should be aware of changes in functionality between versions and plan accordingly.

Overall, I rate it a nine out of ten. 

My team works on commerce services. We use Airflow to synchronize user information or product information from other services. We use the tool for automating data pipelines. We store user history about API calls and show it on a statistics page, like daily or real-time statistics. We use the solution to aggregate API user's data.

Kubernetes from the batch application is the most useful to my team. It uses Python. It is simple. There are not many learning costs. We're using the scheduler. We don't need to care about the batch job every day. We just need to notice when the alerts are firing. It is convenient for us. The product supports many other services, like Kubernetes. I saw some custom applications and programs. The solution integrates very well with other products.

The documents do not precisely define the function of the operators. I had to do some experiments to understand the function of the operators. The documentation must be improved. Some parts of the documentation do not precisely explain the parameters and functions. We often need to do experiments to understand how they work.

I have been using the solution for one and a half years.

I rate the tool’s stability a nine out of ten.

I rate the tool’s scalability a six or seven out of ten. We haven’t horizontally scaled the solution. At least 20% of the teams in my organization are using Airflow to do some batch jobs. There are around 300 users.

I rate the ease of setup an eight out of ten. The product is deployed on the cloud. We release Airflow on Kubernetes. The deployment takes less than five minutes. We use a deployment tool made by our company to deploy the solution.

I am also using Apache Kafka.

I will recommend the product to others. The UI is very simple and easy to learn. There are a lot of users of the product. We can find information easily on Google. Overall, I rate the tool an eight out of ten.

We use Apache Airflow for the automation and orchestration of model deployment, training, and feature engineering steps. It is a model lifecycle management tool.

We have an integration with Apache Airflow in our portal for messaging. We use group and transformation data from Redshift to Tesco, and then create a call flow to the router. This is a source of data leakage, such as data engineering and machine learning, especially in a HIPAA environment. We need to check the evolution steps in the pipeline. In production, we only have two cases. Sometimes, we need customer data not in the database, which we get from object storage. The call flow from Redshift to Tesco involves transforming the data and then generating it with the router or Kibana router for the policy. The data is then transformed and sent to the dashboard or data warehouse.

Airflow is a pipeline for transferring code by clients, but for experimental model experiments, Apache Airflow does not have any solution. There is a need for more features on experimental evolution steps.

I have been using Apache Airflow for one and a half years.

The product is stable. I rate the solution’s stability an eight out of ten.

20 users are using this solution in our organization. I rate the solution’s scalability an eight out of ten.

The initial setup is not complex and can be done by two people. However, open-source prime solutions have some difficulties. We can schedule Apache Airflow on Kubernetes. Space limitations and installation issues may arise, as we do not have full control over Kubernetes cluster resources, and our administration is limited. I rate the initial setup a six out of ten, where one is difficult, and ten is easy.

I recommend Apache Airflow because it is still profitable and can be used with multiple systems and servers, Kubernetes systems, and dashboard systems. You can use it to get social media and other data, but it can be expensive. Overall, I rate the solution a nine out of ten.