Apache Airflow Reviews

Name: Apache Airflow
Brand: Apache
Rating: 4 (34 reviews)

3.9 out of 5

34 reviews
96% willing to recommend

708 followers

What is Apache Airflow?

Apache Airflow is an open-source workflow management system (WMS) that is primarily used to programmatically author, orchestrate, schedule, and monitor data pipelines as well as workflows. The solution makes it possible for you to manage your data pipelines by authoring workflows as directed acyclic graphs (DAGs) of tasks. By using Apache Airflow, you can orchestrate data pipelines over object stores and data warehouses, run workflows that are not data-related, and can also create and manage scripted data pipelines as code (Python).

Get the Apache Airflow Buyer's Guide and find out what your peers are saying about Apache Airflow, Informatica Intelligent Data Management Cloud (IDMC), Camunda and more!

Apache Airflow is the #2 ranked solution in BPM Software. PeerSpot users give Apache Airflow an average rating of 7.8 out of 10. Apache Airflow is most commonly compared to Informatica Intelligent Data Management Cloud (IDMC): Apache Airflow vs Informatica Intelligent Data Management Cloud (IDMC). Apache Airflow is popular among the large enterprise segment, accounting for 73% of users researching this solution on PeerSpot. The top industry researching this solution are professionals from a financial services firm, accounting for 24% of all views.

Helped 801,394 peers since 2012

Featured reviews

Damian Bukowski

Program Python at Santander Bank Polska

The only thing I would like Apache to do is to introduce an integration of the database from Oracle because it currently supports Postgres primarily in MySQL. Oracle is something that many companies use, like a production database, for which you have to pay since it is not free and offers more extended support. With Apache Airflow, even though it uses Python and Python has modules that include Oracle databases, it'll be safer and more convenient to do it through Apache Airflow and not through Python scripts. I want to see Apache Airflow have more integrations with more production-based databases since it is an area where the product lacks currently.

Read full review

SUDHIR KUMAR RATHLAVATH

Student at University of South Florida

One improvement could be the inclusion of a plugin with a drag-and-drop feature. This graphical feature would be beneficial when dealing with connectivity and integration services like connecting to BigQuery or other systems. As a first-time user, although the documentation is available, it would be more user-friendly to have a drag-and-drop interface within the portal. Users could simply drag and drop components to create a pseudo-code, making it more flexible and intuitive. Therefore, I suggest having a drag-and-drop feature for a more user-friendly experience and better code management. Moreover, for admins, there should be improved logging capabilities because Apache Airflow does have logging, but it's limited to some database data. It would be better if everything goes into the server where it's hosted. Probably on the interface level. If something goes well for the developers.

Read full review

Miodrag Milojevic

Senior Data Archirect at Yettel

We leverage Apache Airflow to orchestrate our data pipelines, primarily due to the multitude of data sources we manage. These sources vary in nature, with some delivering streaming data, while others follow different protocols such as FTP or utilize landing areas. We utilize Airflow for orchestrating tasks such as data ingestion, transformation, and preparation, ensuring that the data is formatted appropriately for further processing. Typically, this involves tasks like normalization, enrichment, and structuring the data for consumption by tools like Spark or other similar platforms in our ecosystem. The scheduling and monitoring functionalities enhance our data processing workflows. While the interface could be more user-friendly, proficiency in scheduling and monitoring can be attained through practice and skill development. The scalability of Apache Airflow effectively accommodates our increasing data processing demands without issue. While occasional server problems may arise, particularly in this aspect, overall, the product remains reliably stable. It offers a straightforward means to orchestrate numerous data sources efficiently, thanks to its user-friendly interface. Learning to use it is relatively quick and straightforward, although some experimentation, practice, and training may be required to master certain aspects.

Read full review

Apache Airflow mindshare

As of September 2024, the mindshare of Apache Airflow in the Business Process Management (BPM) category stands at 7.8%, down from 9.6% compared to the previous year, according to calculations based on PeerSpot user engagement data.

Business Process Management (BPM)

Key learnings from peers

Last updated Sep 9, 2024

Valuable Features

"I worked on a project at a leading German bank for two years, successfully migrating large applications with hundreds of jobs."
"To increase efficiency, it's quite simple to add dbt tasks to an Apache Airflow pipeline or orchestration file. With the tool, you can specify dependencies."
"Apache Airflow can be integrated or used to run multiple files."

Room for Improvement

"Apache Airflow improved workflow efficiency, but we had to find solutions for large workflows. For instance, a monthly workflow with 1200 jobs had to be split into three to four pieces as it struggled with large job numbers. Loading a workflow with 500 jobs could take 10 minutes, which wasn't acceptable."
"I would like to see workflow integration across the servers."
"The in-built package dependencies in Python have some issues in Apache Airflow, making it an area that needs improvement."

Pricing

"Apache Airflow is open-source and free. Hyperscalers like Google (with Composer), Azure, and AWS offer managed Airflow services."
"I use the tool's open-source version."
"It is an open-source tool. There are no additional fees or charges associated with the product."

These insights are based on the in-depth reviews provided by peers to help you make a better buying decision.

Download our Apache Airflow Buyer's Guide for additional reliable information.

Review data by company size

By reviewers

By visitors reading reviews

Top industries

By visitors reading reviews

Financial Services Firm

24%

Computer Software Company

14%

Manufacturing Company

Retailer

Healthcare Company

Government

Insurance Company

Educational Organization

Energy/Utilities Company

University

Comms Service Provider

Media Company

Real Estate/Law Firm

Construction Company

Non Profit

Hospitality Company

Recreational Facilities/Services Company

Wholesaler/Distributor

Logistics Company

Transportation Company

Pharma/Biotech Company

Legal Firm

Consumer Goods Company

Outsourcing Company

Performing Arts

Compare Apache Airflow with alternative products

Learn more about Apache Airflow

Apache Airflow Features

Apache Airflow has many valuable key features. Some of the most useful ones include:

Smart sensor: In Apache Airflow, tasks are executed sequentially. The smart sensors are executed in bundles, and therefore consume fewer resources.

Dockerfile: By using Apache Airflow’s dockerfile feature, you can run your business’s Airflow code without having to document and automate the process of running Airflow on a server.

Scalability: Because Apache Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers, you can easily scale it.

Plug-and-play operators: With Apache Airflow, you can choose from several plug-and-play operators that are ready to execute your tasks on many third-party services.

Apache Airflow Benefits

There are many benefits to implementing Apache Airflow. Some of the biggest advantages the solution offers include:

User friendly: Using Apache Airflow requires minimal python knowledge to get started.

Intuitive user interface: The Apache Airflow user interface enables you to visualize pipelines running in production, monitor progress, and also troubleshoot issues when needed.

Easy integration: Apache Airflow can easily be integrated with cloud platforms (Google, AWS, Azure, etc).

Visual DAGs: Apache Airflow’s visual DAGs provide data lineage, which facilitates debugging of data flows and also aids in auditing and data governance.

Flexibility: Apache Airflow provides you with several ways to make DAG objects more flexible. At runtime, a context variable is passed to each workflow execution, which is quickly incorporated into an SQL statement that includes the run ID, execution date, and last and next run times.

Multiple deployment options: With Apache Airflow, you have several options for deployment, including self-service, open source, or a managed service.

Several data source connections: Apache Airflow can connect to a variety of data sources, including APIs, databases, data warehouses, and more.

Reviews from Real Users

Below are some reviews and helpful feedback written by PeerSpot users currently using the Apache Airflow solution.

A Senior Solutions Architect/Software Architect says, “The product integrates well with other pipelines and solutions. The ease of building different processes is very valuable to us. The difference between Kafka and Airflow, is that it's better for dealing with the specific flows that we want to do some transformation. It's very easy to create flows.”

An Assistant Manager at a comms service provider mentions, “The best part of Airflow is its direct support for Python, especially because Python is so important for data science, engineering, and design. This makes the programmatic aspect of our work easy for us, and it means we can automate a lot.”

A Senior Software Engineer at a pharma/biotech company comments that he likes Apache Airflow because it is “Feature rich, open-source, and good for building data pipelines.”

Apache Airflow was previously known as Airflow.