Try our new research platform with insights from 80,000+ expert users

AWS Batch vs Apache NiFi comparison

 

Comparison Buyer's Guide

Executive Summary
 

Categories and Ranking

Apache NiFi
Ranking in Compute Service
8th
Average Rating
7.8
Number of Reviews
11
Ranking in other categories
No ranking in other categories
AWS Batch
Ranking in Compute Service
6th
Average Rating
9.0
Number of Reviews
4
Ranking in other categories
No ranking in other categories
 

Mindshare comparison

As of November 2024, in the Compute Service category, the mindshare of Apache NiFi is 8.0%, up from 5.9% compared to the previous year. The mindshare of AWS Batch is 18.1%, down from 21.4% compared to the previous year. It is calculated based on PeerSpot user engagement data.
Compute Service
 

Featured Reviews

Arjun Pandey - PeerSpot reviewer
Oct 25, 2023
Good monitoring, metrics capabilities and provides ability to design processors with a single click
The good thing about Apache NiFi is that it has a concept called a flow file, and there's something called a flow file processor. The processor is the building block of your entire job. They have close to 500 processors for each purpose. For example, for reading from Kafka, Ni-Fi has a processor called "consumer Kafka". To write to S3, they have a processor called "put S3". Now, if I read from Kafka and write my own application, I'd need to ensure the library I'm using tracks my messages. I'd also need to handle any failures by rereading messages and ensuring acknowledgment. But all this complexity is already handled by Apache processor. They have around 500 processors, with a community investing significant effort into developing them. I can design your processor with a single click, export the entire workflow, and import it. The format is actionable, so NiFi is immediately set up. It's also distributed in nature so that I can scale it across nodes based on the workload. These nodes share their state. If one node goes down during processing, that data might be lost, but any subsequent data is safe. Such occurrences are rare. In essence, if you want a quick solution, Apache NiFi is a strong contender. There are other solutions like AirFlow and some paid pipeline options. AirFlow is open-source but can be complicated. For ETL or ERT solutions, there are pricier options. But if I need a pipeline that I can monitor step by step, Apache NiFi is a good choice. It integrates with Prometheus metrics, allowing me to embed them in my workflow. There's also a processor for integration with Slack, and I can receive notifications when the workflow is completed or fails. Another feature I appreciate is "back pressure," which NiFi handles automatically. It maintains its own queue and addresses back-pressure issues. If, for instance, an upstream entity isn't fast enough, items get stored in a queue, managed internally by NiFi's back pressure algorithm.
Larry Singh - PeerSpot reviewer
Nov 7, 2023
User-friendly, good customization and offers exceptional scalability, allowing users to run jobs ranging from 32 cores to over 2,000 cores
The main drawback to using AWS Batch would be the cost. It will be more expensive in some cases than using an HPC. It's more amenable to cases where you have spot requirements. So, for instance, you don't exactly know how much compute resources you'll need and when you'll need them. So it's much better for that flexibility. But if you're going to be running jobs consistently and using the compute cluster consistently for a lot of time, and it's not going to have a lot of downtime, then the HPC system might be a better alternative. So, really, it boils down to cost versus usage trade-offs. It's going to be more expensive for a lot of people. In future releases, I would like to see anything that could help make it easier to set up your initial system. And besides improving the GUI a little bit, the interface to it, making it a little bit more descriptive and having more information at your fingertips, so if you could point to the help of what the different features are, you can get quick access to that. That might help. With most of the AWS services, the difficulty really is getting information and knowledge about the system and seeing examples. So, seeing examples of how it's being used under multiple use cases would be the best way to become familiar with it. And some of that would just come with experience. You have to just use it and play with it. But in terms of the system itself, it's not that difficult to set up or use.

Quotes from Members

We asked business professionals to review the solutions they use. Here are some excerpts of what they said:
 

Pros

"It's an automated flow, where you can build a flow from source to destination, then do the transformation in between."
"Visually, this is a good product."
"The initial setup is very easy."
"The most valuable features of this solution are ease of use and implementation."
"We can integrate the tool with other applications easily."
"The most valuable feature has been the range of clients and the range of connectors that we could use."
"The initial setup is very easy. I would rate my experience with the initial setup a ten out of ten, where one point is difficult, and ten points are easy."
"The user interface is good and makes it easy to design very popular workflows."
"We can easily integrate AWS container images into the product."
"AWS Batch's deployment was easy."
"There is one other feature in confirmation or call confirmation where you can have templates of what you want to do and just modify those to customize it to your needs. And these templates basically make it a lot easier for you to get started."
"AWS Batch manages the execution of computing workload, including job scheduling, provisioning, and scaling."
 

Cons

"We run many jobs, and there are already large tables. When we do not control NiFi on time, all reports fail for the day. So it's pretty slow to control, and it has to be improved."
"The overall stability of this solution could be improved. In a future release, we would like to have access to more features that could be used in a parallel way. This would provide more freedom with processing."
"More features must be added to the product."
"There is room for improvement in integration with SSO. For example, NiFi does not have any integration with SSO. And if I want to give some kind of rollback access control across the organization. That is not possible."
"The use case templates could be more precise to typical business needs."
"The tool should incorporate more tutorials for advanced use cases. It has tutorials for simple use cases."
"There are some claims that NiFi is cloud-native but we have tested it, and it's not."
"There should be a better way to integrate a development environment with local tools."
"The main drawback to using AWS Batch would be the cost. It will be more expensive in some cases than using an HPC. It's more amenable to cases where you have spot requirements."
"The solution should include better and seamless integration with other AWS services, like Amazon S3 data storage and EC2 compute resources."
"When we run a lot of batch jobs, the UI must show the history."
"AWS Batch needs to improve its documentation."
 

Pricing and Cost Advice

"The solution is open-source."
"It's an open-source solution."
"We use the free version of Apache NiFi."
"I used the tool's free version."
"AWS Batch's pricing is good."
"The pricing is very fair."
"AWS Batch is a cheap solution."
report
Use our free recommendation engine to learn which Compute Service solutions are best for your needs.
814,649 professionals have used our research since 2012.
 

Top Industries

By visitors reading reviews
Financial Services Firm
19%
Computer Software Company
16%
Manufacturing Company
8%
Retailer
6%
Financial Services Firm
27%
Computer Software Company
13%
Manufacturing Company
7%
University
5%
 

Company Size

By reviewers
Large Enterprise
Midsize Enterprise
Small Business
No data available
 

Questions from the Community

What needs improvement with Apache NiFi?
The tool should incorporate more tutorials for advanced use cases. It has tutorials for simple use cases.
Which is better, AWS Lambda or Batch?
AWS Lambda is a serverless solution. It doesn’t require any infrastructure, which allows for cost savings. There is no setup process to deal with, as the entire solution is in the cloud. If you use...
What do you like most about AWS Batch?
AWS Batch manages the execution of computing workload, including job scheduling, provisioning, and scaling.
 

Comparisons

 

Also Known As

No data available
Amazon Batch
 

Overview

 

Sample Customers

Macquarie Telecom Group, Dovestech, Slovak Telekom, Looker, Hastings Group
Hess, Expedia, Kelloggs, Philips, HyperTrack
Find out what your peers are saying about AWS Batch vs. Apache NiFi and other solutions. Updated: October 2024.
814,649 professionals have used our research since 2012.