Try our new research platform with insights from 80,000+ expert users

Apache Hadoop vs Dremio comparison

 

Comparison Buyer's Guide

Executive Summary
 

Categories and Ranking

Apache Hadoop
Average Rating
7.8
Number of Reviews
39
Ranking in other categories
Data Warehouse (6th)
Dremio
Average Rating
8.6
Reviews Sentiment
5.9
Number of Reviews
6
Ranking in other categories
Cloud Data Warehouse (10th), Data Science Platforms (8th)
 

Featured Reviews

Sushil Arya - PeerSpot reviewer
Provides ease of integration with the IT workflow of a business
When working with Kafka, I saw that the data came in an incremental order. The incremental data processing part is still not very effective in Apache Hadoop. If the data is already there, it can be processed very effectively, especially if the data is coming in every second. If you want to know the location of some data every second, then such data is not processed effectively in Apache Hadoop. I can say that one of the features where improvements are required revolves around the licensing cost of the tool. If the tool can build some licensing structures in a pay-per-use manner, organizations can get the look and feel of Apache Hadoop. Apache Hadoop can offer a licensing structure of the product that can be seen as similar to how AWS operates. Apache Hadoop can look into the capability of processing incremental data. The tool's setup process can be a scope of improvement. Also, it is not very simple because while doing the setup, we need to do all the server settings, including port listing and firewall configurations. If we look at other products on the market, then they can be made simpler. There are certain shortcomings when it comes to the product's technical support part, making it an area where improvements are required. The time frame for the resolution is an area that needs to be improved. The overall communication part of the technical support team also needs improvement.
MikeWalker - PeerSpot reviewer
It enables you to manage changes more effectively than any other platform.
Dremio enables you to manage changes more effectively than any other data warehouse platform. There are two things that come into play. One is data lineage. If you are looking at data in Dremio, you may want to know the source and what happened to it along the way or how it may have been transformed in the data pipeline to get to the point where you're consuming it. There's another thing called data providence. They're tied together. Data providence allows you to go back and recreate the data at any particular point in time. It's extremely important for compliance and governance issues because data changes all time. How did it change? What was it three days or months ago? You may have made some decisions based on data that was three months old, so you might need to revisit those. It's essential for things like machine learning and deep learning, where you are generating AI models off data. When the model stops working or doesn't work as expected, you need to figure out why. You have to go back and adjust the datasets used to train the model. We do that through an open-source project called Nessie, which is their basis for providing data lineage and data province capabilities. It's super powerful. Arrow is another open-source project for storing data in memory and performing data query operations. Data sits on a disk in one format. If you want to do anything with data, you have to load it into your computer and put it into memory so you can work with it. Arrow provides a format in memory that enables the whole library to perform various operations on that data. Every vendor has its own way of representing data in memory. They've latched onto an industry standard and developed it so it's open. Now people can use the exact same format in memory to do operations and use the library set to perform functions on data. New developers can decide if they want to develop their own memory format or use one that's already there. Data transfer is a massive problem when you're working with large datasets, doing advanced analytics, and trying to train machine learning or deep learning models. What happens often is companies downsample their data sets to do training on models because transferring and managing data on a deep learning or machine learning platform is too much.

Quotes from Members

We asked business professionals to review the solutions they use. Here are some excerpts of what they said:
 

Pros

"The most important feature is its ability to handle large volumes. Some of our customers have really large volumes, and it is capable of handling their data in terms of the core volume and daily incremental volume. So, its processing power and speed are most valuable."
"Its integration is Hadoop's best feature because that allows us to support different tools in a big data platform."
"The best thing about this solution is that it is very powerful and very cheap."
"It's good for storing historical data and handling analytics on a huge amount of data."
"​​Data ingestion: It has rapid speed, if Apache Accumulo is used."
"They have integrated other tools as well, like Power BI and Oracle BI, both on Azure, for reporting. Oracle BI is difficult to integrate."
"Apache Hadoop is crucial in projects that save and retrieve data daily. Its valuable features are scalability and stability. It is easy to integrate with the existing infrastructure."
"The most valuable features are powerful tools for ingestion, as data is in multiple systems."
"Dremio enables you to manage changes more effectively than any other data warehouse platform. There are two things that come into play. One is data lineage. If you are looking at data in Dremio, you may want to know the source and what happened to it along the way or how it may have been transformed in the data pipeline to get to the point where you're consuming it."
"The most valuable feature of Dremio is it can sit on top of any other data storage, such as Amazon S3, Azure Data Factory, SGFS, or Hive. The memory competition is good. If you are running any kind of materialized view, you'd be running in memory."
"We primarily use Dremio to create a data framework and a data queue."
"Dremio allows querying the files I have on my block storage or object storage."
"Everyone uses Dremio in my company; some use it only for the analytics function."
"Dremio gives you the ability to create services which do not require additional resources and sterilization."
 

Cons

"It could be more user-friendly."
"I mentioned it definitely, and this is probably the only feature we can improve a little bit because the terminal and coding screen on Hadoop is a little outdated, and it looks like the old C++ bio screen. If the UI and UX can be improved slightly, I believe it will go a long way toward increasing adoption and effectiveness."
"The key shortcoming is its inability to handle queries when there is insufficient memory. This limitation can be bypassed by processing the data in chunks."
"It requires a great deal of learning curve to understand. The overall Hadoop ecosystem has a large number of sub-products. There is ZooKeeper, and there are a whole lot of other things that are connected. In many cases, their functionalities are overlapping, and for a newcomer or our clients, it is very difficult to decide which of them to buy and which of them they don't really need. They require a consulting organization for it, which is good for organizations such as ours because that's what we do, but it is not easy for the end customers to gain so much knowledge and optimally use it."
"The main thing is the lack of community support. If you want to implement a new API or create a new file system, you won't find easy support."
"It would be good to have more advanced analytics tools."
"I would like to see more direct integration of visualization applications."
"There is a lack of virtualization and presentation layers, so you can't take it and implement it like a radio solution."
"I cannot use the recursive common table expression (CTE) in Dremio because the support page says it's currently unsupported."
"It shows errors sometimes."
"We've faced a challenge with integrating Dremio and Databricks, specifically regarding authentication. It is not shaking hands very easily."
"Dremio takes a long time to execute large queries or the executing of correlated queries or nested queries. Additionally, the solution could improve if we could read data from the streaming pipelines or if it allowed us to create the ETL pipeline directly on top of it, similar to Snowflake."
"Dremio doesn't support the Delta connector. Dremio writes the IT support for Delta, but the support isn't great. There is definitely room for improvement."
"They have an automated tool for building SQL queries, so you don't need to know SQL. That interface works, but it could be more efficient in terms of the SQL generated from those things. It's going through some growing pains. There is so much value in tools like these for people with no SQL experience. Over time, Dermio will make these capabilities more accessible to users who aren't database people."
 

Pricing and Cost Advice

"The product is open-source, but some associated licensing fees depend on the subscription level."
"If my company can use the cloud version of Apache Hadoop, particularly the cloud storage feature, it would be easier and would cost less because an on-premises deployment has a higher cost during storage, for example, though I don't know exactly how much Apache Hadoop costs."
"The price could be better. Hortonworks no longer exists, and Cloudera killed the free version of Hadoop."
"The price of Apache Hadoop could be less expensive."
"This is a low cost and powerful solution."
"​There are no licensing costs involved, hence money is saved on the software infrastructure​."
"Do take into consider that data storage and compute capacity scale differently and hence purchasing a "boxed" / 'all-in-one" solution (software and hardware) might not be the best idea."
"For any big enterprise the costs can be handled, and it is suitable for big enterprises because the scale of data is large. For medium and small enterprises, the tool is on the high-price side."
"Dremio is less costly competitively to Snowflake or any other tool."
"Right now the cluster costs approximately $200,000 per month and is based on the volume of data we have."
report
Use our free recommendation engine to learn which Cloud Data Warehouse solutions are best for your needs.
816,406 professionals have used our research since 2012.
 

Top Industries

By visitors reading reviews
Financial Services Firm
32%
Computer Software Company
11%
University
7%
Energy/Utilities Company
6%
Financial Services Firm
32%
Computer Software Company
10%
Manufacturing Company
8%
Retailer
4%
 

Company Size

By reviewers
Large Enterprise
Midsize Enterprise
Small Business
No data available
 

Questions from the Community

What do you like most about Apache Hadoop?
It's primarily open source. You can handle huge data volumes and create your own views, workflows, and tables. I can also use it for real-time data streaming.
What is your experience regarding pricing and costs for Apache Hadoop?
The product is open-source, but some associated licensing fees depend on the subscription level. While it might be free for students, organizations typically need to pay for their subscriptions. Th...
What needs improvement with Apache Hadoop?
Hadoop lacks OLAP capabilities. I recommend adding a Delta Lake feature to make the data compatible with ACID properties. Also, video and audio streaming import issues could be improved to ensure p...
What do you like most about Dremio?
Dremio allows querying the files I have on my block storage or object storage.
What is your experience regarding pricing and costs for Dremio?
Every tool has a value based on its visualization, and the pricing is worth its value.
What needs improvement with Dremio?
Dremio's interface is good, but it has a few limitations. I cannot do a lot of things with ANSI SQL or basic SQL. I cannot use the recursive common table expression (CTE) in Dremio because the supp...
 

Comparisons

 

Learn More

Video not available
 

Overview

 

Sample Customers

Amazon, Adobe, eBay, Facebook, Google, Hulu, IBM, LinkedIn, Microsoft, Spotify, AOL, Twitter, University of Maryland, Yahoo!, Cornell University Web Lab
UBS, TransUnion, Quantium, Daimler, OVH
Find out what your peers are saying about Apache Hadoop vs. Dremio and other solutions. Updated: October 2024.
816,406 professionals have used our research since 2012.