Apache Hadoop vs BigQuery comparison

"The most valuable features are powerful tools for ingestion, as data is in multiple systems."

"The scalability of Apache Hadoop is very good."

"Its flexibility in handling and storing large volumes of data is particularly beneficial, as is its resilience, which ensures data redundancy and fault tolerance."

"​​Data ingestion: It has rapid speed, if Apache Accumulo is used."

"Hadoop is extensible — it's elastic."

"It's open-source, so it's very cost-effective."

"The platform's quick data processing capabilities have been instrumental in supporting our AI-driven projects."

"The best thing about this solution is that it is very powerful and very cheap."

"The product is serverless. We only need to write SQL queries to analyze the data. We need to pay based on the number of queries. The retrieval time is very less. Even if you write large queries, the tool is able to bring back data in a few seconds."

"The initial setup is straightforward."

"It stands out in efficiently handling internal actions without the need for manual intervention in tasks like building cubes and defining final dimensions."

"BigQuery can be used for any type of company. It has the capability of building applications and storing data. It can be used for OLTP or OLAP. It has many other products within the Google space."

"The feature called calibrating the capacity is valuable."

"BigQuery allows for very fast access, and it is efficient in handling large datasets compared to other SQL databases."

"The interface is what I find particularly valuable."

"The main thing I like about BigQuery is storage. We did an on-premise BigQuery migration with trillions of records. Usually, we have to deal with insufficient storage on-premises, but in BigQuery, we don't get that because it's like cloud storage, and we can have any number of records. That is one advantage. The next major advantage is the column length. We have some limits on column length on-premises, like 10,000, and we have to design it based on that. However, with BigQuery, we don't need to design the column length at all. It will expand or shrink based on the records it's getting. I can give you a real-life example based on our migration from on-premises to GCP. There was a dimension table with a general number of records, and when we queried that on-premises, like in Apache Spark or Teradata, it took around half an hour to get those records. In BigQuery, it was instant. As it's very fast, you can get it in two or three minutes. That was very helpful for our engineers. Usually, we have to run a query on-premises and go for a break while waiting for that query to give us the results. It's not the case with BigQuery because it instantly provides results when we run it. So, that makes the work fast, it helps a lot, and it helps save a lot of time. It also has a reasonable performance rate and smart tuning. Suppose we need to perform some joins, BigQuery has a smart tuning option, and it'll tune itself and tell us the best way a query can be done in the backend. To be frank, the performance, reliability, and everything else have improved, even the downtime. Usually, on-premise servers have some downtime, but as BigQuery is multiregional, we have storage in three different locations. So, downtime is also not getting impacted. For example, if the Atlantic ocean location has some downtime, or the server is down, we can use data that is stored in Africa or somewhere else. We have three or four storage locations, and that's the main advantage."

"The solution is very expensive."

"The upgrade path should be improved because it is not as easy as it should be."

"Hadoop's security could be better."

"We would like to have more dynamics in merging this machine data with other internal data to make more meaning out of it."

"It needs better user interface (UI) functionalities."

"It requires a great deal of learning curve to understand. The overall Hadoop ecosystem has a large number of sub-products. There is ZooKeeper, and there are a whole lot of other things that are connected. In many cases, their functionalities are overlapping, and for a newcomer or our clients, it is very difficult to decide which of them to buy and which of them they don't really need. They require a consulting organization for it, which is good for organizations such as ours because that's what we do, but it is not easy for the end customers to gain so much knowledge and optimally use it."

"The key shortcoming is its inability to handle queries when there is insufficient memory. This limitation can be bypassed by processing the data in chunks."

"The integration with Apache Hadoop with lots of different techniques within your business can be a challenge."

"The product’s performance could be much faster."

"The process of migrating from Datastore to BigQuery should be improved."

"There is a limitation when copying data directly from BigQuery; it only supports up to ten MB when copying data to the clipboard."

"The price could be better. Compared to competing solutions, BigQuery is expensive. It's only suitable for enterprise customers, not small and medium-sized businesses, as they cannot afford this kind of solution. In the next release, it would be better if they improved their AI bot. Although machine learning and artificial intelligence are doing wonders, there is still a lot of room to enhance them."

"BigQuery should integrate with other tools, such as Cloud Logging and Local Studio, to enhance its capabilities further and enable powerful and innovative analyses."

"We'd like to see more local data residency."

"Instead of connecting directly to BigQuery, we connect to GCP, Cloud Run, and then to BigQuery, which is a long process."

"I would like to see version-based implementation and a fallback arrangement for data stored in BigQuery storage. These are some features I'm interested in."

"The price of Apache Hadoop could be less expensive."

"It's reasonable, but there's room for improvement in cost-effectiveness."

"This is a low cost and powerful solution."

"The price could be better. Hortonworks no longer exists, and Cloudera killed the free version of Hadoop."

"​There are no licensing costs involved, hence money is saved on the software infrastructure​."

"For any big enterprise the costs can be handled, and it is suitable for big enterprises because the scale of data is large. For medium and small enterprises, the tool is on the high-price side."

"Do take into consider that data storage and compute capacity scale differently and hence purchasing a "boxed" / 'all-in-one" solution (software and hardware) might not be the best idea."

"The product is open-source, but some associated licensing fees depend on the subscription level."

"Its cost structure operates on a pay-as-you-go model."

"The price is a bit high but the technology is worth it."

"BigQuery is inexpensive."

"The pricing is good and there are no additional costs involved."

"The solution's pricing is cheaper compared to other solutions."

"I have tried my own setup using my Gmail ID, and I think it had a $300 limit for free for a new user. That's what Google is offering, and we can register and create a project."

"The solution is pretty affordable and quite cheap in comparison to PDP or Cloudera."

"BigQuery pricing can increase quickly. It's a high-priced solution."