AWS Lake Formation vs BigQuery comparison

"In the shortest form, what I appreciated about AWS Lake Formation was that the schema definition and data cataloging were quite good."

"We use this to reduce latency from minutes to seconds, as we aim for real-time visibility into patient healthcare monitoring."

"The LF-Tag system with granular permissions was key to the project as a functionality of AWS Lake Formation."

"The most important advantage in using AWS Lake Formation is its ability to connect the data lake to the other technologies in AWS. This is what I advise my clients."

"It is seamlessly integrated within the AWS ecosystem, making it straightforward to manage access patterns for AWS-native services."

"I can easily move data from cold storage to regular storage."

"We have observed measurable benefits in terms of cost savings, time savings, resource savings, and efficiency improvements in our workflows."

"The integration of AWS Lake Formation with the IAM for authentication and authorization is very good; I didn't have any problems in the setup and thought it was simple."

"BigQuery has a very nice interface that you can easily learn if you know SQL."

"The most valuable features of this solution, in my opinion, are speed and performance, as well as cost-effectiveness."

"The main thing I like about BigQuery is storage. We did an on-premise BigQuery migration with trillions of records. Usually, we have to deal with insufficient storage on-premises, but in BigQuery, we don't get that because it's like cloud storage, and we can have any number of records. That is one advantage. The next major advantage is the column length. We have some limits on column length on-premises, like 10,000, and we have to design it based on that. However, with BigQuery, we don't need to design the column length at all. It will expand or shrink based on the records it's getting. I can give you a real-life example based on our migration from on-premises to GCP. There was a dimension table with a general number of records, and when we queried that on-premises, like in Apache Spark or Teradata, it took around half an hour to get those records. In BigQuery, it was instant. As it's very fast, you can get it in two or three minutes. That was very helpful for our engineers. Usually, we have to run a query on-premises and go for a break while waiting for that query to give us the results. It's not the case with BigQuery because it instantly provides results when we run it. So, that makes the work fast, it helps a lot, and it helps save a lot of time. It also has a reasonable performance rate and smart tuning. Suppose we need to perform some joins, BigQuery has a smart tuning option, and it'll tune itself and tell us the best way a query can be done in the backend. To be frank, the performance, reliability, and everything else have improved, even the downtime. Usually, on-premise servers have some downtime, but as BigQuery is multiregional, we have storage in three different locations. So, downtime is also not getting impacted. For example, if the Atlantic ocean location has some downtime, or the server is down, we can use data that is stored in Africa or somewhere else. We have three or four storage locations, and that's the main advantage."

"The initial setup is straightforward."

"The product’s most valuable feature is its ability to manage the database on the cloud."

"The solution is very useful nowadays for keeping a huge number of records."

"The best features of BigQuery for me are the fact that it's low-code, no-code; you don't have to be a data scientist to really utilize the tool."

"The product is serverless. We only need to write SQL queries to analyze the data. We need to pay based on the number of queries. The retrieval time is very less. Even if you write large queries, the tool is able to bring back data in a few seconds."

"Information about the pricing, cost, and setup cost of the AWS solutions would be beneficial."

"The initial onboarding process is challenging because creating a plan takes a month to a month and a half to build out."

"I think AWS Lake Formation could improve by enforcing the least privilege by design, moving from ad hoc grants to role-based access controls."

"You need to have data experience to use the product."

"AWS Lake Formation's pricing could be cheaper."

"The solution could make improvements around orchestration and doing some automation stuff on AWS front automation. It would be useful if we could use automation to build images and use hardened images which are CIS compliant."

"In our current setup, everyone from my business unit uses the same consumer account. When access is requested for a table, everyone using that business unit account receives access. This could present a security concern, though it benefits new team members who automatically receive all necessary access permissions."

"For the end-users, it's not as user-friendly as it could be."

"We'd like to see more local data residency."

"The product could benefit from improvements in user-friendliness, particularly in terms of the user interface."

"The main challenges are in the areas of performance and cost optimizations."

"I noticed recently it's more expensive now."

"I would like to see version-based implementation and a fallback arrangement for data stored in BigQuery storage. These are some features I'm interested in."

"There is a limitation when copying data directly from BigQuery; it only supports up to ten MB when copying data to the clipboard."

"There is a good amount of documentation out there, but they're consistently making changes to the platform, and, like, their literature hasn't been updated on some plans."

"There are some limitations in the query latency compared to what it was three years ago."

"AWS Lake Formation is a bit expensive."

"We are above the free threshold, so we are paying around 40 euros per month for BigQuery."

"BigQuery pricing can increase quickly. It's a high-priced solution."

"The price is a bit high but the technology is worth it."

"The product operates on a pay-for-use model. Costs include storage and query execution, which can accumulate based on data volume and complexity."

"The solution is pretty affordable and quite cheap in comparison to PDP or Cloudera."

"The product’s pricing could be more flexible for end users."

"BigQuery is inexpensive."

"The pricing is good and there are no additional costs involved."