AWS Lake Formation vs BigQuery comparison

"The main benefits that I have seen from using AWS Lake Formation are related to FinOps because you have control of your data and can track your costs since AWS Lake Formation is integrated into a unique platform, which is AWS Cloud Service."

"We have observed measurable benefits in terms of cost savings, time savings, resource savings, and efficiency improvements in our workflows."

"It is seamlessly integrated within the AWS ecosystem, making it straightforward to manage access patterns for AWS-native services."

"The solution has many features that are applicable to events such as audits."

"AWS Lake Formation significantly improves the structure of the data mesh, making it superior to previous structures we used."

"We use this to reduce latency from minutes to seconds, as we aim for real-time visibility into patient healthcare monitoring."

"The LF-Tag system with granular permissions was key to the project as a functionality of AWS Lake Formation."

"AWS Lake Formation lets you see all your data and tables on one screen."

"The main thing I like about BigQuery is storage. We did an on-premise BigQuery migration with trillions of records. Usually, we have to deal with insufficient storage on-premises, but in BigQuery, we don't get that because it's like cloud storage, and we can have any number of records. That is one advantage. The next major advantage is the column length. We have some limits on column length on-premises, like 10,000, and we have to design it based on that. However, with BigQuery, we don't need to design the column length at all. It will expand or shrink based on the records it's getting. I can give you a real-life example based on our migration from on-premises to GCP. There was a dimension table with a general number of records, and when we queried that on-premises, like in Apache Spark or Teradata, it took around half an hour to get those records. In BigQuery, it was instant. As it's very fast, you can get it in two or three minutes. That was very helpful for our engineers. Usually, we have to run a query on-premises and go for a break while waiting for that query to give us the results. It's not the case with BigQuery because it instantly provides results when we run it. So, that makes the work fast, it helps a lot, and it helps save a lot of time. It also has a reasonable performance rate and smart tuning. Suppose we need to perform some joins, BigQuery has a smart tuning option, and it'll tune itself and tell us the best way a query can be done in the backend. To be frank, the performance, reliability, and everything else have improved, even the downtime. Usually, on-premise servers have some downtime, but as BigQuery is multiregional, we have storage in three different locations. So, downtime is also not getting impacted. For example, if the Atlantic ocean location has some downtime, or the server is down, we can use data that is stored in Africa or somewhere else. We have three or four storage locations, and that's the main advantage."

"The most valuable aspect of BigQuery is its ability to handle high data workloads without causing friction with our online systems."

"The feature called calibrating the capacity is valuable."

"It's straightforward to set up."

"As a cloud solution, it's easy to set up."

"The query tool is scalable and allows for petabytes of data."

"The integrated data storage features are good."

"BigQuery's querying capabilities are very optimized for large datasets."

"Rather than creating an additional hundred tools, optimizing a tool to have a centralized location to do governance would be beneficial."

"The solution could make improvements around orchestration and doing some automation stuff on AWS front automation. It would be useful if we could use automation to build images and use hardened images which are CIS compliant."

"The main challenge we faced with AWS Lake Formation was related to cross-account sharing. Granting access to other AWS accounts for tables or databases in a different AWS account was somewhat difficult."

"Lake Formation could enhance its capabilities in audit logs, real-time monitoring, and advanced data governance."

"If I could improve AWS Lake Formation, I would add more integrations with SageMaker."

"It falls short when it comes to more granular access control, such as cell-level or row-level entitlements which is a significant drawback for organizations that require precise control over who can access specific rows of data."

"Athena can be a bit clunky when writing queries, indicating a potential enhancement point for easier user interaction with query tools such as DataGrip using provided driver JARs."

"I think AWS Lake Formation could improve by enforcing the least privilege by design, moving from ad hoc grants to role-based access controls."

"We'd like to have more integrations with other technologies."

"There are some limitations in the query latency compared to what it was three years ago."

"When I execute a query, the dashboard doesn't always present the output seamlessly. Troubleshooting requires opening each pipeline individually, which is time-consuming."

"The process of migrating from Datastore to BigQuery should be improved."

"An area for improvement in BigQuery is its UI because it's not working very well. Pricing for the solution is also very high."

"It would be helpful if they could provide some dashboards where you can easily view charts and information."

"There is a good amount of documentation out there, but they're consistently making changes to the platform, and, like, their literature hasn't been updated on some plans."

"When it comes to queries or the code being executed in the data warehouse, the management of this code, like integration with the GitHub repository or the GitLab repository, is kind of complicated, and it's not so direct."

"AWS Lake Formation is a bit expensive."

"The product’s pricing could be more flexible for end users."

"One terabyte of data costs $20 to $22 per month for storage on BigQuery and $25 on Snowflake. Snowflake is costlier for one terabyte, but BigQuery charges based on how much data is inserted into the tables. BigQuery charges you based on the amount of data that you handle and not the time in which you handle it. This is why the pricing models are different and it becomes a key consideration in the decision of which platform to use."

"The solution's pricing is cheaper compared to other solutions."

"BigQuery is inexpensive."

"The pricing is good and there are no additional costs involved."

"BigQuery pricing can increase quickly. It's a high-priced solution."

"The price is a bit high but the technology is worth it."

"I have tried my own setup using my Gmail ID, and I think it had a $300 limit for free for a new user. That's what Google is offering, and we can register and create a project."