AWS Lake Formation vs BigQuery comparison

"A favorite feature of AWS Lake Formation is that it provides us with visibility into who has access to a particular table or database in Glue."

"We have observed measurable benefits in terms of cost savings, time savings, resource savings, and efficiency improvements in our workflows."

"It is seamlessly integrated within the AWS ecosystem, making it straightforward to manage access patterns for AWS-native services."

"AWS Lake Formation significantly improves the structure of the data mesh, making it superior to previous structures we used."

"The integration of AWS Lake Formation with the IAM for authentication and authorization is very good; I didn't have any problems in the setup and thought it was simple."

"The most valuable features of AWS Lake Formation were the access model itself, as it allows implementation of filters, Blueprints, and row-level and column-level security to mask data that shouldn't be accessed by certain entities, enabling granular control without exposing PII data."

"The main benefits that I have seen from using AWS Lake Formation are related to FinOps because you have control of your data and can track your costs since AWS Lake Formation is integrated into a unique platform, which is AWS Cloud Service."

"The LF-Tag system with granular permissions was key to the project as a functionality of AWS Lake Formation."

"It's straightforward to set up."

"It has a well-structured suite of complimentary tools for data integration and so forth."

"The scalability is excellent, as it can handle large datasets and scale up pretty easily as the data volume grows while remaining very reliable with no bugs, glitches, crashes, or freezes."

"The product is serverless. We only need to write SQL queries to analyze the data. We need to pay based on the number of queries. The retrieval time is very less. Even if you write large queries, the tool is able to bring back data in a few seconds."

"When integrating their system into the cloud-based solutions, we were able to increase their efficiency and overall productivity twice compared with their on-premises option."

"The query tool is scalable and allows for petabytes of data."

"The main thing I like about BigQuery is storage. We did an on-premise BigQuery migration with trillions of records. Usually, we have to deal with insufficient storage on-premises, but in BigQuery, we don't get that because it's like cloud storage, and we can have any number of records. That is one advantage. The next major advantage is the column length. We have some limits on column length on-premises, like 10,000, and we have to design it based on that. However, with BigQuery, we don't need to design the column length at all. It will expand or shrink based on the records it's getting. I can give you a real-life example based on our migration from on-premises to GCP. There was a dimension table with a general number of records, and when we queried that on-premises, like in Apache Spark or Teradata, it took around half an hour to get those records. In BigQuery, it was instant. As it's very fast, you can get it in two or three minutes. That was very helpful for our engineers. Usually, we have to run a query on-premises and go for a break while waiting for that query to give us the results. It's not the case with BigQuery because it instantly provides results when we run it. So, that makes the work fast, it helps a lot, and it helps save a lot of time. It also has a reasonable performance rate and smart tuning. Suppose we need to perform some joins, BigQuery has a smart tuning option, and it'll tune itself and tell us the best way a query can be done in the backend. To be frank, the performance, reliability, and everything else have improved, even the downtime. Usually, on-premise servers have some downtime, but as BigQuery is multiregional, we have storage in three different locations. So, downtime is also not getting impacted. For example, if the Atlantic ocean location has some downtime, or the server is down, we can use data that is stored in Africa or somewhere else. We have three or four storage locations, and that's the main advantage."

"Its integration with other tools like Atlan through a Google Chrome extension is highly beneficial."

"In our current setup, everyone from my business unit uses the same consumer account. When access is requested for a table, everyone using that business unit account receives access. This could present a security concern, though it benefits new team members who automatically receive all necessary access permissions."

"The initial onboarding process is challenging because creating a plan takes a month to a month and a half to build out."

"In our experience what could be improved are not the support, performance or monitoring, but at a managerial level, the very expensive professional services of AWS."

"Athena can be a bit clunky when writing queries, indicating a potential enhancement point for easier user interaction with query tools such as DataGrip using provided driver JARs."

"The solution could make improvements around orchestration and doing some automation stuff on AWS front automation."

"Lake Formation could enhance its capabilities in audit logs, real-time monitoring, and advanced data governance."

"You need to have data experience to use the product."

"In our current setup, everyone from my business unit uses the same consumer account. When access is requested for a table, everyone using that business unit account receives access. This could present a security concern, though it benefits new team members who automatically receive all necessary access permissions."

"It would be beneficial to integrate additional tools, particularly from a business intelligence perspective."

"The price could be better. Compared to competing solutions, BigQuery is expensive."

"The main challenges are in the areas of performance and cost optimizations."

"It would be beneficial if BigQuery could be made more affordable."

"An area for improvement in BigQuery is its UI because it's not working very well. Pricing for the solution is also very high."

"There is a good amount of documentation out there, but they're consistently making changes to the platform, and, like, their literature hasn't been updated on some plans."

"There are some limitations in the query latency compared to what it was three years ago."

"As a product, BigQuery still requires a lot of maturity to accommodate other use cases and to be widely acceptable across other organizations."

"AWS Lake Formation is a bit expensive."

"The pricing is adaptable, ensuring that organizations can tailor their usage and costs based on their specific requirements and configurations within the Google Cloud Platform."

"BigQuery is inexpensive."

"We are above the free threshold, so we are paying around 40 euros per month for BigQuery."

"The solution's pricing is cheaper compared to other solutions."

"The platform is inexpensive."

"Price-wise, I think that is very reasonable."

"The pricing appears to be competitive for the intended usage scenarios we have in mind."

"I have tried my own setup using my Gmail ID, and I think it had a $300 limit for free for a new user. That's what Google is offering, and we can register and create a project."