AWS Lake Formation vs BigQuery comparison

"It is seamlessly integrated within the AWS ecosystem, making it straightforward to manage access patterns for AWS-native services."

"In the shortest form, what I appreciated about AWS Lake Formation was that the schema definition and data cataloging were quite good."

"I can easily move data from cold storage to regular storage."

"The solution is quite good at handling analytics. It's done a good job at helping us centralize them."

"The most important advantage in using AWS Lake Formation is its ability to connect the data lake to the other technologies in AWS. This is what I advise my clients."

"The features and capabilities of AWS Lake Formation that I have found most valuable are that it is really convenient to see all the different data assets that were configured and understand who has and what type of service has or does not have access to those services."

"The integration of AWS Lake Formation with the IAM for authentication and authorization is very good; I didn't have any problems in the setup and thought it was simple."

"AWS Lake Formation lets you see all your data and tables on one screen."

"The integrated data storage features are good."

"The product is serverless. We only need to write SQL queries to analyze the data. We need to pay based on the number of queries. The retrieval time is very less. Even if you write large queries, the tool is able to bring back data in a few seconds."

"The main thing I like about BigQuery is storage. We did an on-premise BigQuery migration with trillions of records. Usually, we have to deal with insufficient storage on-premises, but in BigQuery, we don't get that because it's like cloud storage, and we can have any number of records. That is one advantage. The next major advantage is the column length. We have some limits on column length on-premises, like 10,000, and we have to design it based on that. However, with BigQuery, we don't need to design the column length at all. It will expand or shrink based on the records it's getting. I can give you a real-life example based on our migration from on-premises to GCP. There was a dimension table with a general number of records, and when we queried that on-premises, like in Apache Spark or Teradata, it took around half an hour to get those records. In BigQuery, it was instant. As it's very fast, you can get it in two or three minutes. That was very helpful for our engineers. Usually, we have to run a query on-premises and go for a break while waiting for that query to give us the results. It's not the case with BigQuery because it instantly provides results when we run it. So, that makes the work fast, it helps a lot, and it helps save a lot of time. It also has a reasonable performance rate and smart tuning. Suppose we need to perform some joins, BigQuery has a smart tuning option, and it'll tune itself and tell us the best way a query can be done in the backend. To be frank, the performance, reliability, and everything else have improved, even the downtime. Usually, on-premise servers have some downtime, but as BigQuery is multiregional, we have storage in three different locations. So, downtime is also not getting impacted. For example, if the Atlantic ocean location has some downtime, or the server is down, we can use data that is stored in Africa or somewhere else. We have three or four storage locations, and that's the main advantage."

"BigQuery has a very nice interface that you can easily learn if you know SQL."

"BigQuery's querying capabilities are very optimized for large datasets."

"BigQuery excels at structuring data, performing predictions, and conducting insightful analyses and it leverages machine learning and artificial intelligence capabilities, powered by Google's Duarte AI."

"It's similar to a Hadoop cluster, except it's managed by Google."

"When integrating their system into the cloud-based solutions, we were able to increase their efficiency and overall productivity twice compared with their on-premises option."

"AWS Lake Formation's pricing could be cheaper."

"I haven't seen any measurable benefits from using AWS Lake Formation, such as time saving, resource saving, or efficiency improvements."

"It falls short when it comes to more granular access control, such as cell-level or row-level entitlements which is a significant drawback for organizations that require precise control over who can access specific rows of data."

"You need to have data experience to use the product."

"In our current setup, everyone from my business unit uses the same consumer account. When access is requested for a table, everyone using that business unit account receives access. This could present a security concern, though it benefits new team members who automatically receive all necessary access permissions."

"Rather than creating an additional hundred tools, optimizing a tool to have a centralized location to do governance would be beneficial."

"I think AWS Lake Formation could improve by enforcing the least privilege by design, moving from ad hoc grants to role-based access controls."

"Despite following official documentation, configuration problems persisted, requiring weeks of support from multiple AWS engineers to resolve."

"There is a limitation when copying data directly from BigQuery; it only supports up to ten MB when copying data to the clipboard."

"The solution hinges on Google patterns so continued improvement is important."

"The primary hurdle in this migration lies in the initial phase of moving substantial volumes of data to cloud-based platforms."

"BigQuery can be very expensive if not used properly. Introducing AI tools that would allow you to optimize the data extraction process is an area of serious improvement."

"The product could benefit from improvements in user-friendliness, particularly in terms of the user interface."

"When I execute a query, the dashboard doesn't always present the output seamlessly. Troubleshooting requires opening each pipeline individually, which is time-consuming."

"I understand that Snowflake has made some improvements on its end to further reduce costs, so I believe BigQuery can catch up."

"When it comes to queries or the code being executed in the data warehouse, the management of this code, like integration with the GitHub repository or the GitLab repository, is kind of complicated, and it's not so direct."

"AWS Lake Formation is a bit expensive."

"The tool has competitive pricing."

"The price is a bit high but the technology is worth it."

"The pricing is adaptable, ensuring that organizations can tailor their usage and costs based on their specific requirements and configurations within the Google Cloud Platform."

"The pricing is good and there are no additional costs involved."

"The platform is inexpensive."

"One terabyte of data costs $20 to $22 per month for storage on BigQuery and $25 on Snowflake. Snowflake is costlier for one terabyte, but BigQuery charges based on how much data is inserted into the tables. BigQuery charges you based on the amount of data that you handle and not the time in which you handle it. This is why the pricing models are different and it becomes a key consideration in the decision of which platform to use."

"The solution is pretty affordable and quite cheap in comparison to PDP or Cloudera."

"Its cost structure operates on a pay-as-you-go model."