Spark SQL Reviews

Name: Spark SQL
Brand: Apache
Rating: 3.9 (14 reviews)

3.9 out of 5

14 reviews
85% willing to recommend

173 followers

What is Spark SQL?

Spark SQL is a Spark module for structured data processing. Unlike the basic Spark RDD API, the interfaces provided by Spark SQL provide Spark with more information about the structure of both the data and the computation being performed. There are several ways to interact with Spark SQL including SQL and the Dataset API. When computing a result the same execution engine is used, independent of which API/language you are using to express the computation. This unification means that developers can easily switch back and forth between different APIs based on which provides the most natural way to express a given transformation.

Get the Spark SQL Buyer's Guide and find out what your peers are saying about Spark SQL, SAP HANA, Apache Spark and more!

Spark SQL is the #4 ranked solution in top Hadoop solutions. PeerSpot users give Spark SQL an average rating of 7.8 out of 10. Spark SQL is most commonly compared to SAP HANA: Spark SQL vs SAP HANA. Spark SQL is popular among the large enterprise segment, accounting for 72% of users researching this solution on PeerSpot. The top industry researching this solution are professionals from a financial services firm, accounting for 25% of all views.

Helped 815,854 peers since 2012

Featured reviews

SurjitChoudhury

Data engineer at Cocos pt

My experience with the initial setup of Spark SQL was relatively smooth. Understanding the system wasn't overly difficult because the data was structured in databases, and we could use notebooks for coding in Python or Java. Configuring networks and running scripts to load data into the database were routine tasks that didn't pose significant challenges. The flexibility to use different languages for coding and the ability to process data using key-value pairs in Python made the setup adaptable. Once we received the source data, processing it in SparkSQL involved writing scripts to create dimension and fact tables, which became a standard part of our workflow. Setting up Spark SQL was reasonably quick, but sometimes we face performance issues, especially during data loading into the SQL Server data warehouse. Sequencing notebooks for efficient job runs is crucial, and managing complex tasks with multiple notebooks requires careful tracking. Exploring ways to optimize this process could be beneficial. However, once you are familiar with the database architecture and project tools, understanding and adapting to the system become more straightforward.

Read full review

Lucas Dreyer

Data Engineer at BBD

It takes a bit of time to get used to using this solution versus Panda as it has a steep learning curve. You need quite a high level of skill with SQL in general to use this solution. If SQL is not someone's primary language, they might find it difficult to get used to. This solution could be improved if there was a bridge between Panda and Spark SQL such as translating from Panda operations to SQL and then working with those queries that are generated. In a future release, it would be useful to have a real time dashboard versus batch updates to Power BI.

Read full review

Sahil Taneja

Principal Consultant/Manager at Tenzing

We are using PySpark for big data processing, like multiple competitors of stock. We process it in in-memory using data frames and Spark SQL. We are using it along with the database to process the big data, especially the special Azure data. We are using PySpark. Databricks itself provides an…

Read full review

Spark SQL mindshare

As of November 2024, the mindshare of Spark SQL in the Hadoop category stands at 9.9%, down from 12.3% compared to the previous year, according to calculations based on PeerSpot user engagement data.

Hadoop

Key learnings from peers

Valuable Features

"Spark SQL's efficiency in managing distributed data and its simplicity in expressing complex operations make it an essential part of our data pipeline."
"I find the Thrift connection valuable."
"One of Spark SQL's most beautiful features is running parallel queries to go through enormous data."

Room for Improvement

"In terms of improvement, the only thing that could be enhanced is the stability aspect of Spark SQL."
"I've experienced some incompatibilities when using the Delta Lake format."
"It would be useful if Spark SQL integrated with some data visualization tools."

Pricing

"We don't have to pay for licenses with this solution because we are working in a small market, and we rely on open-source because the budgets of projects are very small."
"We use the open-source version, so we do not have direct support from Apache."
"The on-premise solution is quite expensive in terms of hardware, setting up the cluster, memory, hardware and resources. It depends on the use case, but in our case with a shared cluster which is quite large, it is quite expensive."

These insights are based on the in-depth reviews provided by peers to help you make a better buying decision.

Download our Spark SQL Buyer's Guide for additional reliable information.