Spark SQL Reviews

Name: Spark SQL
Brand: Apache
Rating: 3.9 (14 reviews)

3.9 out of 5

14 reviews
85% willing to recommend

173 followers

What is Spark SQL?

Spark SQL is a Spark module for structured data processing. Unlike the basic Spark RDD API, the interfaces provided by Spark SQL provide Spark with more information about the structure of both the data and the computation being performed. There are several ways to interact with Spark SQL including SQL and the Dataset API. When computing a result the same execution engine is used, independent of which API/language you are using to express the computation. This unification means that developers can easily switch back and forth between different APIs based on which provides the most natural way to express a given transformation.

Get the Spark SQL Buyer's Guide and find out what your peers are saying about Spark SQL, Apache Spark, Amazon EMR and more!

Spark SQL is the #4 ranked solution in top Hadoop solutions. PeerSpot users give Spark SQL an average rating of 7.8 out of 10. Based on the analysis of the 14 most recent Spark SQL reviews, the overall sentiment is positive, with a sentiment score of 7.6. (Among the highest in the category). Spark SQL is most commonly compared to Apache Spark: Spark SQL vs Apache Spark. Spark SQL is popular among the large enterprise segment, accounting for 73% of users researching this solution on PeerSpot. The top industry researching this solution are professionals from a financial services firm, accounting for 26% of all views.

Helped 816,660 peers since 2012

Featured reviews

SurjitChoudhury

Data engineer at Cocos pt

My experience with the initial setup of Spark SQL was relatively smooth. Understanding the system wasn't overly difficult because the data was structured in databases, and we could use notebooks for coding in Python or Java. Configuring networks and running scripts to load data into the database were routine tasks that didn't pose significant challenges. The flexibility to use different languages for coding and the ability to process data using key-value pairs in Python made the setup adaptable. Once we received the source data, processing it in SparkSQL involved writing scripts to create dimension and fact tables, which became a standard part of our workflow. Setting up Spark SQL was reasonably quick, but sometimes we face performance issues, especially during data loading into the SQL Server data warehouse. Sequencing notebooks for efficient job runs is crucial, and managing complex tasks with multiple notebooks requires careful tracking. Exploring ways to optimize this process could be beneficial. However, once you are familiar with the database architecture and project tools, understanding and adapting to the system become more straightforward.

Read full review

Lucas Dreyer

Data Engineer at BBD

It takes a bit of time to get used to using this solution versus Panda as it has a steep learning curve. You need quite a high level of skill with SQL in general to use this solution. If SQL is not someone's primary language, they might find it difficult to get used to. This solution could be improved if there was a bridge between Panda and Spark SQL such as translating from Panda operations to SQL and then working with those queries that are generated. In a future release, it would be useful to have a real time dashboard versus batch updates to Power BI.

Read full review

Sahil Taneja

Principal Consultant/Manager at Tenzing

Spark SQL can improve the documentation they have provided. It can be a bit unclear at times. They could improve the documentation a bit more so that we can understand it more easily. Moreover, they could improve SparkUI to have more advanced versions of the performance and the queries and all.

Read full review

Spark SQL mindshare

As of November 2024, the mindshare of Spark SQL in the Hadoop category stands at 9.9%, down from 12.3% compared to the previous year, according to calculations based on PeerSpot user engagement data.

Hadoop

Key learnings from peers

Last updated Nov 21, 2024

Valuable Features

"Spark SQL's efficiency in managing distributed data and its simplicity in expressing complex operations make it an essential part of our data pipeline."
"I find the Thrift connection valuable."
"One of Spark SQL's most beautiful features is running parallel queries to go through enormous data."

Room for Improvement

"In terms of improvement, the only thing that could be enhanced is the stability aspect of Spark SQL."
"I've experienced some incompatibilities when using the Delta Lake format."
"It would be useful if Spark SQL integrated with some data visualization tools."

Pricing

"We don't have to pay for licenses with this solution because we are working in a small market, and we rely on open-source because the budgets of projects are very small."
"We use the open-source version, so we do not have direct support from Apache."
"The on-premise solution is quite expensive in terms of hardware, setting up the cluster, memory, hardware and resources. It depends on the use case, but in our case with a shared cluster which is quite large, it is quite expensive."