We performed a comparison between Apache Spark and Spark SQL based on real PeerSpot user reviews.
Find out in this report how the two Hadoop solutions compare in terms of features, pricing, service and support, easy of deployment, and ROI."The memory processing engine is the solution's most valuable aspect. It processes everything extremely fast, and it's in the cluster itself. It acts as a memory engine and is very effective in processing data correctly."
"We use it for ETL purposes as well as for implementing the full transformation pipelines."
"The tool's most valuable feature is its speed and efficiency. It's much faster than other tools and excels in parallel data processing. Unlike tools like Python or JavaScript, which may struggle with parallel processing, it allows us to handle large volumes of data with more power easily."
"One of the key features is that Apache Spark is a distributed computing framework. You can help multiple slaves and distribute the workload between them."
"The deployment of the product is easy."
"The most valuable feature of Apache Spark is its flexibility."
"Its scalability and speed are very valuable. You can scale it a lot. It is a great technology for big data. It is definitely better than a lot of earlier warehouse or pipeline solutions, such as Informatica. Spark SQL is very compliant with normal SQL that we have been using over the years. This makes it easy to code in Spark. It is just like using normal SQL. You can use the APIs of Spark or you can directly write SQL code and run it. This is something that I feel is useful in Spark."
"ETL and streaming capabilities."
"The speed of getting data."
"The team members don't have to learn a new language and can implement complex tasks very easily using only SQL."
"The solution is easy to understand if you have basic knowledge of SQL commands."
"This solution is useful to leverage within a distributed ecosystem."
"It is a stable solution."
"Data validation and ease of use are the most valuable features."
"Overall the solution is excellent."
"One of Spark SQL's most beautiful features is running parallel queries to go through enormous data."
"Apache Spark provides very good performance The tuning phase is still tricky."
"The initial setup was not easy."
"At times during the deployment process, the tool goes down, making it look less robust. To take care of the issues in the deployment process, users need to do manual interventions occasionally."
"Technical expertise from an engineer is required to deploy and run high-tech tools, like Informatica, on Apache Spark, making it an area where improvements are required to make the process easier for users."
"Apart from the restrictions that come with its in-memory implementation. It has been improved significantly up to version 3.0, which is currently in use."
"They could improve the issues related to programming language for the platform."
"More ML based algorithms should be added to it, to make it algorithmic-rich for developers."
"The solution needs to optimize shuffling between workers."
"The solution needs to include graphing capabilities. Including financial charts would help improve everything overall."
"In terms of improvement, the only thing that could be enhanced is the stability aspect of Spark SQL."
"It would be beneficial for aggregate functions to include a code block or toolbox that explains its calculations or supported conditional statements."
"Being a new user, I am not able to find out how to partition it correctly. I probably need more information or knowledge. In other database solutions, you can easily optimize all partitions. I haven't found a quicker way to do that in Spark SQL. It would be good if you don't need a partition here, and the system automatically partitions in the best way. They can also provide more educational resources for new users."
"It takes a bit of time to get used to using this solution versus Pandas as it has a steep learning curve."
"There should be better integration with other solutions."
"It would be useful if Spark SQL integrated with some data visualization tools."
"In the next update, we'd like to see better performance for small points of data. It is possible but there are better tools that are faster and cheaper."
Apache Spark is ranked 1st in Hadoop with 60 reviews while Spark SQL is ranked 4th in Hadoop with 14 reviews. Apache Spark is rated 8.4, while Spark SQL is rated 7.8. The top reviewer of Apache Spark writes "Reliable, able to expand, and handle large amounts of data well". On the other hand, the top reviewer of Spark SQL writes "Offers the flexibility to handle large-scale data processing". Apache Spark is most compared with Spring Boot, AWS Batch, SAP HANA, Cloudera Distribution for Hadoop and Azure Stream Analytics, whereas Spark SQL is most compared with IBM Db2 Big SQL, Netezza Analytics, SAP HANA and HPE Ezmeral Data Fabric. See our Apache Spark vs. Spark SQL report.
See our list of best Hadoop vendors.
We monitor all Hadoop reviews to prevent fraudulent reviews and keep review quality high. We do not post reviews by company employees or direct competitors. We validate each review for authenticity via cross-reference with LinkedIn, and personal follow-up with the reviewer when necessary.