Spark SQL Reviews

Our company uses the solution to create pipelines and data sets. The ETL process transforms the data and certain written aggregations convert the raw data to data sets. The data sets are then exported to tables for dashboards. 

The solution is easy to understand if you have basic knowledge of SQL commands.

Projects sit within the Spark scope and there are multiple options for data sets such as closed, private, or public. 

It is easy to perform analytics because data is contained within each use case. For example, you request data for a particular use case, receive the data link, and import it for analytics. 

It would be beneficial for aggregate functions to include a code block or toolbox that explains calculations or supported conditional statements. Multiple functions come within an aggregate so it is important to understand them. When you are trying to do something new, it would be easier and quite unique to get information within the solution rather than having to search the web. 

For example, once you select an aggregate it tells you what type of functions the solution can perform and includes a code block explaining its calculations. Or, a certain conditional statement gives you a second option or explains other types of statements the solution performs as part of a rule-level function. 

I have been using the solution for fourteen months. 

The solution is stable. 

The scalability depends on administrative rights. Every use case has certain allocated resources so a use case demanding scalability or extensive use can have additional resources allocated to it. 

I have not experienced any issues with the solution so have not needed technical support. 

I have been using SQL to extract data throughout my five-year career. 

The setup is very straightforward so I rate it a ten out of ten. 

We implemented the solution in-house. 

The solution is bundled with Palantir Foundry at no extra charge. 

Our company gives us the freedom to use Python R, PySpark, or SQL languages so we have many tools available. Our team includes 17 developers and 25% of them use the solution. 

The solution is way better than Oracle SQL because Oracle takes a lot of effort to understand and use. 

The solution is similar to the format of MS SQL. With MS, there are defined data sources that place restrictions on what you are supposed to use. Sometimes we had to make sure we had a way through the restrictions. For example, if we didn't have access to a physical table then we had to create a duplicate instance or view of it. We could see the values but couldn't manipulate them because we didn't have access to the physical table. The effect of MS restrictions is based on the complexity of a project and any privacy-related data constraints.

For the solution, use cases sit within the Spark scope so you get multiple options for creating them. You can individually set each use case as closed, private, or public. You can run analytics for each use case because the data is contained within it. This process is much easier when compared to Oracle SQL or MS SQL. 

The solution is very similar to the generic Spark and SQL language. 

I rate the solution an eight out of ten. 

Our primary use case is for building a data pipeline and data analytics. 

The performance is one of the most important features. It has an API to process the data in a functional manner. 

I would like to have the ability to process data without the overhead. To use the same API to process both terabytes data and be able to process one GB of data. 

I have been using Spark SQL for around four years. 

It is very stable.

It is scalable. I use it on and off. I use it mostly daily. 

From an infrastructure perspective, it was easy for us to set up because we used some cloud services. But on-premise requires more setup. There is a learning curve. If you're not a programmer there is a learning curve. It requires more effort to learn more complex steps. 

I deployed it by myself. We use cloud so we are able to do it. 

The amount of people required for deployment will depend. One person is enough for AWS but not in other places. 

If you know how to do it, the deployment can be done in minutes. 

I would rate Spark SQL a nine out of ten. 

My advice would be to read Databricks books about Spark. It's a good source of knowledge. 

In the next update, we'd like to see better performance for small points of data. It is possible but there are better tools that are faster and cheaper. 

The primary use case of this solution is to function within a distributed ecosystem. Spark is part of EMR, a Hadoop distribution, and is one of the tools in the ecosystem. You are not working with Hadoop in a vacuum—you leverage Spark, Hive, HBase—because it is just a distributed ecosystem. It has no value within itself. 

This solution can be deployed both on the cloud and on Cloudera distributions. 

This solution is useful to leverage within a distributed ecosystem. 

This solution could be improved by adding monitoring and integration for the EMR. 

We have been working with Spark SQL for a few years. We are an outsourcing and consulting company, so it's not for our use—we mostly work with clients. 

This solution is stable. 

This solution is scalable. 

The installation is straightforward because it's a cloud-based solution. 

We implement this solution for customers ourselves. 

There is no license or subscription for this solution. 

I rate this solution an eight out of ten and would recommend it to others. 

We use it to gather all the transaction data. We have Hadoop and Spark in our system, and we use some easy process flows for transport. 

It is a stable solution. 

Being a new user, I am not able to find out how to partition it correctly. I probably need more information or knowledge. In other database solutions, you can easily optimize all partitions. I haven't found a quicker way to do that in Spark SQL. It would be good if you don't need a partition here, and the system automatically partitions in the best way. They can also provide more educational resources for new users.

I have been using this solution for two months.

Its scalability is okay. We are a big organization. 

Being a new user, I would rate Spark SQL a four out of ten. 

I am using this solution for data validation and writing queries.

Data validation and ease of use are the most valuable features.

There should be better integration with other solutions.

I have been using the solution for approximately two years.

The solution has been stable.

I have found the solution to be scalable. We have 20 people using the solution in my organization and we plan to increase usage.

The solution is open-sourced and free.

I rate Spark SQL a ten out of ten.

The primary use is to process big data. We were connecting into and we were applying sentiment analysis via hardware.

In the next release, maybe the visualization of some command-line features could be added.

The stability was fine. It behaved as expected.

The scalability of the solution is good.

Technical support has been fine.

We previously used Apache Hadoop.

The initial setup was fine. If somebody knows what to expect it's okay.

We've just started using this solution. We were using it until recently on a research basis, just to measure the performance, the cost, and so on and so forth. Many things could be improved, but are okay up till now, I'm happy with. I would recommend the product.

I would rate this solution eight out of ten.