Spark SQL's efficiency in managing distributed data and its simplicity in expressing complex operations make it an essential part of our data pipeline.
In the next update, we'd like to see better performance for small points of data. It is possible but there are better tools that are faster and cheaper.