Apache Spark Reviews

The most valuable feature is the Fault Tolerance and easy binding with other processes like Machine Learning, graph analytics. The community is growing and hence executing ML in a distributed fashion is quite good.

Previously we were using Hadoop MapReduce to reduce the Google Ngrams (3TB), which took us approximately five days on our cluster. After using Spark, we were able to accomplish this task within hours.

This product is already improving as the community is developing it rapidly. More ML based algorithms should be added to it, to make it algorithmic-rich for developers.

Two and a half years.

No, I did not encounter any problems with the stability. It is also quite backwards compatible.

No I did not as of now, it is quite scalable. Using simple scripts you can add as many workers as you want.

This is a very good product for the big data analytics and integrates well with other parts like Machine Learning and graph analytics.

Spark Streaming, which allows you to construct event-driven information systems and respond to the events in near-real time.

Apache Spark’s ability to perform batch processing at one second or less intervals is the most transformative and less pervasive for any data processing application. The ingested data can also be validated and verified for quality early in the data pipeline.

Apache Spark as a data processing engine has come a long way since its inception. Although you are able to perform complex transformations using Spark libraries, the support for SQL to perform transformations is still limited. You can alleviate some of these limitations by running Spark within Hadoop ecosystem and by leveraging the fairly evolved HiveQL.

I've used it for 16 months.

The enterprise scale deployment of Apache Spark is slightly involved to derive its full potential of stability, scalability and security. However, some Hadoop vendors like Cloudera have integrated Spark data processing engine into their Hadoop platforms and have made it easier to deploy, scale and secure.

This is an open source technology and is dependent on community support. The Apache Spark community is vibrant and it is easy to find answers to questions. The enterprises can also get commercial support from Hadoop vendors such as Cloudera. I recommend enterprises to inspect Hadoop vendors’ commitment to open source as well as their ability to curate Apache Spark technology into the rest of the ecosystem before signing up for a commercial support or subscription.

The initial set-up is straightforward as long as you have picked a right Hadoop distribution.

I recommend engaging an experienced Hadoop vendor during the planning and initial implementation phases of the project. You will be able to avoid any potential pitfalls or reduce overall project time by having a Hadoop expert guiding you during the initial stages of the project.

I evaluated some other technologies such as Samza but community backing for Apache Spark stood out.

I also suggest having a Chief Technologist who has extensive experience in architecting several Big Data solutions. They should be able to communicate in business as well as technology language. Their expertise should range from infrastructure to application development and have command of Hadoop technologies.

Streaming telematics data.

It's a better MR, supports streaming and micro-batch, and supports Spark ML and Spark SQL.

It supports streaming and micro-batch.

Better data lineage support.

• RDDs
• DataFrames
• Machine learning libraries

Faster time to parse and compute data. It makes web-based queries for plotting data easier.

It needs to be simpler to use the machine learning algorithms supported by Octave (example polynomial regressions, polynomial interpolation).

I've been using it for one year.

There have been no issues with the deployment.

There have been no issues with the stability.

There have been no issues with the scalability.

We still rely on user forums for my answers. We do not use a commercial product yet.

The initial set-up was easy. I have not explored using this on AWS clusters.

We did an in-house implementation and development for our regression tool.

The ROI will be an in-house product to do machine learning analytics on data obtained from customer.

We did not evaluate any other products.

It's easy to use and has a learning curve.

I am using Apache Spark for the data transition from databases. We have customers who have one database as a data lake.

The most valuable feature of Apache Spark is its ease of use.

Apache Spark can improve the use case scenarios from the website. There is not any information on how you can use the solution across the relational databases toward multiple databases.  

I have been using Apache Spark for approximately 18 months.

Apache Spark is stable.

We are using Apache Spark across multiple nodes and it is scalable.

We have approximately five people using this solution.

The technical support from Apache Spark is very good.

I rate Apache Spark an eight out of ten.

DataFrame: Spark SQL gives the leverage to create applications more easily and with less coding effort.

We developed a tool for data ingestion from HDFS->Raw->L1 layer with data quality checks, putting data to elastic search, performing CDC.

Dynamic DataFrame options are not yet available.

One and a half years.

No.

No.

Spark gives the flexibility for developing custom applications.

It allows the loading and investigation of very lard data sets, has MLlib for machine learning, Spark streaming, and both the new and old dataframe API.

We're able to perform data discovery on large datasets without too much difficulty.

It needs better documentation as well as examples for all the Spark libraries. That would be very helpful in maximizing its capabilities and results.

I've used it for over nine months now.

I haven't encountered any issues with deployment.

There have been no stability issues.

I haven't had any scalability issues. It scales better than Python and R.

I haven't had to use customer service.

I haven't had to use technical support.

I previously used Python and R, but neither of these scaled particularly well.

The initial setup was complex. It was not easy getting the correct version and dependencies set up.

I implemented it in-house on my own!

It's open-source, so ROI is inapplicable.

Learn Scala as this will greatly reduce the pain in starting off with Spark.

\Spark Streaming, Spark SQL and MLib in that order.

We have been using Spark to do a lot of batch and stream processing of inbound data from Apache Kafka. Scaling Spark on YARN is still an issue but we are getting acceptable performance.

Like I said scalability is still an issue, also stability. Spark on Yarn still doesn't seem to have programming submission api, so have to rely on spark-submit script to run jobs on YARN. Scala vs Java API have performance differences which will require sometimes to code in Scala.

Have Scala developers at hand. Base Java competency will not be enough during optimization rounds.