Databricks Reviews

I am working as a data engineer at Fractal. On a daily basis, I work on Azure Cloud, and I use Databricks frequently. We have EDF pipelines and utilize Synapse for our daily tasks.

Databricks offers various courses that I can use, whether it's PySpark, Scala, or R. I can leverage all these courses in a single notebook, which is beneficial for clients as they can access various tools in one place whenever needed. This is quite significant. 

I usually work with PySpark based on client requirements. After coding, I feed the Databricks notebooks into the ADF pipeline for updates. Databricks' capability to process data in parallel enhances data processing speed. Furthermore, I can connect our Databricks notebook directly with Power BI and other visualization tools like Qlik. Once we develop code, it allows us to transform raw data into visualizations for clients using analysis diagrams, which is very helpful.

We work on three platforms. Databricks is hosted on Azure for us, so we work with ADFS, Azure Data Factory, and also the AWS Cloud. We work for some customers.

The notebooks and the ability to share them with collaborators are valuable, as multiple developers can use a single cluster. This reduces costs. The scheduling part is managed by Databricks itself, for example, when it is idle, it will automatically turn off. All these features are handled by Databricks, reducing costs. We do not need to schedule separately. 

For example, on AWS EC2, we have to create a Lambda function or use System Manager templates to schedule EC2 and EMRs. Here, it is taken care of, saving significant resources. 

Additionally, notebooks can be shared within the development team which saves effort. Developers can share their notebooks. Git and Azure DevOps integration on the Databricks side is also very helpful.

I usually handle data ingestion and create warehouses. I also assist other teams, such as analytics, to create reports or perform other tasks.

Having one solution for everything, from data engineering to machine learning, is beneficial since everything comes under one hood.

Our company uses the solution for big data and as an interface for analytics. 

We also create custom APIs to get data and provide SQL endpoints so users can access it over traditional tools like JDVC or ODBC. 

We use the solution on AWS and Azure. The data lake is wide open for departmental use. We have ten departments and two or three people from each department access the solution. 

The platform as a service allows us to ramp up a new database pretty fast. We deploy some of the infrastructure as a code. End users can access data immediately and connect with Power BI for reporting. 

I use Databricks for various purposes, including data engineering, MLOps, machine learning training and deployment, the entire ML cycle, and dashboards. It serves different purposes for different projects.

Unity Catalog is a feature I am currently using extensively. I am migrating many projects to Unity Catalog. MLflow, which I use for model registering and creating the lineage of models, is also valuable. 

Additionally, Databricks serves as a single platform for conducting the entire end-to-end lifecycle of machine learning models or AI ops. I don't need to switch between various tools, making it an all-encompassing solution for development and research. I use the lake house and utilize features effectively.

I primarily use Databricks to process large-scale data sets with Apache Spark. My main use case is processing large data sets, such as 600 GB or 800 GB.

Databricks integrates natively with Apache Spark, which I use as a processing engine for large-scale datasets. This native integration is one of its strengths. Another strength is that the platform makes it very easy to manage resources. For example, setting up a cluster of five or fifteen nodes is straightforward with Databricks. The notebook environment is also excellent, making it easy to perform various tasks.

We use this solution for finding anomalies and applying the rules to the streaming data.

There are around 50 people using this solution in my organization, including data scientists.

The ability to stream data and the windowing feature are valuable. There are a number of targeted integration points, so that is a difference between Stream Analytics and Databricks. The integrations input or output are better in Databricks. It's accessible to use any of the Python or even Java. I can use the third party, deploy it, and use it.

We work with clients in the insurance space mostly. Insurance companies need to process claims. Their claim systems run under Databricks, where we do multiple transformations of the data. 

The elasticity of the solution is excellent.

The storage, etc., can be scaled up quite easily when we need it to.

It's easy to increase performance as required.

The solution runs on Spark very well.