We are a consulting house and we employ solutions based on our customers' needs. We don't generally use products internally.
I am a certified data engineer from Microsoft and have worked on the Azure platform, which is why I have experience with Databricks. Now that Microsoft has launched Synapse, I think that there will be more use cases.
You can spin up an Azure Databricks clustered, and integrating with it is seamless.
The integration with Python and the notebooks really helps.
There is definitely room for improvement.
This is the type of solution where you need to have people with technical expertise to use it. Other products are self-service and can be employed by end-users. Databricks is not geared towards the end-user, but rather it is for data engineers or data scientists. I'm not sure whether Databricks is working towards it, or not.
It would be nice if it were more user-friendly, where you don't have to rely on Power BI or a visualization tool. I know that there is integration in the notebook where you can do it, but still, the relationships and semantics make it more difficult. It would be better to do it right in Databricks. You could put them within the portal and I don't have to log out and bring that into Power BI and then visualize.
We have not done any major implementation yet, although I think it's stable to an extent. I can't comment on it in terms of benchmark and experiencing any issues. It works seamlessly in the places where I've used it.
Our implementations have been small and we haven't needed to scale as of yet.
Databricks can help you to build a data lake, and it's something that they need to help make more popular. People are slowly understanding it because if you look, there are lots of data lakes that people are trying to create. I'm not intimate with it, but the concept seems complicated. I think they need to write up something where videos can explain it better. What I have seen on YouTube is quite complicated for an end-user to understand.
The initial setup is easy. It's not difficult when you are used to Azure.
I am based in South Africa, where it is expensive adapting to the cloud, and then there is the price for the tool itself.
The cost is difficult to estimate. I've got customers who went to the cloud and then they realized that the costs were more, compared to what they used to be on-premises. Also, because our exchange rate is so weak, I would always advocate that prices being lower is better, although I don't know how feasible it is.
From a purely technical perspective, I would rate Databricks and eight out of ten. However, there is a failure in terms of user adoption. After I look at other products, including Synapse, those are better. I still feel that Databricks is quite complicated for the average person.
I would rate this solution a five out of ten.