Dremio Reviews

I use Dremio for proof of concept purposes. I haven't used it in a real-time project, however, I explore Dremio as a data virtualization application in the ecosystem. It is relatively new, possibly a one-year or two-year-old system.

It's almost similar, yet it's better than Starburst in spinning up or connecting to the new source since it's on SaaS. It is a similar experience between the based application and cloud-based application. You just get the source, connect the data, get visualization, get connected, and do whatever you want. 

They say data reflection is one way where they do the caching and all that. Starburst also does the caching. In Starburst, you have a data product. Here, the data product comes from a reflection perspective. The y are working on a columnar memory map, columnar computation. That will have some edge in computation.

I can visualize traffic from BI and Tableau on the same page and have my tables and schema on the same page. The data link comprises everything. If I want one structure, I connect it to a big table in the hive and the data team that could read my SQL work on my tables, schemas, table structures and everything all in one place. Dermio is as good as any other Presto engine.

Everyone uses Dremio in my company; some use it only for the analytics function.

Dremio is a platform that enables you to perform high-performance queries from a data lake. It helps you manage that data in a sophisticated way. The use cases are broad, but it allows you to make extremely good use of data in a data lake. It gives you data warehouse capabilities with data lake data.

Dremio enables you to manage changes more effectively than any other data warehouse platform. There are two things that come into play. One is data lineage. If you are looking at data in Dremio, you may want to know the source and what happened to it along the way or how it may have been transformed in the data pipeline to get to the point where you're consuming it. 

There's another thing called data providence. They're tied together. Data providence allows you to go back and recreate the data at any particular point in time. It's extremely important for compliance and governance issues because data changes all time. How did it change? What was it three days or months ago? You may have made some decisions based on data that was three months old, so you might need to revisit those.

It's essential for things like machine learning and deep learning, where you are generating AI models off data. When the model stops working or doesn't work as expected, you need to figure out why. You have to go back and adjust the datasets used to train the model. We do that through an open-source project called Nessie, which is their basis for providing data lineage and data province capabilities. It's super powerful.

Arrow is another open-source project for storing data in memory and performing data query operations. Data sits on a disk in one format. If you want to do anything with data, you have to load it into your computer and put it into memory so you can work with it. Arrow provides a format in memory that enables the whole library to perform various operations on that data. Every vendor has its own way of representing data in memory.

They've latched onto an industry standard and developed it so it's open. Now people can use the exact same format in memory to do operations and use the library set to perform functions on data. New developers can decide if they want to develop their own memory format or use one that's already there.

Data transfer is a massive problem when you're working with large datasets, doing advanced analytics, and trying to train machine learning or deep learning models. What happens often is companies downsample their data sets to do training on models because transferring and managing data on a deep learning or machine learning platform is too much.

We use Dremio for financial data analytics and as a data lake. We connect Dremio with Oracle, Docker, MySQL, and utilize it for Power BI. 

Additionally, we use it to process data from MongoDB, although we face occasional challenges with NoSQL integration.

Dremio is very easy to use for building queries. It's easy to connect Dremio to various databases and data sources like Oracle and MySQL. It is also very flexible, providing us with scalability and integration capabilities effortlessly.

I have used this solution as an ETL tool to create data marks on data lakes for bridging. I have used it as a greater layer for ad-hoc queries and for some services which do not require sub-second latency to credit data from very big data lakes. I have also used it to manage simple ad-hoc queries similar to Athena, Presto or BigQuery. 

We do not have a large number of people using this solution because it's mainly setup as a service to service integration. We integrated a big workload when we started using Dremio and this was very expensive. The migration is still in progress. As soon as this migration is finished, we plan to migrate ad-hoc queries from our analytical team. 

The native error interface has been most valuable. With Apache Airflow, which Dremio supports, you don't need to digitalize data. Dremio gives you the ability to create services which do not require additional resources and serialization. 

We have been using it to build one of our frameworks. We primarily use Dremio to create a data framework and a data queue. It's being used in combination with DBT and Databricks.

We're still in the exploration phase with Dremio, so it's a bit early to determine its most valuable feature. We're currently deploying it across different departments for various use cases and learning from these internal applications.

We use Dremio for data engineering.

Dremio has resolved my data lineage and data governance problems. The solution has also resolved the data availability for a different range of users, which used to be a problem.

We were using Dremio as a data lake query engine tool. We were creating our PDSs and VDSs on top of our S3 buckets, and our data lake and the data-scientist teams were using the data for further processing. We didn't use it for any ETL jobs. We were using it as a data-lake tool.

The most valuable feature of Dremio is it can sit on top of any other data storage, such as Amazon S3, Azure Data Factory, SGFS, or Hive. The memory competition is good. If you are running any kind of materialized view, you'd be running in memory.