Matillion ETL Reviews

While loading data into Snowflake, I encountered an issue with the key due to the file's large size and a record count in the billions. Loading the data with a Python script was taking a long time, so I decided to explore other options. This is when I discovered Matillion ETL, which I had not heard of before. I learned more about it and used some of its features, including the Material Data Loader, to load the data into Snowflake. Using Matillion ETL, I was able to load around 770 million records in just five to ten minutes. This was a successful use case, and I have also used Matillion ETL for loading data from Amazon S3 to Snowflake and for other data-loading tasks, including connectivity to on-premise servers and different cloud platforms.

I have used on-premise and cloud deployments of this solution.

The most valuable feature of Matillion ETL is the ETL. The solution is open-source which provides advantages, such as good performance and high efficiency. Additionally, it supports three data types which eliminates predefining the data, and we can write script models in Python.

We use it primarily for transferring data into our cloud data warehouse and conducting research. We rely on Matillion ETL for our data integration and transformation needs, finding its user-friendly interface and robust capabilities highly effective. Essentially, we've built a cloud-based data platform using Matillion ETL to seamlessly extract data from various sources, perform necessary transformations, and store it in our cloud environment.

In our finance-focused projects, we predominantly utilize Matillion for data warehousing tasks, particularly in the realms of finance and micro-lending. While our projects may not involve big data volumes, they often entail handling intricate datasets that require sophisticated processing.

The cloud-native design of Matillion has significantly impacted our video data workflows, providing a unified view across multiple data sources and serving as a centralized and secure platform. It offers a seamless experience, consolidating various functionalities into one interface, which simplifies our tasks considerably. Moving forward, we're eager to explore their SaaS offering, particularly for Redshift, once it becomes generally available. This scalable solution eliminates concerns about version upgrades, maintenance, and patching, offering a hassle-free experience.

Its versatility is impressive, as it allows us to seamlessly transition between different cloud platforms without being locked into a specific provider.

Matillion EDR is used for data loading. It extracts data from various sources, stages it in a data warehouse environment, and then performs orchestration and transformation jobs to automate processes across different layers of the data warehouse.

It serves as a development environment to build data pipelines. This is part of the entire data integration process, encompassing extract, transform, load (ETL) actions. It's used to run transformations on ETL data, preparing it for consumption.

We had a customer with on-premise systems. We extracted data from these systems and staged it in S3 or Azure Blob Storage. Then, Matillion picked it up for processing. The key advantage here was the speed of development.

I use Matillion ETL for wrangling or transforming data from sources like S3 and Databricks.

The most valuable feature of Matillion ETL is the UI experience in which you can drag and drop most of the transformation.

• It enabled an enterprise data-warehouse to be set up and operated, quickly and cheaply.
• The pipeline UI provides a means to present solutions to analysts and non-tech management for review and agreement.

• It works well with AWS Redshift: I have used Informatica, SnapLogic, and Talend and they do not work well with Redshift.
• Mid-pipeline data sampling: Without changes to pipelines, one can quickly and easily track down errors.
• Automatic database-object verification: A fundamental part of how Matillion works is ensuring objects, tables, columns, views, and other metadata are ready and available for use if, and when, a pipeline is started.
• Run-time parameters: These enable pipelines to be organized and modularized with minimum effort.
• An array of source-data components: Pulling data from wide tables, such as SFDC, can be setup in seconds. Using Sqoop to pull data to S3, for example, can take hours or even weeks to get right. It is worth noting that Matillion does not cost more if you need more source-data components (SFDC, Marketo, Google AdWords, RDS, MongoDB, etc.).
• A wide array of in-database DDLs and UDFs accessible from ETL are able to process unstructured data easily, without having to resort to EMR/Hadoop solutions.
• Integration with other AWS services: It can use Amazon SQS or SNS very easily to extend capabilities, such as doing micro-batch imports (near real-time updates) from source-systems.
• Python components using Boto and other libraries running on Matillion’s EC2: Matillion can push data from a data warehouse, via REST APIs, to target systems like DynamoDB and Marketo.

We use the solution to make data transfers between our source systems and Snowflake. It's our data analytics architecture.

The product has a good user interface. I don’t have to write too many codes. I can design or create general data transfer and technical jobs through simple texts. If I want to repeat a job, I can do it without creating multiple pipelines. I can create multiple jobs with a single pipeline. It is a fantastic feature of the product.

We primarily use Matillion ETL to effectively manage data's movement, ingestion, and transformation through pipelines. We have specific use cases that involve different types of data, but they all fall under the general bracket of data movement.

We've got customers that have used Matillion to stand up entire and data platforms. We have customers that use it on a daily basis to perform heavy data movements and pipelines. We have a whole load of different case studies for different customers for different technologies. For example, Optio used Matillion to actually build a full new data platform and those pipelines. 

We have some very unique use cases for the solution. We are interfacing between the on-premises database and on-cloud database– Oracle and Snowflake. It is a very complex process wherein we had to ask for help from Matillion’s engineers to build it out. We're looking up our on-premises database servers for information to build our cloud database. Once we get everything copied in Snowflake, we go back to Oracle on-premises. We have created this bridge to use till we switch to AWS full-time. But right now, we don't have that in the book. So the best way to move forward is to make sure we're taking a solution that could bridge that gap, and that's Matillion.

Matillion’s ability to bridge the gap between on-premises and on-cloud helps us prepare for the future while building our platform. If you plan to switch to AWS from on-premises, using Matillion allows you to keep everything in sync as you port the data. They also have a CRM sync function that allows you to make changes to the on-premises database and automatically sync it up with the material inside the cloud system.