Azure Data Factory Reviews

I am a solution architect and work with the presales team. I generally provide solutions, study Azure Data Factory, and conduct POCs. I use Azure Data Factory as an ETL tool. I have been using it for about three years.

Azure Data Factory is a low code, no code platform, which is helpful. It provides many prebuilt functionalities that assist in building data pipelines. 

Also, it facilitates easy transformation with all required functionalities for analytics. Furthermore, it connects to different sources out-of-the-box, making integration much easier. The monitoring is very thorough, though a more readable version would be appreciable.

Our company uses the solution as a data pipeline. We get information outside the cloud from our factory such as data relating to production. We categorize it, clean it up, and transfer it to a database and data model. From there, we analyze the data using BI and other things. We gather information in data lake products like Microsoft Synapse and Microsoft Data Lake. 

We have two to three administrators who use the solution in a quite standard, mainstream way with nothing extreme. They handle administration, security, and development. 

It is difficult to define the total number of users because that depends on the number of data factory agents. We built the solution to have a different data factory agent for every customer. For example, if we have ten customers then we have ten users. We hope to increase usage but growth depends on our marketing efforts and how well we sell our products. 

The data factory agent is quite good and programming or defining the value of jobs, processes, and activities is easy. We have the agent installed on-premises in order to gather information.  

The cloud includes all kinds of API connections so we can easily gather information from other services. 

The solution seamlessly integrates with the Azure infrastructure. 

The current use is for extracting data from Google Analytics into Azure SQL Database as a source for our EDW.  Extracting from GA was problematic with SSIS. 

The larger use case is to assess the viability of the tool for larger use in our organization as a replacement for SSIS for our EDW and also as an orchestration agent to replace SQL Agent for firing SSIS packages using Azure SSIS-IR.

The initial rollout was to solve the immediate problem while assessing its ability to be used for other purposes within the organization. And also establish the development and administration pipeline process.  

ADF allowed us to extract Google Analytics data (via BigQuery) without purchasing an adapter.  

It has also helped with establishing how our team can operate within Azure using both PaaS and IaaS resources and how those can interact. Rolling out a small data factory has forced us to understand more about all of Azure and how ADF needs to rely upon and interact with other Azure resources.

It provides a learning ground for use of DevOps Git along with managing ARM templates as well as driving the need to establish best practices for CI.  

We are a partner and operate as a reseller. Our main area is as a service provider, working with businesses in building their data pipelines, data engineering, and data analytics solutions. We often work with clients on their infrastructure, with Data Factory being a key component more commonly in these situations.

The interface of Azure Data Factory is very usable with a more interactive visual experience, making it easier for people who are not as experienced in coding to work with. The scheduling management of pipelines and how the environment is set up allows for effective troubleshooting and identifying issues quickly. Alerts and warnings make it easy to diagnose pipeline errors.

We mainly use it to migrate data from on-premises sources to the cloud, such as Oracle and Cisco servers. It's a good solution for integrations within the Azure environment, and it connects well with other Azure data products. However, for external configurations, we use Informatica Cloud or Informatica Data Accelerator (IDA).

For automation, we primarily rely on Snowflake and Informatica. Our strategy is not to depend on a single tool. When it's strictly on-premises to cloud, we use ADF. 

Otherwise, Informatica is more mature and integrates well with various third-party products. We also use Snowflake copy commands to load data into Snowflake. Azure Data Factory doesn't fully meet your automation requirements.

We use Informatica for pipelines that were originally in SSIS, but for new pipelines and ETL processes, we choose either Informatica or Snowflake scripts.

We have multiple banking applications running on SSIS pipelines. We're in the process of upgrading them to a hybrid cloud architecture. For that, we use Azure Data Factory to move data from on-premises to the cloud – mainly for back-end database operations and ETL transformations.

We primarily use it to load data from an on-premises SQL Server to either Blob storage or an Azure SQL data warehouse. For other integrations, especially those outside of Azure, we tend to use Informatica Cloud Services (ICS).

For structured data loading, we use it However, we use Informatica for unstructured or semi-structured data. We also use Snowflake for ETL processes and sometimes for streaming. 

In my opinion, ADF isn't as suitable for streaming – for streaming, Snowflake streamlets or Informatica structured streaming are more reliable. ADF works well for batch processing, though.

I find the most valuable feature in Azure Data Factory to be its ability to handle large datasets. The data is more scalable.

There is a problem with the integration with third-party solutions, particularly with SAP.

I can describe a scenario where I was tasked with developing a beta data trace and integration system. GitLab served as the data integration platform responsible for creating pipelines to extract data from various sources, including on-premises and cloud-based systems. Alongside GitLab Data Factory, we utilized Azure Logic Apps for orchestration and Azure Key Vault for securely storing data-related information. This combination enabled us to manage data extraction, transformation, and loading efficiently. The solution also involved Azure Data Lake for further data transformations, culminating in a comprehensive data processing engine that I played a key role in implementing.

The workflow automation features in GitLab, particularly its low code/no code approach, are highly beneficial for accelerating development speed. This feature allows for quick creation of pipelines and offers customization options for integration needs, making it versatile for various use cases. GitLab supports a wide range of connectors, catering to a majority of integration needs. Azure Data Factory's virtual enterprise and monitoring capabilities, the visual interface of GitLab makes it user-friendly and easy to teach, facilitating adoption within teams. While the monitoring capabilities are sufficient out of the box, they may not be as comprehensive as dedicated enterprise monitoring tools. GitLab's monitoring features are manageable for production use, with the option to integrate log analytics or create custom dashboards if needed.

The data flow feature in Azure Data Factory within GitLab is valuable for data transformation tasks, especially for those who may not have expertise in writing complex code. It simplifies the process of data manipulation and is particularly useful for individuals unfamiliar with Spark coding. While there could be improvements for more flexibility, overall, the data flow feature effectively accomplishes its purpose within GitLab's ecosystem.

The primary use case is integrating data from different ERP systems and loading it into Azure Synapse for reporting. We use Power BI for the reporting side of it.

We also have customers who are migrating to Azure Data Factory and we are assisting them with making the transition.

It is easy to integrate.

I do not foresee any issues with security.