Approaching Data Quality in Today's World

reviewer1925439

Manager at HCL Technologies

Published:Jul 26, 2022

Why Data Quality Is Important

Every organization needs good data quality to ensure the right business decisions are made. These days organizations are adopting new data cleansing strategies to elevate and ensure the quality of enterprise data used for analytical purposes.

Data Quality can also be defined as a measurement of how to fit a data set is to serve the specific needs of an organization. High data quality will result in trusted business decisions.

The success of data-driven initiatives for enterprise organizations depends largely on the quality of data available for analysis. The axiom can be summarized simply as garbage-in, garbage-out, low-quality data that is inaccurate, inconsistent, or incomplete often results in low validity data analytics that can lead to poor business decision making.

To fix data quality issues, an organization should implement data cleaning strategies to ensure high-quality data feeds into data analytics applications and business intelligence initiatives. Implementation of an effective data cleansing strategy should expect more accurate insights, increased productivity, and increased business efficiency.

Data Quality Reality

An Experian report identified that companies globally feel that 26% of their data is inaccurate.
Gartner.com suggest that organizations lose between $10 – $14 Million USD annually due to poor data.
MIT Sloan reported that employees spend half of their time coping with managing data quality tasks.

Impact of Data Quality

80% of a data mining effort will be spent on addressing data quality issues.
Poor Data Quality results in inaccurate data mining outcomes and poor business decisions.
Reports based on poor data quality can result in Lost revenue and or reputational damage.

Data Quality Dimensions

Data cleansing is the process of identifying and correcting issues that impact the overall quality of a data set across five dimensions of data quality below:

Accuracy – Ensuring that the recorded data values are as close as possible to the “true” values.
Completeness – Ensuring that all required data is present in the data set.
Consistency – Ensuring that data values are consistent within the same data set and/or between data sets.
Uniformity – Ensuring that data is specified to a uniform standard, including things like units of measure and significant figures.
Validity – Ensuring that data conforms to predefined business rules.

Data Cleansing Use Cases

B2B Data Cleansing

Compared to business-to-consumer (B2C) sales, business-to-business (B2B) sales are usually characterized by a higher price point, a longer sales cycle with more customer touchpoints, and multiple customer stakeholders. To manage this complexity, organizations that sell B2B use a customer relationship management (CRM) software tool to collect, store, and organize structured data about prospective customers.

Sales teams use data from the CRM to manage relationships with prospective customers at every step in the marketing/sales funnel. As a result, sales agents depend on the accuracy and completeness of CRM data to be productive in their roles. When CRM data is incomplete, inaccurate, or duplicated, agents waste time manually searching for phone numbers and email addresses instead of generating high-quality conversations with prospects.

An effective data cleansing strategy for our B2B example might include tactics like:

Removing duplicate CRM entries
Removing incorrect or outdated contact information
Standardizing data between marketing and sales teams
Appending missing contact information from other sources

Log Data Cleansing

Applications, network devices, and endpoints all generate log data that can be analyzed to support IT functions like network security and application performance monitoring. Log data is machine-generated and written into log files, usually as unstructured or semi-structured text data. Before this data can be effectively analyzed, it must be captured, stored, parsed into a machine-readable format and cleaned to ensure high data quality.

An effective data cleansing strategy for our log data example might include tactics like:

Identifying and parsing log data automatically (using a data platform)
Removing duplicate logs to save storage space
Removing or selectively retaining logs with a specific status code
Standardizing the format of log data from multiple sources

Transactional data cleansing

Organizations generate transactional data whenever they make a purchase or complete the sale of a product or service. Transactional data includes information about the customer (personal data, payment card information, etc.), the product or service being sold (name, price, SKU number, etc.), as well as transaction metadata (sale ID, timestamp, etc.).

Business analysts and accountants rely on high-quality transactional data to develop insights that help the organization better understand the behavior of its customers, identify high-performing products and services, and measure its financial results.

An effective data cleansing strategy for our transactional data example might include tactics like:

Removing credit card information to comply with the PCI DSS standard

Anonymizing data to protect consumer privacy

Converting coded data fields into a human-readable format

Standardizing transactional data formats across multiple revenue channels

Data Cleansing Strategies

Build a business case with a problem defined for strategic data cleansing

A business case needs to be built by properly defining what business outcomes an organization expects and for what duration of time. Once the case is created there should be a proper connection needs to be established between data quality improvements and enhanced business results.

Building the business case for data cleansing within your organization requires a clear understanding of your strategic business goals and how those goals might be supported by enhanced data quality. You’ll also need to identify KPIs that can be used to measure the performance of data cleansing initiatives and estimate the financial impact of improving the quality of your data.

Realistic Data Quality Plan with achievable time limits and measurements

A data quality plan should identify which types of data will be targeted and the biggest quality issues present in those data sets. It should identify which data cleansing tactics and techniques will be applied and which software tools will support the process. Your plan should also establish roles and responsibilities, along with a clear definition of success for your data cleansing initiative. There should also be measurement metrics that need to be decided based on certain time intervals and how much data quality needs to be achieved in what time period. For example, if there are 100K vendor records, there should be a clear plan defining if 60K records will be cleansed in the next 4 months of time and this will impact decision-making at a certain level and is directly proportional to the business benefits.

Business Validation Rules application to data sets

Standardizing data as it is captured is one of the easiest ways to enhance the consistency and uniformity of data collected by your organization. This means applying data entry standards, such as requiring specific data fields to be completed in a valid format before the data is submitted to your organization or added to a database.
This improves data quality by validating it at the point of entry. Information like phone numbers, emails, and credit card numbers can be validated by software or authenticated by the user in real-time to reduce the number of false entries and preserve the integrity and usability of data sets.

Choosing the right data cleansing tools and techniques

Need to dig deep to understand what kind of data is causing issues, what are the different data sets, how data quality can be improved upon, etc. One should answers the questions below:

What data fields are most important for this data’s intended purpose?

Are required data fields often missing? How should we address that (by appending data, removing the entry, sourcing it from elsewhere, etc.)?

How should data in each field be formatted?

How should similar data from multiple sources be standardized or normalized?

There is a number of tools available in the market for pursuing data quality operations on different data sets. However, it becomes crucial to thoroughly study what tool can be best utilized for the purpose. There are cloud-based and on-premise versions available. Some are best suited for SAP-driven landscape and connectors. So, it's very important to decide the tool's purpose to execute the quality rules and generate reports.

Utilize cloud storage for cleansing purposes

Organizations that store data in the cloud can use software solutions to clean, prepare, and transform data directly in cloud storage buckets.

While traditional databases use a schema-on-write approach that makes it complex and time-consuming to clean and process data, look for a solution with a schema-on-read approach that gives you the ability to apply customized data cleansing strategies to data directly in the cloud storage with no need for data movement or reindexing. With the ability to strategically cleanse data in cloud storage buckets, organizations can save time and money in the data cleansing process while accelerating time-to-insights and maximizing the value of their data.

Automate Cleansing process

Software technologies that automate the data cleansing process help organizations accelerate the development of insights and reduce the cost of maintaining high-quality data sets.

Data cleansing automation can be executed using Regular Expression (RegEx) functions, scripts that check for patterns in strings of text and execute predefined operations on them. Regex expressions can be used to clean and transform data in a variety of different ways, ensuring its quality and preparing it for use in business analytics applications.

To conclude:

It’s time to start building the business case for data cleansing, documenting a data quality plan, and investing in technologies that automate the data cleaning process and accelerate time-to-insights for data.

References: