Fat fingers and failed businesses - Managing Data Quality

The quality of a company's data is an often overlooked facet of many business operations. This article introduces the concept of data quality, the impacts to your business, why you should care, and what you can do about it. Read on to see how a recent client's exit strategy valuation was facing a dramatic reduction because of preventable data quality issues:

Defining data quality

Symptoms of poor data quality

• Information in the wrong fields. e.g. mailing addresses stored in physical address fields and vice-versa, email addresses in phone number fields
• Business process information in descriptive fields. e.g. using the comments field to keep track of where a particular file/customer/application is at in it's workflow
• Information simply not recorded. e.g. discounts or rebates provided to clients not recorded quantitatively during the initial transaction
• Inconsistencies of data entry. e.g. NSW vs New South Wales, 20 vs 0.20 (for percentages)
• Same or similar information stored in multiple locations. e.g. in a multitude of spreadsheets or databases
• Colloquial information. Sometimes, in the heat of the moment, we record subjective commentary which could later be seen by the wrong person.

Business implications of poor data quality

1. Staff downtime: Even though there are software tools available to cleanse data; time is required to investigate and amend incorrect, missing or ambiguous information. This time can have a direct cost to the business through lost employee productivity.
2. Strategic: While data is being cleansed; some business initiatives may need to go on hold.
3. Business workflow: Too often, we have seen business workflow information stored in the comments fields on a particular item or record. These include an item's current status, who is handling it, what needs to happen next, etc. By storing this information descriptively it is practically impossible to get an overview of the business' operations, and to provide auditability of processes.
4. Marketing: As any sales & marketing professional will attest to; the better targeted a campaign the more effective it will be. So, if customer contact information, as well as their purchasing or service activity with the firm, is not captured accurately; it becomes difficult to segment the customer database for targeted marketing efforts.
5. Compliance: Correctly recording the dates, amounts, and GST implications of each and every financial transaction.

6. Business intelligence: Regardless of the tools used companies looking to implement reporting for strategic analysis or for day-to-day business transparency are often hampered by not having their raw data available or reliable in the first place. No reporting tool, however sophisticated, can report on data which does not exist.
7. Technology: Vendors may have scripts and tools available but there may be cost implications for migrating or upgrading your software. This cost may be in purchasing licenses for tools or in consulting hours to configure them to run over your databases.
8. Cashflow timing: Businesses who do not make the capture of clean data an integral part of ongoing business operations and the associated incremental on-costs, are in fact deferring the inevitable data cleanup to a lump-sum project cost.
9. Financial reporting: It may be difficult to report on the true profitability of product and service sales if discounts, rebates, and incentives are not captured consistently and accurately.
10. Exit strategy: Unfortunately, we have seen the value proposition of a client's exit strategy reduced because the core data of the business was impaired and could not be integrated into the purchaser's systems without significant cost.

Sources of poor data quality

• No clear, unambiguous definition of data quality for the department or organisation
• Lack of awareness by staff on the importance of good data practices
• Lack of management awareness and enforcement of data policies
• System limitations on entering quality data
• Lack of caring or engagement by staff, or outsource partners
• Low data quality from external parties
• Corruptions during system upgrades or migrations

Strategies to address data quality

1. The first step in any data quality strategy is to define what good data looks like for your organisation. This includes, for any particular context, defining the minimum information to be captured, which fields or systems should be used for the data, and guidance on what should be entered where there is potential for variation (e.g. must enter percentages as a value between 0.0 and 1.0 - say 0.33 for 33%)
2. Secondly, it is important to understand how you are going to use the data as it will influence how resources are deployed to address any issues. When faced with setting policies or a data cleanup effort; the highest priority items, such as customer contact details or core product specifications, will need to be worked on first.

• Training may be required to ensure staff are well versed in data quality policies, and are aware of the business implications of non-compliance. If your systems are not able to provide input validation; your staff become the last line of defence, and so must be educated on how to help the business by recording data of a high quality.
• Most businesses do not have the luxury of starting with a clean slate, and will need to resolve existing data issues. Data generally needs to be corrected with a combination of technology tools and manual intervention. Tools can help identify bad records (e.g. this email address has no @ symbol), make rudimentary corrections (New South Wales = NSW), and fill in standard or easily assumed data (if State is one of NSW, QLD, VIC etc, then Country = "Australia"). Tools however, cannot create data which does not exist, and cannot extract real data from descriptive comments. For this, staff or external consultant time is needed to resolve.

• There are many trade-off's to be considered when deciding how to undertake a data cleanup. Depending on the situation, businesses may choose to undertake a "big bang" approach to data cleansing or a more incremental approach. Other trade-off's may include quarantining business processes while the data is being cleansed, taking steps to mitigate the risks of wholesale updates to financial data, and being able to report on both clean and unclean data concurrently. When working with large datasets or high risk data; it is wise to get some expert help with the process.
• Ensure data from external sources is clean before importing it into your systems. Where you can influence your suppliers, create data guidelines for information coming into your company. Alternatively, consider having your IT provider create or customise a script or tool that verifies the data meets minimum quality standards before importing.
• Do not wait for the business to suffer significantly from poor data quality. Put data cleanup plans in place, and execute them. Business owners, or the firm's management, will need to hold all relevant stakeholders accountable for their commitments to meeting the milestones of the plans.

Summary

So, what strategies do you have in place to manage your data quality?