Data Quality Assessment Manager – Quality Control
Business Intelligence takes guesswork out of decision making. However without enterprise data quality, you are taking a shot in the dark. Organizations can no longer ignore poor data quality when the writing is on the wall. Data quality is all about confidence in a data driven world that has become more and more complex.
Data, data everywhere but not a single insight to use. Living in an information age where we have to make decisions quickly, and often with precision, what separates the winners from losers is the ability to leverage the data with confidence. Today many organizations use some form of Business Intelligence tool or tools, for obvious reasons, yet they’re far away from achieving the desired results. These tools can turn data into information, but bad data is still the root-cause of bad information.
The surprising ways poor data quality affects your business
If your BI insights are not hitting the mark, data quality is the one thing standing between you and your business success. While it is difficult to quantify without the proper tools, experts estimate that hundreds of millions of dollars are poorly spent, misallocated and simply lost due to poor or unknown dataquality. “Poor data quality also effects operational efficiency, risk mitigation and agility by compromising the decisions made in each of these areas” according to Gartner.
From another view, among the biggest challenges is regulatory compliance and transparency which is now becoming the top concern for many organizations. Thanks to regulatory compliance including Basel III, Solvency, and SOX for example, data accuracy and the ability to demonstrate a mastery of your data has now become critical. Organizations have got to become data-driven to survive, and the quality with which they drive the business will determine their level of success.
What poor or unknown data quality means for your business
Organizations are realizing the impact of poor data quality problems, but in many cases, they are far from understanding the magnitude or scope of the issue. Some have to come believe that a problem exists, but do not have any clue about the impact of data quality on their business, or a viable plan to fix it. If Gartner findings are any indication of the impact, “Poor data quality is a primary reason for 40% of all business initiatives failing toachieve their targeted benefits.
Understanding the business impact of poor quality data
Gartner predicts that, by 2017, a full 33 percent of Fortune 100 organizations will experience an information crisis, due to their inability to effectively value, govern and trust their enterprise information – which of course is rooted in their data assets and the data assets they acquire.
Big Data challenges and opportunities: Making sense of it all
Meeting the challenges of big data involves a lot of questionsin terms of how to get most out of the data analytics and how to leverage them to make better decisions. These questions and moreover their answers, are vital and should be at the top of mind for all Business leaders.
Barriers to data-based business decision making
Decision making is often not precision, but rather “Directionally Correct” because of the barriers to accessing quality data or understanding the quality of the data we own. Often bad data quality acts as a barrier to effective business decision making, not because we know it is bad, but we sense that it is – without a way to prove or disprove that instinct nominally. Ultimately , poor data quality will have a significant impact on decision making due to uncertainty and this, over time, will negatively impact an organization’s cost of doing business and drive down the bottom line.
Tracing the root of data quality problems
The key to resolving data quality issues lies in understanding the root-cause of the problem. We need to get to the root of the problem to arrive at a solution – start with the end in mind. Below are some of the most common causes of data quality issues, and where issues begin:
- Purchased or Unknown Data
- Data drawn from multiple places with no lineage
- Poor data handling practices internally, making bad data worse
- Not adhering to data entry standards, or governance and stewardship program
- Post Migration issues – good or even mediocre data becomes worse through bad practices.
What is the role of IT in fixing data quality problem?
IT will play a big role in fixing many data quality issues – but in many cases are limited by the tools or lack of tools that they have. Smart organizations will understand the need to invest in a robust solution or combination of solutions including data sourcing, profiling, cleansing, matching, scoring, and data mapping with true automated enterprise lineage and impact analysis capabilities.
We don’t know what we don’t know:
“Many companies are trying to improve data quality ‘by hand’ or with home-grown solutions and tactics. Often this comes in the form of writing their own correction or cleansing routines, and scripting, and usually doing so independently rather than part of an organized program with consistency and an eye toward a program level approach. The best approach will involve using purpose-built tools to pursue these efforts. While this recommendation comes from industry analysts at Gartner, there are many solutions not included in their analysis and quadrants, often due to the cost to be a part of their programs.
What is Data quality assessment and why it is important?
Data quality starts with an effective Data quality assessment. DQA, as it is also known, will help analyze data using different techniques like Pattern analysis, Length analysis, Form analysis, Case analysis, Value Filtering, etc.. These combined techniques give the user a full view of the data that can help in making an assessment – whether anecdotal or empirical – so that they can start to score the data in terms of its quality, and its value to the organization. The assessment results can help determine the accuracy, completeness, consistency, precision, reliability, uniqueness and validity of the data – which is the first step in trusting the data.
How to choose a right data quality assessment:
Choosing the right data quality assessment methodology can be an arduous task. There are a variety of data quality tools on the market, most make similar promises, and come with very a very large cost of entry. Often these tools are “standalone” products that don’t integrate with other tools and components of the architecture, making it difficult to leverage any results and perpetuate the process into the integration practice.
Enterprise level Data Quality Assessment products should bring together all aspects of the enterprise data management process, with the assessment, scoring and dashboards that make the next step “actually fixing the data” and integrated, organized, and efficient approach. This combination will not only make the process more efficient, but will provide a feedback loop that allows users to quantify and visualize all efforts made to improve the overall quality of the data – rather than a snapshot that takes as much work to gather the second or third time as it did the first.
Precision decision making that drives companies to success cannot come through poor quality data. Unless a company can definitively prove the quality of their data, they are working in a way that can be considered ‘directionally correct’ at best. Survival in today’s data driven market will require an enterprise approach to resolving data quality issues that includes a “full-circle” approach to profiling, scoring, and correcting the data issues at hand. The purpose of data quality is to help businesses derive massive value from their data. The question that needs to be asked is not “how much will it cost to procure the tools to fix our data quality?” but rather “How much is our lack of data quality awareness costing us?”