Blog Administrator | Analyzing Data Quality, Data Management, Data Quality | accuracy, completeness, data management, data quality, data quality managers, data quality standards, Metadata

By Elliot King

 If you can’t measure a process, you can’t improve upon it. That’s one of the
ironclad quality initiatives. Edward Deming’s revolutionary insight was that if
you can measure a process or outcome continually, you have created the
opportunity to improve it continually.

But measurement is only a starting point to quality improvement. Those
measurements have to be assessed according to specific standards and then their
impact on business operations has to be assessed. Only at that point can
managers determine whether investing in remediation is appropriate or cost
effective.

In many data quality improvement programs, data scorecards are the tools of
choice for data managers to place raw statistics into meaningful contexts. From
that point, different constituencies can identify problem areas and determine
what actions should be taken.

To understand the idea of a data scorecard, a baseball analogy actually works
here. Broken down into its granular bits, a baseball game consists of strikes
and balls, hits and outs.

But in most contexts (although not all), to know that there were 210 pitches, 13
hits and 8 runs scored has little value. That data has be to combined and
grouped correctly to have meaning. And different stakeholders–the players, the
manager, the front office, the opposing teams–may want that data in different
formats.

For data scorecards, the process of measuring and aggregating raw statistics
into useful combinations is called a hierarchical rollup. It could look like
this. At the most granular level, data is measured against the metadata
associated with the database. Are fields the right length? Is there missing
data? These statistics are of interest to database managers.

The next step is to assess whether the data meets data quality standards such as
accuracy and completeness. This is a concern to data quality managers. Does the
data conform to business rules? Business analysts need to know that. And what is
the impact of data quality on process outcomes? As that point, management can
determine what actions are prudent.

In baseball, a glance at a scorecard reveals the outcome of the game and who
played poorly as well as offering the chance to drill down deeper to the
underlying raw data if need be. A data scorecard plays the same role in data
quality improvement.