Data Quality Assessment: Value Domain Compliance

Melissa Team | Analyzing Data, Analyzing Data Quality, Data Profiling, Data Quality, Data Quality Assessment, USPS | , , , ,

By David Loshin

To continue the review of techniques for using column value analysis for assessing data quality, we can build on a concept I brought up in my last post about format and pattern analysis and the reasonableness of data values, namely whether the set of values that appear in the column complies with the set of allowable data values altogether.
Read More

Data Quality Assessment: Value and Pattern Frequency

Blog Administrator | Analyzing Data, Analyzing Data Quality, Data Profiling, Data Quality, Data Quality Assessment | , , , , , ,

By David Loshin

Once we have started our data quality assessment process by performing column value analysis, we can reach out beyond the scope of the types of null value analysis we discussed in the previous blog post. Since our column analysis effectively tallies the number of each value that appears in the column, we can use this frequency distribution of values to identify additional potential data flaws by considering a number of different aspects of value frequency (as well as lexicographic ordering), including:

  • Range Analysis, which looks at the values, and allows the analyst to consider whether they can be ordered so as to determine whether the values are constrained within a well-defined range.
Read More

Data Quality Assessment: Sparsity and Nullness

Blog Administrator | Analyzing Data, Analyzing Data Quality, Data Quality, Data Quality Assessment | , , , , , ,

By David Loshin

The first set of data quality assessment techniques that use column value frequency analysis focuses on the relationship of the population of values to the business processes that consume the data. The intent is to understand how the relative population of the column is associated with defined (or implicit) business rules, and then isolate and validate those rules.
Read More

Ask First, Fix Later

Blog Administrator | Address Quality, Analyzing Data, Analyzing Data Quality, Data Audit, Data Quality, Data Quality Assessment | , , , , , , ,

By Elliot King

Like the Boston Red Sox breaking their fans’ hearts, almost inevitably (stress on the almost) you will discover that some percentage of your data is wrong. The realization that you have data quality problems may come about for few reasons: 1) you’ve looked under the hood of your data systems by conducting a data assessment or 2) a data audit revealed that the data you have is not what you think you have.
Read More

Data Quality Assessment: Column Value Analysis

Blog Administrator | Analyzing Data, Analyzing Data Quality, Data Cleansing, Data Enrichment, Data Profiling, Data Quality, Data Quality Assessment | , , , , ,

By David Loshin

In recent blog series, I have shared some thoughts about methods used for data quality and data correction/cleansing. This month, I’d like to share some thoughts about data quality assessment, and the techniques that analysts use to review potential anomalies that present themselves.
Read More