Understanding Data Quality

Why data quality matters

Bad data quality leads to:

• Wrong decisions
• Broken dashboards
• Unreliable ML models
• Loss of trust

Data preprocessing is the discipline of making data usable and trustworthy.

Key dimensions of data quality

1) Completeness

Are required values missing?

• df.isna().sum()
• Missing percentage per column

2) Accuracy

Do values match reality?

• Negative ages
• Impossible dates (future DOB)
• Wrong units (₹ vs paise)

3) Consistency

Do values follow the same format across rows?

• “delhi”, “Delhi”, ” DELHI ”
• Mixed currencies

4) Validity

Do values fit the allowed set/range?

• status should be one of: closed
• rating must be 1–5

5) Uniqueness

Are there duplicates?

• Duplicate rows
• Duplicate IDs

Practical quality checks (Pandas)

Quick quality checks

# shape
print(df.shape)
 
# schema
print(df.dtypes)
 
# missing
print(df.isna().sum().sort_values(ascending=False).head(20))
 
# duplicates
print("Duplicate rows:", df.duplicated().sum())
 
# basic stats
print(df.describe(include="all"))

Typical preprocessing outputs

• Cleaned columns (trimmed text, fixed casing)
• Correct dtypes (numeric/date)
• Handled missing values
• Outlier strategy selected (remove/cap/keep)
• Encoded categories for modeling

Rule of thumb

Always document:

• What you changed
• Why you changed it
• What assumptions you made

Visualize it

Think of data quality as a pipeline of checks. Raw data only becomes “trustworthy” once it has passed through each dimension below.

diagram Data quality workflow mermaid

  flowchart LR
  A["Raw data"] --> B["Completeness check"]
  B --> C["Accuracy check"]
  C --> D["Consistency check"]
  D --> E["Validity & uniqueness check"]
  E --> F["Trustworthy dataset"]

Raw data passes through a series of checks before it can be trusted for analysis or modeling.

From the book

McKinney points out that loading, cleaning, transforming, and rearranging data “are often reported to take up 80% or more of an analyst’s time.” Checking data quality isn’t a detour from analysis — for most real projects, it is most of the work.

🧪 Try It Yourself

Exercise 1 – Count missing values

pch.exerciseTag pch.exerciseTitle

Exercise 2 – Find duplicate rows

pch.exerciseTag pch.exerciseTitle

Exercise 3 – Check validity of a range

pch.exerciseTag pch.exerciseTitle

Next

Once you know where the quality problems are, the next step is usually fixing dtypes — head to Outlier Detection (IQR Method) to start finding the extreme values hiding in your numbers.