Why data quality matters
Bad data quality leads to:
- Wrong decisions
- Broken dashboards
- Unreliable ML models
- Loss of trust
Data preprocessing is the discipline of making data usable and trustworthy.
Key dimensions of data quality
1) Completeness
Are required values missing?
df.isna().sum()- Missing percentage per column
2) Accuracy
Do values match reality?
- Negative ages
- Impossible dates (future DOB)
- Wrong units (₹ vs paise)
3) Consistency
Do values follow the same format across rows?
- "delhi", "Delhi", " DELHI "
- Mixed currencies
4) Validity
Do values fit the allowed set/range?
statusshould be one of: closedratingmust be 1–5
5) Uniqueness
Are there duplicates?
- Duplicate rows
- Duplicate IDs
Practical quality checks (Pandas)
# shape
print(df.shape)
# schema
print(df.dtypes)
# missing
print(df.isna().sum().sort_values(ascending=False).head(20))
# duplicates
print("Duplicate rows:", df.duplicated().sum())
# basic stats
print(df.describe(include="all"))Typical preprocessing outputs
- Cleaned columns (trimmed text, fixed casing)
- Correct dtypes (numeric/date)
- Handled missing values
- Outlier strategy selected (remove/cap/keep)
- Encoded categories for modeling
Rule of thumb
Always document:
- What you changed
- Why you changed it
- What assumptions you made
Visualize it
Think of data quality as a pipeline of checks. Raw data only becomes "trustworthy" once it has passed through each dimension below.
flowchart LR A["Raw data"] --> B["Completeness check"] B --> C["Accuracy check"] C --> D["Consistency check"] D --> E["Validity & uniqueness check"] E --> F["Trustworthy dataset"]
🧪 Try It Yourself
Exercise 1 – Count missing values
Tab indents. Press Escape, then Tab, to leave the editor. Control plus Enter runs the code; Control plus Shift plus Enter submits it.
Press Run to see what your code prints.Exercise 2 – Find duplicate rows
Tab indents. Press Escape, then Tab, to leave the editor. Control plus Enter runs the code; Control plus Shift plus Enter submits it.
Press Run to see what your code prints.Exercise 3 – Check validity of a range
Tab indents. Press Escape, then Tab, to leave the editor. Control plus Enter runs the code; Control plus Shift plus Enter submits it.
Press Run to see what your code prints.Check your understanding
Which dimension of data quality is violated by city values delhi, Delhi and DELHI?
Consistency asks whether values follow the same format across rows.
Show answer
A — Consistency — Consistency asks whether values follow the same format across rows.
A rating column must be 1 to 5 but contains 9. Which dimension fails?
Validity checks that values fit the allowed set or range, like status being open or closed.
Show answer
A — Validity — Validity checks that values fit the allowed set or range, like status being open or closed.
Which call counts duplicate rows?
duplicated() flags repeated rows and sum() counts them. That checks uniqueness.
Show answer
A — df.duplicated().sum() — duplicated() flags repeated rows and sum() counts them. That checks uniqueness.
Negative ages or a future date of birth are examples of which problem?
Accuracy asks whether values match reality, including wrong units like rupees versus paise.
Show answer
A — Accuracy — Accuracy asks whether values match reality, including wrong units like rupees versus paise.
Next
Once you know where the quality problems are, the next step is usually fixing dtypes — head to Outlier Detection (IQR Method) to start finding the extreme values hiding in your numbers.