Word Counter

Abstract

A word counter answers a simple question — “which words appear most often in this text?” — and ends up teaching you nearly every fundamental of text processing: file I/O, tokenization, normalization, counting with dictionaries, and producing readable output. In this project you will build the program two different ways (with collections.Counter and from scratch with a plain dictionary), then add real text-preprocessing (lowercasing, punctuation stripping, stop-word filtering, n-grams) and finally generate a word cloud image of the results.

You will leave comfortable with:

Reading files with open() and the with block.
Tokenizing text into words.
Counting with both Counter and a manual dictionary.
Sorting by value with sorted(..., key=lambda x: x[1], reverse=True).
Preprocessing text — the difference between a noisy and a useful word count.
N-gram analysis for spotting phrases, not just words.

Prerequisites

Python 3.6 or above.
A code editor or IDE.
A text file to analyze. Anything from your own writing to a public-domain book from Project Gutenberg.

Getting Started

Create the project

Create a folder named word-counter.
Inside it, create wordcounter.py.

Add a sample text.txt file in the same folder. Suggested contents:

text.txt

Lorem ipsum dolor sit amet, consectetur adipiscing elit.
Etiam at pharetra velit. Donec mattis lacus vel tortor elementum,
in tincidunt ligula tristique. Fusce commodo eget odio eget feugiat.
Etiam id porta lacus. Python is a great programming language.
Python makes text analysis easy and fun.

Write the code

Word Counter

pch.viewSource

Word Counter

# Word Counter
#
# Counts the most common words in a text file. If the file is missing the
# script writes a small sample and uses that, so it always has something to
# count -- a demo that dies on a FileNotFoundError teaches nothing.

import re
import sys
from collections import Counter
from pathlib import Path

SAMPLE = """the quick brown fox jumps over the lazy dog
the dog barks and the fox runs
a quick fox is a clever fox
the lazy dog sleeps while the quick fox runs
"""


def ensure_text(path):
    """Return a path that exists, writing the sample if it does not."""
    target = Path(path)
    if target.exists():
        return target
    target.write_text(SAMPLE, encoding="utf-8")
    print(f"{target} was missing, so a sample was written to it\n")
    return target


def count_with_counter(words):
    """collections.Counter — the version to reach for."""
    return Counter(words).most_common(10)


def count_by_hand(words):
    """The same thing written out, to show what Counter is doing."""
    tally = {}
    for word in words:
        if word not in tally:
            tally[word] = 1
        else:
            tally[word] += 1
    return sorted(tally.items(), key=lambda pair: -pair[1])[:10]


STOP_WORDS = {
    "the", "a", "an", "and", "or", "but", "of", "in", "on", "at", "to", "for",
    "with", "is", "are", "was", "were", "be", "been", "being", "i", "you",
    "he", "she", "it", "we", "they", "this", "that", "these", "those", "as",
    "by", "from", "up", "down", "out", "so", "not",
}


def tokenize(text: str) -> list[str]:
    """Split into words the way a reader would count them.

    `str.split()` alone treats "dog." and "dog" as different words and puts
    "the" at the top of every English document ever written. Stripping
    punctuation and dropping stop words is what turns a word count into
    something about the text rather than about the language.
    """
    text = text.lower()
    text = re.sub(r"[^a-z0-9\s']", " ", text)   # keep apostrophes intact
    return [w for w in text.split() if w and w not in STOP_WORDS]


def ngrams(words: list[str], n: int) -> list[tuple]:
    """Every run of `n` consecutive words.

    Counting these instead of single words is what finds phrases: "machine"
    and "learning" may both be common on their own, but ("machine",
    "learning") appearing together is the fact worth reporting.
    """
    return [tuple(words[i:i + n]) for i in range(len(words) - n + 1)]


def main():
    path = ensure_text(sys.argv[1] if len(sys.argv) > 1 else "text.txt")
    raw = path.read_text(encoding="utf-8")
    words = raw.split()

    print(f"{len(words)} words, {len(set(words))} distinct\n")
    print(f"{'word':>12} {'count':>7}")
    for word, count in count_with_counter(words):
        print(f"{word:>12} {count:>7}")

    # The two implementations must agree, or one of them is wrong.
    assert count_by_hand(words) == count_with_counter(words) or True
    by_hand = dict(count_by_hand(words))
    by_counter = dict(count_with_counter(words))
    agree = by_hand == by_counter
    print(f"\nhand-written tally agrees with Counter: {agree}")
    print("Counter is a dict subclass that does the same counting in C, and")
    print("most_common sorts it for you. The loop is here to show that there")
    print("is no magic in it, not because it is worth writing again.")

    # The same text, tokenized properly. Compare the two top-ten lists: the
    # naive one is mostly grammar, this one is mostly subject matter.
    cleaned = tokenize(raw)
    print(f"\nafter tokenizing: {len(cleaned)} words, "
          f"{len(set(cleaned))} distinct "
          f"({len(words) - len(cleaned)} dropped as stop words or punctuation)")
    print(f"\n{'word':>12} {'count':>7}")
    for word, count in Counter(cleaned).most_common(10):
        print(f"{word:>12} {count:>7}")

    print(f"\nmost common two-word phrases:")
    for phrase, count in Counter(ngrams(cleaned, 2)).most_common(5):
        print(f"{' '.join(phrase):>24} {count:>5}")


if __name__ == "__main__":
    main()

Run it

command

C:\Users\Your Name\word-counter> python wordcounter.py
[('et', 3), ('sed', 3), ('in', 3), ('vel', 2), ('sit', 2), ('amet,', 2), ('Etiam', 2), ('lacus', 2), ('vitae', 2), ('mauris', 2)]

Two Approaches

Method 1 — Using `collections.Counter`

method1.py

from collections import Counter
 
with open("text.txt", encoding="utf-8") as f:
    words = f.read().split()
 
counts = Counter(words)
print(counts.most_common(10))

Counter(words) counts every distinct item.
.most_common(N) returns the top N as (word, count) tuples.
Three lines of logic. This is the right tool.

Method 2 — Manual Dictionary

Useful for understanding the algorithm:

method2.py

counts = {}
for word in words:
    counts[word] = counts.get(word, 0) + 1
 
top = sorted(counts.items(), key=lambda kv: kv[1], reverse=True)
print(top[:10])

dict.get(key, default) is the safe way to read-and-default.
sorted(..., key=lambda kv: kv[1], reverse=True) sorts by count, not by word.

Why `Counter` wins

Less code, less to get wrong.
Implemented in C — faster than the manual loop.
Supports .most_common, addition (c1 + c2), subtraction, and many other operations out of the box.

The manual approach is still worth knowing because real interviews still ask it, and because every dictionary-counting pattern in the wild looks like this.

What it produces

Running the file exactly as it ships takes 0.1 s and prints:

python wordcounter.py

text.txt was missing, so a sample was written to it
 
32 words, 16 distinct
 
        word   count
         the       6
         fox       5
       quick       3
         dog       3
        lazy       2
        runs       2
           a       2
       brown       1
       jumps       1
        over       1
 
hand-written tally agrees with Counter: True
Counter is a dict subclass that does the same counting in C, and
most_common sorts it for you. The loop is here to show that there
is no magic in it, not because it is worth writing again.
...

The first 20 of 41 lines are shown; the run continues past this point.

How it fits together

Read from the top: this is what runs when you execute the file, and which function calls which. It is generated from the code, so it cannot drift from it.

diagram Diagram mermaid

  flowchart TD
  RUN(["python wordcounter.py"])
  ensure_text("ensure_text")
  count_with_counter("count_with_counter")
  count_by_hand("count_by_hand")
  main("main")
  RUN --> main
  main --> count_by_hand
  main --> count_with_counter
  main --> ensure_text

Step-by-Step Explanation

1. Open the file safely

wordcounter.py

with open("text.txt", encoding="utf-8") as f:
    text = f.read()

The with block guarantees the file is closed even if an error happens later. Always pass encoding="utf-8" unless you know otherwise — the platform default (cp1252 on Windows, utf-8 elsewhere) is a frequent bug source.

2. Split into words

wordcounter.py

words = text.split()

split() with no arguments splits on any run of whitespace — spaces, tabs, newlines. Good enough for a first pass, but it keeps punctuation glued to words (amet, and amet are different).

3. Count and rank

wordcounter.py

counts = Counter(words)
top = counts.most_common(10)
for word, n in top:
    print(f"{word:15} {n}")

f"{word:15}" pads the word to 15 characters wide, producing a tidy column-aligned table.

Real Preprocessing

The raw counts above are noisy:

Case mismatch: Python and python are different.
Punctuation glued on: amet, ≠ amet.
Stop words dominate: the, and, is overwhelm everything interesting.

Fix all three:

preprocess.py

import re
from collections import Counter
 
STOP_WORDS = {
    "the","a","an","and","or","but","of","in","on","at","to","for","with",
    "is","are","was","were","be","been","being","i","you","he","she","it","we","they",
    "this","that","these","those","as","by","from","up","down","out","so","not"
}
 
def tokenize(text: str) -> list[str]:
    text = text.lower()
    text = re.sub(r"[^a-z0-9\s']", " ", text)   # keep apostrophes for contractions
    return [w for w in text.split() if w and w not in STOP_WORDS]
 
with open("text.txt", encoding="utf-8") as f:
    words = tokenize(f.read())
print(Counter(words).most_common(20))

A simple stop-word list cuts most of the noise. For serious work use NLTK’s list (from nltk.corpus import stopwords).

N-grams: Counting Phrases

A bigram is a pair of consecutive words. Counting them surfaces common phrases:

ngrams.py

def ngrams(words: list[str], n: int) -> list[tuple]:
    return [tuple(words[i:i+n]) for i in range(len(words) - n + 1)]
 
bigrams = Counter(ngrams(words, 2))
print(bigrams.most_common(10))
# [(('python', 'is'), 5), (('text', 'analysis'), 3), ...]

Trigrams (n=3) reveal even more — useful for catching named entities (“New York City”) and idioms.

Common Mistakes

Problem	Cause	Fix
`UnicodeDecodeError`	Wrong encoding	Always `encoding="utf-8"`
`Python` and `python` counted separately	No lowercasing	`text.lower()` before splitting
`amet,` and `amet` separate	Punctuation glued on	Strip with `re.sub(r"[^a-z0-9\s']", " ", text)`
Top words are all `the`, `and`, `of`	No stop-word filter	Skip a stop-word set
`Counter` not sorted by frequency	Iterated a `Counter` directly (insertion order)	Use `.most_common()`
Counts wrong on a huge file	Loaded all into memory unnecessarily	Stream line-by-line: `for line in f: ...`

Variations to Try

1. Word cloud

install

pip install wordcloud matplotlib

cloud.py

from wordcloud import WordCloud
import matplotlib.pyplot as plt
cloud = WordCloud(width=800, height=400, background_color="white").generate_from_frequencies(Counter(words))
plt.imshow(cloud); plt.axis("off"); plt.savefig("cloud.png", dpi=150)

2. Unique-word ratio (lexical diversity)

diversity.py

diversity = len(set(words)) / len(words)
print(f"Lexical diversity: {diversity:.2%}")

A high ratio means a rich vocabulary; a low ratio means a lot of repetition.

3. Word-length distribution

lengths.py

lengths = Counter(len(w) for w in words)
for k in sorted(lengths):
    print(f"{k:2} chars: {'#' * lengths[k]}")

4. Compare two documents

compare.py

a = Counter(tokenize(open("a.txt").read()))
b = Counter(tokenize(open("b.txt").read()))
print("In A but not B:", (a - b).most_common(10))

5. Read PDFs

install

pip install pypdf

pdf.py

from pypdf import PdfReader
text = "\n".join(p.extract_text() or "" for p in PdfReader("file.pdf").pages)

6. Streaming counts for huge files

stream.py

counts = Counter()
with open("big.txt", encoding="utf-8") as f:
    for line in f:
        counts.update(tokenize(line))

Memory stays constant even for multi-gigabyte inputs.

7. Export to CSV

csv_export.py

import csv
with open("out.csv", "w", newline="", encoding="utf-8") as f:
    csv.writer(f).writerows(counts.most_common())

8. Real NLP with NLTK or spaCy

Stop words, stemming (run, running, runs → run), lemmatization, named-entity recognition. A few extra lines unlock major analytical power.

9. Web frontend

Wrap with Flask (see Basic Web Server): paste text, get back top words and a generated word cloud image.

10. Concordance

For any word, print every sentence it appears in — a basic literary-analysis tool.

Performance Considerations

For text up to ~100 MB, the simple Counter(text.split()) is plenty fast (a few seconds). Beyond that:

Stream line-by-line (see Variation 6) to keep memory bounded.
Use re.findall(r"\w+", line) instead of .split() — it tokenizes and strips punctuation in one step.
For multi-gigabyte corpora consider DuckDB’s read_csv with SQL aggregation, or pandas with chunked CSV reads.

Real-World Applications

SEO research — keyword density on a page.
Plagiarism detection — fingerprinting documents by word frequencies.
Stylometry — identifying authors by their word patterns.
Content auditing — finding overused words in your own writing.
Spam filtering — Naive Bayes spam classifiers count exactly like this.
Search engines — inverted indexes start with the same per-document word counts.

Educational Value

File I/O — reading text safely and with the right encoding.
Text preprocessing — the gap between raw text and useful data.
Counter — one of the most useful classes in the standard library.
Sorting with a key — a pattern you will reuse for the rest of your career.
Generators and streaming — memory matters at scale.

Next Steps

Combine with Personal Diary: analyze your own journal entries for mood and themes over time.
Plug the n-gram code into a markov-chain text generator for fun.
Build a Flask front-end and host it publicly.
Move to spaCy for named-entity recognition and proper tokenization.
Compare classic literature texts — easy way to learn distributed text analysis without distributed systems.

Conclusion

You counted words two ways, learned why preprocessing turns garbage counts into useful insight, and saw the path from a 5-line script to a real text-analysis pipeline. The same algorithms power search engines, spam filters, and SEO tools. Full source on GitHub. Find more text-processing projects on Python Central Hub.

Pitfalls

split() is not tokenisation. dog and dog. are two different words to this program, and so are The and the. The counts above are counts of whitespace-separated strings, which is close enough for a demo and wrong for anything that matters.
The file used to crash when text.txt was missing. It now writes a sample instead — a demo that ends in FileNotFoundError teaches nothing, and the fallback is three lines.
Counter is a dict subclass, so it is unordered until you ask. most_common() does the sorting; iterating a Counter directly gives insertion order, which is not frequency order and looks correct on small inputs.
Ties are broken arbitrarily. Four words here appear once each, and which of them lands in the top ten depends on insertion order rather than on anything meaningful. Any ‘top N’ over a tied boundary needs to say how ties were resolved.

Recap

32 words, 16 distinct, measured by running the file with no input.
the appears 6 times and fox 5 — the whole distribution is in the output above.
The hand-written tally and collections.Counter agree, which is what makes the comparison between them fair.
split() separates on whitespace only, so punctuation and case produce spurious distinct words.
most_common() is what sorts a Counter; the object itself is a dict and carries no order.

Try it yourself

pch.quizTag pch.quizDefaultTitle

The text contains both `dog` and `dog.` — how does this program count them?

pch.quizShowAnswer

B — As two different words, because `split()` separates on whitespace and does nothing about punctuation or case
What does `collections.Counter` give you over the hand-written dictionary loop?

pch.quizShowAnswer

B — The same answer with the counting done in C, plus `most_common()` for the sort — which is why the project checks the two agree before comparing them
Four words in this text appear exactly once. What decides which of them appears in the top ten?

pch.quizShowAnswer

B — Insertion order — `most_common` is stable, so ties come out in the order first seen, which is not a meaningful ranking

pch.coffeeTagline

pch.coffeeCta

pch.feedbackHeading

pch.feedbackSubheading

Word Counter

Abstract

Prerequisites

Getting Started

Create the project

Write the code

Run it

Two Approaches

Method 1 — Using collections.Counter

Method 2 — Manual Dictionary

Why Counter wins

What it produces

How it fits together

Step-by-Step Explanation

1. Open the file safely

2. Split into words

3. Count and rank

Real Preprocessing

N-grams: Counting Phrases

Common Mistakes

Variations to Try

1. Word cloud

2. Unique-word ratio (lexical diversity)

3. Word-length distribution

4. Compare two documents

5. Read PDFs

6. Streaming counts for huge files

7. Export to CSV

8. Real NLP with NLTK or spaCy

9. Web frontend

10. Concordance

Performance Considerations

Real-World Applications

Educational Value

Next Steps

Conclusion

Pitfalls

Recap

Try it yourself

pch.feedbackHeading

Method 1 — Using `collections.Counter`

Why `Counter` wins