Abstract

A word counter answers a simple question — “which words appear most often in this text?” — and ends up teaching you nearly every fundamental of text processing: file I/O, tokenization, normalization, counting with dictionaries, and producing readable output. In this project you will build the program two different ways (with collections.Countercollections.Counter and from scratch with a plain dictionary), then add real text-preprocessing (lowercasing, punctuation stripping, stop-word filtering, n-grams) and finally generate a word cloud image of the results.

You will leave comfortable with:

• Reading files with open()open() and the withwith block.
• Tokenizing text into words.
• Counting with both CounterCounter and a manual dictionary.
• Sorting by value with sorted(..., key=lambda x: x[1], reverse=True)sorted(..., key=lambda x: x[1], reverse=True).
• Preprocessing text — the difference between a noisy and a useful word count.
• N-gram analysis for spotting phrases, not just words.

Prerequisites

• Python 3.6 or above.
• A code editor or IDE.
• A text file to analyze. Anything from your own writing to a public-domain book from Project Gutenberg.

Getting Started

Create the project

1. Create a folder named word-counterword-counter.
2. Inside it, create wordcounter.pywordcounter.py.
3. Add a sample text.txttext.txt file in the same folder. Suggested contents:
   text.txt

   Lorem ipsum dolor sit amet, consectetur adipiscing elit.
   Etiam at pharetra velit. Donec mattis lacus vel tortor elementum,
   in tincidunt ligula tristique. Fusce commodo eget odio eget feugiat.
   Etiam id porta lacus. Python is a great programming language.
   Python makes text analysis easy and fun.

   text.txt

   Lorem ipsum dolor sit amet, consectetur adipiscing elit.
   Etiam at pharetra velit. Donec mattis lacus vel tortor elementum,
   in tincidunt ligula tristique. Fusce commodo eget odio eget feugiat.
   Etiam id porta lacus. Python is a great programming language.
   Python makes text analysis easy and fun.

Write the code

# Word Counter
 
# Import the Counter class from the collections module
from collections import Counter
 
# Open the file in read mode
text = open('text.txt', 'r')
 
# Use the read method to read the file contents
allWords = text.read()
 
# Use split method to create a list of words from the text
words = allWords.split()
 
# Create a Counter object
counter = Counter(words)
 
# Use the most_common method to print the 10 most common words
print(counter.most_common(10))
 
# Close the file
text.close()
 
 
# Alternative solution
words2 = {}
 
# Loop through the list of words
for word in words:
    # If the word is not in the dictionary, add it
    if word not in words2:
        words2[word] = 1
    # If the word is in the dictionary, increment its value
    else:
        words2[word] += 1
        
# Sort the dictionary by value in descending order
sorted_words = sorted(words2.items(), key=lambda x: x[1], reverse=True)
 
# Print the 10 most common words
print(sorted_words[:10])
 
# Close the file
text.close()

# Word Counter
 
# Import the Counter class from the collections module
from collections import Counter
 
# Open the file in read mode
text = open('text.txt', 'r')
 
# Use the read method to read the file contents
allWords = text.read()
 
# Use split method to create a list of words from the text
words = allWords.split()
 
# Create a Counter object
counter = Counter(words)
 
# Use the most_common method to print the 10 most common words
print(counter.most_common(10))
 
# Close the file
text.close()
 
 
# Alternative solution
words2 = {}
 
# Loop through the list of words
for word in words:
    # If the word is not in the dictionary, add it
    if word not in words2:
        words2[word] = 1
    # If the word is in the dictionary, increment its value
    else:
        words2[word] += 1
        
# Sort the dictionary by value in descending order
sorted_words = sorted(words2.items(), key=lambda x: x[1], reverse=True)
 
# Print the 10 most common words
print(sorted_words[:10])
 
# Close the file
text.close()

Run it

C:\Users\Your Name\word-counter> python wordcounter.py
[('et', 3), ('sed', 3), ('in', 3), ('vel', 2), ('sit', 2), ('amet,', 2), ('Etiam', 2), ('lacus', 2), ('vitae', 2), ('mauris', 2)]

C:\Users\Your Name\word-counter> python wordcounter.py
[('et', 3), ('sed', 3), ('in', 3), ('vel', 2), ('sit', 2), ('amet,', 2), ('Etiam', 2), ('lacus', 2), ('vitae', 2), ('mauris', 2)]

Two Approaches

Method 1 — Using collections.Countercollections.Counter

from collections import Counter
 
with open("text.txt", encoding="utf-8") as f:
    words = f.read().split()
 
counts = Counter(words)
print(counts.most_common(10))

from collections import Counter
 
with open("text.txt", encoding="utf-8") as f:
    words = f.read().split()
 
counts = Counter(words)
print(counts.most_common(10))

• Counter(words)Counter(words) counts every distinct item.
• .most_common(N).most_common(N) returns the top N as (word, count)(word, count) tuples.
• Three lines of logic. This is the right tool.

Method 2 — Manual Dictionary

Useful for understanding the algorithm:

counts = {}
for word in words:
    counts[word] = counts.get(word, 0) + 1
 
top = sorted(counts.items(), key=lambda kv: kv[1], reverse=True)
print(top[:10])

counts = {}
for word in words:
    counts[word] = counts.get(word, 0) + 1
 
top = sorted(counts.items(), key=lambda kv: kv[1], reverse=True)
print(top[:10])

• dict.get(key, default)dict.get(key, default) is the safe way to read-and-default.
• sorted(..., key=lambda kv: kv[1], reverse=True)sorted(..., key=lambda kv: kv[1], reverse=True) sorts by count, not by word.

Why CounterCounter wins

• Less code, less to get wrong.
• Implemented in C — faster than the manual loop.
• Supports .most_common.most_common, addition (c1 + c2c1 + c2), subtraction, and many other operations out of the box.

The manual approach is still worth knowing because real interviews still ask it, and because every dictionary-counting pattern in the wild looks like this.

Step-by-Step Explanation

1. Open the file safely

with open("text.txt", encoding="utf-8") as f:
    text = f.read()

with open("text.txt", encoding="utf-8") as f:
    text = f.read()

The withwith block guarantees the file is closed even if an error happens later. Always pass encoding="utf-8"encoding="utf-8" unless you know otherwise — the platform default (cp1252 on Windows, utf-8 elsewhere) is a frequent bug source.

2. Split into words

words = text.split()

words = text.split()

split()split() with no arguments splits on any run of whitespace — spaces, tabs, newlines. Good enough for a first pass, but it keeps punctuation glued to words (amet,amet, and ametamet are different).

3. Count and rank

counts = Counter(words)
top = counts.most_common(10)
for word, n in top:
    print(f"{word:15} {n}")

counts = Counter(words)
top = counts.most_common(10)
for word, n in top:
    print(f"{word:15} {n}")

f"{word:15}"f"{word:15}" pads the word to 15 characters wide, producing a tidy column-aligned table.

Real Preprocessing

The raw counts above are noisy:

• Case mismatch: PythonPython and pythonpython are different.
• Punctuation glued on: amet,amet, ≠ ametamet.
• Stop words dominate: thethe, andand, isis overwhelm everything interesting.

Fix all three:

import re
from collections import Counter
 
STOP_WORDS = {
    "the","a","an","and","or","but","of","in","on","at","to","for","with",
    "is","are","was","were","be","been","being","i","you","he","she","it","we","they",
    "this","that","these","those","as","by","from","up","down","out","so","not"
}
 
def tokenize(text: str) -> list[str]:
    text = text.lower()
    text = re.sub(r"[^a-z0-9\s']", " ", text)   # keep apostrophes for contractions
    return [w for w in text.split() if w and w not in STOP_WORDS]
 
with open("text.txt", encoding="utf-8") as f:
    words = tokenize(f.read())
print(Counter(words).most_common(20))

import re
from collections import Counter
 
STOP_WORDS = {
    "the","a","an","and","or","but","of","in","on","at","to","for","with",
    "is","are","was","were","be","been","being","i","you","he","she","it","we","they",
    "this","that","these","those","as","by","from","up","down","out","so","not"
}
 
def tokenize(text: str) -> list[str]:
    text = text.lower()
    text = re.sub(r"[^a-z0-9\s']", " ", text)   # keep apostrophes for contractions
    return [w for w in text.split() if w and w not in STOP_WORDS]
 
with open("text.txt", encoding="utf-8") as f:
    words = tokenize(f.read())
print(Counter(words).most_common(20))

A simple stop-word list cuts most of the noise. For serious work use NLTK’s list (from nltk.corpus import stopwordsfrom nltk.corpus import stopwords).

N-grams: Counting Phrases

A bigram is a pair of consecutive words. Counting them surfaces common phrases:

def ngrams(words: list[str], n: int) -> list[tuple]:
    return [tuple(words[i:i+n]) for i in range(len(words) - n + 1)]
 
bigrams = Counter(ngrams(words, 2))
print(bigrams.most_common(10))
# [(('python', 'is'), 5), (('text', 'analysis'), 3), ...]

def ngrams(words: list[str], n: int) -> list[tuple]:
    return [tuple(words[i:i+n]) for i in range(len(words) - n + 1)]
 
bigrams = Counter(ngrams(words, 2))
print(bigrams.most_common(10))
# [(('python', 'is'), 5), (('text', 'analysis'), 3), ...]

Trigrams (n=3n=3) reveal even more — useful for catching named entities (“New York City”) and idioms.

Common Mistakes

Variations to Try

1. Word cloud

pip install wordcloud matplotlib

pip install wordcloud matplotlib

from wordcloud import WordCloud
import matplotlib.pyplot as plt
cloud = WordCloud(width=800, height=400, background_color="white").generate_from_frequencies(Counter(words))
plt.imshow(cloud); plt.axis("off"); plt.savefig("cloud.png", dpi=150)

from wordcloud import WordCloud
import matplotlib.pyplot as plt
cloud = WordCloud(width=800, height=400, background_color="white").generate_from_frequencies(Counter(words))
plt.imshow(cloud); plt.axis("off"); plt.savefig("cloud.png", dpi=150)

2. Unique-word ratio (lexical diversity)

diversity = len(set(words)) / len(words)
print(f"Lexical diversity: {diversity:.2%}")

diversity = len(set(words)) / len(words)
print(f"Lexical diversity: {diversity:.2%}")

A high ratio means a rich vocabulary; a low ratio means a lot of repetition.

3. Word-length distribution

lengths = Counter(len(w) for w in words)
for k in sorted(lengths):
    print(f"{k:2} chars: {'#' * lengths[k]}")

lengths = Counter(len(w) for w in words)
for k in sorted(lengths):
    print(f"{k:2} chars: {'#' * lengths[k]}")

4. Compare two documents

a = Counter(tokenize(open("a.txt").read()))
b = Counter(tokenize(open("b.txt").read()))
print("In A but not B:", (a - b).most_common(10))

a = Counter(tokenize(open("a.txt").read()))
b = Counter(tokenize(open("b.txt").read()))
print("In A but not B:", (a - b).most_common(10))

5. Read PDFs

pip install pypdf

pip install pypdf

from pypdf import PdfReader
text = "\n".join(p.extract_text() or "" for p in PdfReader("file.pdf").pages)

from pypdf import PdfReader
text = "\n".join(p.extract_text() or "" for p in PdfReader("file.pdf").pages)

6. Streaming counts for huge files

counts = Counter()
with open("big.txt", encoding="utf-8") as f:
    for line in f:
        counts.update(tokenize(line))

counts = Counter()
with open("big.txt", encoding="utf-8") as f:
    for line in f:
        counts.update(tokenize(line))

Memory stays constant even for multi-gigabyte inputs.

7. Export to CSV

import csv
with open("out.csv", "w", newline="", encoding="utf-8") as f:
    csv.writer(f).writerows(counts.most_common())

import csv
with open("out.csv", "w", newline="", encoding="utf-8") as f:
    csv.writer(f).writerows(counts.most_common())

8. Real NLP with NLTK or spaCy

Stop words, stemming (runrun, runningrunning, runsruns → runrun), lemmatization, named-entity recognition. A few extra lines unlock major analytical power.

9. Web frontend

Wrap with Flask (see Basic Web Server): paste text, get back top words and a generated word cloud image.

10. Concordance

For any word, print every sentence it appears in — a basic literary-analysis tool.

Performance Considerations

For text up to ~100 MB, the simple Counter(text.split())Counter(text.split()) is plenty fast (a few seconds). Beyond that:

• Stream line-by-line (see Variation 6) to keep memory bounded.
• Use re.findall(r"\w+", line)re.findall(r"\w+", line) instead of .split().split() — it tokenizes and strips punctuation in one step.
• For multi-gigabyte corpora consider DuckDB’s read_csvread_csv with SQL aggregation, or pandaspandas with chunked CSV reads.

Real-World Applications

• SEO research — keyword density on a page.
• Plagiarism detection — fingerprinting documents by word frequencies.
• Stylometry — identifying authors by their word patterns.
• Content auditing — finding overused words in your own writing.
• Spam filtering — Naive Bayes spam classifiers count exactly like this.
• Search engines — inverted indexes start with the same per-document word counts.

Educational Value

• File I/O — reading text safely and with the right encoding.
• Text preprocessing — the gap between raw text and useful data.
• CounterCounter — one of the most useful classes in the standard library.
• Sorting with a key — a pattern you will reuse for the rest of your career.
• Generators and streaming — memory matters at scale.

Next Steps

• Combine with Personal Diary: analyze your own journal entries for mood and themes over time.
• Plug the n-gram code into a markov-chain text generator for fun.
• Build a Flask front-end and host it publicly.
• Move to spaCy for named-entity recognition and proper tokenization.
• Compare classic literature texts — easy way to learn distributed text analysis without distributed systems.

Conclusion

You counted words two ways, learned why preprocessing turns garbage counts into useful insight, and saw the path from a 5-line script to a real text-analysis pipeline. The same algorithms power search engines, spam filters, and SEO tools. Full source on GitHub. Find more text-processing projects on Python Central Hub.