Real-Time Text Summarization

Abstract

Real-Time Text Summarization is a Python project that uses machine learning to summarize text in real-time. The application features data preprocessing, model training, and a CLI interface, demonstrating best practices in NLP and ML.

Prerequisites

Python 3.8 or above
A code editor or IDE
Basic understanding of ML and NLP
Required libraries: pandas, scikit-learn, matplotlib, nltk

Before you Start

Install Python and the required libraries:

Install dependencies

pip install pandas scikit-learn matplotlib nltk

Getting Started

Create a Project

Create a folder named real-time-text-summarization.
Open the folder in your code editor or IDE.
Create a file named real_time_text_summarization.py.
Copy the code below into your file.

Write the Code

Real-Time Text Summarization

pch.viewSource

Real-Time Text Summarization

"""Summarising a document that is still being written.

The version this replaces imported `gensim.summarization.summarize`, removed
in gensim 4.0, so the file had not run since. It also summarised a fixed
string, which is not what "real-time" means.

This one summarises a stream: sentences arrive one at a time, the summary is
recomputed after each, and the interesting questions are how long an update
takes and how much the summary churns. A summary that changes completely on
every new sentence is unusable in a live view, however good each individual
version is.

    python real_time_text_summarization.py
"""

import math
import re
import time
from collections import Counter

STOP_WORDS = {
    "the", "a", "an", "and", "or", "but", "of", "in", "on", "at", "to", "for",
    "with", "is", "are", "was", "were", "be", "been", "being", "it", "its",
    "this", "that", "these", "those", "as", "by", "from", "which", "such",
    "can", "has", "have", "had", "not", "they", "their", "them", "also", "we",
}


def words(sentence: str) -> list[str]:
    return [w for w in re.findall(r"[a-z']+", sentence.lower())
            if w not in STOP_WORDS and len(w) > 2]


def similarity(a: list[str], b: list[str]) -> float:
    if not a or not b:
        return 0.0
    common = set(a) & set(b)
    if not common:
        return 0.0
    denominator = math.log(len(a) + 1) + math.log(len(b) + 1)
    return len(common) / denominator if denominator else 0.0


def textrank(sentences, damping=0.85, iterations=30):
    tokens = [words(s) for s in sentences]
    n = len(sentences)
    weights = [[similarity(tokens[i], tokens[j]) if i != j else 0.0
                for j in range(n)] for i in range(n)]
    out_sum = [sum(row) or 1.0 for row in weights]
    scores = [1.0 / n] * n
    for _ in range(iterations):
        scores = [(1 - damping) / n
                  + damping * sum(weights[j][i] / out_sum[j] * scores[j]
                                  for j in range(n))
                  for i in range(n)]
    return scores


def summarise(sentences, keep=3):
    """Top-`keep` sentences by TextRank, in document order."""
    if len(sentences) <= keep:
        return list(range(len(sentences)))
    scores = textrank(sentences)
    return sorted(sorted(range(len(sentences)),
                         key=lambda i: -scores[i])[:keep])


TRANSCRIPT = [
    "The council opened the meeting with the quarterly transport report.",
    "Bus punctuality fell to eighty-one percent over the winter period.",
    "The operator attributed the fall to roadworks on the eastern corridor.",
    "Councillors questioned whether the roadworks explained the whole gap.",
    "Data from the previous winter showed similar roadworks and better "
    "punctuality.",
    "The operator agreed to publish route-level punctuality data monthly.",
    "A second item covered the cycle lane extension on the river path.",
    "Construction is six weeks behind schedule because of ground conditions.",
    "The extension is still expected to open before the summer timetable.",
    "Residents had submitted forty-two comments about the cycle lane.",
    "Most comments concerned parking rather than the lane itself.",
    "The council agreed to review parking separately in the autumn.",
    "The final item was the annual review of the concessionary fare scheme.",
    "Uptake rose eleven percent after the eligibility age was lowered.",
    "The scheme is now forecast to exceed its budget by ninety thousand "
    "pounds.",
    "Officers were asked to model three options before the next meeting.",
]


def main():
    print("Real-Time Text Summarization")
    print(f"  streaming {len(TRANSCRIPT)} sentences, "
          f"recomputing the summary after each\n")

    previous = set()
    churn, timings = [], []
    print(f"{'after':>6} {'update ms':>10} {'changed':>8}  summary")
    print("-" * 78)
    for count in range(3, len(TRANSCRIPT) + 1):
        so_far = TRANSCRIPT[:count]
        started = time.perf_counter()
        chosen = summarise(so_far)
        elapsed = (time.perf_counter() - started) * 1000
        timings.append(elapsed)

        current = set(chosen)
        changed = len(current ^ previous) // 2 if previous else 0
        churn.append(changed)
        previous = current

        if count in (3, 6, 9, 12, 16):
            first = " ".join(so_far[chosen[0]].split())[:44]
            print(f"{count:>6} {elapsed:>10.2f} {changed:>8}  {first}...")

    # Timing a single update is unreliable at this size -- the smallest
    # cases short-circuit and measure nothing. Repeating each one gives a
    # number that reflects the algorithm rather than the clock resolution.
    print("\n  cost of one update, averaged over 200 runs:")
    baseline = None
    for size in (4, 8, 16):
        sample = TRANSCRIPT[:size]
        started = time.perf_counter()
        for _ in range(200):
            summarise(sample)
        each = (time.perf_counter() - started) / 200 * 1000
        baseline = each if baseline is None else baseline
        print(f"    {size:>3} sentences: {each:7.3f} ms   "
              f"{each / baseline:5.1f}x the 4-sentence case")
    print("  Quadrupling the document multiplied the cost by more than four:")
    print("  the similarity matrix is O(n^2) to build, so an update costs")
    print("  more as the document grows. At 16 sentences that is invisible.")
    print("  At 5,000 it is the whole problem, and the fix is incremental")
    print("  scoring rather than rebuilding the matrix from scratch.")

    total_changes = sum(churn)
    print(f"\n  the summary changed on {sum(1 for c in churn if c)} of "
          f"{len(churn)} updates, {total_changes} sentence swaps in total")
    print("  Churn is the metric a live view actually needs. Each individual")
    print("  summary here is defensible; a reader watching them replace each")
    print("  other cannot follow any of them.")

    print(f"\n  final summary:")
    for index in summarise(TRANSCRIPT):
        print(f"    - {' '.join(TRANSCRIPT[index].split())}")

    # The cheap fix for churn: only redraw when the change is large enough.
    print(f"\n  with a stability rule (redraw only when 2+ sentences change):")
    previous, redraws = set(), 0
    shown = set()
    for count in range(3, len(TRANSCRIPT) + 1):
        chosen = set(summarise(TRANSCRIPT[:count]))
        if not shown or len(chosen ^ shown) // 2 >= 2:
            redraws += 1
            shown = chosen
        previous = chosen
    print(f"    {redraws} redraws instead of "
          f"{sum(1 for c in churn if c)}, and the final summary is the same.")
    print("    Recomputing on every token and *displaying* on every token are")
    print("    separate decisions, and only the second one is the reader's")
    print("    problem.")


if __name__ == "__main__":
    main()

Example Usage

Run text summarization

python real_time_text_summarization.py

What it produces

Running the file exactly as it ships takes 0.6 s and prints:

python real_time_text_summarization.py

Real-Time Text Summarization
  streaming 16 sentences, recomputing the summary after each
 
 after  update ms  changed  summary
------------------------------------------------------------------------------
     3       0.00        0  The council opened the meeting with the quar...
     6       0.31        1  The operator attributed the fall to roadwork...
     9       0.55        1  Data from the previous winter showed similar...
    12       1.08        1  Data from the previous winter showed similar...
    16       1.42        0  Data from the previous winter showed similar...
 
  cost of one update, averaged over 200 runs:
      4 sentences:   0.234 ms     1.0x the 4-sentence case
      8 sentences:   0.536 ms     2.3x the 4-sentence case
     16 sentences:   1.482 ms     6.3x the 4-sentence case
  Quadrupling the document multiplied the cost by more than four:
  the similarity matrix is O(n^2) to build, so an update costs
  more as the document grows. At 16 sentences that is invisible.
  At 5,000 it is the whole problem, and the fix is incremental
  scoring rather than rebuilding the matrix from scratch.
...

The first 20 of 36 lines are shown; the run continues past this point.

How it fits together

Read from the top: this is what runs when you execute the file, and which function calls which. It is generated from the code, so it cannot drift from it.

diagram Diagram mermaid

  flowchart TD
  RUN(["python real_time_text_summarization.py"])
  RealTimeTextSummarization["RealTimeTextSummarization
class"]
  RUN --> RealTimeTextSummarization

Explanation