Skip to main content

Text and HTML Processing

This codebase provides a robust set of utilities for manipulating text and HTML, focusing on safety, correct truncation of markup, and intelligent conversion of plain text to hyperlinked content. These utilities are central to how the framework handles user-generated content and administrative displays.

Safe Strings and HTML Output​

The foundation of HTML processing in this project is the distinction between "safe" and "unsafe" strings. This is managed primarily through the SafeString class.

SafeString​

Located in django/utils/safestring.py, SafeString is a subclass of the standard Python str that also inherits from SafeData. When a string is wrapped in SafeString, it signals to the template engine and other output utilities that the content has already been sanitized or contains intentional HTML that should not be escaped.

A critical implementation detail is how SafeString handles concatenation. If you add a SafeString to a normal str, the result is a standard (unsafe) str to prevent accidental injection:

from django.utils.safestring import SafeString, mark_safe

# mark_safe returns a SafeString instance
header = mark_safe("<h1>Welcome</h1>")
content = " <script>alert('xss')</script>"

# Result is a standard str, which will be escaped on output
final_output = header + content 
isinstance(final_output, SafeString) # False

Text Truncation​

The Truncator class in django/utils/text.py provides sophisticated logic for shortening text by either character or word count. It is designed to handle both plain text and complex HTML.

Character and Word Truncation​

Truncator wraps a string and provides two primary methods: chars() and words().

from django.utils.text import Truncator

text = "The quick brown fox jumps over the lazy dog"
truncator = Truncator(text)

# Truncate to 20 characters (includes the ellipsis)
print(truncator.chars(20)) # "The quick brown fox…"

# Truncate to 4 words
print(truncator.words(4))  # "The quick brown fox…"

HTML-Aware Truncation​

A standout feature of Truncator is its ability to truncate HTML without breaking the markup. By passing html=True, the utility uses internal parsers (TruncateCharsHTMLParser or TruncateWordsHTMLParser) to:

  1. Count only the visible text, ignoring the length of HTML tags.
  2. Ensure that any tags opened within the truncated segment are correctly closed.
html_content = "<div><p>Hello <b>world</b>, this is a test.</p></div>"
truncator = Truncator(html_content)

# Truncate to 10 visible characters
# Result: "<div><p>Hello <b>wor…</b></p></div>"
print(truncator.chars(10, html=True))

Linkification with Urlizer​

The Urlizer class in django/utils/html.py is responsible for finding URLs and email addresses in plain text and converting them into HTML <a> tags.

Punctuation Handling​

Urlizer implements complex logic in trim_punctuation() to distinguish between a URL and the punctuation surrounding it. It recognizes trailing periods, commas, and balanced parentheses or brackets, ensuring that a link like (see https://example.com) correctly identifies the URL without including the closing parenthesis.

Protocol Assumptions and HTTPS​

The handle_word() method determines the protocol for schemeless URLs (e.g., www.google.com).

  • Current Behavior: It defaults to http:// unless the URLIZE_ASSUME_HTTPS setting is enabled.
  • Future Change: The codebase includes a RemovedInDjango70Warning, indicating that the default will switch to https:// in Django 7.0.
# Usage via the urlize wrapper
from django.utils.html import urlize

# Standard linkification
urlize("Visit djangoproject.com") 
# '<a href="http://djangoproject.com" rel="nofollow">djangoproject.com</a>'

# With truncation and nofollow disabled
urlize("https://verylongurl.com/path", trim_url_limit=15, nofollow=False)
# '<a href="https://verylongurl.com/path">https://verylon…</a>'

HTML Tag Stripping​

For scenarios where HTML must be removed entirely, the codebase provides MLStripper in django/utils/html.py.

MLStripper​

MLStripper is a subclass of html.parser.HTMLParser. Unlike a simple regex-based approach, it properly parses the HTML structure, collecting data and entities while discarding tags.

It is used internally by the strip_tags function, which includes a safety mechanism (MAX_STRIP_TAGS_DEPTH = 50) to prevent infinite loops or DoS attacks from maliciously nested tags.

from django.utils.html import strip_tags

unsafe_html = "<b>Bold</b><script>alert('xss')</script><i>Italic</i>"
# Result: "Boldalert('xss')Italic"
clean_text = strip_tags(unsafe_html)

Note that MLStripper preserves character entities (like &amp;) and numeric references (like &#123;) by implementing handle_entityref and handle_charref, ensuring that the resulting text remains readable.