Parsing Multipart Form Data and File Uploads

The multipart parsing system in this codebase provides a robust, memory-efficient way to process multipart/form-data requests. It is designed to handle large file uploads by streaming data through a series of specialized iterators and handlers rather than loading entire files into memory.

The core logic resides in django/http/multipartparser.py, centered around the MultiPartParser class and a "Lazy Stream" architecture.

The Lazy Stream Architecture

Efficiently parsing multipart data requires the ability to look ahead for boundaries without permanently consuming data from the underlying stream. This is achieved through two primary classes: LazyStream and BoundaryIter.

LazyStream

The LazyStream class wraps a data producer (typically a ChunkIter reading from the raw request) and adds an unget() method. This allows the parser to "push back" bytes onto the front of the stream if it reads too far—for example, when it encounters a boundary.

# django/http/multipartparser.py

class LazyStream:
    def unget(self, bytes):
        """
        Place bytes back onto the front of the lazy stream.
        Future calls to read() will return those bytes first.
        """
        if not bytes:
            return
        self._update_unget_history(len(bytes))
        self.position -= len(bytes)
        self._leftover = bytes + self._leftover

To prevent malicious requests from causing infinite loops (by repeatedly forcing the parser to "unget" the same data), LazyStream tracks its _unget_history and raises a SuspiciousMultipartForm exception if it detects a pattern of excessive rollbacks.

BoundaryIter

BoundaryIter is a producer that yields bytes until it encounters a specific multipart boundary. It uses a "rollback" mechanism (calculated as len(boundary) + 6) to ensure it doesn't accidentally consume a partial boundary as data. When a boundary is found, it uses stream.unget() to return the post-boundary bytes to the main stream and then terminates its own iteration.

The Parsing Flow

The MultiPartParser.parse() method orchestrates the extraction of form fields and files. It returns a tuple of two MultiValueDict objects: one for POST data and one for FILES.

1. Initialization and Handlers

When MultiPartParser is initialized, it extracts the boundary from the CONTENT_TYPE header and determines the optimal chunk size based on the registered upload_handlers.

2. The Parser Loop

The _parse() method iterates over the input using the Parser class. The Parser yields a tuple of (item_type, meta_data, field_stream) for each part of the multipart message.

# django/http/multipartparser.py

for item_type, meta_data, field_stream in Parser(stream, self._boundary):
    # ... logic to handle FIELD vs FILE ...

• item_type: Identifies if the part is a standard form field (FIELD) or a file upload (FILE).
• meta_data: A dictionary of headers for that specific part (e.g., Content-Disposition, Content-Type).
• field_stream: A LazyStream containing only the data for this specific part.

3. Handling Fields

If the part is a FIELD, the parser reads the field_stream and appends the value to the _post QueryDict. It enforces DATA_UPLOAD_MAX_MEMORY_SIZE during this read to prevent memory exhaustion from oversized text fields.

4. Handling Files and Upload Handlers

If the part is a FILE, the parser interacts with the configured upload_handlers. The process follows a specific lifecycle for each file:

1. handler.new_file(): Called with the field name, filename, and content type.
2. Chunked Streaming: The parser iterates over field_stream in chunks. Each chunk is passed to handler.receive_data_chunk(chunk, counters[i]).
3. handler.file_complete(): Once the field_stream for that part is exhausted, handle_file_complete() is called. If a handler returns a file-like object, it is added to the FILES dictionary.

Security and Constraints

The parser implements several safeguards to protect the server from resource exhaustion and malicious input:

• Field Limits: DATA_UPLOAD_MAX_NUMBER_FIELDS limits the total number of form fields (including those in multipart data) to prevent HashDoS attacks.
• File Limits: DATA_UPLOAD_MAX_NUMBER_FILES restricts the number of files that can be uploaded in a single request.
• Memory Limits: DATA_UPLOAD_MAX_MEMORY_SIZE limits the total size of the request body (excluding file data handled by streaming handlers).
• Filename Sanitization: The sanitize_file_name method strips path separators (like ../) and non-printable characters from uploaded filenames to prevent directory traversal attacks.

# Example of sanitization logic in MultiPartParser
file_name = file_name.rsplit("/")[-1]
file_name = file_name.rsplit("\\")[-1]
file_name = "".join([char for char in file_name if char.isprintable()])

Integration with HttpRequest

In practice, developers rarely interact with MultiPartParser directly. Instead, it is invoked by django.http.request.HttpRequest when the POST or FILES attributes are first accessed on a multipart request.

# django/http/request.py (Conceptual usage)
def _load_post_and_files(self):
    if self.content_type == 'multipart/form-data':
        parser = MultiPartParser(self.META, self, self.upload_handlers, self.encoding)
        self._post, self._files = parser.parse()

This integration ensures that multipart data is parsed lazily and only when needed, while still providing a consistent API for accessing form data and uploaded files.