Data Serialization Formats

The serialization framework in this codebase provides a unified interface for converting Django querysets into various data formats and reconstructing model instances from those formats. This is primarily used for creating fixtures, exporting data, and inter-process communication.

The system is built around a central registry in django.core.serializers that maps format names (like json, xml, yaml) to specific serializer modules.

Core Serialization Interface

The primary entry points are the serialize and deserialize functions located in django/core/serializers/__init__.py.

Serializing Data

The serialize function takes a format name and a queryset, returning the serialized data as a string (or writing it to a stream).

from django.core import serializers
from myapp.models import MyModel

# Serialize to a JSON string
data = serializers.serialize("json", MyModel.objects.all())

# Serialize to a file-like object (stream)
with open("data.xml", "w") as f:
    serializers.serialize("xml", MyModel.objects.all(), stream=f)

Deserializing Data

The deserialize function returns an iterator of DeserializedObject instances. These objects are wrappers around unsaved model instances.

for deserialized_obj in serializers.deserialize("json", data):
    # The actual model instance is in .object
    print(deserialized_obj.object)
    # You must call .save() to persist it to the database
    deserialized_obj.save()

The Serialization Lifecycle

All serializers inherit from the abstract Serializer class in django/core/serializers/base.py. This class implements a template pattern that defines the lifecycle of serializing a queryset:

1. start_serialization(): Initializes the output (e.g., opening a root XML tag or a JSON array).
2. start_object(obj): Called at the beginning of each model instance.
3. Field Handling:
   • handle_field(obj, field): Processes standard fields (integers, strings, etc.).
   • handle_fk_field(obj, field): Processes ForeignKey relationships.
   • handle_m2m_field(obj, field): Processes ManyToManyField relationships.
4. end_object(obj): Finalizes the object representation.
5. end_serialization(): Closes the output.

Supported Formats

Python Objects

The PythonSerializer in django/core/serializers/python.py is the foundation for many other formats. It converts a queryset into a list of standard Python dictionaries. It is marked as internal_use_only = True because its output is intended to be consumed by other serializers (like JSON or YAML) rather than used directly as a public format.

JSON and JSON Lines

The JSONSerializer (django/core/serializers/json.py) extends the Python serializer. It uses DjangoJSONEncoder to handle Django-specific types like datetime, Decimal, and UUID that the standard json module cannot process.

The JSONLSerializer (django/core/serializers/jsonl.py) implements the JSON Lines format, where each object is serialized as a single JSON object on a new line. This is particularly useful for large datasets as it allows for line-by-line processing.

YAML

The YAML implementation in django/core/serializers/pyyaml.py requires the PyYAML library. It uses a custom DjangoSafeDumper to ensure that Decimal and time objects are represented as strings, improving interoperability with other languages that might consume the YAML.

XML

Unlike the others, the XMLSerializer (django/core/serializers/xml_serializer.py) does not inherit from the Python serializer. It uses SimplerXMLGenerator to produce a structured XML document. It handles nested natural keys by rolling them out as sub-elements within the <field> tag.

Natural Keys

Natural keys allow objects to be identified by a unique set of attributes rather than a database-specific primary key (like an auto-incrementing ID). This is critical for moving data between databases where IDs might conflict.

To use natural keys during serialization, pass the following flags to serialize:

• use_natural_foreign_keys=True: Uses the natural_key() method of related models to represent foreign keys.
• use_natural_primary_keys=True: Omits the pk field in the output if the model itself defines a natural_key() method.

The Deserializer uses Model.objects.get_by_natural_key() to resolve these references back into database IDs during the import process.

DeserializedObject and Persistence

When you iterate over a deserializer, you receive DeserializedObject instances (django/core/serializers/base.py).

The save() method on this class is unique:

• It calls models.Model.save_base(..., raw=True).
• Setting raw=True tells Django to skip custom save() logic and signals to listeners (via pre_save/post_save) that the data is being loaded from a serialized source.
• It handles the saving of Many-to-Many relationships after the primary object has been saved.

Advanced Configuration

Custom Serializers

You can register custom serialization formats using the SERIALIZATION_MODULES setting. This dictionary maps the format name to the module path containing the Serializer and Deserializer classes.

# settings.py
SERIALIZATION_MODULES = {
    "csv": "myapp.serializers.csv_serializer",
}

Large Datasets

For large querysets, the serialize method supports a progress_output parameter. If provided with a stream (like sys.stdout), it uses the ProgressBar class from django/core/serializers/base.py to provide visual feedback during the process.

import sys
serializers.serialize(
    "json", 
    LargeModel.objects.all(), 
    progress_output=sys.stdout, 
    object_count=LargeModel.objects.count()
)