Skip to main content

Querying and Managers

This tutorial walks you through the process of retrieving data from your database using the core querying components of this framework: Managers and QuerySets. You will learn how to create custom query logic, chain filters lazily, and optimize data retrieval for large datasets.

Prerequisites​

To follow this tutorial, you should have a basic Django environment configured. The examples below use standard model fields and assume you are working within a Django application.

Step 1: Defining a Custom Manager​

A Manager is the interface through which database query operations are provided to model classes. By default, every model has an objects manager, but you can define your own to encapsulate common query logic.

In django/db/models/manager.py, the Manager class is the standard implementation. You can extend it to add domain-specific methods.

from django.db import models

class PersonManager(models.Manager):
    def get_fun_people(self):
        # self refers to the manager instance
        return self.filter(fun=True)

class Person(models.Model):
    first_name = models.CharField(max_length=30)
    fun = models.BooleanField(default=False)
    
    # Assign the custom manager
    objects = PersonManager()

By overriding or adding methods to PersonManager, you create a reusable entry point for queries. Note that Manager methods typically return a QuerySet, allowing for further chaining.

Gotcha: Managers are only accessible via the model class (e.g., Person.objects). Attempting to access them via a model instance (e.g., my_person.objects) will raise an AttributeError, as enforced by the ManagerDescriptor in django/db/models/manager.py.

Step 2: Leveraging the Lazy QuerySet API​

When you call a method like filter() or exclude() on a manager, it returns a QuerySet. In django/db/models/query.py, the QuerySet class is designed to be "lazy"—it does not hit the database until it is absolutely necessary.

# This line does NOT execute a database query
query = Person.objects.filter(first_name__startswith="A").exclude(fun=False)

# The query is executed only when you evaluate it
for person in query:
    print(person.first_name)

The QuerySet uses a method called _chain() to return a new instance for every filter operation. Evaluation is triggered by:

  • Iterating over the QuerySet (calls __iter__).
  • Slicing with a step (e.g., query[::2]).
  • Calling len(query).
  • Calling bool(query).

Once evaluated, the results are stored in _result_cache to avoid redundant database hits.

Step 3: Creating Chainable Logic with Custom QuerySets​

A common pattern in this codebase is to define logic on a QuerySet subclass and then promote it to the Manager level using as_manager(). This allows you to chain your custom methods together.

from django.db import models

class PersonQuerySet(models.QuerySet):
    def active(self):
        return self.filter(is_active=True)
    
    def seniors(self):
        return self.filter(age__gte=65)

class Person(models.Model):
    is_active = models.BooleanField(default=True)
    age = models.IntegerField()
    
    # Promote QuerySet methods to the Manager
    objects = PersonQuerySet.as_manager()

# Now you can chain custom methods!
active_seniors = Person.objects.active().seniors()

The as_manager() method in QuerySet uses Manager.from_queryset() to dynamically create a manager class that proxies all public QuerySet methods.

Step 4: Optimizing Data Retrieval​

Sometimes you don't need full model instances. You can use values() or values_list() to change the format of the returned data. This modifies the _iterable_class used by the QuerySet.

# Returns a QuerySet of dictionaries (uses ValuesIterable)
names_dict = Person.objects.values("first_name")

# Returns a QuerySet of tuples (uses ValuesListIterable)
names_list = Person.objects.values_list("first_name", flat=True)

For very large datasets where caching the entire result set in memory is undesirable, use the iterator() method. This bypasses the _result_cache and fetches rows as needed.

# Efficiently process 10,000+ records
for person in Person.objects.all().iterator(chunk_size=100):
    # Process each person without loading all into memory
    pass

When using iterator() with prefetch_related(), you must provide a chunk_size to ensure the framework can correctly manage the prefetching logic.

Summary of Results​

By the end of this tutorial, you have:

  1. Created a Custom Manager to encapsulate specific business logic.
  2. Utilized the Lazy API of QuerySet to build complex queries without premature execution.
  3. Implemented Chainable QuerySets using as_manager() for a DRY (Don't Repeat Yourself) architecture.
  4. Applied Performance Optimizations like values() and iterator() to handle data efficiently.

For next steps, explore django/db/models/query.py to see how advanced methods like annotate() and aggregate() allow for complex database-level calculations.