Overview

Relevant source files

• .bumpversion.cfg
• .cookiecutterrc
• README.rst
• django_celery_results/__init__.py
• docs/includes/introduction.txt

This document provides an introduction to the django-celery-results project, a Django application that enables storing Celery task and group results using the Django ORM or cache framework. It serves as a foundation for understanding the purpose, architecture, and key components of the system before diving into more specific documentation.

Purpose and Scope

Django-celery-results allows you to store and retrieve Celery task and group results using Django's Object-Relational Mapping (ORM) system or Django's cache framework. The primary goals of this extension are to:

1. Provide a persistent storage solution for Celery task results
2. Allow querying of task results using Django's familiar ORM interface
3. Support visualization and management through Django's admin interface
4. Enable storage of both individual task and group task results
5. Support Celery chord operations through database synchronization

For detailed installation instructions, please see Installation and Configuration. For specific usage examples, refer to Usage.

Sources: README.rst15-21 docs/includes/introduction.txt10-14

System Architecture

Core Components

Django-Celery-Results System Architecture

Django-celery-results consists of several interconnected components:

1. Models: Define the structure for storing task and group results

   • TaskResult: Stores information about individual tasks
   • GroupResult: Stores information about task groups
   • ChordCounter: Used for chord synchronization

2. Backends: Provide storage mechanisms for Celery results

   • DatabaseBackend: Stores results in the database using Django ORM
   • CacheBackend: Stores results in Django's cache framework

3. Managers: Handle database operations for models

   • TaskResultManager: Manages operations on TaskResult
   • GroupResultManager: Manages operations on GroupResult

4. Admin: Provide Django admin interface for result management

   • TaskResultAdmin: Admin interface for TaskResult model
   • GroupResultAdmin: Admin interface for GroupResult model

5. Views: Provide API endpoints for accessing result data

   • Task status views
   • Group status views

Sources: README.rst16-21 docs/includes/introduction.txt12-14

Data Flow

Task Result Storage and Retrieval Flow

The diagram above illustrates the typical flow of data through the system:

1. A Celery task is executed by a Celery worker
2. When the task completes, its result is stored using a backend
3. The backend processes the result (encoding it and collecting extended properties)
4. The manager creates or updates a model instance in the database
5. Later, the task result can be retrieved following the reverse path, with the content being decoded before returning

Sources: README.rst16-21 docs/includes/introduction.txt10-14

Key Models

Model Relationships

Django-celery-results defines three primary models:

1. TaskResult: Stores individual task results with the following key fields:

   • task_id: Unique identifier for the task (primary key)
   • status: Current status of the task (e.g., SUCCESS, FAILURE)
   • result: The actual result of the task
   • date_created: When the task record was created
   • date_started: When the task started execution (if tracking is enabled)
   • date_done: When the task was completed
   • traceback: Error traceback (if the task failed)
   • task_name: Name of the task function
   • worker: Name of the worker that executed the task
   • task_args: Arguments passed to the task (if extended result storage is enabled)
   • task_kwargs: Keyword arguments passed to the task (if extended result storage is enabled)
   • periodic_task_name: Name of the periodic task (if applicable)
   • content_type: Content type of the result (e.g., application/json)
   • content_encoding: Encoding of the result (e.g., utf-8)
   • meta: Additional metadata about the task

2. GroupResult: Stores information about task groups:

   • group_id: Unique identifier for the group (primary key)
   • result: The combined results of the tasks in the group
   • date_created: When the group record was created
   • date_done: When the group completed
   • content_type: Content type of the result
   • content_encoding: Encoding of the result

3. ChordCounter: Used for chord synchronization:

   • group_id: Identifier for the group (chord header, primary key)
   • sub_tasks: Tasks that are part of the chord header
   • count: Number of uncompleted tasks

Sources: README.rst18-21 docs/includes/introduction.txt12-14

Storage Backends

Result Backend Configuration

Django-celery-results provides two types of result backends:

Database Backend

The primary backend that stores task and group results in the database using Django ORM. This provides persistent storage that survives application restarts and allows for complex queries.

Configuration:

Additional configuration options:

Cache Backend

An alternative backend that stores results in Django's cache framework. This is typically faster but less permanent than the database backend.

Configuration:

Sources: README.rst16-21

Version and Compatibility

• Current version: 2.6.0
• Requires Celery 4.0.0 or newer
• Supports Django versions 3.2 and above (checks performed in django_celery_results/__init__.py35-36)

Version information is stored as:

And parsed into a structured format:

Key features include:

• Support for storing task arguments and keyword arguments
• Support for GroupResult and chord synchronization
• Periodic task name tracking from django-celery-beat
• Integration with Django admin interface
• MySQL compatibility with configurable key length

Sources: django_celery_results/__init__.py12-31 .bumpversion.cfg1-8

Use Cases

Task Result Storage and Retrieval

Django-celery-results allows you to:

• Store task results persistently in a database
• Query task results using Django's ORM
• Filter results by status, task name, or other attributes
• Display and manage results through Django's admin interface

Group Result Management

The extension also supports:

• Storing results of task groups
• Tracking completion of all tasks in a group
• Retrieving combined results of group tasks

Chord Synchronization

For Celery chords (a group of tasks followed by a callback):

• Track completion of all tasks in the chord header
• Automatically trigger the callback when all header tasks complete
• Provide reliable chord execution even with transient workers

Sources: README.rst16-20 docs/includes/introduction.txt10-14 Changelog13-23

Integration with Existing Systems

Integration Architecture

Django-celery-results integrates with:

1. Django:

   • Added to INSTALLED_APPS in Django settings:

   • Requires running migrations:

     python manage.py migrate django_celery_results
     

   • Uses Django's ORM for database access
   • Provides Django admin integration
   • Uses Django's cache framework (for cache backend)
   • Has special configuration for MySQL:

2. Celery:

   • Acts as a result backend for Celery
   • Stores results of Celery tasks and groups
   • Supports Celery's chord synchronization mechanism
   • Used by specifying in settings:

   • Tracks periodic tasks from django-celery-beat

Sources: README.rst15-102 docs/includes/introduction.txt10-17

Conclusion

Django-celery-results provides a robust solution for storing and retrieving Celery task and group results using Django's familiar ORM interface. It offers both database and cache storage options, supports advanced Celery features like chords, and integrates seamlessly with Django's admin interface. This makes it an excellent choice for Django applications that need persistent and queryable storage for their Celery task results.

For more detailed information about specific components, please refer to the other pages in this documentation.

Sources: README.rst15-20 docs/includes/introduction.txt10-17 Changelog9-23