Models

To store indexed data in the database, you need to define models that are Python classes representing database tables. DipDup uses customized Tortoise ORM to manage models and transactions.

DipDup

Our storage layer is based on Tortoise ORM. This library is fast, flexible, and has a syntax familiar to Django users. We have extended it with some useful features like a copy-on-write rollback mechanism, caching, and more.

Before we dive into the details, here's an important note:

Use Tortoise ORM docs as a reference. We will describe only DipDup-specific features here.

Defining models

Project models should be placed in the models directory in the project root. By default, the __init__.py module is created on project initialization, but you can use any structure you want. Models from nested packages will be discovered automatically.

Here's an example containing all available fields:

import enum

from dipdup import fields
from dipdup.models import Model


class ExampleModel(Model):
    id = fields.IntField(primary_key=True)
    array = fields.ArrayField()
    big_int = fields.BigIntField()
    binary = fields.BinaryField()
    boolean = fields.BooleanField()
    decimal = fields.DecimalField(10, 2)
    date = fields.DateField()
    datetime = fields.DatetimeField()
    enum_ = fields.EnumField(enum.Enum)
    float = fields.FloatField()
    int_enum = fields.IntEnumField(enum.IntEnum)
    int_ = fields.IntField()
    json = fields.JSONField()
    small_int = fields.SmallIntField()
    text = fields.TextField()
    time_delta = fields.TimeDeltaField()
    time = fields.TimeField()
    uuid = fields.UUIDField()

    relation: fields.ForeignKeyField['ExampleModel'] = fields.ForeignKeyField(
        'models.ExampleModel', related_name='reverse_relation'
    )
    m2m_relation: fields.ManyToManyField['ExampleModel'] = fields.ManyToManyField(
        'models.ExampleModel', related_name='reverse_m2m_relation'
    )

    created_at = fields.DatetimeField(auto_now_add=True)
    updated_at = fields.DatetimeField(auto_now=True)

    relation_id: int
    m2m_relation_ids: list[int]

    class Meta:
        abstract = True

Pay attention to the imports: field and model classes must be imported from the dipdup package instead of tortoise to make our extensions work.

Some limitations are applied to model names and fields to avoid ambiguity in the GraphQL API:

• Table names must be in snake_case
• Model fields must be in snake_case
• Model fields must differ from the table name

Basic usage

Now you can use these models in hooks and handlers.

import demo_tezos_dao.models as models
from demo_tezos_dao.types.registry.tezos_parameters.propose import ProposeParameter
from demo_tezos_dao.types.registry.tezos_storage import RegistryStorage
from dipdup.context import HandlerContext
from dipdup.models.tezos import TezosTransaction


async def on_propose(
    ctx: HandlerContext,
    propose: TezosTransaction[ProposeParameter, RegistryStorage],
) -> None:
    dao = await models.DAO.get(address=propose.data.target_address)
    await models.Proposal(dao=dao).save()

Visit Tortoise ORM docs for more examples.

Caching

Some models can be cached to avoid unnecessary database queries. Use the CachedModel base class for this purpose. It's a drop-in replacement for dipdup.models.Model with additional methods to manage the cache.

• cached_get — get a single object from the cache or the database
• cached_get_or_none — the same, but a None result is also cached
• cache — cache a single object

See the demo_evm_uniswap project for real-life examples.

Differences from Tortoise ORM

This section describes the differences between DipDup and Tortoise ORM. You most likely won't notice them, but it's better to be aware of them.

Fields

We use different column types for some fields to avoid unnecessary reindexing for minor schema changes. Some fields also behave slightly differently for the sake of performance.

• TextField can be indexed and used as a primary key. We can afford this since MySQL is not supported.
• DecimalField is stored as DECIMAL(x,y) both in SQLite and PostgreSQL. In Tortoise ORM it's VARCHAR(40) in SQLite for some reason. DipDup doesn't have an upper bound for precision.
• EnumField is stored in TEXT column in DipDup. There's no need in VARCHAR in SQLite and PostgreSQL. You can still add max_length directive for additional validation, but it won't affect the database schema.

DipDup also provides ArrayField for native array support in PostgreSQL.

Querysets

Querysets are not copied between chained calls. Consider the following example:

await dipdup.models.Index.filter().order_by('-level').first()

In Tortoise ORM, each subsequent call creates a new queryset using an expensive copy.copy() call. In DipDup, it's the same queryset, so it's much faster.

Transactions

DipDup manages transactions automatically for indexes, opening one for each level. You cannot open another one. Entering a transaction context manually with in_transaction() will return the same active transaction. For hooks, there's the atomic flag in the configuration.