Models

To store indexed data in the database, you need to define models, that are Python classes that represent database tables. DipDup uses a custom ORM to manage models and transactions.

DipDup ORM

Our storage layer is based on Tortoise ORM. This library is fast, flexible, and has a syntax familiar to Django users. We have extended it with some useful features like a copy-on-write rollback mechanism, caching, and more. We plan to make things official and fork Tortoise ORM under a new name, but it's not ready yet. For now, let's call our implementation DipDup ORM.

Before we begin to dive into the details, here's an important note:

You can use Tortoise ORM docs as a reference. We will describe only DipDup-specific features here.

Defining models

Project models should be placed in the models directory in the project root. By default, the __init__.py module is created on project initialization, but you can use any structure you want. Models from nested packages will be discovered automatically.

Here's an example containing all available fields:

import enum

from dipdup import fields
from dipdup.models import Model


class ExampleModel(Model):
    id = fields.IntField(primary_key=True)
    array = fields.ArrayField()
    big_int = fields.BigIntField()
    binary = fields.BinaryField()
    boolean = fields.BooleanField()
    decimal = fields.DecimalField(10, 2)
    date = fields.DateField()
    datetime = fields.DatetimeField()
    enum_ = fields.EnumField(enum.Enum)
    float = fields.FloatField()
    int_enum = fields.IntEnumField(enum.IntEnum)
    int_ = fields.IntField()
    json = fields.JSONField()
    small_int = fields.SmallIntField()
    text = fields.TextField()
    time_delta = fields.TimeDeltaField()
    time = fields.TimeField()
    uuid = fields.UUIDField()

    relation: fields.ForeignKeyField['ExampleModel'] = fields.ForeignKeyField(
        'models.ExampleModel', related_name='reverse_relation'
    )
    m2m_relation: fields.ManyToManyField['ExampleModel'] = fields.ManyToManyField(
        'models.ExampleModel', related_name='reverse_m2m_relation'
    )

    created_at = fields.DatetimeField(auto_now_add=True)
    updated_at = fields.DatetimeField(auto_now=True)

    relation_id: int
    m2m_relation_ids: list[int]

    class Meta:
        abstract = True

Pay attention to the imports: field and model classes must be imported from dipdup package instead of tortoise to make our extensions work.

Some limitations are applied to model names and fields to avoid ambiguity in GraphQL API:

• Table names must be in snake_case
• Model fields must be in snake_case
• Model fields must differ from the table name

Basic usage

Now you can use these models in hooks and handlers.

import demo_tezos_dao.models as models
from demo_tezos_dao.types.registry.tezos_parameters.propose import ProposeParameter
from demo_tezos_dao.types.registry.tezos_storage import RegistryStorage
from dipdup.context import HandlerContext
from dipdup.models.tezos import TezosTransaction


async def on_propose(
    ctx: HandlerContext,
    propose: TezosTransaction[ProposeParameter, RegistryStorage],
) -> None:
    dao = await models.DAO.get(address=propose.data.target_address)
    await models.Proposal(dao=dao).save()

Visit Tortose ORM docs for more examples.

Caching

Some models can be cached to avoid unnecessary database queries. Use CachedModel base class for this purpose. It's a drop-in replacement for dipdup.models.Model, but with additional methods to manage the cache.

• cached_get — get a single object from the cache or the database
• cached_get_or_none — the same, but None result is also cached
• cache — cache a single object

See demo_evm_uniswap project for real-life examples.

Differences from Tortoise ORM

This section describes the differences between DipDup and Tortoise ORM. Most likely won't notice them, but it's better to be aware of them.

Fields

We use different column types for some fields to avoid unnecessary reindexing for minor schema changes. Some fields also behave slightly differently for the sake of performance.

• TextField can be indexed and used as a primary key. We can afford this since MySQL is not supported.
• DecimalField is stored as DECIMAL(x,y) both in SQLite and PostgreSQL. In Tortoise ORM it's VARCHAR(40) in SQLite for some reason. DipDup ORM doesn't have an upper bound for precision.
• EnumField is stored in TEXT column in DipDup ORM. There's no need in VARCHAR in SQLite and PostgreSQL. You can still add max_length directive for additional validation, but it won't affect the database schema.

We also have ArrayField for native array support in PostgreSQL.

Querysets

Querysets are not copied between chained calls. Consider the following example:

await dipdup.models.Index.filter().order_by('-level').first()

In Tortoise ORM each subsequent call creates a new queryset using an expensive copy.copy()` call. In DipDup ORM it's the same queryset, so it's much faster.

Transactions

DipDup manages transactions automatically for indexes opening one for each level. You can't open another one. Entering a transaction context manually with in_transaction() will return the same active transaction. For hooks, there's the atomic flag in the configuration.