To store indexed data in the database, you need to define models, that are Python classes that represent database tables. DipDup uses a custom ORM to manage models and transactions.
Our storage layer is based on Tortoise ORM. This library is fast, flexible, and has a syntax familiar to Django users. We have extended it with some useful features like a copy-on-write rollback mechanism, caching, and more. We plan to make things official and fork Tortoise ORM under a new name, but it's not ready yet. For now, let's call our implementation DipDup ORM.
Before we begin to dive into the details, here's an important note:
You can use Tortoise ORM docs as a reference. We will describe only DipDup-specific features here.
Project models should be placed in the
models directory in the project root. By default, the
__init__.py module is created on project initialization, but you can use any structure you want. Models from nested packages will be discovered automatically.
Here's an example containing all available fields:
import enum from dipdup import fields from dipdup.models import Model class ExampleModel(Model): id = fields.IntField(pk=True) array = fields.ArrayField() big_int = fields.BigIntField() binary = fields.BinaryField() boolean = fields.BooleanField() decimal = fields.DecimalField(10, 2) date = fields.DateField() datetime = fields.DatetimeField() enum_ = fields.EnumField(enum.Enum) float = fields.FloatField() int_enum = fields.IntEnumField(enum.IntEnum) int_ = fields.IntField() json = fields.JSONField() small_int = fields.SmallIntField() text = fields.TextField() time_delta = fields.TimeDeltaField() time = fields.TimeField() uuid = fields.UUIDField() relation: fields.ForeignKeyField['ExampleModel'] = fields.ForeignKeyField( 'models.ExampleModel', related_name='reverse_relation' ) m2m_relation: fields.ManyToManyField['ExampleModel'] = fields.ManyToManyField( 'models.ExampleModel', related_name='reverse_m2m_relation' ) created_at = fields.DatetimeField(auto_now_add=True) updated_at = fields.DatetimeField(auto_now=True) relation_id: int m2m_relation_ids: list[int] class Meta: abstract = True
Pay attention to the imports: field and model classes must be imported from
dipdup package instead of
tortoise to make our extensions work.
Some limitations are applied to model names and fields to avoid ambiguity in GraphQL API:
- Table names must be in snake_case
- Model fields must be in snake_case
- Model fields must differ from the table name
Now you can use these models in hooks and handlers.
import demo_dao.models as models from demo_dao.types.registry.tezos_parameters.propose import ProposeParameter from demo_dao.types.registry.tezos_storage import RegistryStorage from dipdup.context import HandlerContext from dipdup.models.tezos_tzkt import TzktTransaction async def on_propose( ctx: HandlerContext, propose: TzktTransaction[ProposeParameter, RegistryStorage], ) -> None: dao = await models.DAO.get(address=propose.data.target_address) await models.Proposal(dao=dao).save()
Visit Tortose ORM docs for more examples.
Some models can be cached to avoid unnecessary database queries. Use
CachedModel base class for this purpose. It's a drop-in replacement for
dipdup.models.Model, but with additional methods to manage the cache.
cached_get— get a single object from the cache or the database
cached_get_or_none— the same, but None result is also cached
cache— cache a single object
demo_uniswap project for real-life examples.
This section describes the differences between DipDup and Tortoise ORM. Most likely won't notice them, but it's better to be aware of them.
We use different column types for some fields to avoid unnecessary reindexing for minor schema changes. Some fields also behave slightly differently for the sake of performance.
TextFieldcan be indexed and used as a primary key. We can afford this since MySQL is not supported.
DecimalFieldis stored as
DECIMAL(x,y)both in SQLite and PostgreSQL. In Tortoise ORM it's
VARCHAR(40)in SQLite for some reason. DipDup ORM doesn't have an upper bound for precision.
EnumFieldis stored in
TEXTcolumn in DipDup ORM. There's no need in
VARCHARin SQLite and PostgreSQL. You can still add
max_lengthdirective for additional validation, but it won't affect the database schema.
We also have
ArrayField for native array support in PostgreSQL.
Querysets are not copied between chained calls. Consider the following example:
In Tortoise ORM each subsequent call creates a new queryset using an expensive
copy.copy()` call. In DipDup ORM it's the same queryset, so it's much faster.
DipDup manages transactions automatically for indexes opening one for each level. You can't open another one. Entering a transaction context manually with
in_transaction() will return the same active transaction. For hooks, there's the
atomic flag in the configuration.