Models
To store indexed data in the database, you need to define models that are Python classes representing database tables. DipDup uses customized Tortoise ORM to manage models and transactions.
DipDup
Our storage layer is based on Tortoise ORM. This library is fast, flexible, and has a syntax familiar to Django users. We have extended it with some useful features like a copy-on-write rollback mechanism, caching, and more.
Before we dive into the details, here's an important note:
Use Tortoise ORM docs as a reference. We will describe only DipDup-specific features here.
Defining models
Project models should be placed in the models
directory in the project root. By default, the __init__.py
module is created on project initialization, but you can use any structure you want. Models from nested packages will be discovered automatically.
Here's an example containing all available fields:
import enum
from dipdup import fields
from dipdup.models import Model
class ExampleModel(Model):
id = fields.IntField(primary_key=True)
array = fields.ArrayField()
big_int = fields.BigIntField()
binary = fields.BinaryField()
boolean = fields.BooleanField()
decimal = fields.DecimalField(10, 2)
date = fields.DateField()
datetime = fields.DatetimeField()
enum_ = fields.EnumField(enum.Enum)
float = fields.FloatField()
int_enum = fields.IntEnumField(enum.IntEnum)
int_ = fields.IntField()
json = fields.JSONField()
small_int = fields.SmallIntField()
text = fields.TextField()
time_delta = fields.TimeDeltaField()
time = fields.TimeField()
uuid = fields.UUIDField()
relation: fields.ForeignKeyField['ExampleModel'] = fields.ForeignKeyField(
'models.ExampleModel', related_name='reverse_relation'
)
m2m_relation: fields.ManyToManyField['ExampleModel'] = fields.ManyToManyField(
'models.ExampleModel', related_name='reverse_m2m_relation'
)
created_at = fields.DatetimeField(auto_now_add=True)
updated_at = fields.DatetimeField(auto_now=True)
relation_id: int
m2m_relation_ids: list[int]
class Meta:
abstract = True
Pay attention to the imports: field and model classes must be imported from the dipdup
package instead of tortoise
to make our extensions work.
Some limitations are applied to model names and fields to avoid ambiguity in the GraphQL API:
- Table names must be in snake_case
- Model fields must be in snake_case
- Model fields must differ from the table name
Basic usage
Now you can use these models in hooks and handlers.
import demo_tezos_dao.models as models
from demo_tezos_dao.types.registry.tezos_parameters.propose import ProposeParameter
from demo_tezos_dao.types.registry.tezos_storage import RegistryStorage
from dipdup.context import HandlerContext
from dipdup.models.tezos import TezosTransaction
async def on_propose(
ctx: HandlerContext,
propose: TezosTransaction[ProposeParameter, RegistryStorage],
) -> None:
dao = await models.DAO.get(address=propose.data.target_address)
await models.Proposal(dao=dao).save()
Visit Tortoise ORM docs for more examples.
Caching
Some models can be cached to avoid unnecessary database queries. Use the CachedModel
base class for this purpose. It's a drop-in replacement for dipdup.models.Model
with additional methods to manage the cache.
cached_get
— get a single object from the cache or the databasecached_get_or_none
— the same, but a None result is also cachedcache
— cache a single object
See the demo_evm_uniswap
project for real-life examples.
Differences from Tortoise ORM
This section describes the differences between DipDup and Tortoise ORM. You most likely won't notice them, but it's better to be aware of them.
Fields
We use different column types for some fields to avoid unnecessary reindexing for minor schema changes. Some fields also behave slightly differently for the sake of performance.
TextField
can be indexed and used as a primary key. We can afford this since MySQL is not supported.DecimalField
is stored asDECIMAL(x,y)
both in SQLite and PostgreSQL. In Tortoise ORM it'sVARCHAR(40)
in SQLite for some reason. DipDup doesn't have an upper bound for precision.EnumField
is stored inTEXT
column in DipDup. There's no need inVARCHAR
in SQLite and PostgreSQL. You can still addmax_length
directive for additional validation, but it won't affect the database schema.
DipDup also provides ArrayField
for native array support in PostgreSQL.
Querysets
Querysets are not copied between chained calls. Consider the following example:
await dipdup.models.Index.filter().order_by('-level').first()
In Tortoise ORM, each subsequent call creates a new queryset using an expensive copy.copy()
call. In DipDup, it's the same queryset, so it's much faster.
Transactions
DipDup manages transactions automatically for indexes, opening one for each level. You cannot open another one. Entering a transaction context manually with in_transaction()
will return the same active transaction. For hooks, there's the atomic
flag in the configuration.