F.A.Q.
This page contains answers to the most popular questions about DipDup guts. If you landed here - congrats, you're writing a pretty advanced indexer!
Indexing
How to index similar but not identical contracts as a single entity?
Multiple contracts can provide the same interface (like FA1.2 and FA2 standard tokens) but have a different storage structure. If you try to use the same typename for them, indexing will fail. However, you can modify typeclasses manually. Modify types/<typename>/storage.py
file and comment out unique fields that are not important for your index:
class ContractStorage(BaseModel):
class Config:
extra = Extra.ignore
common_ledger: dict[str, str]
# unique_field_foo: str
# unique_field_bar: str
Don't forget Extra.ignore
Pydantic hint, otherwise storage deserialization will fail. To restore the original typeclass, remove modified file and run dipdup init
again. You can also add --force
flag to overwrite all ABIs and typeclasses.
How to use off-chain datasources?
DipDup provides convenient helpers to process off-chain data like market quotes or IPFS metadata. Follow the tips below to use them most efficiently.
- Do not perform off-chain requests in handers until necessary. Handlers need to be as fast as possible not to block the database transaction. Use hooks instead, enriching indexed data on-demand.
- Use generic
http
datasource for external APIs instead of plainaiohttp
requests. It makes available the same features DipDup uses for internal requests: retry with backoff, rate limiting, Prometheus integration etc. - Database tables that store off-chain data can be marked as immune, to speed up reindexing.
How to process indexes in a specific order?
Indexes of all kinds are fully independent. They are processed in parallel, have their own message queues, and don't share any state. It is one of the essential DipDup concepts, so there's no "official" way to manage the order of indexing.
Avoid using sync primitives like asyncio.Event
or asyncio.Lock
in handlers. Indexing will be stuck forever, waiting for the database transaction to complete.
Instead, save raw data in handlers and process it later with hooks when all conditions are met. For example, process data batch only when all indexes in the dipdup_index
table have reached a specific level.
Database
How to perform database migrations?
DipDup does not provide any tooling for database migrations. The reason is that schema changes almost always imply reindexing when speaking about indexers. However, you can perform migrations yourself using any tool you like. First, disable schema hash check in config:
advanced:
reindex:
schema_modified: ignore
You can also use the schema approve
command for a single schema change.
To determine what manual modifications you need to apply after changing models.py
, you can compare raw SQL schema before and after the change.
- timestamp = fields.DatetimeField()
+ timestamp = fields.DatetimeField(auto_now=True)
dipdup schema export > old-schema.sql
# ...modify `models.py` here...
dipdup schema export > new-schema.sql
diff old-schema.sql new-schema.sql
76c76
< "timestamp" TIMESTAMP NOT NULL,
---
> "timestamp" TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
Now you can prepare and execute an ALTER TABLE
query manually or using SQL hooks.
I get schema_modified
error, but I didn't change anything
schema_modified
error, but I didn't change anythingDipDup compares the current schema hash with the one stored in the database. If they don't match, it throws an error. If models were not modified, most likely the reason is incorrect model definitions. e.g. if you define a timestamp field like this…
timestamp = fields.DatetimeField(default=datetime.utcnow())
…schema will be different every time you run DipDup, because datetime.utcnow()
is evaluated on a module import.
$ dipdup schema export > schema.sql
$ dipdup schema export > same-schema.sql
$ diff schema.sql same-schema.sql
116c116
< "timestamp" TIMESTAMP NOT NULL DEFAULT '2023-04-19T21:16:31.183036',
---
> "timestamp" TIMESTAMP NOT NULL DEFAULT '2023-04-19T21:16:36.231221',
You need to make the following change in models.py:
< timestamp = fields.DatetimeField(default=datetime.utcnow())
> timestamp = fields.DatetimeField(auto_now=True)
We plan to improve field classes in future releases to accept callables as default values.
Why am I getting decimal.InvalidOperation
error?
decimal.InvalidOperation
error?If your models contain DecimalField
s, you may encounter this error when performing arithmetic operations. It's because the value is too big to fit into the current decimal context.
class Token(Model):
id = fields.TextField(pk=True)
volume = fields.DecimalField(decimal_places=18, max_digits=76)
...
Default decimal precision in Python is 28 digits. DipDup tries to increase it automatically guessing the value from the schema. It works in most cases, but not for really big numbers. You can increase the precision manually in config.
advanced:
decimal_precision: 128
Don't forget to reindex after this change. When decimal context precision is adjusted you'll get a warning in the logs.
WARNING dipdup.database Decimal context precision has been updated: 28 -> 128