Config
Developing a DipDup indexer begins with creating a YAML config file of a specific format. DipDup expects it to be located in the project root and named dipdup.yaml
. However, you can provide any path with a -c
CLI option.
Structure
Config consists of multiple top-level mappings. In the table below they are grouped by sections for convenience, but in the config file, they can be in any order. The only required section is Header.
See Config reference guide for the full list of available options.
Header | spec_version * | DipDup project specification; currently "3.0" |
package * | Python package name | |
Inventory | database | Database configuration |
contracts | Contracts that need to be indexed | |
datasources | Datasources to gather information from | |
Indexes | indexes | Index definitions |
templates | Also index definitions, but with placeholders that make them reusable | |
Hooks | hooks | Callbacks to run manually or by schedule |
jobs | Schedules for hooks | |
Integrations | api | Internal API configuration |
hasura | Hasura GraphQL Engine configuration | |
sentry | Sentry configuration | |
prometheus | Prometheus configuration | |
Miscellaneous | advanced | Tunables that affect framework behavior |
custom | Mapping of user-defined values; neither typed nor validated | |
logging | Configure logging verbosity |
Merging multiple files
DipDup allows you to customize the configuration for a specific environment or workflow. It works similarly to docker-compose anchors but only for top-level sections. If you want to override a nested property, you need to recreate a whole top-level section. To merge several DipDup config files, provide the -c
command-line option multiple times:
dipdup -c dipdup.yaml -c configs/dipdup.sqlite.yaml run
Use config export
and config env
commands to check the resulting config used by DipDup.
Environment variables
For the list of variables to configure DipDup see this page.
DipDup supports compose-style variable expansion with an optional default value. Use this feature to store sensitive data outside of the configuration file and make your app fully declarative. If a required variable is not set, DipDup will fail with an error. You can use these placeholders anywhere throughout the configuration file.
database:
kind: postgres
host: ${POSTGRES_HOST:-localhost}
password: ${POSTGRES_PASSWORD}
There are multiple ways to pass environment variables to DipDup:
- Export them in the shell before running DipDup
- Create the env file and pass it to DipDup with the
-e
CLI option
For every config file in the deploy
project directory, DipDup will create a corresponding .env.default
file with all the variables used in the config. Copy it, remove the .default
suffix and fill in the values.
POSTGRES_HOST=localhost
POSTGRES_PASSWORD=
You can use dipdup -e .env config export --unsafe
command to ensure that env variables resolve as expected, but avoid sharing the output with third parties.
Contract typenames
typename
is an alias for the particular contract script, meaning that two contracts sharing the same code can have the same type name. If this field is not set, a contract alias will be used instead.
contracts:
kusd_dex_mainnet:
kind: tezos
address: KT1CiSKXR68qYSxnbzjwvfeMCRburaSDonT2
typename: quipu_fa12
tzbtc_dex_mainnet:
kind: tezos
address: KT1N1wwNPqT5jGhM91GQ2ae5uY8UzFaXHMJS
typename: quipu_fa12
If multiple contracts you index have the same interface but different code, see F.A.Q. to learn how to avoid conflicts.
Reindexing
In some cases, DipDup can't proceed with indexing without a full wipe. Several reasons trigger reindexing:
reason | description |
---|---|
manual | Reindexing triggered manually from callback with ctx.reindex . |
migration | Applied migration requires reindexing. Check release notes before switching between major DipDup versions to be prepared. |
rollback | Reorg message received from datasource and can not be processed. |
config_modified | One of the index configs has been modified. |
schema_modified | Database schema has been modified. Try to avoid manual schema modifications in favor of SQL scripts. |
It is possible to configure desirable action on reindexing triggered by a specific reason.
action | description |
---|---|
exception (default) | Raise ReindexingRequiredError and quit with error code. The safest option since you can trigger reindexing accidentally, e.g., by a typo in config. Don't forget to set up the correct restart policy when using it with containers. |
wipe | Drop the whole database and start indexing from scratch. Be careful with this option! |
ignore | Ignore the event and continue indexing as usual. It can lead to unexpected side-effects up to data corruption; make sure you know what you are doing. |
To configure actions for each reason, add the following section to the DipDup config:
advanced:
reindex:
manual: wipe
migration: exception
rollback: ignore
config_modified: exception
schema_modified: exception
Advanced options
Flags related to the project are set in the advanced
section of the config (most likely in dipdup.yaml
).
flag | description |
---|---|
early_realtime | Establish realtime connection and start collecting messages while sync is in progress (faster, but consumes more RAM). |
decimal_precision | Overwrite precision if it's not guessed correctly based on project models. |
postpone_jobs | Do not start job scheduler until all indexes reach the realtime state. |
rollback_depth | A number of levels to keep for rollback. |
unsafe_sqlite | Disable journaling and data integrity checks. Use only for testing. |