Config
Developing a DipDup indexer begins with creating a YAML config file of a specific format. DipDup expects it to be located in the project root and named dipdup.yaml
. However, you can provide any path with a -c
CLI option.
Structure
Config consists of multiple top-level mappings. In the table below they are grouped by sections for convenience, but in the config file, they can be in any order. The only required section is Header.
See Config reference guide for the full list of available options.
Header | spec_version * | DipDup project specification; currently "3.0" |
package * | Python package name | |
Inventory | database | Database configuration |
contracts | Contracts that need to be indexed | |
datasources | Datasources to gather information from | |
Indexes | indexes | Index definitions |
templates | Also index definitions, but with placeholders that make them reusable | |
Hooks | hooks | Callbacks to run manually or by schedule |
jobs | Schedules for hooks | |
Integrations | api | Internal API configuration |
hasura | Hasura GraphQL Engine configuration | |
sentry | Sentry configuration | |
prometheus | Prometheus configuration | |
Miscellaneous | advanced | Tunables that affect framework behavior |
custom | Mapping of user-defined values; neither typed nor validated | |
logging | Configure logging verbosity |
Config merging
DipDup allows you to customize the configuration for specific environments or workflows. It works similarly to Docker Compose anchors but only for top-level sections. If you want to override a nested property, you need to recreate the entire top-level section.
Environment-specific configurations
You can organize your configuration files in a configs/
directory within your project.
Base template contains three environment-specific configurations:
my-dipdup-project/
├── dipdup.yaml # Root config `database` and `hasura` sections
├── configs/
│ ├── dipdup.sqlite.yaml # SQLite configuration override
│ ├── dipdup.compose.yaml # Docker Compose configuration override
│ └── dipdup.swarm.yaml # Docker Swarm configuration override
To merge multiple DipDup config files, provide the -c
command-line option multiple times. Root config must be specified first.
# Merge root config with SQLite-specific settings
dipdup -c . -c configs/dipdup.sqlite.yaml run
# Using a shorthand
dipdup -C sqlite run
# Combining multiple configuration overrides
dipdup -C sqlite -C mainnet run
Creating custom configurations
Config merging can be easily extended. Some ideas include:
dipdup.testnet.yaml
to overridedatasources
andcontracts
sectionsdipdup.debug.yaml
to overridelogging
andsentry
sections
Exporting the merged configuration
You can use the config export
command to see the merged configuration that will be used by DipDup. This is useful for debugging and understanding how different configurations are combined.
# Export the merged configuration
dipdup -C sqlite config export
# Show all available options
dipdup config export -h
Environment variables
DipDup supports compose-style variable expansion in dipdup.yaml
and other configuration files. You can use placeholders in the format ${VARIABLE_NAME}
or ${VARIABLE_NAME:-default_value}
to substitute environment variables at runtime. If a variable is not set and no default value is provided, DipDup will raise an error.
This feature is useful for:
- Keeping sensitive data like API keys or passwords out of your version-controlled configuration files.
- Customizing deployments for different environments (development, staging, production) without altering the base configuration.
You can use these placeholders anywhere throughout the configuration file. For example:
database:
kind: postgres
host: ${POSTGRES_HOST:-localhost}
password: ${POSTGRES_PASSWORD}
There are multiple ways to pass environment variables to DipDup:
- Export them in the shell before running DipDup
- Create the env file and pass it to DipDup with the
-e
CLI option
See Environment Variables for advanced usage, troubleshooting, and built-in variables.
Contract typenames
typename
is an alias for the particular contract script, meaning that two contracts sharing the same code can have the same type name. If this field is not set, a contract alias will be used instead.
contracts:
kusd_dex_mainnet:
kind: tezos
address: KT1CiSKXR68qYSxnbzjwvfeMCRburaSDonT2
typename: quipu_fa12
tzbtc_dex_mainnet:
kind: tezos
address: KT1N1wwNPqT5jGhM91GQ2ae5uY8UzFaXHMJS
typename: quipu_fa12
If multiple contracts you index have the same interface but different code, see F.A.Q. to learn how to avoid conflicts.
Reindexing
In some cases, DipDup can't proceed with indexing without a full wipe. Several reasons trigger reindexing:
reason | description |
---|---|
manual | Reindexing triggered manually from callback with ctx.reindex . |
migration | Applied migration requires reindexing. Check release notes before switching between major DipDup versions to be prepared. |
rollback | Reorg message received from datasource and can not be processed. |
config_modified | One of the index configs has been modified. |
schema_modified | Database schema has been modified. Try to avoid manual schema modifications in favor of SQL scripts. |
It is possible to configure desirable action on reindexing triggered by a specific reason.
action | description |
---|---|
exception (default) | Raise ReindexingRequiredError and quit with error code. The safest option since you can trigger reindexing accidentally, e.g., by a typo in config. Don't forget to set up the correct restart policy when using it with containers. |
wipe | Drop the whole database and start indexing from scratch. Be careful with this option! |
ignore | Ignore the event and continue indexing as usual. It can lead to unexpected side-effects up to data corruption; make sure you know what you are doing. |
To configure actions for each reason, add the following section to the DipDup config:
advanced:
reindex:
manual: wipe
migration: exception
rollback: ignore
config_modified: exception
schema_modified: exception
Advanced options
Flags related to the project are set in the advanced
section of the config (most likely in dipdup.yaml
).
flag | description |
---|---|
early_realtime | Establish realtime connection and start collecting messages while sync is in progress (faster, but consumes more RAM). |
decimal_precision | Overwrite precision if it's not guessed correctly based on project models. |
postpone_jobs | Do not start job scheduler until all indexes reach the realtime state. |
rollback_depth | A number of levels to keep for rollback. |
unsafe_sqlite | Disable journaling and data integrity checks. Use only for testing. |