Creating config
Developing a DipDup indexer begins with creating a YAML config file of a specific format. DipDup expect it to be located in the project root and named dipdup.yaml
. However, you can provide any path with a -c
CLI option.
Structure
Config consists of multiple top-level mappings. In the table below they are grouped by sections for convenience, but in the config file, they can be in any order. The only required section is Header.
Header | spec_version* | DipDup project specification; currently "2.0" |
package* | Python package name | |
Inventory | database | Database configuration |
contracts | Contracts that need to be indexed | |
datasources | Datasources to gather information from | |
Indexes | indexes | Index definitions |
templates | Also index definitions, but with placeholders that make them reusable | |
Hooks | hooks | Callbacks to run manually or by schedule |
jobs | Schedules for hooks | |
Integrations | hasura | Hasura GraphQL Engine configuration |
sentry | Sentry configuration | |
prometheus | Prometheus configuration | |
Miscellanous | advanced | Tunables that affect framework behavior |
custom | Mapping of user-defined values; neither typed nor validated | |
logging | Configure logging verbosity |
Merging multiple files
DipDup allows you to customize the configuration for a specific environment or workflow. It works similarly to docker-compose anchors but only for top-level sections. If you want to override a nested property, you need to recreate a whole top-level section. To merge several DipDup config files, provide the -c
command-line option multiple times:
dipdup -c dipdup.yaml -c configs/dipdup.sqlite.yaml run
Use config export
and config env
commands to check the resulting config used by DipDup.
Environment variables
DipDup supports compose-style variable expansion with an optional default value. Use this feature to store sensitive data outside of the configuration file and make your app fully declarative. If a required variable is not set, DipDup will fail with an error. You can use these placeholders anywhere throughout the configuration file.
database:
kind: postgres
host: ${POSTGRES_HOST:-localhost}
password: ${POSTGRES_PASSWORD}
There are multiple ways to pass environment variables to DipDup:
- Export them in the shell before running DipDup
- Create the env file and pass it to DipDup with the
-e
CLI option
For every config file in the deploy
project directory, DipDup will create a corresponding .env.default
file with all the variables used in the config. Copy it, remove the .default
suffix and fill in the values.
POSTGRES_HOST=localhost
POSTGRES_PASSWORD=
You can use dipdup -e .env config export --unsafe
command to ensure that env variables resolve as expected, but avoid sharing the output with third parties.