Config file reference
advanced
advanced
advanced:
early_realtime: False
merge_subscriptions: False
postpone_jobs: False
reindex:
manual: wipe
migration: exception
rollback: ignore
config_modified: exception
schema_modified: exception
This config section allows users to tune some system-wide options, either experimental or unsuitable for generic configurations.
field | description |
---|---|
reindex | Mapping of reindexing reasons and actions DipDup performs |
scheduler | apscheduler scheduler config |
postpone_jobs | Do not start job scheduler until all indexes are in realtime state |
early_realtime | Establish realtime connection immediately after startup |
merge_subscriptions | Subscribe to all operations instead of exact channels |
CLI flags have priority over self-titled AdvancedConfig
fields.
contracts
A list of the contracts you can use in the index definitions. Each contract entry has two fields:
address
— either originated or implicit account address encoded in base58.typename
— an alias for the particular contract script, meaning that two contracts sharing the same code can have the same type name.
contracts:
kusd_dex_mainnet:
address: KT1CiSKXR68qYSxnbzjwvfeMCRburaSDonT2
typename: quipu_fa12
tzbtc_dex_mainnet:
address: KT1N1wwNPqT5jGhM91GQ2ae5uY8UzFaXHMJS
typename: quipu_fa12
kusd_token_mainnet:
address: KT1K9gCRgaLRFKTErYt1wVxA3Frb9FjasjTV
typename: kusd_token
tzbtc_token_mainnet:
address: KT1PWx2mnDueood7fEmfbBDKx1D9BAnnXitn
typename: tzbtc_token
If the typename
field is not set, a contract alias will be used instead.
Contract entry does not contain information about the network, so it's a good idea to include the network name in the alias. This design choice makes possible a generic index parameterization via templates. See index templates for details.
If multiple contracts you index have the same interface but different code, see faq to learn how to avoid conflicts.
custom
An arbitrary YAML object you can use to store internal indexer configuration.
package: my_indexer
...
custom:
foo: bar
Access or modify it from any callback:
ctx.config.custom['foo'] = 'buzz'
database
DipDup supports several database engines for development and production. The obligatory field kind
specifies which engine has to be used:
sqlite
postgres
(and compatible engines)
Database engines article may help you choose a database that better suits your needs.
SQLite
path
field must be either path to the .sqlite3 file or :memory:
to keep a database in memory only (default):
database:
kind: sqlite
path: db.sqlite3
field | description |
---|---|
kind | always 'sqlite' |
path | Path to .sqlite3 file, leave default for in-memory database |
PostgreSQL
Requires host
, port
, user
, password
, and database
fields. You can set schema_name
to values other than public
, but Hasura integration won't be available.
database:
kind: postgres
host: db
port: 5432
user: dipdup
password: ${POSTGRES_PASSWORD:-changeme}
database: dipdup
schema_name: public
field | description |
---|---|
kind | always 'postgres' |
host | Host |
port | Port |
user | User |
password | Password |
database | Database name |
schema_name | Schema name |
immune_tables | List of tables to preserve during reindexing |
connection_timeout | Connection timeout in seconds |
You can also use compose-style environment variable substitutions with default values for secrets and other fields. See dipdup environment variables.
Immune tables
You might want to keep several tables during schema wipe if the data in them is not dependent on index states yet heavy. A typical example is indexing IPFS data — changes in your code won't affect off-chain storage, so you can safely reuse this data.
database:
immune_tables:
- ipfs_assets
immune_tables
is an optional array of table names that will be ignored during schema wipe. Once an immune table is created, DipDup will never touch it again; to change the schema of an immune table, you need to perform a migration manually. Check schema export
output before doing this to ensure the resulting schema is the same as Tortoise ORM would generate.
datasources
A list of API endpoints DipDup uses to retrieve indexing data to process.
A datasource config entry is an alias for the endpoint URI; there's no network mention. Thus it's good to add a network name to the datasource alias, e.g. tzkt_mainnet
.
tzkt
datasources:
tzkt:
kind: tezos.tzkt
url: ${TZKT_URL:-https://api.tzkt.io}
http:
retry_count: # retry infinetely
retry_sleep:
retry_multiplier:
ratelimit_rate:
ratelimit_period:
connection_limit: 100
connection_timeout: 60
batch_size: 10000
buffer_size: 0
coinbase
datasources:
coinbase:
kind: coinbase
dipdup-metadata
datasources:
metadata:
kind: tzip_metadata
url: https://metadata.dipdup.net
network: mainnet|ghostnet|mumbainet
ipfs
datasources:
ipfs:
kind: ipfs
url: https://ipfs.io/ipfs
hasura
This optional section used by DipDup executor to automatically configure Hasura engine to track your tables.
hasura:
url: http://hasura:8080
admin_secret: ${HASURA_SECRET:-changeme}
allow_aggregations: false
camel_case: true
rest: true
select_limit: 100
source: default
hooks
Hooks are user-defined callbacks you can execute with a job scheduler or within another callback (with ctx.fire_hook
).
hooks:
calculate_stats:
callback: calculate_stats
atomic: False
args:
major: bool
depth: int
jobs
Add the following section to DipDup config:
jobs:
midnight_stats:
hook: calculate_stats
crontab: "0 0 * * *"
args:
major: True
leet_stats:
hook: calculate_stats
interval: 1337 # in seconds
args:
major: False
If you're unfamiliar with the crontab syntax, an online service crontab.guru will help you build the desired expression.
logging
You can configure an amount of logging output by modifying the logging
field.
logging: DEBUG
At the moment these values are equal to setting dipdup
log level to INFO
, WARNING
or DEBUG
, but this may change in the future.
package
DipDup uses this field to discover the Python package of your project.
package: my_indexer_name
DipDup will search for a module named my_module_name
in PYTHONPATH
starting from the current working directory.
DipDup configuration file is decoupled from the indexer implementation which gives more flexibility in managing the source code.
See project package for details.
prometheus
prometheus:
host: 0.0.0.0
Prometheus integration options
field | description |
---|---|
host | Host to bind to |
port | Port to bind to |
update_interval | Interval to update some metrics in seconds |
sentry
sentry:
dsn: https://...
environment: dev
debug: False
field | description |
---|---|
dsn | DSN of the Sentry instance |
environment | Environment to report to Sentry (informational only) |
debug | Catch warning messages and more context |
spec_version
The DipDup specification version defines the format of the configuration file and available features.
spec_version: 2.0
This table shows which specific SDK releases support which DipDup file versions.
spec_version value | Supported DipDup versions |
---|---|
1.0 | >=1.0, <2.0 |
1.1 | >=2.0, <3.0 |
1.2 | >=3.0, <7.0 |
2.0 | >=7.0 |
If you're getting MigrationRequiredError
after updating the framework, run the dipdup migrate
command to perform project migration.
At the moment, spec_version
has not changed for a very long time. Consider recreating the package from scratch and migrating logic manually if you have another value in your configuration file.
templates
indexes:
foo:
template: bar
first_level: 12341234
template_values:
network: mainnet
templates:
bar:
kind: tezos.tzkt.operations
datasource: tzkt_<network> # resolves into `tzkt_mainnet`
...
field | description |
---|---|
kind | always template |
name | Name of index template |
template_values | Values to be substituted in template (<key> → value ) |
first_level | Level to start indexing from |
last_level | Level to stop indexing at (DipDup will terminate at this level) |