Hooks
Hooks are user-defined callbacks not linked to any index. There are two types of hooks:
- System hooks are called on system-wide events like process restart.
- User hooks are called either with the
ctx.fire_hookmethod or by the job scheduler.
System hooks
Every DipDup project has multiple system hooks; they fire on system-wide events and, like regular hooks, are not linked to any index. Names of those hooks are reserved; you can't use them in config. System hooks are not atomic and can't be fired manually or with a job scheduler.
You can also put SQL scripts in corresponding sql/on_* directories to execute them like with regular hooks.
on_restart
This hook executes right before starting indexing. It allows configuring DipDup in runtime based on data from external sources. Datasources are already initialized at execution and available at ctx.datasources. You can, for example, configure logging here or add contracts and indexes in runtime instead of from static config.
SQL scripts in sql/on_restart directory may contain CREATE OR REPLACE VIEW or similar non-destructive operations.
on_reindex
This hook fires after the database are re-initialized after reindexing (wipe) before starting indexing.
Helpful in modifying schema with arbitrary SQL scripts before indexing. For example, you can to change the database schema in ways that are not supported by the DipDup ORM, e.g., to create a composite primary key.
on_synchronized
This hook fires when every active index reaches a realtime state. Here you can clear caches internal caches or do other cleanups.
on_index_rollback
Fires when one of the index datasources received a chain reorg message.
Since version 6.0 this hook performs a database-level rollback by default. If you want to process rollbacks manually, remove ctx.rollback call and implement custom logic in this callback.
User hooks
Let's assume we want to calculate some statistics on-demand to avoid blocking an indexer with heavy computations. Add the following lines to the DipDup config:
hooks:
calculate_stats:
callback: calculate_stats
atomic: False
args:
major: bool
depth: int
Values of args mapping are used as type hints in a signature of a generated callback. The following callback stub will be created on init:
from dipdup.context import HookContext
async def calculate_stats(
ctx: HookContext,
major: bool,
depth: int,
) -> None:
await ctx.execute_sql_script('calculate_stats')
By default, hooks execute SQL scripts from the corresponding subdirectory of sql/. Remove or comment out the ctx.execute_sql call to prevent it.
To trigger the hook, call the ctx.fire_hook method from any callback:
await ctx.fire_hook('calculate_stats', major=True, depth=10)
Atomicity and blocking
The atomic option defines whether the hook callback will be wrapped in a single SQL transaction or not. With atomic: true, everything the hook writes is committed together or rolled back on error.
DipDup talks to the database over a single connection. Level transactions of all indexes, index state updates and SQL executed by hooks and jobs take turns on it, so the option also decides how long a hook keeps indexing waiting:
- An atomic hook holds the connection from start to finish. Every index waits for the whole hook, including the time it spends outside SQL, like HTTP requests or
asyncio.sleep. - A non-atomic hook takes the connection only while each of its SQL statements runs, and indexing continues between them. A long statement still pauses every index until it finishes, so
atomic: falsealone doesn't help with it: for example, a job runningREFRESH MATERIALIZED VIEWon a large view stops indexing for as long as the refresh takes.
Calling an atomic hook from a handler with the default wait=True doesn't give the hook a transaction of its own. On levels within rollback_depth of the head, it fails with ValueError: Transaction is already started. On older levels, it runs as a savepoint inside the level transaction on PostgreSQL and fails with no such savepoint on SQLite. To run the hook in its own transaction, use the wait=False option:
async def handler(ctx: HandlerContext, ...) -> None:
await ctx.fire_hook('atomic_hook', wait=False)
This hook will be executed when the current transaction is committed.
To keep indexing while a long statement runs, pass dedicated_connection=True where the hook executes the script. The same goes for jobs, since a job fires a hook:
async def refresh_views(ctx: HookContext) -> None:
await ctx.execute_sql_script('refresh_views', dedicated_connection=True)
The scripts then run in autocommit on a separate short-lived connection. What that changes:
- The scripts don't see uncommitted changes of the current level and aren't rolled back with it. So DipDup refuses this option inside a transaction: in handlers, atomic hooks and hooks they fire with
wait=True. - Each statement commits on its own;
BEGINandROLLBACKin the script don't group them. Put the statements that must apply together into a singleDOblock. - The connection is separate, but the tables are not. If the scripts write or lock rows that indexing writes too, indexing waits for them, and crossing writes can fail the level with a deadlock error.
- Don't use the option in hooks whose SQL must stay in order with indexing, like
on_restart,on_reindexandon_index_rollback. - The connection is opened on each call, without the startup retries of
database.connection_timeout: if the database can't be reached, the call fails. - On SQLite, there is no second connection: the scripts run on the shared one, so indexing still waits for them. Inside a transaction the option is refused, as on PostgreSQL.
Outside a transaction, on either connection, SET in a script doesn't reach the next statement: each statement takes the connection from a pool that runs RESET ALL when it is returned. Set the parameter in the statement that needs it, like DO $$ BEGIN PERFORM set_config('work_mem', '256MB', true); REFRESH MATERIALIZED VIEW my_view; END $$;, or for the role with ALTER ROLE ... SET, which a dedicated connection picks up on the next call and the shared one after a restart.
Arguments typechecking
DipDup will ensure that arguments passed to the hooks have the correct types when possible. CallbackTypeError exception will be raised otherwise. Values of an args mapping in a hook config should be either built-in types or __qualname__ of external type like decimal.Decimal. Generic types are not supported: hints like Optional[int] = None will be correctly parsed during codegen but ignored on type checking.
Job scheduler
Jobs are schedules to run specific hooks by crontab or interval. They are defined in the jobs config section.
jobs:
midnight_stats:
hook: calculate_stats
crontab: "0 0 * * *"
args:
major: True
minute_stats:
hook: calculate_stats
interval: 60
args:
major: False
You can also run hook as a daemon, implying that it will never finish execution. This is useful for hooks that perform some kind of background processing, like sending notifications or updating caches.
jobs:
daemon_stats:
hook: calculate_stats
daemon: True
If you're unfamiliar with the crontab syntax, an online service crontab.guru will help you build the desired expression.
apscheduler configuration
DipDup utilizes apscheduler library to run hooks according to schedules in jobs config section. In the following example, apscheduler will spawn up to three instances of the same job every time the trigger is fired, even if previous runs are in progress:
advanced:
scheduler:
apscheduler.job_defaults.coalesce: True
apscheduler.job_defaults.max_instances: 3
See apscheduler docs for details.
Note that you can't use executors from apscheduler.executors.pool module - ConfigurationError exception will be raised.