Release Notes: 8.0

Welcome to DipDup 8.0 release notes!

This major release contains lots of new features and improvements both for existing users and newcomers. Key highlights include Starknet support, updated Python 3.12 environment, improved performance, handler batching, new CLI commands. As always, developer experience is at the top of our list, so coding with DipDup 8.0 is more enjoyable than ever. After three months in beta stage, we consider the 8.0 branch polished enough for a stable release. This article will guide you through the most significant changes and show you how to upgrade your projects.

Starknet support

GM, Starknet! 🐺

Starknet is a permissionless zero-knowledge (ZK) rollup on Ethereum, allowing developers to scale their dapps without compromising on security and composability of the Ethereum ecosystem.

We welcome Starknet to the large family of supported networks! DipDup 8.0 introduces a new index kind starknet.events and new datasources starknet.subsquid, starknet.node to work with Starknet events.

Starknet smart contracts are written in Cairo language. It's not EVM-compatible, but many concepts are similar. To start indexing Starknet events, you need to add a new index definition to the project config, then place the contract ABI to abi/<typename>/cairo_abi.json file and run dipdup init command to generate Python types and handler stubs. You can use Starkscan explorer to get the ABI and other information about the contract. We are going to add support for automatic fetching ABIs from node in the future.

Follow the Starknet Quickstart guide to get started or run dipdup new and choose demo_starknet_events template.

Updated Python 3.12 environment

DipDup indexers run in unified Python environment kept stable between releases to simplify deployment and maintenance. DipDup 8.0 introduces a major update including the following notable changes:

• Python 3.12 with the latest language features and performance improvements.
• pydantic 2.9 with significantly faster (de)serialization and powerful validators.
• tortoise-orm 0.21.6 with better Pydantic integration and a bunch of bugfixes and optimizations.
• web3-py 7.2 with the latest EIP and RPC changes.

Make sure to visit the docs of corresponding libraries to learn about the important changes.

New config specification

DipDup 8.0 introduces an updated configuration specification for better flexibility and extensibility. Previously, every index definition was linked to a single "index datasource", which in turn could be linked to one or more complementary ones. This approach appeared to be limiting, and also confusing, since Subsquid and node RPC datasources could be used interchangeably despite the difference in indexing speed.

In the new spec version 3.0, an index can have any number of attached datasources. DipDup will choose the most suitable one for each stage of the process. For load balancing purposes, if multiple node datasources are attached, a random one will be chosen for each request. When applicable, DipDup will consider the order of datasources in the config file. The naming convention for index kinds has been updated to reflect these changes. They now consist of two parts: network and data type, without the datasource one.

spec_version: 3.0  # <- was `2.0`
package: demo_evm_events

indexes:
  eth_usdt_events:
    kind: evm.events  # <- was `evm.subsquid.events`
    datasources:  # <- replacing `datasource` key
      - subsquid
      - etherscan
      - evm_node
      - another_evm_node

JSONSchema of DipDup config specification was uploaded to SchemaStore catalog. That means config file validation and auto-completion are available in major IDEs without additional configuration.

Handler batching

DipDup 8.0 introduces a new batch handler to modify higher-level indexing logic. Examples could be skipping whole blocks by condition or recalculating some data between fixed intervals. Currently, the number of matched handlers in a single batch equals block, but the size of handler batch (and therefore database transaction) is going to be configurable in the future.

async def batch(
    ctx: HandlerContext,
    handlers: Iterable[MatchedHandler],
) -> None:
    for handler in handlers:
        await ctx.fire_matched_handler(handler)

Migration from 7.5

Existing projects require manual migration, but some steps are automated. Please follow the steps below to upgrade to 8.0.

• Make sure you have Python 3.12 installed; which python3.12 command will help you to check that.
• Update the current DipDup installation. Run dipdup self uninstall, then curl -Lsf https://dipdup.io/install.py | python3.12.
• Enter the project directory, but do not activate the virtual environment. Run the dipdup migrate command. It will update your config files and generate a new package structure. Modules requiring manual migration will be moved to <module>.old path; leave them as is for now. Review and commit the changes.
• Run dipdup init --base --force command to update pyproject.toml and other metadata files. Recreate and enter the virtual environment. For PDM run rm -rf .venv pdm.lock && pdm venv create python3.12 && pdm install && $(pdm venv activate). For Poetry run rm -rf .venv poetry.lock && poetry install && poetry shell. Review and commit the changes.
• Move the callback function bodies from <module>.old to <module> files. Run make all to ensure that everything works as expected. Fix arrors if any, review and commit the changes for the last time.

DipDup 7.5 release is going to be supported for several months after the stable release of 8.0. During this period, we will provide bug fixes and security updates, but no new features will be added. End-of-Life date is going to be announced in advance.

Changes since 7.x

Added

• cli: Added --pre flag to self group commands to install pre-release versions.
• cli: Added --raw option to config export command to dump config preserving the original structure.
• cli: Added -C option, a shorthand for -c . -c configs/dipdup.<name>.yaml.
• cli: Added package verify command to check the package consistency.
• cli: Added full project migration support for 3.0 spec.
• cli: Added spec_version 3.0 support to migrate command.
• config: Publish JSON schemas for config validation and autocompletion.
• database: Added dipdup_status view to the schema.
• env: Added DIPDUP_JSON_LOG environment variable to enable JSON logging.
• env: Added DIPDUP_LOW_MEMORY variable to reduce the size of caches and buffers.
• env: Added DIPDUP_PACKAGE_PATH environment variable to override discovered package path.
• package: Added built-in batch handler to modify higher-level indexing logic.
• starknet.events: Added starknet.events index kind to process Starknet events.
• starknet.node: Added Starknet node datasource for last mile indexing.
• starknet.subsquid: Added starknet.subsquid datasource to fetch historical data from Subsquid Archives.
• tezos.operations: Added sr_cement operation type to process Smart Rollup Cemented Commitments.

Fixed

• cli: Don't save reports for successful test runs.
• cli: Don't update existing installation in self install command unless asked to.
• cli: Fixed --pre installer flag.
• cli: Fixed env files not being loaded in some commands.
• cli: Fixed errors raised when the project package is invalid.
• cli: Fixed progress estimation when there are indexes with last_level option set.
• cli: Import some dependencies on demand to reduce memory footprint.
• cli: Improved logging of indexer status.
• config: Allow sentry.dsn to be empty string.
• config: Fixed (de)serialization of hex strings in config.
• config: Fixed setting logging levels according to the config.
• database: Fixed concurrency issue when using get_or_create method.
• evm.events: Fixed matching logs when filtering by topic0.
• evm.events: Improve fetching event batches from node.
• evm.subsquid: Fixed typo in iter_events method name.
• evm: Fixed crash when contract ABI contains overloaded methods.
• install: Fixed reinstalling package when --force flag is used.
• models: Fixed CachedModel preloading.
• models: Fixed setting default value for Meta.maxsize.
• package: Create package in-place if cwd equals package name.
• performance: Add index name to fetcher and realtime queues.
• performance: Fixed estimation indexing speed in levels per second.
• starknet.events: Fixed filtering events by key.
• subsquid: Fixed missing entry in dipdup_head internal table.
• tezos.big_maps: Fixed logging status message in skip_history mode.
• tezos.big_maps: Respect order of handlers in skip_history mode.
• tezos.operations: Fixed sr_cement operation index subscription.
• yaml: Fixed indentation and formatting of generated YAML files.

Changed

• api: /performance endpoint response format has been changed.
• config: Index configs accept datasources list instead of datasource field.
• config: Index kinds have been renamed and grouped by the network.
• config: Index template values now can be any JSON-serializable object.
• config: When filtering EVM transactions by signature, use signature field instead of method.
• context: Signatures of fire_handler and fire_hook methods have changed.
• context: ctx.logger is a regular logging.Logger instead of pre-configured FormattedLogger.
• deps: Python 3.12 is now required to run DipDup.
• performance: All time intervals are now measured in seconds.
• performance: Several metrics have been renamed and new ones have been added.

Removed

• config: Removed advanced.skip_version_check flag; use DIPDUP_NO_VERSION_CHECK environment variable.
• config: abi index config field has been removed; add abi.etherscan datasource(s) to the datasources list instead.
• config: node_only index config flag has been removed; add evm.node datasource(s) to the datasources list instead.
• database: Removed dipdup_head_status view; use dipdup_status view instead.

Performance

• database: Set synchronous=NORMAL and journal_mode=WAL pragmas for on-disk SQLite databases.

Other

• demos: Demo projects have been renamed to reflect the new config structure.
• deps: Use uvloop to improve asyncio performance.
• deps: datamodel-code-generator updated to 0.25.
• deps: pyarrow updated to 16.0.
• deps: pydantic updated to 2.2.
• deps: sentry-sdk updated to 2.1.
• deps: tortoise-orm updated to 0.20.1.
• deps: tortoise-orm updated to 0.21.2.
• deps: web3 updated to 6.18.

See you soon! 👋

DipDup is a free open-source software created by the community and for the community. Join our socials to discuss this release, ask any questions, or participate in development.

Twitter | Discord | GitHub