Release Notes: 7.0

Welcome, developers! Today we introduce 7.0, the most significant major release for DipDup in terms of both changes and developer hours. The new framework architecture allows to easily integrate new blockchains and data sources. EVM support is the first step in this direction; more to come soon. Also we have focused on improving developer experience, so you can initialize, extend and maintain DipDup projects with minimal effort. Finally, updated docs and new demo projects won't let you get lost.

Key highlights:

• Support for EVM-compatible blockchains.
• Updated project package structure with configs and deployment recipes.
• Storage layer improvements; first steps path to DipDup ORM.
• A Python 3.11 environment, updated Docker images, and better performance.
• New convenient CLI commands.

Join our socials to discuss this release and ask any questions!

Twitter | Discord | GitHub

EVM support

Now DipDup supports EVM-compatible blockchains in addition to Tezos. The new index allows you to process contract events from Ethereum and other EVM-compatible networks. DipDup uses historical data from Subsquid Network, real-time data from RPC nodes, and ABIs from Etherscan. All you need is to define an index in your config and implement handlers for each event.

We have two demo projects for EVM: a very basic USDt price indexer and a more complex one for Uniswap v3 protocol. Run the dipdup new command, choose "EVM" on the first page, then a template to use.

Project package

The project package structure was updated to become more consistent and easier to extend. Every package now have a fixed structure with directories for code, configs, SQL, etc. It allows discovery of package contents and increases the overall readability.

The DipDup package now includes three new sections:

• models section replaces models.py module as a source of DipDup ORM models. You can use any structure inside; models will be discovered automatically.
• configs directory contains files to extend the root config with environment-specific settings like database connection or logging. Keep these settings separated from the root config to make it more readable and declarative.
• deploy directory contains Dockerfiles, Compose stack definitions and other deployment recipes. Also, there are .env.default files for each config in the configs directory. Use them as a template never to miss or commit an environment variable again.

The new dipdup package tree command allows inspecting the package structure and ensuring that everything is in place.

See the Package page in the docs.

DipDup ORM

We continue to improve our storage layer based on Tortoise ORM. Currently, it's a patchset, not a separate library, but we refer to it as DipDup ORM. 7.0 release brings improved query performance, better schema uniformity between database engines, and new field types.

DipDup models must use classes from dipdup.fields and dipdup.models. Here's a shortened typical definition:

from dipdup import fields
from dipdup.models import Model


class User(Model):
    name = fields.TextField(pk=True)

See the Models page in the docs.

Developer experience

Our interactive installer was updated for better compatibility with pyenv and other Python tooling. The new command for creating new projects now has a more convenient interface and includes EVM templates.

# Install DipDup
curl -Lsf https://dipdup.io/install.py | python3

# Create a new project interactively. That's all!
dipdup new

Package and config discovery was improved, so managing a project is a bit easier now. You can omit default the dipdup.y[a]ml filename. Also, a package root can be a workdir now (optional; creates a magic symlink).

dipdup -c . -c configs/dipdup.sqlite.yaml config export

Installer commands were moved to the dipdup self command group. There you can switch the release channel or uninstall DipDup at any time:

dipdup self update
dipdup self install --force --ref next
dipdup self uninstall

Starting 7.0 we use PDM as a default package manager. It's a swiss-knife to deal with Python's packaging pain with great PEP compatibility and a lot of features. Also, it can run scripts from pyproject.toml as npm does. DipDup projects now use a single, well PEP-compliant pyproject.toml for everything code-related. Explore it to find useful scripts for development and deployment.

Poetry and GNU Make were removed from the default template, but you can still use them if you like.

See the Installation page in the docs.

Environment and performance

DipDup projects now run on Python 3.11. Performance improvements introduced in this release are crucial for indexers. Also, you can have fun with the latest syntax like pattern matching and union types.

We have improved pre-fetching and caching data during indexing to increase the indexing speed.

Docker images are now based on Debian 12. They are simple, secure and easy to extend - just run pip as a default user. Alpine images are no longer published due to the lack of support in one of the libraries we depend on, but migration should be seamless.

See the Docker page in the docs.

Miscellaneous

Here are some other notable changes not covered above:

• dipdup_meta internal table was added to the schema. You can store arbitrary JSON there and use it in your code, but don't touch records with dipdup_ prefix. Survives reindexing. See the Internal tables page.
• Multiple feature flags were added for experimental and rarely used features. See the Feature flags page. metadata_interface flag was removed, now it's always enabled.
• We no longer accept crash reports. Enabling them required adding crash_reporting: True to the config. Set up your own Sentry instance; it's easy!
• Saved crash- and performance reports in the home directory can be viewed with new report ls and report show commands. If you want to open an issue and share this report, just drop us a message on GitHub or Discord.
• You can use long syntax to configure logging, a mapping of logger name and loglevel. See the Logging page.
• YAML files in templates and examples use .yaml extension instead of .yml as recommended by the YAML and Compose specs, but you can use any.
• report command has been renamed to report ls for consistency with other command groups.

Future of DipDup 6.5

The previous version of the framework is powering dozens of APIs in production. We want to give those projects enough time for migration or to wait a bit while the current branch is being ironed out. Given that, DipDup 6.5 will be supported until March 2024. We will continue to release bugfixes and security updates until that date. You can find the old documentation here, but some pages may be outdated. If you've found such a page or other issue, please drop us a message.

Of course, we encourage you to migrate to 7.0 soon to explore all the cool stuff in this release.

Migration guide

Migration from 6.5 requires several manual actions described further. First, make sure that your project is running the latest release, currently 6.5.10. Then, perform the steps below.

• Install Python 3.11 with pyenv or your favorite tool. Make sure that python3.11 is available in your shell. Install DipDup 7.0 with our script or manually.
• Run the dipdup new command. Choose [none] on the first page, then demo_blank template (also used as package name in examples below). Answer the rest of the questions as you like.
• Move your old root config to the demo_blank/dipdup.yaml path. Update its contents according to the Config changes section below.
• Enter the package directory and run the dipdup init command. It will generate a bunch of typeclasses and callback stubs following the structure of the old package. Now is a good time to commit your changes.
• Move function bodies of every callback in hooks and handlers directories to the corresponding stubs. If necessary, update type annotations to match with autogenerated imports.
• Move model definitions from the models.py module to the freshly created models/__init__.py file preserving imports.
• Move SQL scripts, GraphQL queries and Hasura metadata to the new package.
• Ensure that everything is in place running the dipdup package tree command. Then, run pdm all script or commands from pyproject.toml to check that everything is working as expected.

If you have any questions or issues with migration, please open a ticket on GitHub or reach us on Discord.

Config changes

• Update the spec_version field from 1.2 to 2.0.
• Update kind fields of contract, datasource and index definitions using the table below.

See you soon! 👋

DipDup is a free open-source software created by the community and for the community. Join our socials to discuss this release, ask any questions, or participate in development.

Twitter | Discord | GitHub