Release Notes: 7.0
Welcome, developers! Today we introduce 7.0, the most significant major release for DipDup in terms of both changes and developer hours. The new framework architecture allows to easily integrate new blockchains and data sources. EVM support is the first step in this direction; more to come soon. Also we have focused on improving developer experience, so you can initialize, extend and maintain DipDup projects with minimal effort. Finally, updated docs and new demo projects won't let you get lost.
- Support for EVM-compatible blockchains.
- Updated project package structure with configs and deployment recipes.
- Storage layer improvements; first steps path to DipDup ORM.
- A Python 3.11 environment, updated Docker images, and better performance.
- New convenient CLI commands.
Join our socials to discuss this release and ask any questions!
Now DipDup supports EVM-compatible blockchains in addition to Tezos. The new index allows you to process contract events from Ethereum and other EVM-compatible networks. DipDup uses historical data from Subsquid Network, real-time data from RPC nodes, and ABIs from Etherscan. All you need is to define an index in your config and implement handlers for each event.
We have two demo projects for EVM: a very basic USDt price indexer and a more complex one for Uniswap v3 protocol. Run the
dipdup new command, choose "EVM" on the first page, then a template to use.
The project package structure was updated to become more consistent and easier to extend. Every package now have a fixed structure with directories for code, configs, SQL, etc. It allows discovery of package contents and increases the overall readability.
The DipDup package now includes three new sections:
models.pymodule as a source of DipDup ORM models. You can use any structure inside; models will be discovered automatically.
configsdirectory contains files to extend the root config with environment-specific settings like database connection or logging. Keep these settings separated from the root config to make it more readable and declarative.
deploydirectory contains Dockerfiles, Compose stack definitions and other deployment recipes. Also, there are
.env.defaultfiles for each config in the
configsdirectory. Use them as a template never to miss or commit an environment variable again.
dipdup package tree command allows inspecting the package structure and ensuring that everything is in place.
See the Package page in the docs.
We continue to improve our storage layer based on Tortoise ORM. Currently, it's a patchset, not a separate library, but we refer to it as DipDup ORM. 7.0 release brings improved query performance, better schema uniformity between database engines, and new field types.
DipDup models must use classes from
dipdup.models. Here's a shortened typical definition:
from dipdup import fields
from dipdup.models import Model
name = fields.TextField(pk=True)
See the Models page in the docs.
Our interactive installer was updated for better compatibility with pyenv and other Python tooling. The
new command for creating new projects now has a more convenient interface and includes EVM templates.
# Install DipDup
curl -Lsf https://dipdup.io/install.py | python3
# Create a new project interactively. That's all!
Package and config discovery was improved, so managing a project is a bit easier now. You can omit default the
dipdup.y[a]ml filename. Also, a package root can be a workdir now (optional; creates a magic symlink).
dipdup -c . -c configs/dipdup.sqlite.yaml config export
Installer commands were moved to the
dipdup self command group. There you can switch the release channel or uninstall DipDup at any time:
dipdup self update
dipdup self install --force --ref next
dipdup self uninstall
Starting 7.0 we use PDM as a default package manager. It's a swiss-knife to deal with Python's packaging pain with great PEP compatibility and a lot of features. Also, it can run scripts from pyproject.toml as npm does. DipDup projects now use a single, well PEP-compliant pyproject.toml for everything code-related. Explore it to find useful scripts for development and deployment.
Poetry and GNU Make were removed from the default template, but you can still use them if you like.
See the Installation page in the docs.
DipDup projects now run on Python 3.11. Performance improvements introduced in this release are crucial for indexers. Also, you can have fun with the latest syntax like pattern matching and union types.
We have improved pre-fetching and caching data during indexing to increase the indexing speed.
Docker images are now based on Debian 12. They are simple, secure and easy to extend - just run pip as a default user. Alpine images are no longer published due to the lack of support in one of the libraries we depend on, but migration should be seamless.
See the Docker page in the docs.
Here are some other notable changes not covered above:
dipdup_metainternal table was added to the schema. You can store arbitrary JSON there and use it in your code, but don't touch records with
dipdup_prefix. Survives reindexing. See the Internal tables page.
- Multiple feature flags were added for experimental and rarely used features. See the Feature flags page.
metadata_interfaceflag was removed, now it's always enabled.
- We no longer accept crash reports. Enabling them required adding
crash_reporting: Trueto the config. Set up your own Sentry instance; it's easy!
- Saved crash- and performance reports in the home directory can be viewed with new
report showcommands. If you want to open an issue and share this report, just drop us a message on GitHub or Discord.
- You can use long syntax to configure logging, a mapping of logger name and loglevel. See the Logging page.
- YAML files in templates and examples use
.yamlextension instead of
.ymlas recommended by the YAML and Compose specs, but you can use any.
reportcommand has been renamed to
report lsfor consistency with other command groups.
The previous version of the framework is powering dozens of APIs in production. We want to give those projects enough time for migration or to wait a bit while the current branch is being ironed out. Given that, DipDup 6.5 will be supported until March 2024. We will continue to release bugfixes and security updates until that date. You can find the old documentation here, but some pages may be outdated. If you've found such a page or other issue, please drop us a message.
Of course, we encourage you to migrate to 7.0 soon to explore all the cool stuff in this release.
Migration from 6.5 requires several manual actions described further. First, make sure that your project is running the latest release, currently 6.5.10. Then, perform the steps below.
- Install Python 3.11 with pyenv or your favorite tool. Make sure that
python3.11is available in your shell. Install DipDup 7.0 with our script or manually.
- Run the
dipdup newcommand. Choose
[none]on the first page, then
demo_blanktemplate (also used as package name in examples below). Answer the rest of the questions as you like.
- Move your old root config to the
demo_blank/dipdup.yamlpath. Update its contents according to the Config changes section below.
- Enter the package directory and run the
dipdup initcommand. It will generate a bunch of typeclasses and callback stubs following the structure of the old package. Now is a good time to commit your changes.
- Move function bodies of every callback in
handlersdirectories to the corresponding stubs. If necessary, update type annotations to match with autogenerated imports.
- Move model definitions from the
models.pymodule to the freshly created
models/__init__.pyfile preserving imports.
- Move SQL scripts, GraphQL queries and Hasura metadata to the new package.
- Ensure that everything is in place running the
dipdup package treecommand. Then, run
pdm allscript or commands from
pyproject.tomlto check that everything is working as expected.
If you have any questions or issues with migration, please open a ticket on GitHub or reach us on Discord.
- Update the
kindfields of contract, datasource and index definitions using the table below.
See you soon! 👋
DipDup is a free open-source software created by community and for community. Join our socials to discuss this release, ask any questions, or participate in development.