Table of Contents

Quickstart

This page will guide you through the steps to get your first selective indexer up and running in a few minutes without getting too deep into the details.

Let's create an indexer for the USDt token contract. Our goal is to save all token transfers to the database and then calculate some statistics of its holders' activity.

Install DipDup

A modern Linux/macOS distribution with Python 3.11 installed is required to run DipDup.

We have a convenient installer that will install DipDup and all the required dependencies for you. It uses pipx to install DipDup as a CLI application and make it available everywhere for the current user. Run the following command in your terminal:

Terminal

curl -Lsf https://dipdup.io/install.py | python3

If you don't want to use our script, install DipDup manually using commands from the snippet below. You can use any Python package manager you like, but we recommend PDM and provide some useful scripts for it. Here are some spells to get you started:

Terminal

# pipx
pipx install dipdup datamodel-code-generator
# PDM
pdm init --python 3.11 --lib  # use "">=3.11,<3.12" for requires-python
pdm venv create
pdm add "dipdup>=7,<8" --venv
$(pdm venv activate)
# Poetry
poetry init --python ">=3.11,<3.12"
poetry add "dipdup>=7,<8"
poetry shell
# pip + venv
python -m venv .venv
. .venv/bin/activate
echo "dipdup>=7,<8" > requirements.txt
pip install -r requirements.txt -e .

Create a project

DipDup CLI has a built-in project generator. Run the following command in your terminal:

Terminal

dipdup new

For educational purposes, we'll create a project from scratch, so choose [none] network and demo_blank template.

Note

Want to skip a tutorial? Choose EVM-compatible and demo_evm_events instead!

Follow the instructions; the project will be created in the new directory.

Write a configuration file

In the project root, you'll find a file named dipdup.yaml. It's the main configuration file of your indexer. We will discuss it in detail in the Config section; for now just replace its content with the following:

dipdup.yaml

spec_version: 2.0
package: demo_evm_events
datasources:
  ethscan:
    kind: abi.etherscan
  mainnet_node:
    kind: evm.node
    url: https://eth-mainnet.g.alchemy.com/v2/${ALCHEMY_KEY:-''}
    ws_url: wss://eth-mainnet.g.alchemy.com/v2/${ALCHEMY_KEY:-''}
  mainnet_subsquid:
    kind: evm.subsquid
    url: ${ARCHIVE_URL:-https://v2.archive.subsquid.io/network/ethereum-mainnet}
    node: mainnet_node
contracts:
  eth_usdt:
    kind: evm
    address: 0xdac17f958d2ee523a2206206994597c13d831ec7
    typename: eth_usdt
indexes:
  eth_usdt_events:
    kind: evm.subsquid.events
    datasource: mainnet_subsquid
    handlers:
      - callback: on_transfer
        contract: eth_usdt
        name: Transfer

Generate types and stubs

Now it's time to generate typeclasses and callback stubs based on definitions from config. Examples below use demo_evm_events as a package name; yours may differ.

Run the following command:

Terminal

dipdup init

DipDup will create a Python package demo_evm_events with everything you need to start writing your indexer. Use package tree command to see the generated structure:

Terminal

$ dipdup package tree
demo_evm_events [.]
├── abi
│   ├── eth_usdt/abi.json
│   └── eth_usdt/events.json
├── configs
│   ├── dipdup.compose.yaml
│   ├── dipdup.sqlite.yaml
│   ├── dipdup.swarm.yaml
│   └── replay.yaml
├── deploy
│   ├── .env.default
│   ├── Dockerfile
│   ├── compose.sqlite.yaml
│   ├── compose.swarm.yaml
│   ├── compose.yaml
│   ├── sqlite.env.default
│   └── swarm.env.default
├── graphql
├── handlers
│   └── on_transfer.py
├── hasura
├── hooks
│   ├── on_index_rollback.py
│   ├── on_reindex.py
│   ├── on_restart.py
│   └── on_synchronized.py
├── models
│   └── __init__.py
├── sql
├── types
│   └── eth_usdt/evm_events/transfer.py
└── py.typed

That's a lot of files and directories! But don't worry, we will need only models and handlers sections in this guide.

Define data models

DipDup supports storing data in SQLite, PostgreSQL and TimescaleDB databases. We use custom ORM based on Tortoise ORM as an abstraction layer.

First, you need to define a model class. Our schema will consist of a single model Holder with the following fields:


`address`	account address
`balance`	token amount held by the account
`turnover`	total amount of transfer/mint calls
`tx_count`	number of transfers/mints
`last_seen`	time of the last transfer/mint

Here's how to implement this model in DipDup:

models/__init__.py

from dipdup import fields
from dipdup.models import CachedModel
class Holder(CachedModel):
    address = fields.TextField(pk=True)
    balance = fields.DecimalField(decimal_places=6, max_digits=20, default=0)
    turnover = fields.DecimalField(decimal_places=6, max_digits=20, default=0)
    tx_count = fields.BigIntField(default=0)
    last_seen = fields.BigIntField(null=True)
    class Meta:
        maxsize = 2 ** 20

Implement handlers

Everything's ready to implement an actual indexer logic.

Our task is to index all the balance updates. Put some code to the on_transfer handler callback to process matched logs:

handlers/on_transfer.py

from decimal import Decimal
from tortoise.exceptions import DoesNotExist
from demo_evm_events import models as models
from demo_evm_events.types.eth_usdt.evm_events.transfer import Transfer
from dipdup.context import HandlerContext
from dipdup.models.evm_subsquid import SubsquidEvent
async def on_transfer(
    ctx: HandlerContext,
    event: SubsquidEvent[Transfer],
) -> None:
    amount = Decimal(event.payload.value) / (10**6)
    if not amount:
        return
    await on_balance_update(
        address=event.payload.from_,
        balance_update=-amount,
        level=event.data.level,
    )
    await on_balance_update(
        address=event.payload.to,
        balance_update=amount,
        level=event.data.level,
    )
    
async def on_balance_update(
    address: str,
    balance_update: Decimal,
    level: int,
) -> None:
    try:
        holder = await models.Holder.cached_get(pk=address)
    except DoesNotExist:
        holder = models.Holder(
            address=address,
            balance=0,
            turnover=0,
            tx_count=0,
            last_seen=None,
        )
        holder.cache()
    holder.balance += balance_update
    holder.turnover += abs(balance_update)
    holder.tx_count += 1
    holder.last_seen = level
    await holder.save()

And that's all! We can run the indexer now.

Next steps

Run the indexer in-memory:

dipdup run

Store data in SQLite database:

dipdup -c . -c configs/dipdup.sqlite.yaml run

Or spawn a docker-compose stack:

cd deploy
cp .env.default .env
# Edit .env before running
docker-compose up

DipDup will fetch all the historical data and then switch to realtime updates. You can check the progress in the logs.

If you use SQLite, run a query to check the data:

sqlite3 demo_evm_events.sqlite 'SELECT * FROM holder LIMIT 10'

If you run a Compose stack, check open http://127.0.0.1:8080 in your browser to see the Hasura console (exposed port may differ). You can use it to explore the database and build GraphQL queries.

Congratulations! You've just created your first DipDup indexer. Proceed to the Getting Started section to learn more about DipDup configuration and features.

NextQuickstart

Help and tips -> Join our Discord

Ideas or suggestions -> Issue Tracker

DipDup

—ChangelogContributionLicense