Quickstart

This page will guide you through the steps to get your first selective indexer up and running in a few minutes without getting too deep into the details.

A selective blockchain indexer is an application that extracts and organizes specific blockchain data from multiple data sources, rather than processing all blockchain data. It allows users to index only relevant entities, reducing storage and computational requirements compared to full node indexing, and query data more efficiently for specific use cases. Think of it as a customizable filter that captures and stores only the blockchain data you need, making data retrieval faster and more resource-efficient. DipDup is a framework that helps you implement such an indexer.

Let's create an indexer for the tzBTC FA1.2 token contract. Our goal is to save all token transfers to the database and then calculate some statistics of its holders' activity.

Install DipDup

A modern Linux/macOS distribution with Python 3.12 installed is required to run DipDup.

The recommended way to install DipDup CLI is pipx. We also provide a convenient helper script that installs all necessary tools. Run the following command in your terminal:

Terminal
curl -Lsf https://dipdup.io/install.py | python3.12

See the Installation page for all options.

Create a project

DipDup CLI has a built-in project generator. Run the following command in your terminal:

Terminal
dipdup new

Choose From template, then Tezos network and demo_tezos_token template.

Note
Want to skip a tutorial and start from scratch? Choose Blank at the first step instead and proceed to the Config section.

Follow the instructions; the project will be created in the new directory.

Write a configuration file

In the project root, you'll find a file named dipdup.yaml. It's the main configuration file of your indexer. We will discuss it in detail in the Config section; now it has the following content:

dipdup.yaml
spec_version: 3.0
package: demo_tezos_token

contracts:
  tzbtc_mainnet:
    kind: tezos
    address: KT1PWx2mnDueood7fEmfbBDKx1D9BAnnXitn
    typename: tzbtc

datasources:
  tzkt_mainnet:
    kind: tezos.tzkt
    url: https://api.tzkt.io

indexes:
  tzbtc_holders_mainnet:
    template: tzbtc_holders
    values:
      contract: tzbtc_mainnet
      datasource: tzkt_mainnet

templates:
  tzbtc_holders:
    kind: tezos.operations
    datasources:
      - <datasource>
    contracts:
      - <contract>
    handlers:
      - callback: on_transfer
        pattern:
          - destination: <contract>
            entrypoint: transfer
      - callback: on_mint
        pattern:
          - destination: <contract>
            entrypoint: mint

Generate types and stubs

Now it's time to generate typeclasses and callback stubs based on definitions from config. Examples below use demo_tezos_token as a package name; yours may differ.

Run the following command:

Terminal
dipdup init

DipDup will create a Python package demo_tezos_token with everything you need to start writing your indexer. Use package tree command to see the generated structure:

Terminal
$ dipdup package tree
demo_tezos_token [.]
├── abi
├── configs
   ├── dipdup.compose.yaml
   ├── dipdup.sqlite.yaml
   ├── dipdup.swarm.yaml
   └── replay.yaml
├── deploy
   ├── .env.default
   ├── Dockerfile
   ├── compose.sqlite.yaml
   ├── compose.swarm.yaml
   ├── compose.yaml
   ├── sqlite.env.default
   └── swarm.env.default
├── graphql
├── handlers
   ├── on_balance_update.py
   ├── on_mint.py
   └── on_transfer.py
├── hasura
├── hooks
   ├── on_index_rollback.py
   ├── on_reindex.py
   ├── on_restart.py
   └── on_synchronized.py
├── models
   └── __init__.py
├── sql
├── types
   ├── tzbtc/tezos_parameters/mint.py
   ├── tzbtc/tezos_parameters/transfer.py
   └── tzbtc/tezos_storage.py
└── py.typed

That's a lot of files and directories! But don't worry, we will need only models and handlers sections in this guide.

Define data models

DipDup supports storing data in SQLite, PostgreSQL and TimescaleDB databases. We use modified Tortoise ORM library as an abstraction layer.

First, you need to define a model class. DipDup uses model definitions both for database schema and autogenerated GraphQL API. Our schema will consist of a single model Holder with the following fields:

addressaccount address
balancetoken amount held by the account
turnovertotal amount of transfer/mint calls
tx_countnumber of transfers/mints
last_seentime of the last transfer/mint

Here's how to define this model in DipDup:

models/__init__.py
from dipdup import fields
from dipdup.models import Model


class Holder(Model):
    address = fields.TextField(primary_key=True)
    balance = fields.DecimalField(decimal_places=8, max_digits=20, default=0)
    turnover = fields.DecimalField(decimal_places=8, max_digits=20, default=0)
    tx_count = fields.BigIntField(default=0)
    last_seen = fields.DatetimeField(null=True)

Using ORM is not a requirement; DipDup provides helpers to run SQL queries/scripts directly, see Database page.

Implement handlers

Everything's ready to implement an actual indexer logic.

Our task is to index all the balance updates, so we'll start with a helper method to handle them. Create a file named on_balance_update.py in the handlers package with the following content:

handlers/on_balance_update.py
from datetime import datetime
from decimal import Decimal

import demo_tezos_token.models as models


async def on_balance_update(
    address: str,
    balance_update: Decimal,
    timestamp: datetime,
) -> None:
    holder, _ = await models.Holder.get_or_create(address=address)
    holder.balance += balance_update
    holder.turnover += abs(balance_update)
    holder.tx_count += 1
    holder.last_seen = timestamp
    await holder.save()

Three methods of tzBTC contract can alter token balances — transfer, mint, and burn. The last one is omitted in this tutorial for simplicity. Edit corresponding handlers to call the on_balance_update method with data from matched operations:

handlers/on_transfer.py
from decimal import Decimal

from demo_tezos_token.handlers.on_balance_update import on_balance_update
from demo_tezos_token.types.tzbtc.tezos_parameters.transfer import TransferParameter
from demo_tezos_token.types.tzbtc.tezos_storage import TzbtcStorage
from dipdup.context import HandlerContext
from dipdup.models.tezos import TezosTransaction


async def on_transfer(
    ctx: HandlerContext,
    transfer: TezosTransaction[TransferParameter, TzbtcStorage],
) -> None:
    if transfer.parameter.from_ == transfer.parameter.to:
        # NOTE: Internal tzBTC transfer
        return

    amount = Decimal(transfer.parameter.value) / (10**8)
    await on_balance_update(
        address=transfer.parameter.from_,
        balance_update=-amount,
        timestamp=transfer.data.timestamp,
    )
    await on_balance_update(
        address=transfer.parameter.to,
        balance_update=amount,
        timestamp=transfer.data.timestamp,
    )
handlers/on_mint.py
from decimal import Decimal

from demo_tezos_token.handlers.on_balance_update import on_balance_update
from demo_tezos_token.types.tzbtc.tezos_parameters.mint import MintParameter
from demo_tezos_token.types.tzbtc.tezos_storage import TzbtcStorage
from dipdup.context import HandlerContext
from dipdup.models.tezos import TezosTransaction


async def on_mint(
    ctx: HandlerContext,
    mint: TezosTransaction[MintParameter, TzbtcStorage],
) -> None:
    amount = Decimal(mint.parameter.value) / (10**8)
    await on_balance_update(
        address=mint.parameter.to,
        balance_update=amount,
        timestamp=mint.data.timestamp,
    )

And that's all! We can run the indexer now.

Next steps

Run the indexer in memory:

dipdup run

Store data in SQLite database (defaults to /tmp, set SQLITE_PATH env variable):

dipdup -c . -c configs/dipdup.sqlite.yaml run

Or spawn a Compose stack with PostgreSQL and Hasura:

cd deploy
cp .env.default .env
# Edit .env file before running
docker-compose up

DipDup will fetch all the historical data and then switch to realtime updates. You can check the progress in the logs.

If you use SQLite, run this query to check the data:

sqlite3 /tmp/demo_tezos_token.sqlite 'SELECT * FROM holder LIMIT 10'

If you run a Compose stack, open http://127.0.0.1:8080 in your browser to see the Hasura console (an exposed port may differ). You can use it to explore the database and build GraphQL queries.

Congratulations! You've just created your first DipDup indexer. Proceed to the Getting Started section to learn more about DipDup configuration and features.

Help and tips -> Join our Discord
Ideas or suggestions -> Issue Tracker
GraphQL IDE -> Open Playground