Quickstart
This page will guide you through the steps to get your first selective indexer up and running in a few minutes without getting too deep into the details.
Let's create an indexer for the tzBTC FA1.2 token contract. Our goal is to save all token transfers to the database and then calculate some statistics of its holders' activity.
Install DipDup
A modern Linux/macOS distribution with Python 3.11 installed is required to run DipDup.
The easiest way to install DipDup as a CLI application pipx. We have a convenient wrapper script that installs DipDup for the current user. Run the following command in your terminal:
curl -Lsf https://dipdup.io/install.py | python3
See the Installation page for all options.
Create a project
DipDup CLI has a built-in project generator. Run the following command in your terminal:
dipdup new
For educational purposes, we'll create a project from scratch, so choose [none] network and demo_blank template.
Tezos and demo_token instead!Follow the instructions; the project will be created in the new directory.
Write a configuration file
In the project root, you'll find a file named dipdup.yaml. It's the main configuration file of your indexer. We will discuss it in detail in the Config section; for now just replace its content with the following:
spec_version: 2.0
package: demo_token
contracts:
tzbtc_mainnet:
kind: tezos
address: KT1PWx2mnDueood7fEmfbBDKx1D9BAnnXitn
typename: tzbtc
datasources:
tzkt_mainnet:
kind: tezos.tzkt
url: https://api.tzkt.io
indexes:
tzbtc_holders_mainnet:
template: tzbtc_holders
values:
contract: tzbtc_mainnet
datasource: tzkt_mainnet
templates:
tzbtc_holders:
kind: tezos.tzkt.operations
datasource: <datasource>
contracts:
- <contract>
handlers:
- callback: on_transfer
pattern:
- destination: <contract>
entrypoint: transfer
- callback: on_mint
pattern:
- destination: <contract>
entrypoint: mint
Generate types and stubs
Now it's time to generate typeclasses and callback stubs based on definitions from config. Examples below use demo_token as a package name; yours may differ.
Run the following command:
dipdup init
DipDup will create a Python package demo_token with everything you need to start writing your indexer. Use package tree command to see the generated structure:
$ dipdup package tree
demo_token [.]
├── abi
├── configs
│ ├── dipdup.compose.yaml
│ ├── dipdup.sqlite.yaml
│ ├── dipdup.swarm.yaml
│ └── replay.yaml
├── deploy
│ ├── .env.default
│ ├── Dockerfile
│ ├── compose.sqlite.yaml
│ ├── compose.swarm.yaml
│ ├── compose.yaml
│ ├── sqlite.env.default
│ └── swarm.env.default
├── graphql
├── handlers
│ ├── on_balance_update.py
│ ├── on_mint.py
│ └── on_transfer.py
├── hasura
├── hooks
│ ├── on_index_rollback.py
│ ├── on_reindex.py
│ ├── on_restart.py
│ └── on_synchronized.py
├── models
│ └── __init__.py
├── sql
├── types
│ ├── tzbtc/tezos_parameters/mint.py
│ ├── tzbtc/tezos_parameters/transfer.py
│ └── tzbtc/tezos_storage.py
└── py.typed
That's a lot of files and directories! But don't worry, we will need only models and handlers sections in this guide.
Define data models
DipDup supports storing data in SQLite, PostgreSQL and TimescaleDB databases. We use custom ORM based on Tortoise ORM as an abstraction layer.
First, you need to define a model class. Our schema will consist of a single model Holder with the following fields:
address | account address |
balance | token amount held by the account |
turnover | total amount of transfer/mint calls |
tx_count | number of transfers/mints |
last_seen | time of the last transfer/mint |
Here's how to implement this model in DipDup:
from dipdup import fields
from dipdup.models import Model
class Holder(Model):
address = fields.TextField(pk=True)
balance = fields.DecimalField(decimal_places=8, max_digits=20, default=0)
turnover = fields.DecimalField(decimal_places=8, max_digits=20, default=0)
tx_count = fields.BigIntField(default=0)
last_seen = fields.DatetimeField(null=True)
Implement handlers
Everything's ready to implement an actual indexer logic.
Our task is to index all the balance updates, so we'll start with a helper method to handle them. Create a file named on_balance_update.py in the handlers package with the following content:
from datetime import datetime
from decimal import Decimal
import demo_token.models as models
async def on_balance_update(
address: str,
balance_update: Decimal,
timestamp: datetime,
) -> None:
holder, _ = await models.Holder.get_or_create(address=address)
holder.balance += balance_update
holder.turnover += abs(balance_update)
holder.tx_count += 1
holder.last_seen = timestamp
await holder.save()
Three methods of tzBTC contract can alter token balances — transfer, mint, and burn. The last one is omitted in this tutorial for simplicity. Edit corresponding handlers to call the on_balance_update method with data from matched operations:
from decimal import Decimal
from demo_token.handlers.on_balance_update import on_balance_update
from demo_token.types.tzbtc.tezos_parameters.transfer import TransferParameter
from demo_token.types.tzbtc.tezos_storage import TzbtcStorage
from dipdup.context import HandlerContext
from dipdup.models.tezos_tzkt import TzktTransaction
async def on_transfer(
ctx: HandlerContext,
transfer: TzktTransaction[TransferParameter, TzbtcStorage],
) -> None:
if transfer.parameter.from_ == transfer.parameter.to:
# NOTE: Internal tzBTC transfer
return
amount = Decimal(transfer.parameter.value) / (10**8)
await on_balance_update(
address=transfer.parameter.from_,
balance_update=-amount,
timestamp=transfer.data.timestamp,
)
await on_balance_update(
address=transfer.parameter.to,
balance_update=amount,
timestamp=transfer.data.timestamp,
)
from decimal import Decimal
from demo_token.handlers.on_balance_update import on_balance_update
from demo_token.types.tzbtc.tezos_parameters.mint import MintParameter
from demo_token.types.tzbtc.tezos_storage import TzbtcStorage
from dipdup.context import HandlerContext
from dipdup.models.tezos_tzkt import TzktTransaction
async def on_mint(
ctx: HandlerContext,
mint: TzktTransaction[MintParameter, TzbtcStorage],
) -> None:
amount = Decimal(mint.parameter.value) / (10**8)
await on_balance_update(
address=mint.parameter.to,
balance_update=amount,
timestamp=mint.data.timestamp,
)
And that's all! We can run the indexer now.
Next steps
Run the indexer in memory:
dipdup run
Store data in SQLite database:
dipdup -c . -c configs/dipdup.sqlite.yaml run
Or spawn a Compose stack with PostgreSQL and Hasura:
cd deploy
cp .env.default .env
# Edit .env file before running
docker-compose up
DipDup will fetch all the historical data and then switch to realtime updates. You can check the progress in the logs.
If you use SQLite, run this query to check the data:
sqlite3 demo_token.sqlite 'SELECT * FROM holder LIMIT 10'
If you run a Compose stack, check open http://127.0.0.1:8080 in your browser to see the Hasura console (an exposed port may differ). You can use it to explore the database and build GraphQL queries.
Congratulations! You've just created your first DipDup indexer. Proceed to the Getting Started section to learn more about DipDup configuration and features.