Monitoring
It's important to configure your monitoring to know that your indexer is working properly. DipDup provides multiple ways to monitor your indexer:
- Prometheus integration
- Internal database tables and views
- Monitoring API endpoint
- Sentry integration
Monitoring endpoint
You can use API endpoint to get the various indexing stats. First, enable internal API in config:
api: {}
By default, DipDup will listen on 0.0.0.0:46339. Query the /performance endpoint:
curl http://0.0.0.0:46339/performance
Internal tables
DipDup uses a set of internal tables to keep track of indexing stats. See the Internal tables page for details.
Additionally, metrics returned by the monitoring endpoint are stored in the dipdup_meta table in JSON format. You can use dipdup_status database view to get a nice summary.
Via SQL:
sqlite> select * from dipdup_status;
type name level size updated_at
---------- ------------------------- -------- ------ --------------------------------
index eth_usdt_events 8211425 0 2024-09-09 20:33:07.482867+00:00
datasource subsquid 20714000 0 2024-09-09 20:22:38.486122+00:00
queue eth_usdt_events:realtime 0 0 2024-09-09 20:33:04.493736+00:00
queue eth_usdt_events:readahead 0 9171 2024-09-09 20:33:04.493736+00:00
cache model:Holder 0 262144 2024-09-09 20:33:04.493736+00:00
Via Hasura:
curl http://0.0.0.0:8000/api/rest/dipdup_head_status?name=eth_events
You can also create your custom alert endpoints using SQL views and functions; see the SQL scripts page for details.
Sentry
Sentry is an error tracking software that can be used either as a service or on-premise. It dramatically improves the troubleshooting experience and requires nearly zero configuration. To start catching exceptions with Sentry in your project, add the following section in dipdup.yaml config:
sentry:
dsn: https://example.com
environment: dev
debug: False
You can obtain Sentry DSN from the web interface at Settings -> Projects -> <project_name> -> Client Keys (DSN). The cool thing is that if you catch an exception and suspect there's a bug in DipDup, you can share this event with us using a public link (created at Share menu).
Prometheus
DipDup provides integration with the Prometheus monitoring system. To enable the integration and listen on 0.0.0.0:9000, add the following section to the config:
prometheus: {}
The following metrics are exposed under dipdup namespace:
| name | description | type |
|---|---|---|
| dipdup_datasource_head_updated_timestamp | Timestamp of the last head update | Gauge |
| dipdup_datasource_requests | Total number of datasource requests | Counter |
| dipdup_datasource_rollbacks | Number of rollbacks | Counter |
| dipdup_datasource_time_in_requests_seconds | Time spent in datasource requests | Histogram |
| dipdup_http_errors | Number of http errors | Counter |
| dipdup_http_errors_in_row | Number of consecutive failed requests | Gauge |
| dipdup_index_handlers_matched | Index total hits | Counter |
| dipdup_index_levels_to_realtime_total | Number of levels to reach realtime state | Histogram |
| dipdup_index_levels_to_sync_total | Number of levels to reach synced state | Histogram |
| dipdup_index_time_in_callbacks_seconds | Time spent in callbacks | Histogram |
| dipdup_index_time_in_matcher_seconds | Time spent in matcher | Histogram |
| dipdup_index_total_realtime_duration_seconds | Duration of the last index realtime syncronization | Histogram |
| dipdup_index_total_sync_duration_seconds | Duration of the last index syncronization | Histogram |
| dipdup_indexes_total | Number of indexes in operation by status | Gauge |
| dipdup_levels_indexed_total | Total number of levels indexed | Gauge |
| dipdup_levels_nonempty | Total number of nonempty levels indexed | Counter |
| dipdup_levels_nonempty_speed | Nonempty levels per second | Gauge |
| dipdup_levels_speed | Levels per second | Gauge |
| dipdup_levels_speed_average | Average levels per second | Gauge |
| dipdup_levels_total | Total number of levels | Gauge |
| dipdup_metrics_updated_at_timestamp | Timestamp of the last metrics update | Gauge |
| dipdup_objects_indexed | Total number of objects indexed | Counter |
| dipdup_objects_speed | Objects per second | Gauge |
| dipdup_progress | Progress in percents | Gauge |
| dipdup_realtime_at_timestamp | Timestamp of the last realtime update | Gauge |
| dipdup_started_at_timestamp | Timestamp of the DipDup start | Gauge |
| dipdup_synchronized_at_timestamp | Timestamp of the last synchronization | Gauge |
| dipdup_time_left_seconds | Time left estimated until the end | Gauge |
| dipdup_time_passed_seconds | Time passed since the start | Gauge |
| sqd_processor_archive_http_errors_in_row | Number of consecutive failed requests to Subsquid Network | Gauge |
| sqd_processor_chain_height | Current chain height as reported by Subsquid Network | Gauge |
| sqd_processor_last_block | Level of the last processed block from Subsquid Network | Gauge |
Metrics under the sqd namespace are for compatibility with Subsquid Cloud.
Logging
To control the number of logs DipDup produces, set the logging field in config. It can be either a string or a mapping from logger names to logging levels.
# Configure dipdup and package loggers
logging: WARN
# Increase verbosity of some loggers
logging:
dipdup.database: DEBUG
aiosqlite: DEBUG
# Enable ALL logs
logging:
'': DEBUG
By default only dipdup and dipdup_indexer namespace loggers are configured. DipDup loggers are usually named after the corresponding module, e.g., dipdup.database or dipdup.index.
Keep in mind, that excessive logging can affect indexing performance.
If you need your logs in JSON format, use DIPDUP_JSON_LOG=1 environment variable.