Monitoring
Monitoring is essential to ensure your DipDup indexer is healthy, up-to-date, and performing as expected. DipDup provides several ways to monitor your indexer, suitable for both development and production environments.
Monitoring options
- Prometheus integration: Exposes a wide range of metrics for scraping by Prometheus. See the metrics table for a full list of available metrics.
- Internal database tables and views: Track indexing stats directly in your database.
- Monitoring API endpoint: Query the
/performanceendpoint for a summary of performance metrics. See the API reference. - Sentry integration: Capture and track errors and exceptions.
Tip: For production deployments, consider combining Prometheus metrics with alerting tools (like BetterUptime or Grafana) to get notified about issues in real time.
Monitoring endpoint
You can use the API endpoint to get various indexing stats. First, enable the internal API in your config:
api: {}
By default, DipDup will listen on 0.0.0.0:46339. Query the /performance endpoint:
curl http://0.0.0.0:46339/performance
See the API reference for details on the response format.
Internal tables
DipDup uses a set of internal tables to keep track of indexing stats. See the Internal tables page for details.
Additionally, metrics returned by the monitoring endpoint are stored in the dipdup_meta table in JSON format. You can use the dipdup_status database view to get a summary.
sqlite> select * from dipdup_status;
type name level size updated_at
---------- ------------------------- -------- ------ --------------------------------
index eth_usdt_events 8211425 0 2024-09-09 20:33:07.482867+00:00
datasource subsquid 20714000 0 2024-09-09 20:22:38.486122+00:00
queue eth_usdt_events:realtime 0 0 2024-09-09 20:33:04.493736+00:00
queue eth_usdt_events:readahead 0 9171 2024-09-09 20:33:04.493736+00:00
cache model:Holder 0 262144 2024-09-09 20:33:04.493736+00:00
You can also create your custom alert endpoints using SQL views and functions; see the SQL scripts page for details.
Sentry
Sentry is an error tracking software that can be used either as a service or on-premise. It dramatically improves the troubleshooting experience and requires nearly zero configuration. To start catching exceptions with Sentry in your project, add the following section in dipdup.yaml config:
sentry:
dsn: https://example.com
environment: dev
debug: False
You can obtain Sentry DSN from the web interface at Settings -> Projects -> <project_name> -> Client Keys (DSN). The cool thing is that if you catch an exception and suspect there's a bug in DipDup, you can share this event with us using a public link (created at Share menu).
Prometheus
DipDup provides integration with the Prometheus monitoring system. To enable the integration and listen on 0.0.0.0:9000, add the following section to the config:
prometheus: {}
The following metrics are exposed under dipdup namespace:
| name | description | type |
|---|---|---|
| dipdup_datasource_head_updated_timestamp | Timestamp of the last head update | Gauge |
| dipdup_datasource_requests | Total number of datasource requests | Counter |
| dipdup_datasource_rollbacks | Number of rollbacks | Counter |
| dipdup_datasource_time_in_requests_seconds | Time spent in datasource requests | Histogram |
| dipdup_http_errors | Number of http errors | Counter |
| dipdup_http_errors_in_row | Number of consecutive failed requests | Gauge |
| dipdup_index_handlers_matched | Index total hits | Counter |
| dipdup_index_levels_to_realtime_total | Number of levels to reach realtime state | Histogram |
| dipdup_index_levels_to_sync_total | Number of levels to reach synced state | Histogram |
| dipdup_index_time_in_callbacks_seconds | Time spent in callbacks | Histogram |
| dipdup_index_time_in_matcher_seconds | Time spent in matcher | Histogram |
| dipdup_index_total_realtime_duration_seconds | Duration of the last index realtime syncronization | Histogram |
| dipdup_index_total_sync_duration_seconds | Duration of the last index syncronization | Histogram |
| dipdup_indexes_total | Number of indexes in operation by status | Gauge |
| dipdup_levels_indexed_total | Total number of levels indexed | Gauge |
| dipdup_levels_nonempty | Total number of nonempty levels indexed | Counter |
| dipdup_levels_nonempty_speed | Nonempty levels per second | Gauge |
| dipdup_levels_speed | Levels per second | Gauge |
| dipdup_levels_speed_average | Average levels per second | Gauge |
| dipdup_levels_total | Total number of levels | Gauge |
| dipdup_metrics_updated_at_timestamp | Timestamp of the last metrics update | Gauge |
| dipdup_objects_indexed | Total number of objects indexed | Counter |
| dipdup_objects_speed | Objects per second | Gauge |
| dipdup_progress | Progress in percents | Gauge |
| dipdup_realtime_at_timestamp | Timestamp of the last realtime update | Gauge |
| dipdup_started_at_timestamp | Timestamp of the DipDup start | Gauge |
| dipdup_synchronized_at_timestamp | Timestamp of the last synchronization | Gauge |
| dipdup_time_left_seconds | Time left estimated until the end | Gauge |
| dipdup_time_passed_seconds | Time passed since the start | Gauge |
| sqd_processor_archive_http_errors_in_row | Number of consecutive failed requests to Subsquid Network | Gauge |
| sqd_processor_chain_height | Current chain height as reported by Subsquid Network | Gauge |
| sqd_processor_last_block | Level of the last processed block from Subsquid Network | Gauge |
Metrics under the sqd namespace are for compatibility with Subsquid Cloud.
Logging
To control the number of logs DipDup produces, set the logging field in config. It can be either a string or a mapping from logger names to logging levels.
# Configure dipdup and package loggers
logging: WARN
# Increase verbosity of some loggers
logging:
dipdup.database: DEBUG
aiosqlite: DEBUG
# Enable ALL logs
logging:
'': DEBUG
By default only dipdup and <package> namespace loggers are configured. DipDup loggers are usually named after the corresponding module, e.g., dipdup.database or dipdup.index.
Keep in mind, that excessive logging can affect indexing performance.
If you need your logs in JSON format, use DIPDUP_JSON_LOG=1 environment variable.