Monitoring
It's important to configure your monitoring to know that your indexer is working properly. DipDup provides multiple ways to monitor your indexer:
- Prometheus integration
- Internal database tables and views
- Monitoring API endpoint
- Sentry integration
Monitoring endpoint
You can use API endpoint to get the various indexing stats. First, enable internal API in config:
api: {}
By default, DipDup will listen on 0.0.0.0:46339
. Query the /performance
endpoint:
curl http://0.0.0.0:46339/performance
Internal tables
DipDup uses a set of internal tables to keep track of indexing stats. See the Internal tables page for details.
Additionally, metrics returned by the monitoring endpoint are stored in the dipdup_meta
table in JSON format. You can use dipdup_status
database view to get a nice summary.
Via SQL:
sqlite> select * from dipdup_status;
type name level size updated_at
---------- ------------------------- -------- ------ --------------------------------
index eth_usdt_events 8211425 0 2024-09-09 20:33:07.482867+00:00
datasource subsquid 20714000 0 2024-09-09 20:22:38.486122+00:00
queue eth_usdt_events:realtime 0 0 2024-09-09 20:33:04.493736+00:00
queue eth_usdt_events:readahead 0 9171 2024-09-09 20:33:04.493736+00:00
cache model:Holder 0 262144 2024-09-09 20:33:04.493736+00:00
Via Hasura:
curl http://0.0.0.0:8000/api/rest/dipdup_head_status?name=eth_events
You can also create your custom alert endpoints using SQL views and functions; see the SQL scripts page for details.
Sentry
Sentry is an error tracking software that can be used either as a service or on-premise. It dramatically improves the troubleshooting experience and requires nearly zero configuration. To start catching exceptions with Sentry in your project, add the following section in dipdup.yaml
config:
sentry:
dsn: https://example.com
environment: dev
debug: False
You can obtain Sentry DSN from the web interface at Settings -> Projects -> <project_name> -> Client Keys (DSN). The cool thing is that if you catch an exception and suspect there's a bug in DipDup, you can share this event with us using a public link (created at Share menu).
Prometheus
DipDup provides integration with the Prometheus monitoring system. To enable the integration and listen on 0.0.0.0:9000
, add the following section to the config:
prometheus: {}
The following metrics are exposed under dipdup
namespace:
name | description | type |
---|---|---|
dipdup_datasource_head_updated_timestamp | Timestamp of the last head update | Gauge |
dipdup_datasource_requests | Total number of datasource requests | Counter |
dipdup_datasource_rollbacks | Number of rollbacks | Counter |
dipdup_datasource_time_in_requests_seconds | Time spent in datasource requests | Histogram |
dipdup_http_errors | Number of http errors | Counter |
dipdup_http_errors_in_row | Number of consecutive failed requests | Gauge |
dipdup_index_handlers_matched | Index total hits | Counter |
dipdup_index_levels_to_realtime_total | Number of levels to reach realtime state | Histogram |
dipdup_index_levels_to_sync_total | Number of levels to reach synced state | Histogram |
dipdup_index_time_in_callbacks_seconds | Time spent in callbacks | Histogram |
dipdup_index_time_in_matcher_seconds | Time spent in matcher | Histogram |
dipdup_index_total_realtime_duration_seconds | Duration of the last index realtime syncronization | Histogram |
dipdup_index_total_sync_duration_seconds | Duration of the last index syncronization | Histogram |
dipdup_indexes_total | Number of indexes in operation by status | Gauge |
dipdup_levels_indexed_total | Total number of levels indexed | Gauge |
dipdup_levels_nonempty | Total number of nonempty levels indexed | Counter |
dipdup_levels_nonempty_speed | Nonempty levels per second | Gauge |
dipdup_levels_speed | Levels per second | Gauge |
dipdup_levels_speed_average | Average levels per second | Gauge |
dipdup_levels_total | Total number of levels | Gauge |
dipdup_metrics_updated_at_timestamp | Timestamp of the last metrics update | Gauge |
dipdup_objects_indexed | Total number of objects indexed | Counter |
dipdup_objects_speed | Objects per second | Gauge |
dipdup_progress | Progress in percents | Gauge |
dipdup_realtime_at_timestamp | Timestamp of the last realtime update | Gauge |
dipdup_started_at_timestamp | Timestamp of the DipDup start | Gauge |
dipdup_synchronized_at_timestamp | Timestamp of the last synchronization | Gauge |
dipdup_time_left_seconds | Time left estimated until the end | Gauge |
dipdup_time_passed_seconds | Time passed since the start | Gauge |
sqd_processor_archive_http_errors_in_row | Number of consecutive failed requests to Subsquid Network | Gauge |
sqd_processor_chain_height | Current chain height as reported by Subsquid Network | Gauge |
sqd_processor_last_block | Level of the last processed block from Subsquid Network | Gauge |
Metrics under the sqd
namespace are for compatibility with Subsquid Cloud.
Logging
To control the number of logs DipDup produces, set the logging
field in config. It can be either a string or a mapping from logger names to logging levels.
# Configure dipdup and package loggers
logging: WARN
# Increase verbosity of some loggers
logging:
dipdup.database: DEBUG
aiosqlite: DEBUG
# Enable ALL logs
logging:
'': DEBUG
By default only dipdup
and dipdup_indexer
namespace loggers are configured. DipDup loggers are usually named after the corresponding module, e.g., dipdup.database
or dipdup.index
.
Keep in mind, that excessive logging can affect indexing performance.
If you need your logs in JSON format, use DIPDUP_JSON_LOG=1
environment variable.