Monitoring

It's important to configure your monitoring to know that your indexer is working properly. DipDup provides multiple ways to monitor your indexer:

  • Prometheus integration
  • Internal database tables and views
  • Monitoring API endpoint
  • Sentry integration

Monitoring endpoint

You can use API endpoint to get the various indexing stats. First, enable internal API in config:

api: {}

By default, DipDup will listen on 0.0.0.0:46339. Query the /performance endpoint:

curl http://0.0.0.0:46339/performance

Internal tables

DipDup uses a set of internal tables to keep track of indexing stats. See the Internal tables page for details.

Additionally, metrics returned by the monitoring endpoint are stored in the dipdup_meta table in JSON format. You can use dipdup_status database view to get a nice summary.

Via SQL:

sqlite> select * from dipdup_status;
type        name                       level     size    updated_at
----------  -------------------------  --------  ------  --------------------------------
index       eth_usdt_events            8211425   0       2024-09-09 20:33:07.482867+00:00
datasource  subsquid                   20714000  0       2024-09-09 20:22:38.486122+00:00
queue       eth_usdt_events:realtime   0         0       2024-09-09 20:33:04.493736+00:00
queue       eth_usdt_events:readahead  0         9171    2024-09-09 20:33:04.493736+00:00
cache       model:Holder               0         262144  2024-09-09 20:33:04.493736+00:00

Via Hasura:

curl http://0.0.0.0:8000/api/rest/dipdup_head_status?name=eth_events

You can also create your custom alert endpoints using SQL views and functions; see the SQL scripts page for details.

Sentry

Sentry is an error tracking software that can be used either as a service or on-premise. It dramatically improves the troubleshooting experience and requires nearly zero configuration. To start catching exceptions with Sentry in your project, add the following section in dipdup.yaml config:

dipdup.yaml
sentry:
  dsn: https://example.com
  environment: dev
  debug: False

You can obtain Sentry DSN from the web interface at Settings -> Projects -> <project_name> -> Client Keys (DSN). The cool thing is that if you catch an exception and suspect there's a bug in DipDup, you can share this event with us using a public link (created at Share menu).

Prometheus

DipDup provides integration with the Prometheus monitoring system. To enable the integration and listen on 0.0.0.0:9000, add the following section to the config:

prometheus: {}

The following metrics are exposed under dipdup namespace:

namedescriptiontype
dipdup_datasource_head_updated_timestampTimestamp of the last head updateGauge
dipdup_datasource_requestsTotal number of datasource requestsCounter
dipdup_datasource_rollbacksNumber of rollbacksCounter
dipdup_datasource_time_in_requests_secondsTime spent in datasource requestsHistogram
dipdup_http_errorsNumber of http errorsCounter
dipdup_http_errors_in_rowNumber of consecutive failed requestsGauge
dipdup_index_handlers_matchedIndex total hitsCounter
dipdup_index_levels_to_realtime_totalNumber of levels to reach realtime stateHistogram
dipdup_index_levels_to_sync_totalNumber of levels to reach synced stateHistogram
dipdup_index_time_in_callbacks_secondsTime spent in callbacksHistogram
dipdup_index_time_in_matcher_secondsTime spent in matcherHistogram
dipdup_index_total_realtime_duration_secondsDuration of the last index realtime syncronizationHistogram
dipdup_index_total_sync_duration_secondsDuration of the last index syncronizationHistogram
dipdup_indexes_totalNumber of indexes in operation by statusGauge
dipdup_levels_indexed_totalTotal number of levels indexedGauge
dipdup_levels_nonemptyTotal number of nonempty levels indexedCounter
dipdup_levels_nonempty_speedNonempty levels per secondGauge
dipdup_levels_speedLevels per secondGauge
dipdup_levels_speed_averageAverage levels per secondGauge
dipdup_levels_totalTotal number of levelsGauge
dipdup_metrics_updated_at_timestampTimestamp of the last metrics updateGauge
dipdup_objects_indexedTotal number of objects indexedCounter
dipdup_objects_speedObjects per secondGauge
dipdup_progressProgress in percentsGauge
dipdup_realtime_at_timestampTimestamp of the last realtime updateGauge
dipdup_started_at_timestampTimestamp of the DipDup startGauge
dipdup_synchronized_at_timestampTimestamp of the last synchronizationGauge
dipdup_time_left_secondsTime left estimated until the endGauge
dipdup_time_passed_secondsTime passed since the startGauge
sqd_processor_archive_http_errors_in_rowNumber of consecutive failed requests to Subsquid NetworkGauge
sqd_processor_chain_heightCurrent chain height as reported by Subsquid NetworkGauge
sqd_processor_last_blockLevel of the last processed block from Subsquid NetworkGauge

Metrics under the sqd namespace are for compatibility with Subsquid Cloud.

Logging

To control the number of logs DipDup produces, set the logging field in config. It can be either a string or a mapping from logger names to logging levels.

dipdup.yaml
# Configure dipdup and package loggers
logging: WARN

# Increase verbosity of some loggers
logging:
  dipdup.database: DEBUG
  aiosqlite: DEBUG

# Enable ALL logs
logging:
  '': DEBUG

By default only dipdup and dipdup_indexer namespace loggers are configured. DipDup loggers are usually named after the corresponding module, e.g., dipdup.database or dipdup.index.

Keep in mind, that excessive logging can affect indexing performance.

If you need your logs in JSON format, use DIPDUP_JSON_LOG=1 environment variable.

Help and tips -> Join our Discord
Ideas or suggestions -> Issue Tracker
GraphQL IDE -> Open Playground