Generic HTTP

If you need to perform arbitrary requests to APIs not supported by DipDup, use generic HTTP datasource instead of plain aiohttp requests. That way you can use the same features DipDup uses for internal requests: retry with backoff, rate limiting, Prometheus integration etc.

datasources:
  my_api:
    kind: http
    url: https://my_api.local/v1

Then, in your hook or handler code:

api = ctx.get_http_datasource('my_api')
response = await api.request(
    method='get',
    url='hello',
    params={
      'foo': 'bar',
    },
)

All DipDup datasources are inherited from http, so you can send arbitrary requests with any datasource. Let's say you want to fetch the protocol of the chain you're currently indexing (tezos.tzkt datasource doesn't have a separate method for it):

tzkt = ctx.get_tzkt_datasource('tzkt_mainnet')
protocol_json = await tzkt.request(
    method='get',
    url='v1/protocols/current',
)
assert protocol_json['hash'] == 'PtHangz2aRngywmSRGGvrcTyMbbdpWdpFKuS4uMWxg2RaH9i1qx'

Datasource HTTP connection parameters (ratelimit, retry with backoff, etc.) are applied to every request.