Params per consumer module

[abhimanyusinghgaur]

Right now, one can specify params for a module at a global level in a substreams.yaml file. Like in this manifest, I'm filtering Transfer events using the ethereum_common:filtered_events imported module:

specVersion: v0.1.0
package:
  name: erc20
  version: v0.1.0

imports:
  ethereum_common: https://github.com/streamingfast/substreams-foundational-modules/releases/download/ethereum_common-v0.3.0/ethereum-common-v0.3.0.spkg

protobuf:
  files:
    - erc20/v1/types.proto
  importPaths:
    - proto

binaries:
  default:
    type: wasm/rust-v1
    file: ../target/wasm32-unknown-unknown/release/erc20.wasm

params:
  # filter the Transfer events
  ethereum_common:filtered_events: "evt_sig:0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"

modules:
  - name: map_extract_erc20_transfer_events
    kind: map
    initialBlock: 0
    inputs:
      - map: ethereum_common:filtered_events
    output:
      type: proto:erc20.v1.TransferEvents

  - name: store_unique_erc20_transfer_addresses
    kind: store
    updatePolicy: set_if_not_exists
    valueType: bytes # empty bytes always
    inputs:
      - map: map_extract_erc20_transfer_events

  - name: map_extract_erc20_tokens
    kind: map
    inputs:
      - store: store_unique_erc20_transfer_addresses
        mode: deltas
    output:
      type: proto:erc20.v1.Tokens

  - name: store_erc20_tokens
    kind: store
    updatePolicy: set
    valueType: proto:erc20.v1.TokenInfo
    inputs:
      - map: map_extract_erc20_tokens

Now, I want to build another substream that uses the store_erc20_tokens store and also wants to filter on the PoolCreated event on UniswapV3Factory contract. For that, I will need a manifest like this:

specVersion: v0.1.0
package:
  name: uniswap_v3_pool_tokens
  version: v0.1.0

imports:
  ethereum_common: https://github.com/streamingfast/substreams-foundational-modules/releases/download/ethereum_common-v0.3.0/ethereum-common-v0.3.0.spkg
  erc20: ../erc20/erc20-v0.1.0.spkg

params:
  # filter the PoolCreated events on the UniswapV3factory contract
  ethereum_common:filtered_events: "evt_sig:0x783cca1c0412dd0d695e784568c96da2e9c22ff989357a2e8b1d9b2b4e6b7118 && evt_addr:0x1f98431c8ad98523631ae4a59f267346ea31f984"

modules:
  - name: map_uniswap_v3_pool_tokens
    kind: map
    initialBlock: 12369621
    inputs:
      - map: ethereum_common:filtered_events
      - store: store_erc20_tokens
    output:
      type: proto:someProto

The issue here is that the uniswap_v3_pool_tokens substream overrides the params even for the erc20 substream, resulting in wrong output from the store_erc20_tokens store.

My proposal here is that it should be allowed to specify params at the level of a consumer module so that this use case can work as intended. Eg:

modules:
  - name: map_uniswap_v3_pool_tokens
    kind: map
    initialBlock: 12369621
    inputs:
      - map: ethereum_common:filtered_events
      - store: store_erc20_tokens
    output:
      type: proto:someProto
    params:
      ethereum_common:filtered_events: "evt_sig:0x783cca1c0412dd0d695e784568c96da2e9c22ff989357a2e8b1d9b2b4e6b7118 && evt_addr:0x1f98431c8ad98523631ae4a59f267346ea31f984"

This way, a set of independent params can be applied to a source module when it's being used by multiple consumer modules.

[abhimanyusinghgaur]

Hi @abourget, the absence of this feature seriously limits the reuse of the substreams cache. Because of this, I always end up using ethereum_common:all_events and filter things manually every time instead of just using the ethereum_common:filtered_events with a param. If it was possible to use the ethereum_common:filtered_events multiple time with different params in the same substream, then I would be able to reuse a lot of substreams cache instead of running computations everytime from scratch.

Are there any plans to solve it?

[sduchesneau]

There is a solution for that, see this example:

modules:
  - name: map_events
    kind: map
    initialBlock: 24183211
    inputs:
      - params: string
      - source: sf.ethereum.type.v2.Block
    output:
      type: proto:contract.v1.Events

  - name: child
    kind: map
    inputs:
      - map: map_events
    output:
      type: proto:contract.v1.Events

  - name: map_events_two # derived from map_events, but with its own param
    use: map_events
    # note: before substreams CLI release v1.17.10, we need the following 3 lines because of a parsing bug. They can be omitted starting at v1.17.10
    inputs:
      - params: string
      - source: sf.ethereum.type.v2.Block

  - name: child_two # child from map_events_two
    use: child
    inputs:
      - map: map_events_two

  - name: grandchild # this modules uses the output from both child and child_two
    kind: map
    inputs:
      - map: child
      - map: child_two
    output:
      type: proto:contract.v1.Events

network: mainnet

params:
  map_events: 1f98431c8ad98523631ae4a59f267346ea31f984
  map_events_two: 18aaa7115705e8be94bffebde57af9bfc265b998

This example does not use stores, but it does use maps that depend upon other maps with params.
The use of blockFilter is also permitted in this context, by defining the blockfilter on map_events_two directly, for example.

Note that before next release (v1.17.10), you need to explicitly repeat the 'inputs' in the map_events_two. Release v1.17.10 is coming very soon, so this won't be needed afterwards.

[abhimanyusinghgaur]

Thanks, I have tested it with v1.17.11, and it is working as expected.