[Discussion] Improve performance for `updateReferences` in a `transformEntriesToType` migration

[andonimihai]

Expected Behavior

When using updateReferences in a transformEntriesToType migration, i would love to have API calls only for the entries that should be affected.

Actual Behavior

In a transformEntriesToType migration, when using the updateReferences, it fetches all the available entries in the space. While this could work for smaller entry amounts, when there are like 60k+, it takes forever.
(source: https://github.com/contentful/contentful-migration/blob/master/src/lib/fetcher.ts#L14, https://github.com/contentful/contentful-migration/blob/master/src/lib/intent/entry-transform-to-type.ts#L50)

Possible Solution

For each entry that requires to have references updated, use the rest API to fetch only the entries where the current entry is referenced, and update them.

Context

The migrations are running very slow when there are a lot of entries(even though it should migrate 200 entries from 60k)

[alecpm]

A big +1 to this. I was testing a migration using updateReferences that only touched two entries in a space with hundreds of thousands of entries, and it took so long that I assumed it had hung. The migration seemed to result in rate limit errors, and might never have completed as far as I know. I assumed it had hung, but if it fetched every entry, that would explain it.

I ended up making an initial mapping from the original entry ids to the migrated ids and ran a transformEntries call after transformEntriesToType to update the references directly using that mapping.