warning: Received invalid partition id from user, retry won't happen, failing the request

[Liave]

Hi,

We are using bulk insert and recently we started getting this error:

WARN com.microsoft.azure.documentdb.internal.directconnectivity.GoneAndRetryWithRetryPolicy [pool-10-thread-3120] Received invalid partition id from user, retry won't happen, failing the request
2021-05-11T10:10:28.936257043Z com.microsoft.azure.documentdb.DocumentClientException: Invalid partition key range,
2021-05-11T10:10:28.936265143Z RequestStartTimeUTC: "11 May 2021 10:10:21.670", RequestEndTimeUTC: "11 May 2021 10:10:22.903", Number of regions attempted: 1

we also see:
Partition key range is finishing split and is not available for reads/writes.

Does anybody knows what does it mean?

thanks,
Liav

more stacktrace:
2021-05-11T10:10:28.936370743Z at com.microsoft.azure.documentdb.internal.directconnectivity.GatewayAddressCache.resolveServer(GatewayAddressCache.java:190)
2021-05-11T10:10:28.936374743Z at com.microsoft.azure.documentdb.internal.directconnectivity.GatewayAddressCache.resolve(GatewayAddressCache.java:131)
2021-05-11T10:10:28.936378543Z at com.microsoft.azure.documentdb.internal.directconnectivity.ReplicatedResourceClient.resolveAddresses(ReplicatedResourceClient.java:115)
2021-05-11T10:10:28.936382543Z at com.microsoft.azure.documentdb.internal.directconnectivity.ConsistencyWriter.writePrivate(ConsistencyWriter.java:112)
2021-05-11T10:10:28.936386443Z at com.microsoft.azure.documentdb.internal.directconnectivity.ConsistencyWriter.write(ConsistencyWriter.java:76)
2021-05-11T10:10:28.936390243Z at com.microsoft.azure.documentdb.internal.directconnectivity.ReplicatedResourceClient.invoke(ReplicatedResourceClient.java:63)
2021-05-11T10:10:28.936394143Z at com.microsoft.azure.documentdb.internal.directconnectivity.ServerStoreModel$1.apply(ServerStoreModel.java:109)
2021-05-11T10:10:28.936398043Z at com.microsoft.azure.documentdb.internal.RetryUtility.executeStoreClientRequest(RetryUtility.java:152)
2021-05-11T10:10:28.936401843Z at com.microsoft.azure.documentdb.internal.directconnectivity.ServerStoreModel.processMessage(ServerStoreModel.java:126)
2021-05-11T10:10:28.936405643Z at com.microsoft.azure.documentdb.DocumentClient$5.apply(DocumentClient.java:3042)
2021-05-11T10:10:28.936409443Z at com.microsoft.azure.documentdb.internal.RetryUtility.executeDocumentClientRequest(RetryUtility.java:74)
2021-05-11T10:10:28.936413343Z at com.microsoft.azure.documentdb.DocumentClient.doCreate(DocumentClient.java:3047)
2021-05-11T10:10:28.936417144Z at com.microsoft.azure.documentdb.DocumentClient.executeStoredProcedure(DocumentClient.java:1576)
2021-05-11T10:10:28.936421244Z at com.microsoft.azure.documentdb.bulkexecutor.internal.BatchInserter$1.call(BatchInserter.java:186)
2021-05-11T10:10:28.936425044Z at com.microsoft.azure.documentdb.bulkexecutor.internal.BatchInserter$1.call(BatchInserter.java:158)
2021-05-11T10:10:28.936429044Z at com.microsoft.azure.documentdb.repackaged.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
2021-05-11T10:10:28.936433044Z at com.microsoft.azure.documentdb.repackaged.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
2021-05-11T10:10:28.936436844Z at com.microsoft.azure.documentdb.repackaged.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
2021-05-11T10:10:28.936440644Z at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
2021-05-11T10:10:28.936444444Z at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
2021-05-11T10:10:28.936448244Z at java.base/java.lang.Thread.run(Unknown Source)
2021-05-11T10:10:28.938340646Z 2021-05-11 10:10:28,936 WARN com.microsoft.azure.documentdb.bulkexecutor.internal.BatchInserter$1 [pool-10-thread-3120] Received a GoneException on Partition range id 16 | Storing the mini batch and retrying
2021-05-11T10:10:28.950830959Z 2021-05-11 10:10:28,950 WARN com.microsoft.azure.documentdb.internal.directconnectivity.GoneAndRetryWithRetryPolicy [pool-10-thread-3122] Received invalid partition id from user, retry won't happen, failing the request
2021-05-11T10:10:28.950851559Z com.microsoft.azure.documentdb.DocumentClientException: Invalid partition key range,