bulkindexer exposes some config options to tune when to flush etc, but it's currently hardcoded (via bulkChannel and StartDocChannel) that if we're busy flushing, we only allow up 100 new items before any call such as BulkIndexer.Index() starts blocking.
we have an application where we may want to add millions of items at once, and if it takes a while for all that data to be bulkindexed, that's fine, but we don't want our app to be hanging on Index() calls.
would it make sense to add such a buffer to elastigo, or should i do it in my application?
a configurable bulkChannel size would be a good start but that allocates all the space up front, so a buffer that can grow (up to the limit) and shrink as needed might be even better ( though i'm not sure how to implement that elegantly, it may not be worth it.)
bulkindexer exposes some config options to tune when to flush etc, but it's currently hardcoded (via
bulkChannelandStartDocChannel) that if we're busy flushing, we only allow up 100 new items before any call such asBulkIndexer.Index()starts blocking.we have an application where we may want to add millions of items at once, and if it takes a while for all that data to be bulkindexed, that's fine, but we don't want our app to be hanging on Index() calls.
would it make sense to add such a buffer to elastigo, or should i do it in my application?
a configurable bulkChannel size would be a good start but that allocates all the space up front, so a buffer that can grow (up to the limit) and shrink as needed might be even better ( though i'm not sure how to implement that elegantly, it may not be worth it.)