Vector Culling and Equality Checks

[nickna]

@hangy, I saw your comment, but it got hidden in the PR comment thread after the MemoryMappedFiles merged.
TLDR: It's not needed in MemoryMappedList since it does not inherit the Ilist or ICollection interfaces. However, it should be checked for duplication prior to rebuilding the index.

The test case for this Update method is failing after the rebase, because the Contains check only checks for equality based on the Vector.Id.
Should Id, OriginalText or Tags count for equality and the Contains check, or is the equality contract just a.Values == b.Values? This PR's implementation of a memory mapped list is solely based on searching by the Vector.Id, so that other definitions might be slow to search/remove. Also, if anything other than Id is relevant for the Contains check, should a call Add do anything if the Values already exist? 

Contains() (in VectorList) came to be when I wanted interface support with IList and ICollection.
As it works now, it's a compliance function. The root question is: Why would someone call Contains() at all?
Even if it did a true equality check across the board (except for Id), it's not obvious why this is needed.

Unintentionally adding duplicate Vectors (if comparing by Values and OriginalText) would be a good thing to prevent. Duplicate data, aside from being, well, duplicate, can throw off the search tree. However, calling Contains() from within the Add() and AddRange() functions would get O(n) slower.

It would be even better to do this asynchronously, even at the cost of allowing duplicate data for brief periods.
In StartIndexService(), when a modified event is invoked, enumerate through the database and cull duplicate records before rebuilding the index.
On mobile devices, where the background thread is not available, this can be added as an optional function for the VectorDatabase.RebuildSearchIndexAsync() and RebuildSearchIndexesAsync().

What do you think?

[hangy]

The root question is: Why would someone call Contains() at all? Even if it did a true equality check across the board (except for Id), it's not obvious why this is needed.

For tests? But those could probably also work with some Search method. I'm anything but an expert on AI and vector databases, but searching for a specific vector value does not seem like the most common use case in my limited experience.

Unintentionally adding duplicate Vectors (if comparing by Values and OriginalText) would be a good thing to prevent. Duplicate data, aside from being, well, duplicate, can throw off the search tree. However, calling Contains() from within the Add() and AddRange() functions would get O(n) slower.

I agree. A Contains() check from each Add() call will grow slow very fast.

It would be even better to do this asynchronously, even at the cost of allowing duplicate data for brief periods. In StartIndexService(), when a modified event is invoked, enumerate through the database and cull duplicate records before rebuilding the index.

I think that sounds like a good idea. If the indexer thread has to remove dupes and needs to check all vectors in the list, that could take a significant amount of time if there are a lot of vectors. Maybe benchmark how long a single run might take with a large number of vectors? In my mind, it would likely have to lock access to the list for modifications, so that this might look like an online index rebuild with a table lock in an RDBMS.
Come to think of it, the indexer thread may need to lock the list for writing anyways, as most implementations of SearchService.BuildIndex naturally need a consistent input list.

[nickna]

I left this thread hanging...

You bring up a reasonable point about equality. I think true Equality will require three functions:

1. Equals and/or Compare: check Id, Tags, Values, and OriginalText
   Tags can be in any order, whereas Values are naturally order-sensitive.
   One way to keep this simple would be to sort the Tags and then make the same comparison between both arrays.
2. Equality of data sans Id: Tags, Values, OriginalText
   If someone were to create a new Vector with existing data, it would come in with a new Id. This is a real possibility. This function could be used in the aforementioned asynchronous background processing.
3. Amend GetHashCode()

• Sort Tags
• Hash ( Tags, Values, OriginalText)
• Store the hash as a private field so it's only produced once per lifetime -- it would still get recalculated if the Vector is recreated after a DB load unless it's stored on disk.

I'm conflating Compare and Equals because I see them as having the same purpose here.

What do you think?

[hangy]

Store the hash as a private field so it's only produced once per lifetime -- it would still get recalculated if the Vector is recreated after a DB load unless it's stored on disk.

Recalculation would also need to be done in the case of InPlace* methods. Also, at least right now, Vector objects are not fully immutable right now. The Tags and Vector array can be modified by outside callers, without a possibility for the Vector object to know about the modification, making the computed hash code stale.

Generally speaking, your approach sounds correct.