The two storage technologies used in this database are Durable Objects and the Cloudflare Key Value store (KV). There exists a low level abstraction around these technologies in the DurableObject and KeyValueStore folders respectively.
The main feature is the Database, it contains Silos which are either Archives or Collections. Archives implement the KV abstraction and ultimately the Cloudflare KV.
- The Archive is able to store, load and remove documents, loading can be done using the Selection type or an id of the document.
- The Collection utilizes a Durable Object abstraction called Buffer and an Archive for long term storage. The Buffer ensures consistency and transactionality while writing. The buffer stores documents for a short timespan and writes documents to the Archive once deemed stale by the Archivist. The buffer can contain several durable objects to increase the number of simultaneous read and writes, these durable objects will have an added key-shard at the end of their ids, the shards are generated by the ids of the documents.
The Database can be partitioned to store data in different layers. The key for the documents are comprised of typeName/doc/partition_1/partition_2/.../partition_n/dateTime/documentID where n is a natural number. The Buffer will divide the data in different durable objects using typeName/partition_1/partition_2/.../partition_n/shard as key.
figure 1: The architecture of the database.
There exists 3 types of keys in the Database, id, changed and doc; each adds functionality to the database.
The definitions below are based on how they look in the Archive.
The doc-index contains the document the user wants to store/load/update. The structure of the doc-key makes it possible to list a certain partition and query it on created time with a date range.
Definition:
key := type + "/doc/" + partitions.join("/") + "/" + document.created + "/" + document.id
{
[key]: document
}Where type is the name of the document used in the layout when initiating the database, the partitions are all of the partitions added to the initiated database, the document is the value the user wants to store.
The purpose of the id-index is to make it possible to fetch a document while only knowing the type and id of the document. Definition:
{
[type + "/id/" + document.id]: key
}Where the type is the name of the document-type used in the layout when initiating the database and the document is the value the user wants to store and the key is defined above in doc.
The changed-index is multipurpose and is used in the buffer to determine which documents to archive and which archived documents to remove. Its use in the archive is to query documents with a daterange representing when the document was changed. A document can be found if it was changed within the daterange, even if the document was changed again later on. So the change query will find every time the document was changed in the archive, either by being updated by the user or the buffer.
Definition:
{
changed := truncate(document.changed, "minutes")
[type + "/changed/" + changed]: `key_1\n
key_2\n
...
key_n`
}Where the truncate function truncates the changed date of the document to minutes to save all keys of documents changed within that minute, the document is the value the user wants to store in the database, the key_n are on the form of key defined in doc and n is a natural number.
The shard is calculated from the first two characters in the id of the document. Since the id of the document is assumed to be base 64 the first two characters will contain 12 bits of information. By converting the characters to a Uint8Array and then truncating, such that only as many bits needed to describe as many shards configured is used, followed by a conversion back to base 64, a surjective mapping is created from id -> shard.
The Cursor's content should be obscured from the end user, this is done by base 64 encoding the data. Encoding is enough since we're not dealing with security, rather ensuring the end user doesn't consider the Cursor to be mutable.
The Archivist is located in the backend of the buffer. The Archivist should be called by the alarm and will store any stale documents. Stale documents are identified by comparing the changed index with the reconcileAfter time. Stale documents are then written to the KV corresponding to the Archive used by the Collection. The Archivist will delete any documents changed before superimposeFor + the time of the last changed index written to the KV.
The test-worker runs a miniflare environment on jest to mock the behavior of durable objects and the Cloudflare KV. Make sure to write tests for any new features added to the database.