Store original (uploaded) documents

[piroor]

Now we don't store original documents and there are only indexed fields. To support "SourceAttribute" and "DefaultValue" behaviours, we have to store original documents separately.

[piroor]

How should we store the source?

• as groonga columns (vector type)
  • Good:
    • more groonga friendly.
  • Bad:
    • cannot/hard to store unknown fields
• as a JSON string
  • Good:
    • we can store any data easily
  • Bad:
    • hard to merge on every upload, if missing fields in newly uploaded SDF batch is filled by the previous value instead of the default value

[piroor]

Steps to confirm:

1. Create a new domain "test".
2. Define a new fields "name" and "age".
3. Upload a SDF with two items:
   • id=id1, name="foo baz", age=100, type="human" (typewill be ignored)
   • id=id2, name="foo bar" age=10
4. Upload a SDF with one item:
   • id=id2, age=50
5. Define a new field "type"
6. Search by "foo". Then...
   • Does the record id1 have "type=human" value as its result?
   • Does the record id2 have its name?

[piroor]

Result:

• Does the record id1 have "type=human" value as its result? => Yes. Uploaded SDFs are completely stored to the database for reindexing.
• Does the record id2 have its name? => No. Uploaded SDFs are not merged by ACS.