Add Collections Containment RFC

[tomcrane]

See t0030-storage-collection-items and -create-iiif-collection for accompanying Playwright examples.

[donaldgray]

Added some comments ahead of scheduled call on Monday

[donaldgray]

"JSON is King" but we append JSON to it. How will that work? What are the possible headaches?

Do we need some notion of "this is just a iiif collection" - you can't use me as a parent (ie prevent the system from updating items). OR do we maintain specific behaviour until a divergence happens?

[tomcrane]

JSON is King, but yes this does muddy the waters. It must be strictly append-only on addition of a new contained item, and never touches or re-orders the existing members.
I'm still not sure whether there is a distinction between the two container types now that we have this new property.

[tomcrane]

At the moment,

• A Storage Collection would only be a container
• a IIIF Collection can be a Container and a thing with a block of JSON for items that might not be the same as container
• ...a "no parent" ... can't be a container? Or just has a flag that means it never tries to help by appending new contained items to items?

... and do we actually need any of these distinctions or can it all be governed by predictable behaviour?
We still need the public/not-public distinction which only makes sense for Storage Collections (e.g., Leeds /cc/ is not a public resource), but if Storage Collections weren't a thing you'd still use this flag. However, if there is no distinction then you could in theory edit the items (as JSON) of a private collection - and what would be the point of that?

[donaldgray]

Check - what do we do now for storage collections?

[JackLewis-digirati]

right now the public view is just an id, a type and a label, i.e.:

{
   {
      "id": "https://presentation-api.dlcs.digirati.io/26/0001-presentation-iiif-collection_2",
      "type": "Collection",
      "label": {
        "en": [
          "0001-presentation-iiif-collection_5A8gz1"
        ]
      }
    },
    {
      "id": "https://presentation-api.dlcs.digirati.io/26/0005_dlcs_creates_assets_FKbIqA",
      "type": "Manifest",
      "label": {
        "en": [
          "IIIF manifest DLCS resource creation tests - multiple"
        ]
      }
    }
}

private additionally has a behavior block on collections

[donaldgray]

Do we need to differentiate this? Or is storing an external Id or something in the DB enough? What is the DB table that stores this? canvasPaintings but not (containedItems like the prop?)

[tomcrane]

containment is solely the existing hierarchical table(s) - manifests and collections - I don't think canvasPaintings comes into this at all?

aaa contains bbb
aaa contains ccc
bbb contains ddd
bbb contains eee
(all db)

aaa hasMember eee
only exists as a fact in JSON, has no db representation at all
(you can actually say what you like)

aaa contains eee
If you can assert this it can only count as a MOVE operation, and bbb no longer contains eee

[donaldgray]

I don't think canvasPaintings comes into this at all?

My comment was badly worded. I was thinking that we might need a new table; this new table being similar to canvasPaintings (that stores things that are in a Manifest - we might need to store things that are in a Collection).

Requirement of it depends on how saving and automatically adding membership is done.

edit: the original reason for comment was whether we need to differentate between these 3 bullets - to do that I think we'd need to store something about the members of a collection?

[donaldgray]

Any concurrency issues here? ETags will need to be eagerly generated for multiple items on save of child - which will be happening already so should be fine. (edit: This is only seting the ETag for DB generated collections but this is possibly reading + writing to stored JSON)

Unsure whether this is the best behaviour - not sure what I'd expect? Is this something we'd eventually want to be a controllable behaviour?

[tomcrane]

to be discussed for sure

[tomcrane]

Sometimes you might want this to happen, sometimes not.
If controllable it needs additional property, or a header, or something (might also be config).

Balance between complexity and doing the most useful thing most of the time (even if you have to undo it occasionally)

POST /some/parent/collection
Headers: "no-append": "true"
{
// a manifest
}

[donaldgray]

I suspect it will be rare for a IIIF Coll in the platform to run into this - do we want to do an MVP staged impl of this? Or will that be too cumbersome?

[tomcrane]

not sure what you mean

[donaldgray]

The distinction is that the latter are DB powered only. Nothing stored on S3, cheap + easy to append children because we're just reading from DB and projecting to an alternative format on the way out.

With this change do we need to store S3 objects for storage and non-storage collections? Or "json is king" for iiif only and append on the way out? If that was the case we'd need to track which ones had since been reordered and not re-add them.

OR could we project them once at first read. So you can add -> add -> add to a parent and we assemble the manifest on first read. If so - how do we track this?

[tomcrane]

Nothing stored on S3

Yeah that is important. atm you can't edit the items property of a Storage Collection - it's simply a different projection of containedItems, entirely DB driven.

So we could maintain the distinction, or we could say that Nothing stored on S3 until it needs to be (which means you don't have to choose what kind of coll it is up front, only its public/private setting.

[donaldgray]

Option to only include one or the other for API requests?

[donaldgray]

How would this work:

• Create Collection A (empty, no "items") - just a label.
• Add 10 Manifests with parent of Collection A.
  • Collection A now has 10 "items" and 10 "containedItems"
• GET Collection A and remove all "items" then POST.
  • Collection A now has 0 "items" and 10 "containedItems"
• Add 1 Manifest with parent of Collection A
  • Collection A now has 1 "items" and 11 "containedItems"? OR * Collection A now has 0 "items" and 11 "containedItems", because it's diverged and you're on your own?

[tomcrane]

Although it sounds odd, I think the former - Collection A now has 1 "items" and 11 "containedItems"

...because this is then consistent with how (I think at least) it would be more commonly used, when they don't diverge much (or at all), or have no containedItems at all.

[tomcrane]

Non-specific comment

If we keep Storage Collection / IIIF Collection distinction:

• A Storage Collection can be public or private
• A Storage Collection's items property is not editable
• ...neither is any other of its JSON except label
• A Storage Collection has no S3 persistence (it doesn't need it)
• A IIIF Collection is always public
• A IIIF Collection can still be a container, it has a containedItems property too, you can PUT and POST children.
• You can edit the items as JSON - but so can we. If you add a child item, and it isn't already present in the items list (in any position), we will append it. If you delete a child item and it is present in items, we will remove it. You can obviously edit it again if you don't like what we have done. I am assuming that you usually will want this behaviour.

Can you convert a Storage Collection into a IIIF Collection?

If we don't keep it:

• A Collection can be public or private
• A Collection's items property is always editable
• A Collection's JSON is generally editable
• ⚠ A Collection has S3 persistence as soon as its JSON is not what would be automatically generated from the DB ⚠
• A Collection is always a container, it has a containedItems property, you can PUT and POST children.
• You can edit the items as JSON - but so can we. If you add a child item, and it isn't already present in the items list (in any position), we will append it. If you delete a child item and it is present in items, we will remove it. You can obviously edit it again if you don't like what we have done. I am assuming that you usually will want this behaviour.