Skip to content

build_lek_data raises for dashed GSI names when dict-key labels are used #7

@msull

Description

@msull

Summary

build_lek_data in simplesingletable/dynamodb_memory.py can raise RuntimeError("Unsupported index ...") for a query that otherwise works against DynamoDB, when:

  1. The DDB GSI is named with a dash (e.g. gsi-1) — as is common when keeping the index name distinct from the attribute names (gsi1pk / gsi1sk).
  2. The resource's get_gsi_config() uses descriptive dict keys (e.g. "by-owner") rather than the index name as the dict key.

The two conventions look reasonable and read well in code — and writes / first-page queries work correctly, because the dict keys are only ever iterated as .values() for writes, and the index name is passed verbatim to dynamodb.query(IndexName=...). The bug is only reachable on the secondary LEK-construction path that fires when client-side filtering forces SST to synthesize a LastEvaluatedKey (since DDB's own LEK is no longer accurate to the truncated result set).

Reproducer (conceptually)

class MyResource(BaseResource):
    @classmethod
    def get_gsi_config(cls):
        return {
            "by-owner": {  # descriptive label, NOT the index name
                "gsi1pk": lambda self: f"things#{self.owner_id}",
                "gsi1sk": lambda self: self.resource_id,
            },
        }

# Table created in CDK with IndexName="gsi-1" (dashed).
memory.paginated_dynamodb_query(
    key_condition=Key("gsi1pk").eq("things#abc"),
    index_name="gsi-1",                 # passed straight to DDB — correct
    resource_class=MyResource,
    filter_expression=Attr("status").eq("active"),  # filter rejects enough rows
    results_limit=50,
)

If the filter rejects enough rows that SST has to truncate the page itself (line 1549+ in dynamodb_memory.py), it then calls build_lek_data(db_item, "gsi-1", MyResource) which:

  1. Checks if "gsi-1" in gsi_config: — False (dict has "by-owner").
  2. Checks if "gsi-1" in ["gsi1", "gsi2", "gsi3"]: — False (dashed form doesn't match no-dash legacy list).
  3. Falls through to raise RuntimeError("Unsupported index 'gsi-1'").

Why dict keys can't fix this on their own

Setting the dict key to "gsi-1" to match the index name doesn't help either:

"gsi-1": {                                   # matches index_name
    "gsi1pk": lambda self: ...,              # writes still produce gsi1pk
    "gsi1sk": lambda self: ...,
}

build_lek_data then enters the first branch (line 62) and computes:

pk_field = f"{index_name}pk"   # = "gsi-1pk"  ← not a real attribute
if pk_field in db_item:        # False — attribute is "gsi1pk" (no dash)
    lek_data[pk_field] = ...

So no LEK fields get added, and the next query would start from the beginning instead of continuing — silently wrong rather than loud.

The only way to make the current code work with dashed index names is to use "gsi-1" as the dict key AND name the attributes with a dash too ("gsi-1pk" / "gsi-1sk"). That's invasive and breaks the existing convention used in downstream projects.

Suggested fix

In build_lek_data, normalize the index name to derive attribute names regardless of whether the caller (or the dict key) uses dashed or non-dashed form. Something like:

attr_prefix = index_name.replace("-", "")  # "gsi-1" → "gsi1"
if attr_prefix in {"gsi1", "gsi2", "gsi3"}:
    pk_field, sk_field = f"{attr_prefix}pk", f"{attr_prefix}sk"
    if pk_field in db_item:
        lek_data[pk_field] = db_item[pk_field]
    if sk_field in db_item:
        lek_data[sk_field] = db_item[sk_field]
    return lek_data

This makes the helper agnostic to whether the dict-key labels match index_name exactly, which matches the existing flexibility of the writes / queries paths.

Workaround

Until fixed, callers using dashed index names + descriptive labels should avoid relying on SST-generated LEKs from filter-rejecting paginated queries. Queries without FilterExpression are fine — DDB returns the LEK directly (line 1501) and SST passes it through verbatim.

Environment

  • simplesingletable==0.7.x (current at time of report)
  • DynamoDB GSIs named gsi-1 / gsi-2 / gsi-3 with attributes gsi{N}pk / gsi{N}sk

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions