Skip to content

Harden filter validation and prevent delimiter key collisions #239

@shaleenji

Description

@shaleenji

Problem Statement

The filter indexes encode compound keys with : as a delimiter. If user-provided
field names or category values also contain :, distinct logical (field, value) pairs can collide at the byte-key level. That can produce incorrect
filter matches.

Users are also affected by inconsistent validation behavior: malformed JSON,
unsupported field values, schema type mismatches, invalid operators, and invalid
range shapes should be rejected clearly rather than being skipped or producing
undefined filter state.

Proposed Solution

Reject unsafe or invalid filter inputs before reading or writing the filter
indexes:

  • malformed filter JSON;
  • unsupported filter value types;
  • empty field names;
  • schema type mismatches;
  • invalid operators;
  • invalid numeric range shape or order;
  • : in filter field names;
  • : in category values.

These rejections should be caller-fixable validation errors and should propagate
to API callers as HTTP 400 responses.

Alternatives Considered

Switch immediately to a length-prefixed or binary tuple key format. That is more
robust long term, but it is a breaking on-disk format change and needs migration
or rebuild tooling.

Escape : inside user fields and values. That avoids rejecting input, but still
requires a careful compatibility story for existing keys and makes key handling
more complex.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions