Skip to content

[SER-1312] Improve crash grouping for partially symbolicated traces#7711

Open
coskunaydinoglu wants to merge 6 commits into
masterfrom
feature/SER-1312-improve-crash-grouping
Open

[SER-1312] Improve crash grouping for partially symbolicated traces#7711
coskunaydinoglu wants to merge 6 commits into
masterfrom
feature/SER-1312-improve-crash-grouping

Conversation

@coskunaydinoglu

Copy link
Copy Markdown
Contributor

Jira: https://countly.atlassian.net/browse/SER-1312

Summary

Customers (reported by Vakifbank) saw identical partially symbolicated iOS SIGTRAP crashes split into many separate crash groups. The grouping seed produced by groupStrategy() in plugins/crashes/api/parts/stacktrace.js retained per-report noise — frame offsets (+ 50648), bare hex load addresses without a 0x prefix (104a30000, <0000000104a3c5d8>), and column-alignment whitespace — so semantically identical traces hashed to different groups.

This change, gated behind the existing smart_preprocessing setting (no behavior change when it is disabled):

  • Removes symbol/frame offsets as one unit ( + 600, +0x1a4) before the standalone-number rule
  • Removes bare hex sequences (4+ chars, at least one digit required, so English words like "dead"/"face" survive) after the existing 0x/UUID rules
  • Collapses whitespace runs and trims as the final smart-preprocessing step

Bug fix: custom smart_regexes

The custom-regex loop passed the flags string as the replacement: seed.replace(new RegExp(reg), "gim"). Only the first match was replaced, case-sensitively, with the literal text gim injected into the seed. Now: seed.replace(new RegExp(reg, "gim"), "") — all matches removed, exactly as the setting's help text has always documented.

Release-notes note: one-time regrouping

For apps with smart_preprocessing enabled, grouping seeds (and thus hashes) change. New incoming crashes may open new groups instead of joining pre-upgrade groups. Existing groups remain intact and readable; they simply stop accruing new occurrences for affected signatures. No migration or merge of historical groups is performed.

Test plan

  • Unit tests for seed normalization added to plugins/crashes/tests.js (groupStrategy called directly with explicit override settings): iOS SIGTRAP pair equality, bare-hex normalization, offset-as-unit removal, distinct crashes staying distinct, smart_regexes all-match/case-insensitive removal with no gim injection, invalid-regex resilience, smart_preprocessing: false byte-identical passthrough, stacktrace strategy, standalone-id regression — all 9 verified locally against the real module (output in commit d561055's review)
  • ESLint 8.57.1 with repo config: 0 errors, 0 warnings on both changed files
  • E2E smoke (two partially symbolicated tvOS crashes land in one crash group) + full crashes plugin suite: runs in CI (no test environment in the authoring checkout)
  • ReDoS analysis performed on the three new regexes against adversarial inputs (linear, 0–5ms at 200k chars)

https://claude.ai/code/session_01KV52CRKaxGSbBLugwh7XrJ


Generated by Claude Code

claude added 6 commits June 11, 2026 21:24
…essing

- Remove symbol/frame offsets as one unit before the standalone-number
  rule, so partially symbolicated iOS frames normalize cleanly
- Remove bare hex sequences (4+ chars, digit required) not covered by
  the 0x rule, e.g. load addresses and <...> address tokens
- Fix custom smart_regexes: flags were passed as the replacement string,
  so only the first match was replaced (case-sensitively) with the
  literal text "gim"; now all matches are removed with gim flags as
  the config help text documents
- Collapse whitespace runs and trim as the final smart-preprocessing
  step so column alignment differences no longer change the seed

Crashes that differ only by memory addresses, frame offsets, or padding
now produce identical grouping seeds (SER-1312, reported for partially
symbolicated iOS SIGTRAP crashes).
Unit tests call groupStrategy() directly with explicit override
settings: iOS SIGTRAP pair equality, bare-hex normalization, offset
removal as a unit, distinct crashes staying distinct, smart_regexes
all-match case-insensitive removal without gim injection, invalid
regex resilience, smart_preprocessing=false passthrough, stacktrace
strategy, and standalone-id regression. Plus an e2e smoke test that
two partially symbolicated crashes land in one crash group.
Crash ingestion is fire-and-forget: /i returns 200 before the crash
group upsert lands, so asserting immediately after submission is flaky
on slow runners. Wait 100*testScalingFactor ms after each submission
(suite convention) and poll the crashes query until the group count is
stable before asserting. Also URL-encode the crash payloads so the
literal '+ 50648' frame offset reaches the server instead of decoding
to a space, exercising the offset-removal rule as intended. The strict
lengthOf(1) assertion is unchanged.

coskunaydinoglu commented Jun 12, 2026

Copy link
Copy Markdown
Contributor Author

CI status summary for reviewers:

9/10 checks green on 7d65cf3 — including test-api-core, test-api-plugins (runs the new crash-grouping tests), and ui-tests (dashboard).

ui-tests (onboarding)pre-existing failure, not related to this PR: it also fails on master's own runs (e.g. the June 5 master push run) and on recent unrelated PRs. Worth a separate ticket.

Note on the earlier red runs (21:38–23:00 UTC June 11): the same code failed test-api-core + test-api-plugins three times in that window and passed this morning with no code changes; the API suites appear date-boundary sensitive around local-midnight (UTC+2/+3 fixtures vs UTC server date). Also separately worth a flakiness ticket.

The two [SER-1312] Re-trigger CI commits are empty (CI re-runs only) — recommend squash-merge.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants