BE-644: Run subgraph reads in a single READ ONLY, REPEATABLE READ transaction#8996
BE-644: Run subgraph reads in a single READ ONLY, REPEATABLE READ transaction#8996claude[bot] wants to merge 13 commits into
Conversation
…ansaction builder
Context::transaction() is now a synchronous method returning a
TransactionBuilder which begins the transaction when awaited, so call
sites can configure the transaction before starting it:
context.transaction().isolation_level(IsolationLevel::RepeatableRead).read_only().await
The Transaction associated type moves from Context to the new
TransactionBuilder trait (bounded by IntoFuture); the ContextTransaction
alias keeps Migration signatures readable. The Postgres implementation
delegates to tokio-postgres' Client::build_transaction() so all options
compose into the single BEGIN statement, e.g.
`START TRANSACTION ISOLATION LEVEL REPEATABLE READ, READ ONLY`.
Groundwork for BE-644.
PostgresStore::transaction() is now synchronous and returns a PostgresStoreTransactionBuilder which begins the transaction when awaited; existing `store.transaction().await` call sites are unchanged. The builder implements the hash-graph-migrations TransactionBuilder trait (and PostgresStore implements Context, PostgresStore<Transaction> implements Transaction), so isolation level, read-only, and deferrable options can be configured before starting. For a Client-backed store the options are delegated through the new AsClient::begin_transaction to tokio-postgres' build_transaction(), so they compose into the single BEGIN statement. Beginning a transaction on an already-running transaction creates a savepoint; savepoints inherit the enclosing transaction's characteristics, so the options are ignored there (used by the integration-test harness). hash-graph-migrations is promoted from a dev-dependency to a public dependency of hash-graph-postgres-store; dependency diagrams updated accordingly. Part of BE-644.
…saction (BE-644) The subgraph read executes multiple statements: the roots query, one recursive CTE per traversal path, the permission filtering of the traversed edges, and the final vertex read. Under READ COMMITTED each statement gets its own MVCC snapshot, so a concurrent write (e.g. patch_entity on a link's endpoint) committing mid-read retroactively evicts the endpoint's edition at the reader's pinned timestamp: the subgraph then contains the link vertex but drops its has-left-entity/has-right-entity edge, which crashed the frontend (FE-1172). query_entity_subgraph now begins a READ ONLY, REPEATABLE READ transaction (a single BEGIN via the new transaction builder) and runs every statement of the read on that transaction's client, giving the whole read one shared snapshot. EntityStore::query_entity_subgraph takes &mut self so the implementation can begin the transaction; callers (REST handler, type fetcher, tests, benches) are adjusted. DB-backed tests assert the composed BEGIN options take effect from the first statement and that the savepoint fallback inherits the enclosing transaction's characteristics.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #8996 +/- ##
==========================================
- Coverage 59.74% 59.70% -0.05%
==========================================
Files 1370 1370
Lines 135170 135264 +94
Branches 6067 6067
==========================================
Hits 80760 80760
- Misses 53477 53571 +94
Partials 933 933
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Merging this PR will degrade performance by 15.38%
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing |
The RefCell wrappers introduced for the `&mut self` change on `EntityStore::query_entity_subgraph` trip `await_holding_refcell_ref` (and `future_not_send` on `run_benchmark`). Criterion drives one benchmark future at a time to completion on a single thread via `block_on`, so the borrows are never contended and the futures never cross threads; mark the lints as expected with that reasoning.
TimDiekmann
left a comment
There was a problem hiding this comment.
Looks good so far, a few comments.
PR SummaryMedium Risk Overview
Dependency diagrams and lockfile reflect the new postgres-store ↔ migrations edge. Reviewed by Cursor Bugbot for commit 000a7c8. Bugbot is set up for automated code reviews on this repo. Configure here. |
There was a problem hiding this comment.
Pull request overview
This PR fixes a snapshot-consistency anomaly (BE-644) where a subgraph read (query_entity_subgraph) executed as several separate autocommit statements under READ COMMITTED, each with its own MVCC snapshot. A concurrent write committing mid-read could retroactively evict an edge endpoint's edition at the reader's pinned timestamp, producing a subgraph containing a link vertex without its has-left-entity/has-right-entity edge (the root cause of the FE-1172 frontend crash loop). The fix runs the entire subgraph read inside a single READ ONLY, REPEATABLE READ transaction so all statements share one snapshot.
Changes:
- Reworked
hash-graph-migrationsContext::transaction()into a synchronous builder (TransactionBuildertrait withisolation_level/read_only/deferrable) that implementsIntoFuture, plus a newIsolationLevelenum andContextTransactionalias; existingcontext.transaction().awaitcall sites keep compiling. - Added
AsClient::begin_transactionandPostgresStoreTransactionBuilderinhash-graph-postgres-store, implementedContext/Transaction/TransactionBuilder, and splitquery_entity_subgraphinto a transaction-wrapping method plusquery_entity_subgraph_impl; the trait method now takes&mut self, cascading to the REST handler, type fetcher, tests, and benches. - Promoted
hash-graph-migrationsto a public dependency, added DB-backed transaction tests, and regenerated dependency wiring/diagrams.
Reviewed changes
Copilot reviewed 43 out of 43 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
libs/@local/graph/migrations/src/context.rs |
Adds IsolationLevel, TransactionBuilder, ContextTransaction; converts Context::transaction to a synchronous builder-returning method. |
libs/@local/graph/migrations/src/postgres.rs |
Implements PostgresTransactionBuilder delegating to Client::build_transaction; maps IsolationLevel to tokio-postgres. |
libs/@local/graph/migrations/src/{lib,migration}.rs |
Re-exports new items; migration signatures use ContextTransaction. |
libs/@local/graph/migrations/graph-migrations/v001..v010/mod.rs |
Mechanical switch to ContextTransaction<'_, Self::Context>. |
libs/@local/graph/postgres-store/src/store/postgres/pool.rs |
Adds TransactionOptions and AsClient::begin_transaction (composed BEGIN for clients, savepoint fallback for transactions). |
libs/@local/graph/postgres-store/src/store/postgres/mod.rs |
Adds PostgresStoreTransactionBuilder and Context/Transaction/TransactionBuilder impls; re-exports migration types. |
libs/@local/graph/postgres-store/src/store/postgres/knowledge/entity/mod.rs |
Extracts query_entity_subgraph_impl; wraps the read in a READ ONLY, REPEATABLE READ transaction. |
libs/@local/graph/postgres-store/src/store/mod.rs |
Re-exports the new transaction types. |
libs/@local/graph/postgres-store/{Cargo.toml,package.json,docs/dependency-diagram.mmd} |
Promotes hash-graph-migrations to a public dependency. |
libs/@local/graph/store/src/entity/store.rs |
EntityStore::query_entity_subgraph now &mut self, with snapshot-consistency doc. |
libs/@local/graph/type-fetcher/src/store.rs |
Delegating impl updated to &mut self. |
libs/@local/graph/api/src/rest/entity/query/mod.rs |
Acquires mut store for the &mut self call. |
tests/graph/integration/postgres/{lib,sorting}.rs, transaction.rs |
Test harness/tests updated for &mut; new tests assert composed BEGIN options and savepoint inheritance. |
tests/graph/benches/** |
Wrap stores in RefCell to supply the mutable borrow inside benchmark closures. |
Various **/docs/dependency-diagram.mmd |
Regenerated dependency graphs. |
…BE-644) An entity query executes multiple statements: the entity read itself, the optional entity-type resolution, and the permission checks on the returned entities. Under READ COMMITTED each statement gets its own MVCC snapshot, so a write committing mid-read can yield entities, entity types, and permissions reflecting different states of the store. query_entities now begins a READ ONLY, REPEATABLE READ transaction via the transaction builder and runs the whole read on that transaction's client, mirroring the query_entity_subgraph structure: the previous trait-method body moves to an inherent query_entities_impl on the store, and the shared low-level read helper (previously query_entities_impl) is renamed to read_entities_impl. EntityStore::query_entities takes &mut self so the implementation can begin the transaction; search_entities inherits the snapshot through query_entities and takes &mut self as well. Callers (REST handlers, type fetcher, tests, benches) are adjusted with the same patterns as for query_entity_subgraph.
PostgresStore had inherent transaction(), commit(), and rollback() methods that only delegated to (or mirrored) the hash-graph-migrations Context and Transaction trait implementations, each carrying an #[expect(clippy::same_name_method)]. The trait implementations are now the single canonical API: the Context impl constructs the transaction builder directly, and the inherent commit/rollback wrappers are gone. Call sites import the traits; Context and Transaction are re-exported from hash_graph_postgres_store::store alongside TransactionBuilder for external users.
Beginning a transaction with TransactionOptions on a store that was already inside a transaction silently degraded to a savepoint and dropped the options. Prevent this at compile time with a typestate: PostgresStore<C, S: TransactionState = NoTransaction> carries a sealed, zero-sized marker (NoTransaction or InTransaction). The options-bearing transaction builder (Context::transaction) only exists in the NoTransaction state and yields the store in the InTransaction state; nesting is only possible through the options-free begin_transaction(), which issues a plain BEGIN at the root and a savepoint inside a transaction, so there is nothing left to silently ignore. AsClient::begin_transaction and its savepoint fallback are removed entirely; AsClient is back to pure client access and store functions remain callable from both states. The REPEATABLE READ, READ ONLY scope used by query_entities and query_entity_subgraph moves into the sealed BeginReadOnlyTransaction trait, which is dispatched per state: a configured top-level transaction in the NoTransaction state, a savepoint within the enclosing transaction in the InTransaction state. The snapshot-restore write batches and the test harnesses now name the InTransaction state explicitly.
query_entities_impl and query_entity_subgraph_impl only ever run on the store returned by BeginReadOnlyTransaction::begin_read_only_transaction, but lived in the state-generic impl block and carried a BeginReadOnlyTransaction bound they never used. Move them into a PostgresStore<C, InTransaction> impl block so the typestate itself states they execute inside the read transaction. Also deduplicate the BEGIN option composition: the NoTransaction BeginReadOnlyTransaction impl now delegates to the canonical Context::transaction() builder instead of hand-rolling build_transaction(), leaving a single code path which compiles TransactionOptions into the BEGIN statement.
In production every snapshot-consistent read begins its own REPEATABLE READ, READ ONLY transaction: the API only ever calls query_entities / query_entity_subgraph / search_entities on NoTransaction stores. The InTransaction impl of BeginReadOnlyTransaction — a savepoint that inherits the enclosing transaction's isolation, i.e. no single-snapshot guarantee — exists solely for the rollback-isolation test harnesses, which wrap each test in a transaction. Hide it behind a new test-utils cargo feature so the production surface cannot silently take the savepoint path; the harness crates (the graph integration tests, the crate's own suites via a self dev-dependency, and the benches' seeding helpers) opt back in. TODO(BE-688) on the trait tracks removing the impl once the harness uses per-test databases. reindex_entity_cache and has_permission_for_entities move to an inherent impl on the state-generic PostgresStore<C, S> (the EntityStore methods delegate): snapshot restore, entity-type reindexing, and the read _impl helpers call them on InTransaction stores and must keep compiling without the gated impl, since the blanket EntityStore impl is bounded on BeginReadOnlyTransaction as a whole.
- Format the tests/graph manifests with taplo (the inline comments split the key alignment groups) - Ignore the self dev-dependency in sync-turborepo metadata so the generated package.json keeps a Turborepo-valid dependency graph - Add the resulting self-loop dev edge to the dependency diagrams that include hash-graph-postgres-store
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit dc931b0. Configure here.
The inherent `reindex_entity_cache` and `has_permission_for_entities` shared their names with the `EntityStore` methods that delegate to them, relying on inherent-method precedence (guarded only by `#[expect(clippy::same_name_method)]`). Renaming or removing the inherent method would silently turn each delegation into infinite recursion. Follow the file's existing `*_impl` naming (`query_entities_impl`, `read_entities_impl`, ...) so the delegations are unambiguous, and drop the now-unneeded lint expectations.
Benchmark results
|
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 2002 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 1002 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 3314 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 1527 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 2078 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 1033 | Flame Graph |
policy_resolution_medium
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 102 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 52 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 269 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 108 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 133 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 63 | Flame Graph |
policy_resolution_none
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 2 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 2 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 8 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 3 | Flame Graph |
policy_resolution_small
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 52 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 26 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 94 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 27 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 66 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 29 | Flame Graph |
read_scaling_complete
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id;one_depth | 1 entities | Flame Graph | |
| entity_by_id;one_depth | 10 entities | Flame Graph | |
| entity_by_id;one_depth | 25 entities | Flame Graph | |
| entity_by_id;one_depth | 5 entities | Flame Graph | |
| entity_by_id;one_depth | 50 entities | Flame Graph | |
| entity_by_id;two_depth | 1 entities | Flame Graph | |
| entity_by_id;two_depth | 10 entities | Flame Graph | |
| entity_by_id;two_depth | 25 entities | Flame Graph | |
| entity_by_id;two_depth | 5 entities | Flame Graph | |
| entity_by_id;two_depth | 50 entities | Flame Graph | |
| entity_by_id;zero_depth | 1 entities | Flame Graph | |
| entity_by_id;zero_depth | 10 entities | Flame Graph | |
| entity_by_id;zero_depth | 25 entities | Flame Graph | |
| entity_by_id;zero_depth | 5 entities | Flame Graph | |
| entity_by_id;zero_depth | 50 entities | Flame Graph |
read_scaling_linkless
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id | 1 entities | Flame Graph | |
| entity_by_id | 10 entities | Flame Graph | |
| entity_by_id | 100 entities | Flame Graph | |
| entity_by_id | 1000 entities | Flame Graph | |
| entity_by_id | 10000 entities | Flame Graph |
representative_read_entity
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/block/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/book/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/building/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/organization/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/page/v/2
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/person/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/playlist/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/song/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/uk-address/v/1
|
Flame Graph |
representative_read_entity_type
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| get_entity_type_by_id | Account ID: bf5a9ef5-dc3b-43cf-a291-6210c0321eba
|
Flame Graph |
representative_read_multiple_entities
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_property | traversal_paths=0 | 0 | |
| entity_by_property | traversal_paths=255 | 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=0 | 0 | |
| link_by_source_by_property | traversal_paths=255 | 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true |
scenarios
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| full_test | query-limited | Flame Graph | |
| full_test | query-unlimited | Flame Graph | |
| linked_queries | query-limited | Flame Graph | |
| linked_queries | query-unlimited | Flame Graph |

Requested by Tim Diekmann · Slack thread
🌟 What is the purpose of this PR?
Before: a subgraph read (
query_entity_subgraph) executes as several separate autocommit statements underREAD COMMITTED— the roots query, one recursive CTE per traversal path, the permission filtering of the traversed edges, and the final vertex read — each with its own MVCC snapshot. A concurrent write committing between those statements (e.g.patch_entityon a link's endpoint) retroactively evicts the endpoint's edition at the reader's pinned timestamp, so the returned subgraph contains the link vertex but silently drops itshas-left-entity/has-right-entityedge. This crashed the frontend (FE-1172, flaky Playwright run with a captured payload showing a membership link with noHAS_LEFT_ENTITYedge while anupdateEntitywas in flight).After: the whole subgraph read runs inside one
READ ONLY, REPEATABLE READtransaction, so every statement of the read shares a single MVCC snapshot and the anomaly cannot occur.🔗 Related links
🔍 What does this change?
hash-graph-migrations:Context::transaction()is now a synchronous method returning a builder that implementsIntoFuture. TheTransactionassociated type moves fromContextto the newTransactionBuildertrait (isolation_level,read_only,deferrable), and theContextTransactionalias keepsMigrationsignatures readable. Existingcontext.transaction().awaitcall sites keep compiling.Client::build_transaction(), so the options compose into the singleBEGINstatement (START TRANSACTION ISOLATION LEVEL REPEATABLE READ, READ ONLY—START TRANSACTIONis Postgres' spelling ofBEGINwith options).hash-graph-postgres-store:PostgresStore::transaction()returns aPostgresStoreTransactionBuilder;PostgresStoreimplementsContext, the builder implementsTransactionBuilder, andPostgresStore<tokio_postgres::Transaction>implementsTransaction. Options are routed through a newAsClient::begin_transaction; beginning a "transaction" on an already-running transaction creates a savepoint and ignores the options (savepoints inherit the enclosing transaction's characteristics — used by the integration-test harness).query_entity_subgraphbegins aREAD ONLY, REPEATABLE READtransaction and runs the roots query, traversal CTEs, permission filters, and vertex reads on that transaction's client.EntityStore::query_entity_subgraphtakes&mut selfso the implementation can begin the transaction; the REST handler, type fetcher, tests, and benches are adjusted.hash-graph-migrationsis promoted from a dev-dependency to a public dependency ofhash-graph-postgres-store(dependency diagrams andpackage.jsonupdated).BEGINoptions take effect from the first statement (transaction_isolation = 'repeatable read',transaction_read_only = 'on') and that the savepoint fallback inherits the enclosing transaction's characteristics.Pre-Merge Checklist 🚀
🚢 Has this modified a publishable library?
This PR:
📜 Does this require a change to the docs?
The changes in this PR:
🕸️ Does this require a change to the Turbo Graph?
The changes in this PR:
patch_entityvs. loopingqueryEntitySubgraph) is described in BE-644 and intentionally not added here: the integration-test harness runs each test inside a single wrapping transaction on one connection, which cannot express multi-connection concurrency.🐾 Next steps
🛡 What tests cover this?
tests/graph/integration/postgres/transaction.rs— asserts the builder composesREPEATABLE READ, READ ONLYinto theBEGINstatement, the default builder keeps session characteristics, and the savepoint fallback ignores options.query_entity_subgraphthrough the new path (98 passed locally; 3 pre-existingclusteringfailures are environmental — sandbox pgvector < 0.7 lackssubvector()).hash-graph-postgres-storeunit tests (142 passed).❓ How to test this?
cargo test -p hash-graph-integration --test postgresentity-editing/ "user can update values on property table") in a loop, or follow the repro sketch in BE-644.