Skip to content

Fix MySQLRecordManager DataSource connection churn#6570

Open
jianongHe wants to merge 2 commits into
FlowiseAI:mainfrom
jianongHe:fix-mysql-record-manager-datasource
Open

Fix MySQLRecordManager DataSource connection churn#6570
jianongHe wants to merge 2 commits into
FlowiseAI:mainfrom
jianongHe:fix-mysql-record-manager-datasource

Conversation

@jianongHe

Copy link
Copy Markdown

Fixes #6567.

MySQLRecordManager was creating and destroying a TypeORM DataSource for every record-manager operation. Because DataSource owns the connection pool, that caused repeated pool creation during indexing and cleanup flows.

This PR changes MySQLRecordManager to:

  • Lazily initialize and reuse one DataSource per record manager instance.
  • Share the pending initialization promise across concurrent operations.
  • Release QueryRunner instances after each operation.
  • Add close() for explicit DataSource teardown.
  • Close record managers after indexing and document-store vector deletion flows.

Verification:

  • pnpm --dir packages/components exec jest --runInBand
    • 22 test suites passed, 875 tests passed
  • pnpm --dir packages/server exec jest --runInBand
    • 31 test suites passed, 934 tests passed
  • pnpm --filter flowise-components build
  • pnpm --filter "./packages/server" build
  • git diff --check
  • prettier --check on changed files

Note:

  • pnpm test at the monorepo root currently fails in packages/observe due to ExecutionTreeSidebar.test.tsx timing out in the full parallel run.
  • The failing observe test is unrelated to this change and passes when run directly:
    pnpm --dir packages/observe exec jest src/features/executions/components/ExecutionTreeSidebar.test.tsx --runInBand --detectOpenHandles

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors MySQLRecordManager to reuse a single DataSource instance across operations, shares pending initializations across concurrent operations, and ensures query runners are properly released. It also introduces a close method to clean up connections, which is integrated into the indexing and vector store deletion workflows. The review feedback highlights three key improvement opportunities: moving table name sanitization inside the try block in listKeys to prevent query runner leaks, clearing state variables synchronously in close() to avoid concurrent race conditions, and querying the database time directly within update() using the active query runner to prevent connection pool starvation.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines 417 to 419
const tableName = this.sanitizeTableName(this.tableName)

try {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If this.sanitizeTableName throws an error before entering the try block, the queryRunner will be leaked. Moving the table name sanitization inside the try block ensures the query runner is safely released.

        try {
            const tableName = this.sanitizeTableName(this.tableName)

Comment on lines +239 to +245
const dataSource = this.dataSourcePromise ? await this.dataSourcePromise.catch(() => undefined) : this.dataSource
this.dataSourcePromise = undefined
this.dataSource = undefined

if (dataSource?.isInitialized) {
await dataSource.destroy()
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If close() is called concurrently, multiple calls could attempt to await the same dataSourcePromise and subsequently call destroy() on the same DataSource instance, which can lead to errors. Clearing the state variables synchronously before awaiting the promise prevents this race condition.

        const dataSourcePromise = this.dataSourcePromise
        const dataSource = this.dataSource
        this.dataSourcePromise = undefined
        this.dataSource = undefined

        const ds = dataSourcePromise ? await dataSourcePromise.catch(() => undefined) : dataSource
        if (ds?.isInitialized) {
            await ds.destroy()
        }


const updatedAt = await this.getTime()
const { timeAtLeast, groupIds: _groupIds } = updateOptions ?? {}
const updatedAt = await this.getTime()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Calling this.getTime() inside update() checks out a second connection and creates another query runner from the pool, which is inefficient and can lead to pool starvation under load. Since we already have an active queryRunner, we can query the database time directly using it.

Suggested change
const updatedAt = await this.getTime()
const timeRes = await queryRunner.manager.query(`SELECT UNIX_TIMESTAMP(NOW()) AS epoch`)
const updatedAt = Number.parseFloat(timeRes[0].epoch)

@itxaiohanglover

Copy link
Copy Markdown

Solid fix! The mock DataSource and QueryRunner tests are thorough — verifying initialize/destroy counts and connection lifecycle. This addresses the per-operation pool instantiation issue from #6567.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Severe Performance Issue and Potential DoS in MySQLRecordManager due to Per-Operation Connection Pool Instantiation

2 participants