Skip to content

fix(mbi): handle orphaned InnoDB tablespace in temp table creation to prevent ETL crash#3165

Open
vuntz wants to merge 3 commits intodevelopfrom
MON-185830-daily-cron-crash
Open

fix(mbi): handle orphaned InnoDB tablespace in temp table creation to prevent ETL crash#3165
vuntz wants to merge 3 commits intodevelopfrom
MON-185830-daily-cron-crash

Conversation

@vuntz
Copy link
Copy Markdown
Member

@vuntz vuntz commented Feb 13, 2026

Description

The daily ETL (centreonBIETL) crashes with errno: 184 "Tablespace already exists" when mod_bi_tmp_* tables have orphaned InnoDB tablespace files (.ibd) left behind after a database migration (e.g. rsync of /var/lib/mysql). Since all DB connections use die => 1, the failed CREATE TABLE immediately crashes the process.

Root cause: DROP TABLE IF EXISTS is a no-op when the table doesn't exist in the data dictionary, but the orphaned .ibd file on disk causes CREATE TABLE to fail.

Changes to Metrics.pm:

  1. New _recreateTempTable helper: Wraps the DROP + CREATE pattern with multi-step error recovery:

    • First attempt: normal DROP TABLE IF EXISTS + CREATE TABLE
    • If CREATE fails (caught with eval): checks if the table structure still exists in DD (e.g., DROP only partially succeeded) and reuses it via TRUNCATE
    • If table not in DD: retries DROP + CREATE on a fresh connection (the error handler auto-disconnects)
    • If all attempts fail: dies with a clear error message explaining the likely cause and manual fix
  2. Updated all three createTempTable* methods to use _recreateTempTable instead of bare DROP + CREATE calls

  3. Fixed dropTempTableCtimeMinMaxValues: Changed DROP TABLE to DROP TABLE IF EXISTS to prevent crashes when the table doesn't exist

Fixes MON-185830

Type of change

  • Patch fixing an issue (non-breaking change)
  • New functionality (non-breaking change)
  • Breaking change (patch or feature) that might cause side effects breaking part of the Software
  • Updating documentation (missing information, typo...)

Target serie

  • 23.10.x
  • 24.04.x
  • 24.10.x
  • 25.10.x
  • master

How this pull request can be tested ?

  1. Have a working MBI setup
  2. Simulate orphaned tablespace by performing an rsync of /var/lib/mysql while MariaDB is running, or manually create .ibd files for mod_bi_tmp_* tables
  3. Run the daily ETL: /usr/share/centreon-bi/etl/centreonBIETL
  4. Verify the ETL no longer crashes with errno: 184 "Tablespace already exists"
  5. Check logs for WARNING-level recovery messages if the orphaned tablespace was detected

… prevent ETL crash

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@vuntz vuntz requested a review from a team as a code owner February 13, 2026 17:17
@vuntz vuntz requested a review from Evan-Adam February 13, 2026 17:17
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Feb 13, 2026

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one) (1)
  • coderabbit

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch MON-185830-daily-cron-crash

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Feb 13, 2026

Logo
Checkmarx One – Scan Summary & Details1e81e54a-87b6-4c7c-af6f-374c0d0ea997

Great job! No new security vulnerabilities introduced in this pull request

Extract the _recreateTempTable recovery logic from Metrics.pm into a
shared TableUtils::recreate_table() function, then use it in all MBI
lib files that create InnoDB temp tables via bare DROP + CREATE. This
prevents the errno 184 "Tablespace already exists" crash across all
ETL code paths, not just the three in Metrics.pm.

MON-185830
# InnoDB tablespace (.ibd file) may exist after a database migration (e.g.
# rsync of /var/lib/mysql). Without this, CREATE TABLE dies with errno 184
# "Tablespace already exists" and the ETL crashes.
sub recreate_table {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This new feature is used in several places; it would be wise to have the associated unit tests, as several different outputs are possible depending on the inputs.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wrote some unit tests, but not really convinced about the value they bring. Can we discuss about this when you have some time?

@Tpo76
Copy link
Copy Markdown
Contributor

Tpo76 commented Apr 15, 2026

🤖 This is an automatic comment generated by an internal tool.

Complexity Classification: Standard

Justification: Automatic classification failed; defaulting to standard.

Metric Value
WIP Points 2
SLA Track Standard SLA (2 work days)
First review SLA 1 work day

A maintainer can override this classification at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants