Feature Proposal: SQLite Storage + GE History Import

[Reggie-Reuss]

Summary

I've built two features for the Flipping Utilities plugin and would like to contribute them upstream:

1. SQLite Storage (replaces JSON files)

• Trade data stored in ~/.runelite/flipping/flipping.db instead of per-account JSON files
• Each trade saved instantly via WAL-mode SQLite — crash-proof, no more full-file rewrites
• Automatic one-time migration from JSON on first run (originals renamed to .pre-sqlite, not deleted)
• JSON fallback if SQLite initialization fails

2. GE History Import (up to ~5 months of trade history)

• One-click sync: Reads trade history directly from RuneLite's ConfigManager (requires OSRS in-game login)
• Manual import: File chooser for JSON exported from runelite.net/account/grand-exchange
• Smart deduplication prevents double-importing (matched by exact timestamp + fuzzy matching)
• Item names/limits resolved via OSRS Wiki API with RuneLite ItemManager fallback
• Import button added to Stats tab (upload icon next to existing CSV download button)

Motivation

This started after I lost all my Flipping Utilities trade data from a RuneLite crash that corrupted the JSON files mid-write. I documented the data recovery process here: https://github.com/Reggie-Reuss/Runescape_FU_DataRecovery

The current JSON storage rewrites the entire file on every save. If RuneLite crashes mid-write, the file can be corrupted or lost entirely. SQLite's WAL mode makes writes atomic — it either fully saves or doesn't, never half-saves.

The GE history import lets users recover trade history after data loss, or bootstrap a fresh install with months of existing trade data from runelite.net.

Implementation

• 6 new files in a db/ package (SqliteSchema, SqliteDataStore, JsonToSqliteMigrator, GeHistoryImporter, OsrsItemMapper, RuneliteGeHistoryFetcher)
• Modifications to DataHandler, FlippingPlugin, NewOfferEventPipelineHandler, StatsPanel, Icons
• Added sqlite-jdbc:3.46.1.0 dependency
• Import conversion follows the same conventions as GeHistoryTabExtractor (slot=-1, ticksSinceFirstOffer=10)

Cross-Platform Compatibility

The sqlite-jdbc:3.46.1.0 library bundles native SQLite binaries for all platforms (Windows x86/x64/ARM, macOS x64/ARM, Linux multiple architectures). Native libraries are auto-extracted at runtime with zero configuration. It is fully compatible with Java 8 source compatibility. No cross-platform or OS-specific concerns for the 50k+ userbase on Windows, Mac, and Linux.

Testing

Full testing results with 16 test phases documented here:
https://github.com/Reggie-Reuss/flipping-utilities-testing

• 12 passed, 1 partial pass, 2 blocked (require in-game OSRS login via non-Jagex launcher), 1 pending
• 3 bugs found and fixed during testing (dead SettingsPanel UI, Accountwide import duplication, SQLite JDBC corruption detection)
• 0 new test failures introduced (compilation: 0 errors, 2 pre-existing warnings)

Note: Java is not my primary language, and my testing was limited to what I could cover manually. I've been thorough with what I understand is possible, but this would definitely benefit from review by someone more experienced with the codebase and the RuneLite plugin ecosystem.

Bugs Found & Fixed

Bug	Severity	Fix
Import buttons placed in dead SettingsPanel class (never instantiated)	High	Moved to Stats tab
Import into "Accountwide" view causes data duplication	Medium	Blocked with warning dialog
SQLite JDBC silently replaces corrupted db instead of triggering fallback	Low	Header validation before connection

Questions

• Is this something you'd be interested in merging?
• Any preferences on how the PR should be structured (single PR with 2 commits, or separate PRs)?
• Any concerns about the sqlite-jdbc dependency or the implementation approach?

Happy to adjust the implementation based on feedback. I have a working branch ready to submit as a PR whenever you're ready.

[Reggie-Reuss]

@AntonyGarand saw your closed PR and your comments on SQLite in progress with further optimizations than the above methodology, love it! Let me know if you would be interested in the GE history import functionality for new users and to republish new branch with removed SQLite changes and only import, otherwise feel free to close this issue as well!
All the best,
Reggie Reuss