feat(import): import JSON files into a table#1536
Conversation
…d, and a type pop-up
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 08bd721326
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if settings.deleteExistingRows { | ||
| try await sink.deleteAllRowsFromTargetTable() | ||
| } | ||
| if useTransaction { | ||
| try await sink.beginTransaction() |
There was a problem hiding this comment.
When deleteExistingRows and the default transactional rollback mode are enabled, the target table is cleared before beginTransaction() runs, so a later parse/insert failure rolls back only the inserted rows and leaves the pre-existing data permanently deleted. Move the delete inside the same transaction as the import so rollback restores the table contents.
Useful? React with 👍 / 👎.
| switch settings.errorHandling { | ||
| case .stopAndRollback, .stopAndCommit: | ||
| throw PluginImportError.statementFailed(statement: "row \(line)", line: line, underlyingError: error) |
There was a problem hiding this comment.
With Stop and Commit selected and transactions enabled, this branch throws the same error as stopAndRollback, and the outer catch unconditionally calls rollbackTransaction(). That discards all rows that were successfully inserted before the failing JSON row, which is the opposite of the selected stop-and-commit behavior.
Useful? React with 👍 / 👎.
| self.columnMapping = Dictionary( | ||
| columnMapping.map { ($0.key.lowercased(), $0.value) }, | ||
| uniquingKeysWith: { _, last in last } | ||
| ) |
There was a problem hiding this comment.
JSON object keys are case-sensitive, but lowercasing the source field names collapses mappings such as id -> lower_id and ID -> upper_id into one entry. For files with fields that differ only by case, imports can write values to the wrong destination column or generate duplicate target columns instead of honoring the user's mapping.
Useful? React with 👍 / 👎.
# Conflicts: # CHANGELOG.md
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: beb90c22c6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| case let number as NSNumber: | ||
| if CFGetTypeID(number) == CFBooleanGetTypeID() { | ||
| return .text(number.boolValue ? "true" : "false") | ||
| } |
There was a problem hiding this comment.
When importing JSON into a newly created MySQL/MariaDB table, inferType maps all-boolean fields to TINYINT(1), but this conversion stores JSON true as the string parameter "true". The MySQL driver binds .text values as MYSQL_TYPE_STRING, and MySQL coerces the string 'true' to numeric 0, so every true value imported into the inferred boolean column is stored as false. This also affects existing MySQL boolean/tinyint targets; booleans need a dialect-aware representation before binding.
Useful? React with 👍 / 👎.
…esktop framework (#1536)
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 18de96f8ee
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| static func extractRows(from object: Any, targetTable: String?) throws -> [[String: Any]] { | ||
| if let array = object as? [Any] { | ||
| return array.compactMap { $0 as? [String: Any] } |
There was a problem hiding this comment.
When a regular .json file is a top-level array, any element that is not a JSON object is silently removed by compactMap, so inputs like [{"id":1}, null, {"id":2}] complete successfully and report only two imported rows. Since the importer advertises an array of objects, default stop-on-error imports should fail rather than hide malformed rows and make the import look complete.
Useful? React with 👍 / 👎.
| var lineNumber = 0 | ||
| for try await line in url.lines { |
There was a problem hiding this comment.
With Skip and Continue selected for .jsonl/.ndjson, parse errors from an individual line still escape before the row-level error handler is reached, so one malformed JSON line aborts the rest of the file even though later lines could be imported. This makes the continue mode only handle database insert failures, not bad input rows in the line-delimited format.
Useful? React with 👍 / 👎.
1 participant
Adds JSON file import. Today TablePro can export to JSON and import SQL files, but there's no way to import JSON data back in. This closes that gap with a dedicated, native import sheet.
What it does
File ▸ Import → pick a
.json/.jsonl/.ndjsonfile opens a dedicated JSON import sheet (the SQL dialog is unchanged and stays SQL-only). The sheet accepts:[{...}, {...}],{ "table": [ {...} ] }, so an export round-trips back in.Two destinations (segmented control):
generateCreateTableSQL, then imports.Options (on-error, wrap-in-transaction, delete-existing-rows) reuse the plugin's settings. Select-all toggles and validation (unique column names, one field per column) gate the Import button.
Architecture
The import contract was SQL-statement-only (
source.statements()thensink.execute(statement:)). JSON is the first row-based importer, so the contract is extended additively (default-implemented, so no PluginKit ABI bump; stays at 17):PluginImportDataSink:insertRow(_:),targetTable,deleteAllRowsFromTargetTable().ImportFormatPlugin:requiresTargetTable,detectSourceFields(at:targetTable:)(name, sample, inferred type).The plugin parses JSON and yields rows; the app owns the target table and generates parameterized inserts via
SQLStatementGenerator+driver.executeParameterized(zero string concatenation, so JSON values can't inject SQL). Bool-vs-int usesCFBooleanGetTypeID; nested objects and arrays serialize to JSON text. NewJSONImport.tablepluginbundled target, never published to the registry.UI
Native SwiftUI:
Grid/GridRowfor both grids (auto-aligned columns), left-aligned destination controls, and a type pop-up sourced fromcolumnTypesByCategory(for:)(same list the structure editor uses).GridoverTabledeliberately, since editableTextFieldcells inTableare unreliable on macOS.Tests
JSONImportPluginTests: array/NDJSON/wrapper parsing, value conversion (null/bool/number/nested), type inference, export-shape round-trip.SQLStatementGeneratorImportTests: parameterized insert + per-dialect quoting, injection-as-data, null binds.JSONImportTypeMapperTests: inferred-type to SQL type per dialect, with a generic fallback.Known v1 limits
.jsonarray parses the whole file (NDJSON streams).Docs and CHANGELOG updated.