From 7f78019099e5cade252781b4d5f3fb666905cb7b Mon Sep 17 00:00:00 2001 From: Arcadiy Ivanov Date: Sat, 13 Jun 2026 13:12:23 -0400 Subject: [PATCH 1/4] MDEV-38975: HEAP engine BLOB/TEXT/JSON/GEOMETRY support with indexable blob columns Remove the HA_NO_BLOBS restriction from the HEAP engine, allowing the optimizer to keep temporary tables with BLOB/TEXT columns in memory when they fit within max_heap_table_size / tmp_memory_table_size limits. Additionally, advertise HA_CAN_GEOMETRY so explicit CREATE TABLE ... ENGINE=MEMORY with GEOMETRY columns works. Unlike other HEAP blob implementations (e.g. Percona), this patch provides full HASH index support on blob columns, enabling efficient lookups, GROUP BY, and DISTINCT operations directly in HEAP without falling back to disk. Architecture ------------ BLOB data is stored using continuation records -- additional fixed-size records allocated from the same HP_BLOCK that holds regular rows. This reuses existing allocation, free list, and size accounting with minimal structural change, and avoids per-blob my_malloc() calls. The existing single-byte visibility flag is extended into a flags byte with bits for HP_ROW_HAS_CONT, HP_ROW_IS_CONT, HP_ROW_CONT_ZEROCOPY, HP_ROW_SINGLE_REC, and HP_ROW_MULTIPLE_REC. Continuation records are grouped into variable-length runs -- contiguous sequences within a leaf block. Only the first record of each run carries a 10-byte header (next_cont pointer + run_rec_count); inner records are pure payload. Three storage formats, detected by flag bits via inline predicates: Case A (HP_ROW_SINGLE_REC): single record, no header, data at offset 0. Zero-copy read. Case B (HP_ROW_CONT_ZEROCOPY): single run, multiple records. Header in rec 0, data contiguous in rec 1..N-1. Zero-copy read via chain + recbuffer. Case C (HP_ROW_MULTIPLE_REC): one or more runs linked via next_cont. Reassembly into blob_buff required. Run allocation uses a two-phase strategy: (1) peek-then-unlink walk of the free list detecting contiguous groups, (2) tail allocation from HP_BLOCK for remaining data. A Step 3 scavenge fallback walks the entire free list when tail allocation fails. HP_SHARE::total_records tracks all physical records (primary + continuation), while HP_SHARE::records remains the logical count used by hash bucket mapping. Reassembly buffer (HP_INFO::blob_buff) follows the same pattern as InnoDB's blob_heap -- allocated once, grown via my_realloc, freed on heap_reset()/close. Zero-copy cases (A/B) return pointers directly into HP_BLOCK with no copy. Full HASH index key handling for BLOB columns: hp_rec_hashnr(), hp_rec_key_cmp(), hp_key_cmp(), hp_make_key(), hp_hashnr() are extended for HA_BLOB_PART segments. Hash pre-check optimization skips expensive blob materialization when hashes differ. PAD SPACE collation semantics are preserved for blob key comparisons. Field_blob_key (Monty) produces HEAP-native key format (4-byte length + 8-byte data pointer) directly, eliminating key buffer translation between the SQL layer and HEAP engine. SQL layer changes ----------------- pick_engine() (new, extracted from choose_engine()): replaced the blob_fields check with a reclength > HA_MAX_REC_LENGTH guard. choose_engine() calls pick_engine() with the real reclength. pick_engine() is also called early in finalize() with reclength=0 to predict whether the engine will be HEAP, enabling blob-aware GROUP BY key setup that avoids unnecessary m_using_unique_constraint. finalize(): HEAP+blob uses fixed-width rows; GROUP BY key setup sets key_part_flag from field, uses item max_length for blob key sizing. store_length initialized for all GROUP BY key parts. key_type uses field->binary() to determine FIELDFLAG_BINARY vs text collation. DISTINCT key setup skips null-bits helper for HEAP. remove_duplicates(): blob check moved before HEAP check to fall through to remove_dup_with_compare(). Aggregator_distinct::add(): overflow-to-disk conversion via create_internal_tmp_table_from_heap() for non-dup write errors. Expression cache disabled for HEAP+blob (key format incompatibility). FULLTEXT early detection in mysql_derived_prepare(): forces disk engine via TMP_TABLE_FORCE_MYISAM when outer query uses MATCH. Deferred blob chain free (MDEV-39732): heap_delete() saves chain pointers to pending_blob_chains, flushed on next mutation or heap_reset()/close. Prevents dangling zero-copy pointers during binlog_log_row(). REPLACE safety (MDEV-39825): HP_SHARE::write_can_replace flag forces copy mode in hp_read_blobs(), preventing blob data corruption from freed-then-reused continuation records during REPLACE. Geometry GROUP_CONCAT fix (MDEV-39761): downgrade Field_geom to Field_blob for GROUP_CONCAT temp tables in both expression creation paths. Type_handler_geometry::type_handler_for_tmp_table() added. Geometry GROUP BY key fix (MDEV-39871): detect when new_key_field() produced non-blob Field_varstring for a blob column, replace with Field_blob_key. Performance ----------- Non-blob tables: zero regression. Every blob-specific code path is guarded by if (share->blob_count). No new allocations, no format changes, no hash function changes for non-blob keys. Blob tables: eliminates file creation/deletion overhead and page cache management. For single-run blobs (common case), the read path is entirely zero-copy. Limitations ----------- 1. No BTREE indexes on blob columns (HASH only) 2. No partial-key prefix indexing for blobs 3. 2x memory for Case C reads only (A/B are 1x) 4. No blob compression 5. 65,535 records per run (uint16 cap, auto-split) 6. max_heap_table_size applies to continuation records 7. Expression cache disabled for HEAP+blob 8. FULLTEXT forces disk engine Linked bugs fixed: - MDEV-39703: mroonga fulltext test ordering - MDEV-39723: ER_DUP_ENTRY on GROUP BY with blob column - MDEV-39724: crash in hp_is_single_rec with GROUP BY - MDEV-39732: slave crash in hp_free_run_chain on blob replication - MDEV-39761: Field_geom::store() assertion in GROUP_CONCAT - MDEV-39782: RBR ER_KEY_NOT_FOUND on HEAP blob UPDATE - MDEV-39825: blob data corruption on REPLACE into HEAP table - MDEV-39871: crash in my_hash_sort_bin on GROUP BY with geometry Reviewed by: Michael Widenius Monty reviewed the entire patch. Areas where he suggested changes or contributed code: - Field_blob_key class (HEAP-native blob key format, 4-byte length + data pointer) - Duplicate key fix on HEAP-to-Aria conversion - hp_blob_key_length() uint32 fix - hp_rec_hashnr_stored removal - type_handler_for_tmp_table() param cleanup - Type_handler_geometry::type_handler_for_tmp_table() virtual - blob pointer bzero() - find_unique_row() double-materialization fix - Tail reclaim review - Batch tail allocation review - hp_update.c cleanup - Field_blob_compressed temp table fix - row_pack_length() dedup - pack_length_no_ptr() removal - Race condition fix in HEAP - MDEV-39703 mroonga test fix - MDEV-39825 write_can_replace optimization - is_text_key_segment removal (field->binary() simplification) - Documentation (Docs/internal-temporary-tables.txt) Contribution by: Alexander Barkov Type_handler::make_and_init_table_field_ex() -- refactored temp table field creation from inline code in sql_select.cc into type handler virtual methods (sql_type.cc, sql_type_geom.cc), enabling clean per-type-handler field creation for HEAP blob promotion. --- Docs/internal-temporary-tables.txt | 60 + include/heap.h | 49 +- include/my_base.h | 23 +- include/my_compare.h | 4 +- mysql-test/include/mtr_check.sql | 2 +- mysql-test/main/blob_sj_test.result | 31 + mysql-test/main/blob_sj_test.test | 34 + mysql-test/main/column_compression.result | 8 + mysql-test/main/column_compression.test | 1 + mysql-test/main/count_distinct.result | 20 +- mysql-test/main/count_distinct.test | 14 +- mysql-test/main/create.result | 9 +- mysql-test/main/create.test | 3 +- mysql-test/main/cte_recursive.test | 2 + mysql-test/main/distinct.result | 2 +- mysql-test/main/distinct.test | 3 + mysql-test/main/func_group.result | 2 +- mysql-test/main/group_by.result | 4 +- mysql-test/main/group_by.test | 4 +- mysql-test/main/group_min_max_innodb.result | 8 +- mysql-test/main/group_min_max_innodb.test | 1 + mysql-test/main/information_schema.result | 2 +- .../main/information_schema_parameters.result | 2 +- .../main/information_schema_part.result | 2 +- mysql-test/main/information_schema_part.test | 2 +- .../main/information_schema_routines.result | 2 +- mysql-test/main/intersect_all.result | 6 +- mysql-test/main/intersect_all.test | 3 + mysql-test/main/max_session_mem_used.test | 1 - mysql-test/main/metadata.result | 14 +- mysql-test/main/mysql_client_test.result | 2 +- mysql-test/main/select.result | 22 +- mysql-test/main/select.test | 3 +- mysql-test/main/select_jcl6.result | 22 +- mysql-test/main/select_pkeycache.result | 22 +- mysql-test/main/status.result | 10 +- mysql-test/main/temp_table_symlink.result | 3 +- mysql-test/main/temp_table_symlink.test | 6 +- mysql-test/main/tmp_table_error.result | 3 + mysql-test/main/tmp_table_error.test | 3 + mysql-test/main/type_bit.result | 4 +- mysql-test/main/type_bit.test | 2 + mysql-test/main/type_enum.result | 2 +- mysql-test/main/type_set.result | 2 +- mysql-test/suite/funcs_1/r/is_columns.result | 2 +- mysql-test/suite/funcs_1/r/is_events.result | 2 +- mysql-test/suite/funcs_1/r/is_routines.result | 2 +- .../funcs_1/r/is_routines_embedded.result | 8 +- .../suite/funcs_1/r/is_tables_is.result | 104 +- .../funcs_1/r/is_tables_is_embedded.result | 104 +- mysql-test/suite/funcs_1/r/is_triggers.result | 2 +- .../funcs_1/r/is_triggers_embedded.result | 2 +- mysql-test/suite/funcs_1/r/is_views.result | 2 +- .../suite/funcs_1/r/is_views_embedded.result | 2 +- .../funcs_1/r/processlist_priv_no_prot.result | 4 +- .../funcs_1/r/processlist_priv_ps.result | 4 +- .../funcs_1/r/processlist_val_no_prot.result | 2 +- .../suite/funcs_1/r/processlist_val_ps.result | 2 +- mysql-test/suite/heap/blob.result | 812 ++++++++++++ mysql-test/suite/heap/blob.test | 592 +++++++++ mysql-test/suite/heap/blob_big.inc | 235 ++++ mysql-test/suite/heap/blob_big1.result | 253 ++++ mysql-test/suite/heap/blob_big1.test | 8 + mysql-test/suite/heap/blob_big2.result | 253 ++++ mysql-test/suite/heap/blob_big2.test | 9 + mysql-test/suite/heap/blob_big3.result | 253 ++++ mysql-test/suite/heap/blob_big3.test | 10 + mysql-test/suite/heap/blob_dedup.result | 19 + mysql-test/suite/heap/blob_dedup.test | 20 + mysql-test/suite/heap/blob_fallback.result | 57 + mysql-test/suite/heap/blob_fallback.test | 85 ++ mysql-test/suite/heap/blob_find_unique.result | 178 +++ mysql-test/suite/heap/blob_find_unique.test | 202 +++ mysql-test/suite/heap/blob_group_by.result | 62 + mysql-test/suite/heap/blob_group_by.test | 61 + .../suite/heap/blob_group_concat_geom.result | 66 + .../suite/heap/blob_group_concat_geom.test | 71 + mysql-test/suite/heap/blob_ops.result | 191 +++ mysql-test/suite/heap/blob_ops.test | 121 ++ .../suite/heap/blob_replace_overlap.result | 87 ++ .../suite/heap/blob_replace_overlap.test | 94 ++ mysql-test/suite/heap/blob_replication.result | 155 +++ mysql-test/suite/heap/blob_replication.test | 201 +++ .../suite/heap/blob_setvar_groupby.result | 73 ++ .../suite/heap/blob_setvar_groupby.test | 63 + mysql-test/suite/heap/blob_stress.result | 181 +++ mysql-test/suite/heap/blob_stress.test | 305 +++++ mysql-test/suite/heap/blob_truncate.result | 31 + mysql-test/suite/heap/blob_truncate.test | 44 + .../suite/heap/blob_update_overflow.result | 131 ++ .../suite/heap/blob_update_overflow.test | 154 +++ mysql-test/suite/heap/heap_geometry.result | Bin 0 -> 3323 bytes mysql-test/suite/heap/heap_geometry.test | 101 ++ .../suite/innodb_fts/r/innodb-fts-ddl.result | 2 +- mysql-test/suite/innodb_fts/r/misc.result | 10 +- .../suite/innodb_fts/t/innodb-fts-ddl.test | 2 +- mysql-test/suite/innodb_fts/t/misc.test | 10 +- .../transaction_nested_events_verifier.inc | 2 +- .../r/transaction_nested_events.result | 16 +- .../perfschema/t/misc_session_status.test | 1 - .../plugins/r/sql_error_log_withdbinfo.result | 6 +- .../r/tmp_disk_table_size_basic.result | 1 + .../sys_vars/t/tmp_disk_table_size_basic.test | 1 + mysys/my_compare.c | 48 + plugin/type_cursor/plugin.cc | 4 +- plugin/type_xmltype/sql_type_xmltype.cc | 3 +- plugin/type_xmltype/sql_type_xmltype.h | 5 +- sql/create_tmp_table.h | 2 + sql/field.cc | 260 +++- sql/field.h | 145 +- sql/item.cc | 3 +- sql/item.h | 36 +- sql/item_func.h | 9 +- sql/item_strfunc.h | 3 +- sql/item_sum.cc | 52 +- sql/item_sum.h | 14 +- sql/item_timefunc.cc | 3 +- sql/item_windowfunc.cc | 3 +- sql/item_windowfunc.h | 3 +- sql/opt_subselect.cc | 3 +- sql/records.cc | 2 +- sql/sql_delete.cc | 18 +- sql/sql_derived.cc | 48 +- sql/sql_expression_cache.cc | 17 + sql/sql_insert.cc | 5 +- sql/sql_parse.cc | 1 + sql/sql_select.cc | 472 +++++-- sql/sql_show.cc | 27 + sql/sql_table.cc | 2 +- sql/sql_trigger.cc | 4 +- sql/sql_type.cc | 104 +- sql/sql_type.h | 103 +- sql/sql_type_fixedbin.h | 3 +- sql/sql_type_geom.cc | 48 +- sql/sql_type_geom.h | 10 + sql/sql_update.cc | 22 +- sql/table.cc | 5 +- sql/table.h | 24 +- storage/csv/ha_tina.cc | 2 +- storage/heap/CMakeLists.txt | 30 +- storage/heap/_check.c | 116 +- storage/heap/ha_heap.cc | 214 ++- storage/heap/ha_heap.h | 8 +- storage/heap/heapdef.h | 150 ++- storage/heap/hp_blob.c | 930 +++++++++++++ storage/heap/hp_clear.c | 6 +- storage/heap/hp_close.c | 6 + storage/heap/hp_create.c | 95 +- storage/heap/hp_delete.c | 53 +- storage/heap/hp_extra.c | 19 + storage/heap/hp_hash.c | 278 +++- storage/heap/hp_open.c | 13 + storage/heap/hp_rfirst.c | 2 + storage/heap/hp_rkey.c | 3 + storage/heap/hp_rlast.c | 2 + storage/heap/hp_rnext.c | 2 + storage/heap/hp_rprev.c | 2 + storage/heap/hp_rrnd.c | 2 + storage/heap/hp_rsame.c | 2 + storage/heap/hp_scan.c | 57 +- storage/heap/hp_static.c | 4 +- storage/heap/hp_test_concurrent-t.c | 131 ++ storage/heap/hp_test_freelist-t.c | 835 ++++++++++++ storage/heap/hp_test_hash-t.c | 1165 +++++++++++++++++ storage/heap/hp_test_helpers.h | 87 ++ storage/heap/hp_test_key_setup-t.cc | 675 ++++++++++ storage/heap/hp_test_scan-t.c | 545 ++++++++ storage/heap/hp_update.c | 194 ++- storage/heap/hp_write.c | 149 ++- storage/maria/ma_create.c | 5 + storage/maria/ma_unique.c | 2 +- ...rder_boolean_mode_different_against.result | 2 +- ...ulltext_order_boolean_mode_no_where.result | 2 +- ...der_boolean_mode_same_match_against.result | 2 +- ...tural_language_mode_different_match.result | 7 +- ...rder_natural_language_mode_no_where.result | 8 +- ..._order_boolean_mode_different_against.test | 2 +- .../fulltext_order_boolean_mode_no_where.test | 2 +- ...order_boolean_mode_same_match_against.test | 2 +- ...natural_language_mode_different_match.test | 5 +- ..._order_natural_language_mode_no_where.test | 5 +- storage/rocksdb/rdb_datadic.cc | 2 +- 182 files changed, 12429 insertions(+), 800 deletions(-) create mode 100644 Docs/internal-temporary-tables.txt create mode 100644 mysql-test/main/blob_sj_test.result create mode 100644 mysql-test/main/blob_sj_test.test create mode 100644 mysql-test/suite/heap/blob.result create mode 100644 mysql-test/suite/heap/blob.test create mode 100644 mysql-test/suite/heap/blob_big.inc create mode 100644 mysql-test/suite/heap/blob_big1.result create mode 100644 mysql-test/suite/heap/blob_big1.test create mode 100644 mysql-test/suite/heap/blob_big2.result create mode 100644 mysql-test/suite/heap/blob_big2.test create mode 100644 mysql-test/suite/heap/blob_big3.result create mode 100644 mysql-test/suite/heap/blob_big3.test create mode 100644 mysql-test/suite/heap/blob_dedup.result create mode 100644 mysql-test/suite/heap/blob_dedup.test create mode 100644 mysql-test/suite/heap/blob_fallback.result create mode 100644 mysql-test/suite/heap/blob_fallback.test create mode 100644 mysql-test/suite/heap/blob_find_unique.result create mode 100644 mysql-test/suite/heap/blob_find_unique.test create mode 100644 mysql-test/suite/heap/blob_group_by.result create mode 100644 mysql-test/suite/heap/blob_group_by.test create mode 100644 mysql-test/suite/heap/blob_group_concat_geom.result create mode 100644 mysql-test/suite/heap/blob_group_concat_geom.test create mode 100644 mysql-test/suite/heap/blob_ops.result create mode 100644 mysql-test/suite/heap/blob_ops.test create mode 100644 mysql-test/suite/heap/blob_replace_overlap.result create mode 100644 mysql-test/suite/heap/blob_replace_overlap.test create mode 100644 mysql-test/suite/heap/blob_replication.result create mode 100644 mysql-test/suite/heap/blob_replication.test create mode 100644 mysql-test/suite/heap/blob_setvar_groupby.result create mode 100644 mysql-test/suite/heap/blob_setvar_groupby.test create mode 100644 mysql-test/suite/heap/blob_stress.result create mode 100644 mysql-test/suite/heap/blob_stress.test create mode 100644 mysql-test/suite/heap/blob_truncate.result create mode 100644 mysql-test/suite/heap/blob_truncate.test create mode 100644 mysql-test/suite/heap/blob_update_overflow.result create mode 100644 mysql-test/suite/heap/blob_update_overflow.test create mode 100644 mysql-test/suite/heap/heap_geometry.result create mode 100644 mysql-test/suite/heap/heap_geometry.test create mode 100644 storage/heap/hp_blob.c create mode 100644 storage/heap/hp_test_concurrent-t.c create mode 100644 storage/heap/hp_test_freelist-t.c create mode 100644 storage/heap/hp_test_hash-t.c create mode 100644 storage/heap/hp_test_helpers.h create mode 100644 storage/heap/hp_test_key_setup-t.cc create mode 100644 storage/heap/hp_test_scan-t.c diff --git a/Docs/internal-temporary-tables.txt b/Docs/internal-temporary-tables.txt new file mode 100644 index 0000000000000..62d7f59d5bd91 --- /dev/null +++ b/Docs/internal-temporary-tables.txt @@ -0,0 +1,60 @@ +MariaDB is automatically creating internal temporary tables to store +temporary results for a query. + +Examples: +- Distinct handling (writing into a temporary table with a unique key) +- Store the whole result for a final order by. +- Store the result for a group by. Two ways to do this: + a) Update a row value for each row combination based on the group-by key + b) Sort the result in group-by order, write a row to temporary table when group-by + order changes. Sort the resulting temporary tables. + This solution is used when all group-by fields is from the first used non const + table and we there is an ORDER BY to sort the final result +- Materialized tables for things like (select DISTINCT ...) +- Derived tables "SELECT ... FROM (SELECT...)" +- Results for UNION, EXCEPT etc +- Store values and results for "t1.a in (select ... )" +- Temporary storage for result from the SHOW command or when accessing + information_schema. + +The internal temporary tables can be of type memory/heap or Aria. +One can recompile MariaDB to use MyISAM instead of Aria for temporary +tables, but this is mainly kept as a legacy option and not recommended. + +The internal temporary tables are normally first created as memory +tables. If the table overflows, in case of MY_MIN(max_heap_table_size, +tmp_memory_table_size), the table is converted to an Aria/MyISAM +table. + +The code for creating internal temporary tables are in sql/sql_select.cc. +- create_tmp_table() is used for create all internal temporary tables. +- instantiate_tmp_table() which will create the aria or memory table + - If Aria was chosen as a base for the temporary table + create_internal_tmp_table() is called + - Heap tables are created by calling open_tmp_table() which calls + table->file->ha_open() +- create_internal_tmp_table_from_heap() or convert_heap_to_aria_update() + is used to change a heap table to an Aria table in case of overflow. + +The following are the usage restrictions for internal temporary tables that +an engine can rely on to provide optimized performance. + +- There will only be one user for the table + - No need for any locking to protect internal structures. No external_lock() calls. +- Table is created and dropped during the same statement. + - Table has either 0 keys, one unique key or an unique constraint (a set of fields + has to be unique) +- The only read/write patterns are: + - Table scan, possible followed by a read rows based on position (for order by + on temporary table). + - Table scan with duplicate row removal or delete of rows that are failing the + HAVING clause. + - In case of duplicate row removal, the table will be rescanned for each + row starting from one row after where the previous inner scan started + - Read by unique key + - Inserts + In case of duplicate key, one of the following options are done: + - Read conflicting row and update it. Unique key is not updated. + This happens in case of group by to update summary fields. + - Ignore to-be-inserted row (for distinct) + - Truncate followed by new inserts diff --git a/include/heap.h b/include/heap.h index 3fac752abd028..d27d6d7ab4073 100644 --- a/include/heap.h +++ b/include/heap.h @@ -19,6 +19,11 @@ #ifndef _heap_h #define _heap_h + +#ifndef DBUG_OFF +#define EXTRA_HEAP_DEBUG +#endif + #ifdef __cplusplus extern "C" { #endif @@ -105,6 +110,7 @@ typedef struct st_heap_block uint recbuffer; /* Length of one saved record */ ulong records_in_block; /* Records in one heap-block */ ulong last_allocated; /* number of records there is allocated space for */ + ulong high_water_allocated; /* peak last_allocated before hp_shrink_tail() */ size_t alloc_size; /* Allocate blocks of this size */ } HP_BLOCK; @@ -131,6 +137,12 @@ typedef struct st_hp_keydef /* Key definition with open */ uint (*get_key_length)(struct st_hp_keydef *keydef, const uchar *key); } HP_KEYDEF; +typedef struct st_hp_blob_desc +{ + uint offset; /* Byte offset of blob descriptor within record buffer */ + uint packlength; /* 1, 2, 3, or 4: length prefix size */ +} HP_BLOB_DESC; + typedef struct st_heap_share { HP_BLOCK block; @@ -138,27 +150,31 @@ typedef struct st_heap_share ulonglong data_length,index_length,max_table_size; ulonglong auto_increment; ulong min_records,max_records; /* Params to open */ - ulong records; /* records */ + ulong records; /* Logical (primary) record count */ + ulong total_records; /* All active records (primary + blob continuation) */ ulong blength; /* records rounded up to 2^n */ ulong deleted; /* Deleted records in database */ uint key_stat_version; /* version to indicate insert/delete */ uint key_version; /* Updated on key change */ uint file_version; /* Update on clear */ uint reclength; /* Length of one record */ - uint visible; /* Offset to the visible/deleted mark */ + uint visible; /* Offset to the flags byte (active/deleted/continuation) */ uint changed; uint keys,max_key_length; uint currently_disabled_keys; /* saved value from "keys" when disabled */ uint open_count; + uint blob_count; /* Number of blob columns */ + uint auto_key; + uint auto_key_type; /* real type of the auto key segment */ uchar *del_link; /* Link to next block with del. rec */ - char * name; /* Name of "memory-file" */ + HP_BLOB_DESC *blob_descs; /* Array of blob column descriptors */ + char * name; /* Name of "memory-file" */ time_t create_time; THR_LOCK lock; + LIST open_list; my_bool delete_on_close; my_bool internal; /* Internal temporary table */ - LIST open_list; - uint auto_key; - uint auto_key_type; /* real type of the auto key segment */ + my_bool write_can_replace; /* Set by HA_EXTRA_WRITE_CAN_REPLACE */ } HP_SHARE; struct st_hp_hash_info; @@ -169,18 +185,27 @@ typedef struct st_heap_info uchar *current_ptr; struct st_hp_hash_info *current_hash_ptr; ulong current_record,next_block; + /* Hash from last hp_search(), reused by hp_search_next() */ + ulong last_hash_of_key; int lastinx,errkey; int mode; /* Mode of file (READONLY..) */ uint opt_flag,update; + enum ha_rkey_function last_find_flag; uchar *lastkey; /* Last used key with rkey */ uchar *recbuf; /* Record buffer for rb-tree keys */ - enum ha_rkey_function last_find_flag; + uchar *blob_buff; /* Reassembly buffer for blob reads */ + uchar *key_blob_buff; /* Separate buffer for key comparison */ + uchar **pending_blob_chains; /* Chain pointers saved by deferred delete */ TREE_ELEMENT *parents[MAX_TREE_HEIGHT+1]; TREE_ELEMENT **last_pos; uint key_version; /* Version at last read */ uint file_version; /* Version at scan */ uint lastkey_len; + ulonglong blob_buff_len; /* Current allocated size of blob_buff */ + ulonglong key_blob_buff_len; /* Current allocated size of key_blob_buff */ my_bool implicit_emptied; + my_bool has_zerocopy_blobs; /* Last hp_read_blobs produced zero-copy ptrs */ + my_bool has_pending_blob_free; /* pending_blob_chains awaits freeing */ THR_LOCK_DATA lock; LIST open_list; } HP_INFO; @@ -189,14 +214,16 @@ typedef struct st_heap_info typedef struct st_heap_create_info { HP_KEYDEF *keydef; + HP_BLOB_DESC *blob_descs; + ulonglong max_table_size; + ulonglong auto_increment; + ulong max_records; + ulong min_records; uint auto_key; /* keynr [1 - maxkey] for auto key */ uint auto_key_type; uint keys; uint reclength; - ulong max_records; - ulong min_records; - ulonglong max_table_size; - ulonglong auto_increment; + uint blob_count; my_bool with_auto_increment; my_bool internal_table; /* diff --git a/include/my_base.h b/include/my_base.h index 1768be0094075..bfba10db063f4 100644 --- a/include/my_base.h +++ b/include/my_base.h @@ -272,7 +272,10 @@ enum ha_base_keytype { /* Varchar (0-65535 bytes) with length packed with 2 bytes */ HA_KEYTYPE_VARTEXT2=17, /* Key is sorted as letters */ HA_KEYTYPE_VARBINARY2=18, /* Key is sorted as unsigned chars */ - HA_KEYTYPE_BIT=19 + HA_KEYTYPE_BIT=19, + /* blob (length 4 bytes, pointer 8 bytes) used for internal tmp tables */ + HA_KEYTYPE_VARTEXT4=20, /* Key is sorted as letters */ + HA_KEYTYPE_VARBINARY4=21, /* Key is sorted as unsigned chars */ }; #define HA_MAX_KEYTYPE 31 /* Must be log2-1 */ @@ -320,13 +323,15 @@ enum ha_base_keytype { #define HA_UNIQUE_HASH 262144U /* Internal Flag Can be calculated */ -#define HA_INVISIBLE_KEY (2<<18) +#define HA_INVISIBLE_KEY (1LL<<19) +#define HA_NO_KEY_READ (1LL<<20) /* Internal debugging */ +#define HA_BLOB_PART_KEY (1LL<<21) /* Key has blob segments */ /* The combination of the above can be used for key type comparison. */ #define HA_KEYFLAG_MASK (HA_NOSAME | HA_AUTO_KEY | HA_NULL_ARE_EQUAL | \ HA_GENERATED_KEY | HA_UNIQUE_HASH) - /* These flags can be added to key-seg-flag */ +/* These flags can be added to key-seg-flag */ #define HA_SPACE_PACK 1 /* Pack space in key-seg */ #define HA_PART_KEY_SEG 4 /* Used by MySQL for part-key-cols */ @@ -708,4 +713,16 @@ C_MODE_START typedef void (* invalidator_by_filename)(const char * filename); C_MODE_END +static inline longlong read_lowendian(const uchar *from, uint bytes) +{ + switch(bytes) { + case 1: return from[0]; + case 2: return uint2korr(from); + case 3: return uint3korr(from); + case 4: return uint4korr(from); + case 8: return sint8korr(from); + default: DBUG_ASSERT(0); return 0; + } +} + #endif /* _my_base_h */ diff --git a/include/my_compare.h b/include/my_compare.h index 8155d9dfc587c..20612654e64b1 100644 --- a/include/my_compare.h +++ b/include/my_compare.h @@ -57,8 +57,8 @@ typedef struct st_HA_KEYSEG /* Key-portion */ uint16 language; uint8 type; /* Type of key (for sort) */ uint8 null_bit; /* bitmask to test for NULL */ - uint8 bit_start; - uint8 bit_length; /* Length of bit part */ + uint8 bit_start; /* or length of record packing */ + uint8 bit_length; /* Length of bit part or key packing */ } HA_KEYSEG; #define get_key_length(length,key) \ diff --git a/mysql-test/include/mtr_check.sql b/mysql-test/include/mtr_check.sql index 360f7b40bb864..46b420da4ae34 100644 --- a/mysql-test/include/mtr_check.sql +++ b/mysql-test/include/mtr_check.sql @@ -66,7 +66,7 @@ BEGIN collation_name, column_type, column_key, extra, column_comment FROM INFORMATION_SCHEMA.COLUMNS WHERE table_schema='mysql' - ORDER BY columns_in_mysql; + ORDER BY columns_in_mysql, ordinal_position; -- Dump all events, there should be none SELECT * FROM INFORMATION_SCHEMA.EVENTS; diff --git a/mysql-test/main/blob_sj_test.result b/mysql-test/main/blob_sj_test.result new file mode 100644 index 0000000000000..e171ef519bf8a --- /dev/null +++ b/mysql-test/main/blob_sj_test.result @@ -0,0 +1,31 @@ +set @save_optimizer_switch=@@optimizer_switch; +set optimizer_switch='semijoin=on,firstmatch=off,loosescan=off'; +set @blob_len = 16; +set @prefix_len = 6; +set @suffix_len = @blob_len - @prefix_len; +create table t1 (a1 blob(16), a2 blob(16)); +create table t2 (b1 blob(16), b2 blob(16)); +insert into t1 values +(concat('1 - 00', repeat('x', @suffix_len)), concat('2 - 00', repeat('x', @suffix_len))); +insert into t1 values +(concat('1 - 01', repeat('x', @suffix_len)), concat('2 - 01', repeat('x', @suffix_len))); +insert into t1 values +(concat('1 - 02', repeat('x', @suffix_len)), concat('2 - 02', repeat('x', @suffix_len))); +insert into t2 values +(concat('1 - 01', repeat('x', @suffix_len)), concat('2 - 01', repeat('x', @suffix_len))); +insert into t2 values +(concat('1 - 02', repeat('x', @suffix_len)), concat('2 - 02', repeat('x', @suffix_len))); +insert into t2 values +(concat('1 - 03', repeat('x', @suffix_len)), concat('2 - 03', repeat('x', @suffix_len))); +explain extended select left(a1,7), left(a2,7) from t1 where a1 in (select b1 from t2 where b1 > '0'); +id select_type table type possible_keys key key_len ref rows filtered Extra +1 PRIMARY t1 ALL NULL NULL NULL NULL 3 100.00 Using where +1 PRIMARY t2 ALL NULL NULL NULL NULL 3 33.33 Using where; Start temporary; End temporary; Using join buffer (flat, BNL join) +Warnings: +Note 1003 select left(`test`.`t1`.`a1`,7) AS `left(a1,7)`,left(`test`.`t1`.`a2`,7) AS `left(a2,7)` from `test`.`t1` semi join (`test`.`t2`) where `test`.`t2`.`b1` = `test`.`t1`.`a1` and `test`.`t1`.`a1` > '0' +select left(a1,7), left(a2,7) from t1 where a1 in (select b1 from t2 where b1 > '0'); +left(a1,7) left(a2,7) +1 - 01x 2 - 01x +1 - 02x 2 - 02x +drop table t1, t2; +set optimizer_switch=@save_optimizer_switch; diff --git a/mysql-test/main/blob_sj_test.test b/mysql-test/main/blob_sj_test.test new file mode 100644 index 0000000000000..42e13a171ce69 --- /dev/null +++ b/mysql-test/main/blob_sj_test.test @@ -0,0 +1,34 @@ +# +# MDEV-38975: Test semi-join with BLOB columns in HEAP temp tables. +# Covers a bug where HA_BLOB_PART in key_part_flag was uninitialized +# in SJ weedout temp tables, causing incorrect blob key handling. +# Force DuplicateWeedout by disabling other SJ strategies. +# +set @save_optimizer_switch=@@optimizer_switch; +set optimizer_switch='semijoin=on,firstmatch=off,loosescan=off'; +set @blob_len = 16; +set @prefix_len = 6; +set @suffix_len = @blob_len - @prefix_len; + +create table t1 (a1 blob(16), a2 blob(16)); +create table t2 (b1 blob(16), b2 blob(16)); + +insert into t1 values +(concat('1 - 00', repeat('x', @suffix_len)), concat('2 - 00', repeat('x', @suffix_len))); +insert into t1 values +(concat('1 - 01', repeat('x', @suffix_len)), concat('2 - 01', repeat('x', @suffix_len))); +insert into t1 values +(concat('1 - 02', repeat('x', @suffix_len)), concat('2 - 02', repeat('x', @suffix_len))); + +insert into t2 values +(concat('1 - 01', repeat('x', @suffix_len)), concat('2 - 01', repeat('x', @suffix_len))); +insert into t2 values +(concat('1 - 02', repeat('x', @suffix_len)), concat('2 - 02', repeat('x', @suffix_len))); +insert into t2 values +(concat('1 - 03', repeat('x', @suffix_len)), concat('2 - 03', repeat('x', @suffix_len))); + +explain extended select left(a1,7), left(a2,7) from t1 where a1 in (select b1 from t2 where b1 > '0'); +select left(a1,7), left(a2,7) from t1 where a1 in (select b1 from t2 where b1 > '0'); + +drop table t1, t2; +set optimizer_switch=@save_optimizer_switch; diff --git a/mysql-test/main/column_compression.result b/mysql-test/main/column_compression.result index b0e533caf4bcf..708f5a709ef7e 100644 --- a/mysql-test/main/column_compression.result +++ b/mysql-test/main/column_compression.result @@ -3008,6 +3008,14 @@ FROM (t5 JOIN t5 AS tt ON (tt.pk != t5.pk)); DROP TABLE t5; create table t1 (pk int not null, b1 blob compressed, v1 varbinary(100))engine=innodb; insert into t1 values (1,'ufhjdtv','f'),(2,'jdt',''),(3,'d','tvs'); +select t1.v1, t1.b1 from t1 join t1 as tt on (tt.pk != t1.pk) order by t1.v1; +v1 b1 + jdt + jdt +f ufhjdtv +f ufhjdtv +tvs d +tvs d select group_concat(t1.v1, t1.b1 order by 1) from (t1 join t1 as tt on (tt.pk != t1.pk)); group_concat(t1.v1, t1.b1 order by 1) jdt,jdt,fufhjdtv,fufhjdtv,tvsd,tvsd diff --git a/mysql-test/main/column_compression.test b/mysql-test/main/column_compression.test index 826197bb33f24..bba9f5d5ed1a1 100644 --- a/mysql-test/main/column_compression.test +++ b/mysql-test/main/column_compression.test @@ -516,6 +516,7 @@ DROP TABLE t5; create table t1 (pk int not null, b1 blob compressed, v1 varbinary(100))engine=innodb; insert into t1 values (1,'ufhjdtv','f'),(2,'jdt',''),(3,'d','tvs'); +select t1.v1, t1.b1 from t1 join t1 as tt on (tt.pk != t1.pk) order by t1.v1; select group_concat(t1.v1, t1.b1 order by 1) from (t1 join t1 as tt on (tt.pk != t1.pk)); drop table t1; diff --git a/mysql-test/main/count_distinct.result b/mysql-test/main/count_distinct.result index 30f24127982b6..0fa16cfb94169 100644 --- a/mysql-test/main/count_distinct.result +++ b/mysql-test/main/count_distinct.result @@ -31,34 +31,34 @@ isbn city libname a 000 New York New York Public Libra 6 001 New York NYC Lib 1 006 San Fran San Fransisco Public 1 -select t2.isbn,city,t1.libname,count(distinct t1.libname) as a from t3 left join t1 on t3.libname=t1.libname left join t2 on t3.isbn=t2.isbn group by city having count(distinct t1.libname) > 1; -isbn city libname a +select min(t2.isbn),city,min(t1.libname),count(distinct t1.libname) as a from t3 left join t1 on t3.libname=t1.libname left join t2 on t3.isbn=t2.isbn group by city having count(distinct t1.libname) > 1; +min(t2.isbn) city min(t1.libname) a 007 Berkeley Berkeley Public1 2 000 New York New York Public Libra 2 -select t2.isbn,city,t1.libname,count(distinct t1.libname) as a from t3 left join t1 on t3.libname=t1.libname left join t2 on t3.isbn=t2.isbn group by city having count(distinct concat(t1.libname,'a')) > 1; -isbn city libname a +select min(t2.isbn),city,min(t1.libname),count(distinct t1.libname) as a from t3 left join t1 on t3.libname=t1.libname left join t2 on t3.isbn=t2.isbn group by city having count(distinct concat(t1.libname,'a')) > 1; +min(t2.isbn) city min(t1.libname) a 007 Berkeley Berkeley Public1 2 000 New York New York Public Libra 2 -select t2.isbn,city,@bar:=t1.libname,count(distinct t1.libname) as a +select min(t2.isbn),city,@bar:=min(t1.libname),count(distinct t1.libname) as a from t3 left join t1 on t3.libname=t1.libname left join t2 on t3.isbn=t2.isbn group by city having count(distinct t1.libname) > 1; -isbn city @bar:=t1.libname a +min(t2.isbn) city @bar:=min(t1.libname) a 007 Berkeley Berkeley Public1 2 000 New York New York Public Libra 2 SELECT @bar; @bar -Berkeley Public2 -select t2.isbn,city,concat(@bar:=t1.libname),count(distinct t1.libname) as a +New York Public Libra +select min(t2.isbn),city,concat(@bar:=min(t1.libname)),count(distinct t1.libname) as a from t3 left join t1 on t3.libname=t1.libname left join t2 on t3.isbn=t2.isbn group by city having count(distinct t1.libname) > 1; -isbn city concat(@bar:=t1.libname) a +min(t2.isbn) city concat(@bar:=min(t1.libname)) a 007 Berkeley Berkeley Public1 2 000 New York New York Public Libra 2 SELECT @bar; @bar -Berkeley Public2 +New York Public Libra drop table t1, t2, t3; create table t1 (f1 int); insert into t1 values (1); diff --git a/mysql-test/main/count_distinct.test b/mysql-test/main/count_distinct.test index 9f682af3d6367..b141ffb379007 100644 --- a/mysql-test/main/count_distinct.test +++ b/mysql-test/main/count_distinct.test @@ -29,25 +29,19 @@ insert into t1 values ('Berkeley Public1','Berkeley'); insert into t1 values ('Berkeley Public2','Berkeley'); insert into t1 values ('NYC Lib','New York'); select t2.isbn,city,t1.libname,count(t1.libname) as a from t3 left join t1 on t3.libname=t1.libname left join t2 on t3.isbn=t2.isbn group by city,t1.libname; -select t2.isbn,city,t1.libname,count(distinct t1.libname) as a from t3 left join t1 on t3.libname=t1.libname left join t2 on t3.isbn=t2.isbn group by city having count(distinct t1.libname) > 1; -select t2.isbn,city,t1.libname,count(distinct t1.libname) as a from t3 left join t1 on t3.libname=t1.libname left join t2 on t3.isbn=t2.isbn group by city having count(distinct concat(t1.libname,'a')) > 1; +select min(t2.isbn),city,min(t1.libname),count(distinct t1.libname) as a from t3 left join t1 on t3.libname=t1.libname left join t2 on t3.isbn=t2.isbn group by city having count(distinct t1.libname) > 1; +select min(t2.isbn),city,min(t1.libname),count(distinct t1.libname) as a from t3 left join t1 on t3.libname=t1.libname left join t2 on t3.isbn=t2.isbn group by city having count(distinct concat(t1.libname,'a')) > 1; -select t2.isbn,city,@bar:=t1.libname,count(distinct t1.libname) as a +select min(t2.isbn),city,@bar:=min(t1.libname),count(distinct t1.libname) as a from t3 left join t1 on t3.libname=t1.libname left join t2 on t3.isbn=t2.isbn group by city having count(distinct t1.libname) > 1; -# -# Wrong result, see bug#49872 -# SELECT @bar; -select t2.isbn,city,concat(@bar:=t1.libname),count(distinct t1.libname) as a +select min(t2.isbn),city,concat(@bar:=min(t1.libname)),count(distinct t1.libname) as a from t3 left join t1 on t3.libname=t1.libname left join t2 on t3.isbn=t2.isbn group by city having count(distinct t1.libname) > 1; -# -# Wrong result, see bug#49872 -# SELECT @bar; drop table t1, t2, t3; diff --git a/mysql-test/main/create.result b/mysql-test/main/create.result index 2214d91858e69..a6dff66e45acb 100644 --- a/mysql-test/main/create.result +++ b/mysql-test/main/create.result @@ -30,10 +30,7 @@ Note 1051 Unknown table 'test.t1,test.t2' create table t1 (b char(0) not null, index(b)); ERROR 42000: The storage engine MyISAM can't index column `b` create table t1 (a int not null,b text) engine=heap; -ERROR 42000: Storage engine MEMORY doesn't support BLOB/TEXT columns -drop table if exists t1; -Warnings: -Note 1051 Unknown table 'test.t1' +drop table t1; create table t1 (ordid int(8) not null auto_increment, ord varchar(50) not null, primary key (ord,ordid)) engine=heap; ERROR 42000: Incorrect table definition; there can be only one auto column and it must be defined as a key create table not_existing_database.test (a int); @@ -1091,7 +1088,7 @@ t1 CREATE TABLE `t1` ( `INFO_BINARY` blob, `TID` bigint(10) NOT NULL, `TMP_SPACE_USED` bigint(10) NOT NULL -) DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci +) ENGINE=MEMORY DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci drop table t1; create temporary table t1 like information_schema.processlist; show create table t1; @@ -1117,7 +1114,7 @@ t1 CREATE TEMPORARY TABLE `t1` ( `INFO_BINARY` blob, `TID` bigint(10) NOT NULL, `TMP_SPACE_USED` bigint(10) NOT NULL -) DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci +) ENGINE=MEMORY DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci drop table t1; create table t1 like information_schema.character_sets; show create table t1; diff --git a/mysql-test/main/create.test b/mysql-test/main/create.test index 32480918d6621..b015906f1530d 100644 --- a/mysql-test/main/create.test +++ b/mysql-test/main/create.test @@ -30,9 +30,8 @@ create table t2 select auto+1 from t1; drop table if exists t1,t2; --error ER_WRONG_KEY_COLUMN create table t1 (b char(0) not null, index(b)); ---error ER_TABLE_CANT_HANDLE_BLOB create table t1 (a int not null,b text) engine=heap; -drop table if exists t1; +drop table t1; --error ER_WRONG_AUTO_KEY create table t1 (ordid int(8) not null auto_increment, ord varchar(50) not null, primary key (ord,ordid)) engine=heap; diff --git a/mysql-test/main/cte_recursive.test b/mysql-test/main/cte_recursive.test index 5f10d92b5cc5c..244a607fee305 100644 --- a/mysql-test/main/cte_recursive.test +++ b/mysql-test/main/cte_recursive.test @@ -3214,6 +3214,8 @@ show create table t2; --eval insert ignore into t2 $query; drop table t2; set @@sql_mode=""; +# Rows with identical (level, mid) due to overflow have non-deterministic order +--sorted_result --eval $query --eval create table t2 as $query; show create table t2; diff --git a/mysql-test/main/distinct.result b/mysql-test/main/distinct.result index 38371c34a35aa..5cd8625500d53 100644 --- a/mysql-test/main/distinct.result +++ b/mysql-test/main/distinct.result @@ -1192,7 +1192,7 @@ insert into t1 values (1, 'Aa123456', 'abc'), (2, 'Bb7897777', 'def'), (3, 'Cc01287', 'xyz'), (5, 'd12345', 'efg'); select distinct if(sum(a), b, 0) from t1 group by value(c) with rollup; if(sum(a), b, 0) -Aa123456 +SOME_B_VALUE drop table t1; # # end of 10.5 tests diff --git a/mysql-test/main/distinct.test b/mysql-test/main/distinct.test index d51e5093f24bc..24efd0cdb7928 100644 --- a/mysql-test/main/distinct.test +++ b/mysql-test/main/distinct.test @@ -916,6 +916,9 @@ create table t1 (a int, b longtext, c varchar(18)); insert into t1 values (1, 'Aa123456', 'abc'), (2, 'Bb7897777', 'def'), (3, 'Cc01287', 'xyz'), (5, 'd12345', 'efg'); +# ROLLUP row's b value is indeterminate (depends on last group processed), +# which varies by temp table engine (HEAP vs Aria). Mask the value. +--replace_regex /(Aa123456|Bb7897777|Cc01287|d12345)/SOME_B_VALUE/ select distinct if(sum(a), b, 0) from t1 group by value(c) with rollup; drop table t1; diff --git a/mysql-test/main/func_group.result b/mysql-test/main/func_group.result index c5a9a56fc52a0..967f9a51d4222 100644 --- a/mysql-test/main/func_group.result +++ b/mysql-test/main/func_group.result @@ -1819,7 +1819,7 @@ CREATE TABLE t1(f1 YEAR(4)); INSERT INTO t1 VALUES (0000),(2001); (SELECT MAX(f1) FROM t1) UNION (SELECT MAX(f1) FROM t1); Catalog Database Table Table_alias Column Column_alias Type Length Max length Is_null Flags Decimals Charsetnr -def MAX(f1) MAX(f1) 13 4 4 Y 49248 0 63 +def MAX(f1) MAX(f1) 13 4 4 Y 49252 0 63 MAX(f1) 2001 DROP TABLE t1; diff --git a/mysql-test/main/group_by.result b/mysql-test/main/group_by.result index 57a3cf97e94eb..7ca537fb0d462 100644 --- a/mysql-test/main/group_by.result +++ b/mysql-test/main/group_by.result @@ -2526,10 +2526,10 @@ SELECT f3, MIN(f2) FROM t1 GROUP BY f1 LIMIT 1; f3 MIN(f2) blob NULL DROP TABLE t1; -the value below *must* be 1 +the value below *must* be 0 (HEAP supports blobs) show status like 'Created_tmp_disk_tables'; Variable_name Value -Created_tmp_disk_tables 1 +Created_tmp_disk_tables 0 # # Bug #1002146: Unneeded filesort if usage of join buffer is not allowed # (bug mdev-645) diff --git a/mysql-test/main/group_by.test b/mysql-test/main/group_by.test index 8009d976d191e..1da4e46518796 100644 --- a/mysql-test/main/group_by.test +++ b/mysql-test/main/group_by.test @@ -1676,14 +1676,14 @@ DROP TABLE t1, t2; --disable_ps2_protocol --disable_view_protocol --disable_cursor_protocol -FLUSH STATUS; # this test case *must* use Aria temp tables +FLUSH STATUS; CREATE TABLE t1 (f1 INT, f2 decimal(20,1), f3 blob); INSERT INTO t1 values(11,NULL,'blob'),(11,NULL,'blob'); SELECT f3, MIN(f2) FROM t1 GROUP BY f1 LIMIT 1; DROP TABLE t1; ---echo the value below *must* be 1 +--echo the value below *must* be 0 (HEAP supports blobs) show status like 'Created_tmp_disk_tables'; --enable_cursor_protocol --enable_view_protocol diff --git a/mysql-test/main/group_min_max_innodb.result b/mysql-test/main/group_min_max_innodb.result index 87b718f53ed38..c58b33e22e95b 100644 --- a/mysql-test/main/group_min_max_innodb.result +++ b/mysql-test/main/group_min_max_innodb.result @@ -303,10 +303,10 @@ CREATE TABLE t2 (`voter_id` int(10) unsigned NOT NULL DEFAULT '0', insert into t2 values (1,repeat("a",1000)),(2,repeat("a",1000)),(3,repeat("b",1000)),(4,repeat("c",1000)),(4,repeat("b",1000)); SELECT GROUP_CONCAT(t1.language_id SEPARATOR ',') AS `translation_resources`, `d`.`serialized_c` FROM t2 AS `d` LEFT JOIN t1 ON `d`.`voter_id` = t1.`voter_id` GROUP BY `d`.`voter_id` ORDER BY 10-d.voter_id+RAND()*0; translation_resources serialized_c -NULL cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc -NULL bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb -NULL aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa -NULL aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa +NULL # +NULL # +NULL # +NULL # drop table t1,t2; # # MDEV-30143: Segfault on select query using index for group-by and filesort diff --git a/mysql-test/main/group_min_max_innodb.test b/mysql-test/main/group_min_max_innodb.test index 165580a7f8fe4..7d105f2d40d0b 100644 --- a/mysql-test/main/group_min_max_innodb.test +++ b/mysql-test/main/group_min_max_innodb.test @@ -248,6 +248,7 @@ CREATE TABLE t1 (`voter_id` int(11) unsigned NOT NULL, CREATE TABLE t2 (`voter_id` int(10) unsigned NOT NULL DEFAULT '0', `serialized_c` mediumblob) ENGINE=InnoDB DEFAULT CHARSET=utf8; insert into t2 values (1,repeat("a",1000)),(2,repeat("a",1000)),(3,repeat("b",1000)),(4,repeat("c",1000)),(4,repeat("b",1000)); +--replace_column 2 # SELECT GROUP_CONCAT(t1.language_id SEPARATOR ',') AS `translation_resources`, `d`.`serialized_c` FROM t2 AS `d` LEFT JOIN t1 ON `d`.`voter_id` = t1.`voter_id` GROUP BY `d`.`voter_id` ORDER BY 10-d.voter_id+RAND()*0; drop table t1,t2; diff --git a/mysql-test/main/information_schema.result b/mysql-test/main/information_schema.result index 5febd5baea044..9e1a291ff48d5 100644 --- a/mysql-test/main/information_schema.result +++ b/mysql-test/main/information_schema.result @@ -730,7 +730,7 @@ select TABLE_NAME,TABLE_TYPE,ENGINE from information_schema.tables where table_schema='information_schema' limit 2; TABLE_NAME TABLE_TYPE ENGINE -ALL_PLUGINS SYSTEM VIEW Aria +ALL_PLUGINS SYSTEM VIEW MEMORY APPLICABLE_ROLES SYSTEM VIEW MEMORY show tables from information_schema like "T%"; Tables_in_information_schema (T%) diff --git a/mysql-test/main/information_schema_parameters.result b/mysql-test/main/information_schema_parameters.result index 9ea739eb511d0..77393a3524d98 100644 --- a/mysql-test/main/information_schema_parameters.result +++ b/mysql-test/main/information_schema_parameters.result @@ -20,7 +20,7 @@ PARAMETERS CREATE TEMPORARY TABLE `PARAMETERS` ( `DTD_IDENTIFIER` longtext NOT NULL, `ROUTINE_TYPE` varchar(9) NOT NULL, `PARAMETER_DEFAULT` longtext -) DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci +) ENGINE=MEMORY DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci SELECT * FROM information_schema.columns WHERE table_schema = 'information_schema' AND table_name = 'parameters' diff --git a/mysql-test/main/information_schema_part.result b/mysql-test/main/information_schema_part.result index da875a8c186bb..f2f9deb4bb042 100644 --- a/mysql-test/main/information_schema_part.result +++ b/mysql-test/main/information_schema_part.result @@ -61,7 +61,7 @@ partition x2 values less than (5) ( subpartition x21 tablespace t1, subpartition x22 tablespace t2) ); -select * from information_schema.partitions where table_schema="test" order by table_name, partition_name; +select * from information_schema.partitions where table_schema="test" order by table_name, partition_name, subpartition_name; TABLE_CATALOG TABLE_SCHEMA TABLE_NAME PARTITION_NAME SUBPARTITION_NAME PARTITION_ORDINAL_POSITION SUBPARTITION_ORDINAL_POSITION PARTITION_METHOD SUBPARTITION_METHOD PARTITION_EXPRESSION SUBPARTITION_EXPRESSION PARTITION_DESCRIPTION TABLE_ROWS AVG_ROW_LENGTH DATA_LENGTH MAX_DATA_LENGTH INDEX_LENGTH DATA_FREE CREATE_TIME UPDATE_TIME CHECK_TIME CHECKSUM PARTITION_COMMENT NODEGROUP TABLESPACE_NAME def test t1 x1 x11 1 1 RANGE HASH `a` `a` + `b` 1 0 0 0 # 1024 0 # # NULL NULL default NULL def test t1 x1 x12 1 2 RANGE HASH `a` `a` + `b` 1 0 0 0 # 1024 0 # # NULL NULL default NULL diff --git a/mysql-test/main/information_schema_part.test b/mysql-test/main/information_schema_part.test index 3741de611505a..02af5be6d02f8 100644 --- a/mysql-test/main/information_schema_part.test +++ b/mysql-test/main/information_schema_part.test @@ -63,7 +63,7 @@ subpartition by key (a) subpartition x22 tablespace t2) ); --replace_column 16 # 19 # 20 # -select * from information_schema.partitions where table_schema="test" order by table_name, partition_name; +select * from information_schema.partitions where table_schema="test" order by table_name, partition_name, subpartition_name; drop table t1,t2; create table t1 ( diff --git a/mysql-test/main/information_schema_routines.result b/mysql-test/main/information_schema_routines.result index 830fb4d9f927c..a201fdfe5a581 100644 --- a/mysql-test/main/information_schema_routines.result +++ b/mysql-test/main/information_schema_routines.result @@ -36,7 +36,7 @@ ROUTINES CREATE TEMPORARY TABLE `ROUTINES` ( `CHARACTER_SET_CLIENT` varchar(32) NOT NULL, `COLLATION_CONNECTION` varchar(64) NOT NULL, `DATABASE_COLLATION` varchar(64) NOT NULL -) DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci +) ENGINE=MEMORY DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci SELECT * FROM information_schema.columns WHERE table_schema = 'information_schema' AND table_name = 'routines' diff --git a/mysql-test/main/intersect_all.result b/mysql-test/main/intersect_all.result index 81a57414426a4..6585c78f25ecb 100644 --- a/mysql-test/main/intersect_all.result +++ b/mysql-test/main/intersect_all.result @@ -789,13 +789,13 @@ t4 CREATE TABLE `t4` ( drop tables t4; (select a,b from t1) intersect all (select c,d from t2) intersect all (select e,f from t3) union all (select 4,4); a b -4 4 2 2 2 2 +4 4 (select a,b from t1) intersect all (select c,d from t2) intersect all (select e,f from t3) union all (select 4,4) except all (select 2,2); a b -4 4 2 2 +4 4 drop tables t1,t2,t3; create table t1 (a int, b int); create table t2 (c int, d int); @@ -850,9 +850,9 @@ insert into t3 values (3,3); e f 3 3 3 3 +4 4 5 5 6 6 -4 4 explain extended (select e,f from t3) intersect all (select c,d from t2) union all (select a,b from t1) union all (select 4,4); id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY t3 ALL NULL NULL NULL NULL 3 100.00 diff --git a/mysql-test/main/intersect_all.test b/mysql-test/main/intersect_all.test index 6ac8df4be14b4..40412edc01b2e 100644 --- a/mysql-test/main/intersect_all.test +++ b/mysql-test/main/intersect_all.test @@ -110,8 +110,10 @@ show create table t4; drop tables t4; +--sorted_result (select a,b from t1) intersect all (select c,d from t2) intersect all (select e,f from t3) union all (select 4,4); +--sorted_result (select a,b from t1) intersect all (select c,d from t2) intersect all (select e,f from t3) union all (select 4,4) except all (select 2,2); drop tables t1,t2,t3; @@ -151,6 +153,7 @@ explain extended (select a,b from t1) union all (select c,d from t2) intersect a insert into t2 values (3,3); insert into t3 values (3,3); +--sorted_result (select e,f from t3) intersect all (select c,d from t2) union all (select a,b from t1) union all (select 4,4); explain extended (select e,f from t3) intersect all (select c,d from t2) union all (select a,b from t1) union all (select 4,4); diff --git a/mysql-test/main/max_session_mem_used.test b/mysql-test/main/max_session_mem_used.test index d05c05b9987bf..99ab5c26b7aef 100644 --- a/mysql-test/main/max_session_mem_used.test +++ b/mysql-test/main/max_session_mem_used.test @@ -1,6 +1,5 @@ # memory usage is sensitive to valgrind/ps-protocol/embedded source include/not_msan.inc; -# We cannot use valgrind build as it uses more memory than normal build source include/not_valgrind_build.inc; source include/no_protocol.inc; source include/not_embedded.inc; diff --git a/mysql-test/main/metadata.result b/mysql-test/main/metadata.result index 0b94350601f78..5e41fadf05dab 100644 --- a/mysql-test/main/metadata.result +++ b/mysql-test/main/metadata.result @@ -146,7 +146,7 @@ id data data 2 female no select t1.id from t1 union select t2.id from t2; Catalog Database Table Table_alias Column Column_alias Type Length Max length Is_null Flags Decimals Charsetnr -def id id 246 4 1 Y 49152 0 63 +def id id 246 4 1 Y 49156 0 63 id 1 2 @@ -157,7 +157,7 @@ insert into t1 values (2,'two'); set @arg00=1 ; select @arg00 FROM t1 where a=1 union distinct select 1 FROM t1 where a=1; Catalog Database Table Table_alias Column Column_alias Type Length Max length Is_null Flags Decimals Charsetnr -def @arg00 @arg00 8 20 1 Y 49152 0 63 +def @arg00 @arg00 8 20 1 Y 49156 0 63 @arg00 1 select * from (select @arg00) aaa; @@ -167,7 +167,7 @@ def aaa @arg00 @arg00 8 20 1 Y 32768 0 63 1 select 1 union select 1; Catalog Database Table Table_alias Column Column_alias Type Length Max length Is_null Flags Decimals Charsetnr -def 1 1 3 2 1 N 49153 0 63 +def 1 1 3 2 1 N 49157 0 63 1 1 select * from (select 1 union select 1) aaa; @@ -259,16 +259,16 @@ c1 c2 2 2 SELECT v1.c1, v2.c2 FROM v1 JOIN v2 ON c1=c2 GROUP BY v1.c1; Catalog Database Table Table_alias Column Column_alias Type Length Max length Is_null Flags Decimals Charsetnr -def test t1 v1 c1 c1 254 1 1 Y 32768 0 8 -def test t2 v2 c2 c2 254 1 1 Y 0 0 8 +def test t1 v1 c1 c1 254 1 1 Y 32772 0 8 +def test t2 v2 c2 c2 254 1 1 Y 4 0 8 c1 c2 1 1 2 2 3 3 SELECT v1.c1, v2.c2 FROM v1 JOIN v2 ON c1=c2 GROUP BY v1.c1 ORDER BY v2.c2; Catalog Database Table Table_alias Column Column_alias Type Length Max length Is_null Flags Decimals Charsetnr -def test t1 v1 c1 c1 254 1 1 Y 32768 0 8 -def test t2 v2 c2 c2 254 1 1 Y 0 0 8 +def test t1 v1 c1 c1 254 1 1 Y 32772 0 8 +def test t2 v2 c2 c2 254 1 1 Y 4 0 8 c1 c2 1 1 2 2 diff --git a/mysql-test/main/mysql_client_test.result b/mysql-test/main/mysql_client_test.result index ba3b9cd8a8f7a..c568d26130415 100644 --- a/mysql-test/main/mysql_client_test.result +++ b/mysql-test/main/mysql_client_test.result @@ -130,7 +130,7 @@ mysql_stmt_next_result(): 0; field_count: 0 # cat MYSQL_TMP_DIR/test_mdev26145.out.log # ------------------------------------ Catalog Database Table Table_alias Column Column_alias Type Length Max length Is_null Flags Decimals Charsetnr -def MAX(a) MAX(a) 3 11 0 Y 49152 0 63 +def MAX(a) MAX(a) 3 11 0 Y 49156 0 63 # ------------------------------------ diff --git a/mysql-test/main/select.result b/mysql-test/main/select.result index b213a9943553c..1c1317d19229e 100644 --- a/mysql-test/main/select.result +++ b/mysql-test/main/select.result @@ -577,18 +577,18 @@ bedlam 1 bedpost 1 boasted 1 set tmp_memory_table_size=default; -select distinct fld3,repeat("a",length(fld3)),count(*) from t2 group by companynr,fld3 limit 100,10; +select distinct fld3,repeat("a",length(fld3)),count(*) from t2 group by companynr,fld3 order by fld3 limit 100,10; fld3 repeat("a",length(fld3)) count(*) -circus aaaaaa 1 -cited aaaaa 1 -Colombo aaaaaaa 1 -congresswoman aaaaaaaaaaaaa 1 -contrition aaaaaaaaaa 1 -corny aaaaa 1 -cultivation aaaaaaaaaaa 1 -definiteness aaaaaaaaaaaa 1 -demultiplex aaaaaaaaaaa 1 -disappointing aaaaaaaaaaaaa 1 +Baird aaaaa 1 +balled aaaaaa 1 +ballgown aaaaaaaa 1 +Baltimorean aaaaaaaaaaa 1 +bankruptcies aaaaaaaaaaaa 1 +Barry aaaaa 1 +batting aaaaaaa 1 +beaner aaaaaa 1 +beasts aaaaaa 1 +beaters aaaaaaa 1 select distinct companynr,rtrim(space(512+companynr)) from t3 order by 1,2; companynr rtrim(space(512+companynr)) 37 diff --git a/mysql-test/main/select.test b/mysql-test/main/select.test index ca248b26b9109..467ed1071210f 100644 --- a/mysql-test/main/select.test +++ b/mysql-test/main/select.test @@ -1443,7 +1443,8 @@ select distinct fld3,count(*) from t2 group by companynr,fld3 limit 10; set tmp_memory_table_size=0; # force on-disk tmp table select distinct fld3,count(*) from t2 group by companynr,fld3 limit 10; set tmp_memory_table_size=default; -select distinct fld3,repeat("a",length(fld3)),count(*) from t2 group by companynr,fld3 limit 100,10; +# ORDER BY fld3 ensures deterministic LIMIT window regardless of temp table engine +select distinct fld3,repeat("a",length(fld3)),count(*) from t2 group by companynr,fld3 order by fld3 limit 100,10; # # A big order by that should trigger a merge in filesort diff --git a/mysql-test/main/select_jcl6.result b/mysql-test/main/select_jcl6.result index 3a28ea89b0191..af24999b1f38b 100644 --- a/mysql-test/main/select_jcl6.result +++ b/mysql-test/main/select_jcl6.result @@ -589,18 +589,18 @@ bedlam 1 bedpost 1 boasted 1 set tmp_memory_table_size=default; -select distinct fld3,repeat("a",length(fld3)),count(*) from t2 group by companynr,fld3 limit 100,10; +select distinct fld3,repeat("a",length(fld3)),count(*) from t2 group by companynr,fld3 order by fld3 limit 100,10; fld3 repeat("a",length(fld3)) count(*) -circus aaaaaa 1 -cited aaaaa 1 -Colombo aaaaaaa 1 -congresswoman aaaaaaaaaaaaa 1 -contrition aaaaaaaaaa 1 -corny aaaaa 1 -cultivation aaaaaaaaaaa 1 -definiteness aaaaaaaaaaaa 1 -demultiplex aaaaaaaaaaa 1 -disappointing aaaaaaaaaaaaa 1 +Baird aaaaa 1 +balled aaaaaa 1 +ballgown aaaaaaaa 1 +Baltimorean aaaaaaaaaaa 1 +bankruptcies aaaaaaaaaaaa 1 +Barry aaaaa 1 +batting aaaaaaa 1 +beaner aaaaaa 1 +beasts aaaaaa 1 +beaters aaaaaaa 1 select distinct companynr,rtrim(space(512+companynr)) from t3 order by 1,2; companynr rtrim(space(512+companynr)) 37 diff --git a/mysql-test/main/select_pkeycache.result b/mysql-test/main/select_pkeycache.result index b213a9943553c..1c1317d19229e 100644 --- a/mysql-test/main/select_pkeycache.result +++ b/mysql-test/main/select_pkeycache.result @@ -577,18 +577,18 @@ bedlam 1 bedpost 1 boasted 1 set tmp_memory_table_size=default; -select distinct fld3,repeat("a",length(fld3)),count(*) from t2 group by companynr,fld3 limit 100,10; +select distinct fld3,repeat("a",length(fld3)),count(*) from t2 group by companynr,fld3 order by fld3 limit 100,10; fld3 repeat("a",length(fld3)) count(*) -circus aaaaaa 1 -cited aaaaa 1 -Colombo aaaaaaa 1 -congresswoman aaaaaaaaaaaaa 1 -contrition aaaaaaaaaa 1 -corny aaaaa 1 -cultivation aaaaaaaaaaa 1 -definiteness aaaaaaaaaaaa 1 -demultiplex aaaaaaaaaaa 1 -disappointing aaaaaaaaaaaaa 1 +Baird aaaaa 1 +balled aaaaaa 1 +ballgown aaaaaaaa 1 +Baltimorean aaaaaaaaaaa 1 +bankruptcies aaaaaaaaaaaa 1 +Barry aaaaa 1 +batting aaaaaaa 1 +beaner aaaaaa 1 +beasts aaaaaa 1 +beaters aaaaaaa 1 select distinct companynr,rtrim(space(512+companynr)) from t3 order by 1,2; companynr rtrim(space(512+companynr)) 37 diff --git a/mysql-test/main/status.result b/mysql-test/main/status.result index 0931f87cd6b26..94eca99904a54 100644 --- a/mysql-test/main/status.result +++ b/mysql-test/main/status.result @@ -326,12 +326,12 @@ Handler_mrr_key_refills 0 Handler_mrr_rowid_refills 0 Handler_prepare 0 Handler_read_first 0 -Handler_read_key 9 +Handler_read_key 13 Handler_read_last 0 Handler_read_next 0 Handler_read_prev 0 Handler_read_retry 0 -Handler_read_rnd 7 +Handler_read_rnd 6 Handler_read_rnd_deleted 0 Handler_read_rnd_next 23 Handler_rollback 0 @@ -339,17 +339,17 @@ Handler_savepoint 0 Handler_savepoint_rollback 0 Handler_tmp_delete 0 Handler_tmp_update 2 -Handler_tmp_write 7 +Handler_tmp_write 6 Handler_update 0 Handler_write 4 show status like '%_tmp%'; Variable_name Value -Created_tmp_disk_tables 1 +Created_tmp_disk_tables 0 Created_tmp_files 0 Created_tmp_tables 2 Handler_tmp_delete 0 Handler_tmp_update 2 -Handler_tmp_write 7 +Handler_tmp_write 6 Max_tmp_space_used 32768 Rows_tmp_read 44 drop table t1; diff --git a/mysql-test/main/temp_table_symlink.result b/mysql-test/main/temp_table_symlink.result index 1c5c68170ff8a..f87a46aba0cf8 100644 --- a/mysql-test/main/temp_table_symlink.result +++ b/mysql-test/main/temp_table_symlink.result @@ -1,10 +1,11 @@ create table d1 (a int); create temporary table t1 (a int); +set @@max_heap_table_size=16384; create temporary table t2 (a int); Got one of the listed errors create temporary table t3 (a int) engine=Aria; Got one of the listed errors -select * from information_schema.columns where table_schema='test'; +select * from information_schema.columns; Got one of the listed errors flush tables; select * from d1; diff --git a/mysql-test/main/temp_table_symlink.test b/mysql-test/main/temp_table_symlink.test index a0be38d907300..9ada8cd8be1ba 100644 --- a/mysql-test/main/temp_table_symlink.test +++ b/mysql-test/main/temp_table_symlink.test @@ -19,15 +19,17 @@ for (<#sql*.MYI>) { } EOF +# Force Aria usage when selecting from information_schema +set @@max_heap_table_size=16384; + error 1,1030; create temporary table t2 (a int); error 1,1030; create temporary table t3 (a int) engine=Aria; --disable_view_protocol error 1,1030; -select * from information_schema.columns where table_schema='test'; +select * from information_schema.columns; --enable_view_protocol - flush tables; select * from d1; drop temporary table t1; diff --git a/mysql-test/main/tmp_table_error.result b/mysql-test/main/tmp_table_error.result index 3a1a97250014b..2016070b5fcb8 100644 --- a/mysql-test/main/tmp_table_error.result +++ b/mysql-test/main/tmp_table_error.result @@ -3,6 +3,8 @@ create table t1 ( a int primary key, b text ) engine=innodb; +SET @save_tmp_memory_table_size=@@tmp_memory_table_size; +SET tmp_memory_table_size=0; create table t2 as select 1 @@ -2631,4 +2633,5 @@ b as c2626 from t1 ) as tt1; ERROR 0A000: Aria table 'tmp' has too many columns and/or indexes and/or unique constraints. +SET tmp_memory_table_size=@save_tmp_memory_table_size; drop table t1; diff --git a/mysql-test/main/tmp_table_error.test b/mysql-test/main/tmp_table_error.test index dbddaaaa4c794..3e3cea4674993 100644 --- a/mysql-test/main/tmp_table_error.test +++ b/mysql-test/main/tmp_table_error.test @@ -5,6 +5,8 @@ create table t1 ( b text ) engine=innodb; +SET @save_tmp_memory_table_size=@@tmp_memory_table_size; +SET tmp_memory_table_size=0; --replace_regex /'.*'/'tmp'/ --error 140 create table t2 as @@ -2634,4 +2636,5 @@ select b as c2626 from t1 ) as tt1; +SET tmp_memory_table_size=@save_tmp_memory_table_size; drop table t1; diff --git a/mysql-test/main/type_bit.result b/mysql-test/main/type_bit.result index 9bbf7c1931925..8cfcc4eef5c54 100644 --- a/mysql-test/main/type_bit.result +++ b/mysql-test/main/type_bit.result @@ -648,13 +648,13 @@ CREATE TABLE t1 (b BIT); INSERT INTO t1 (b) VALUES (1), (0); SELECT DISTINCT b FROM t1; Catalog Database Table Table_alias Column Column_alias Type Length Max length Is_null Flags Decimals Charsetnr -def test t1 t1 b b 16 1 1 Y 32 0 63 +def test t1 t1 b b 16 1 1 Y 36 0 63 b # # SELECT b FROM t1 GROUP BY b; Catalog Database Table Table_alias Column Column_alias Type Length Max length Is_null Flags Decimals Charsetnr -def test t1 t1 b b 16 1 1 Y 32 0 63 +def test t1 t1 b b 16 1 1 Y 36 0 63 b # # diff --git a/mysql-test/main/type_bit.test b/mysql-test/main/type_bit.test index 5f8b0d0af2f23..14e6fee300526 100644 --- a/mysql-test/main/type_bit.test +++ b/mysql-test/main/type_bit.test @@ -300,10 +300,12 @@ CREATE TABLE t1 (b BIT); INSERT INTO t1 (b) VALUES (1), (0); --enable_metadata --disable_view_protocol +--disable_cursor_protocol --replace_column 1 # SELECT DISTINCT b FROM t1; --replace_column 1 # SELECT b FROM t1 GROUP BY b; +--enable_cursor_protocol --enable_view_protocol --disable_metadata DROP TABLE t1; diff --git a/mysql-test/main/type_enum.result b/mysql-test/main/type_enum.result index d7860a8ec49bd..4113860bf77a7 100644 --- a/mysql-test/main/type_enum.result +++ b/mysql-test/main/type_enum.result @@ -2588,7 +2588,7 @@ t2 CREATE TABLE `t2` ( DROP TABLE t2; SELECT c_int FROM t1 UNION SELECT c_enum FROM t1; Catalog Database Table Table_alias Column Column_alias Type Length Max length Is_null Flags Decimals Charsetnr -def c_int c_int 253 11 0 Y 16384 0 8 +def c_int c_int 253 11 0 Y 16388 0 8 c_int SELECT COALESCE(c_int, c_enum) FROM t1; Catalog Database Table Table_alias Column Column_alias Type Length Max length Is_null Flags Decimals Charsetnr diff --git a/mysql-test/main/type_set.result b/mysql-test/main/type_set.result index 867f6b813f7d2..065f12a64c502 100644 --- a/mysql-test/main/type_set.result +++ b/mysql-test/main/type_set.result @@ -692,7 +692,7 @@ t2 CREATE TABLE `t2` ( DROP TABLE t2; SELECT c_int FROM t1 UNION SELECT c_set FROM t1; Catalog Database Table Table_alias Column Column_alias Type Length Max length Is_null Flags Decimals Charsetnr -def c_int c_int 253 33 0 Y 16384 0 192 +def c_int c_int 253 33 0 Y 16388 0 192 c_int SELECT COALESCE(c_int, c_set) FROM t1; Catalog Database Table Table_alias Column Column_alias Type Length Max length Is_null Flags Decimals Charsetnr diff --git a/mysql-test/suite/funcs_1/r/is_columns.result b/mysql-test/suite/funcs_1/r/is_columns.result index 1a2404d03915d..5d306d1b68274 100644 --- a/mysql-test/suite/funcs_1/r/is_columns.result +++ b/mysql-test/suite/funcs_1/r/is_columns.result @@ -81,7 +81,7 @@ COLUMNS CREATE TEMPORARY TABLE `COLUMNS` ( `IS_SYSTEM_TIME_PERIOD_START` varchar(3) NOT NULL, `IS_SYSTEM_TIME_PERIOD_END` varchar(3) NOT NULL, `CREATE_OPTIONS` varchar(2048) NOT NULL -) DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci +) ENGINE=MEMORY DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci SHOW COLUMNS FROM information_schema.COLUMNS; Field Type Null Key Default Extra TABLE_CATALOG varchar(512) NO NULL diff --git a/mysql-test/suite/funcs_1/r/is_events.result b/mysql-test/suite/funcs_1/r/is_events.result index 7df12ee27717e..59afb2d81f2e9 100644 --- a/mysql-test/suite/funcs_1/r/is_events.result +++ b/mysql-test/suite/funcs_1/r/is_events.result @@ -79,7 +79,7 @@ EVENTS CREATE TEMPORARY TABLE `EVENTS` ( `CHARACTER_SET_CLIENT` varchar(32) NOT NULL, `COLLATION_CONNECTION` varchar(64) NOT NULL, `DATABASE_COLLATION` varchar(64) NOT NULL -) DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci +) ENGINE=MEMORY DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci SHOW COLUMNS FROM information_schema.EVENTS; Field Type Null Key Default Extra EVENT_CATALOG varchar(64) NO NULL diff --git a/mysql-test/suite/funcs_1/r/is_routines.result b/mysql-test/suite/funcs_1/r/is_routines.result index a49cd2b482a6a..b3dd5e7779475 100644 --- a/mysql-test/suite/funcs_1/r/is_routines.result +++ b/mysql-test/suite/funcs_1/r/is_routines.result @@ -94,7 +94,7 @@ ROUTINES CREATE TEMPORARY TABLE `ROUTINES` ( `CHARACTER_SET_CLIENT` varchar(32) NOT NULL, `COLLATION_CONNECTION` varchar(64) NOT NULL, `DATABASE_COLLATION` varchar(64) NOT NULL -) DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci +) ENGINE=MEMORY DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci SHOW COLUMNS FROM information_schema.ROUTINES; Field Type Null Key Default Extra SPECIFIC_NAME varchar(64) NO NULL diff --git a/mysql-test/suite/funcs_1/r/is_routines_embedded.result b/mysql-test/suite/funcs_1/r/is_routines_embedded.result index e37d161a703d8..c32b050019ab8 100644 --- a/mysql-test/suite/funcs_1/r/is_routines_embedded.result +++ b/mysql-test/suite/funcs_1/r/is_routines_embedded.result @@ -94,7 +94,7 @@ ROUTINES CREATE TEMPORARY TABLE `ROUTINES` ( `CHARACTER_SET_CLIENT` varchar(32) NOT NULL, `COLLATION_CONNECTION` varchar(64) NOT NULL, `DATABASE_COLLATION` varchar(64) NOT NULL -) DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci +) ENGINE=MEMORY DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci SHOW COLUMNS FROM information_schema.ROUTINES; Field Type Null Key Default Extra SPECIFIC_NAME varchar(64) NO NULL @@ -197,7 +197,7 @@ sp_6_408002_2 def db_datadict_2 sp_6_408002_2 PROCEDURE NULL NULL NULL NULL NUL SELECT * FROM db_datadict_2.res_6_408002_2; END NULL NULL SQL NO CONTAINS SQL NULL DEFINER YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss root@localhost latin1 latin1_swedish_ci utf8mb4_uca1400_ai_ci add_suppression def mtr add_suppression PROCEDURE NULL NULL NULL NULL NULL NULL NULL NULL SQL BEGIN INSERT INTO test_suppressions (pattern) VALUES (pattern); FLUSH NO_WRITE_TO_BINLOG TABLE test_suppressions; END NULL NULL SQL NO CONTAINS SQL NULL DEFINER YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss root@localhost utf8mb3 utf8mb3_general_ci utf8mb4_uca1400_ai_ci -check_testcase def mtr check_testcase PROCEDURE NULL NULL NULL NULL NULL NULL NULL NULL SQL BEGIN SELECT * FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES WHERE variable_name NOT IN ('timestamp') AND variable_name not like "Last_IO_Err*" AND variable_name != 'INNODB_IBUF_MAX_SIZE' AND variable_name != 'INNODB_LOG_FILE_BUFFERING' AND variable_name != 'INNODB_USE_NATIVE_AIO' AND variable_name != 'INNODB_BUFFER_POOL_LOAD_AT_STARTUP' AND variable_name not like 'GTID%POS' AND variable_name != 'GTID_BINLOG_STATE' AND variable_name != 'THREAD_POOL_SIZE' ORDER BY variable_name; SELECT * FROM INFORMATION_SCHEMA.SCHEMATA ORDER BY BINARY SCHEMA_NAME; SELECT * FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME NOT IN ('mtr_wsrep_notify', 'wsrep_schema') ORDER BY BINARY SCHEMA_NAME; SELECT table_name AS tables_in_test FROM INFORMATION_SCHEMA.TABLES WHERE table_schema='test'; SELECT CONCAT(table_schema, '.', table_name) AS tables_in_mysql FROM INFORMATION_SCHEMA.TABLES WHERE table_schema='mysql' ORDER BY tables_in_mysql; SELECT CONCAT(table_schema, '.', table_name) AS columns_in_mysql, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_octet_length, numeric_precision, numeric_scale, character_set_name, collation_name, column_type, column_key, extra, column_comment FROM INFORMATION_SCHEMA.COLUMNS WHERE table_schema='mysql' ORDER BY columns_in_mysql; SELECT * FROM INFORMATION_SCHEMA.EVENTS; SELECT * FROM INFORMATION_SCHEMA.TRIGGERS WHERE TRIGGER_NAME NOT IN ('gs_insert', 'ts_insert') AND TRIGGER_SCHEMA != 'sys'; SELECT * FROM INFORMATION_SCHEMA.ROUTINES WHERE ROUTINE_SCHEMA != 'sys'; SHOW STATUS LIKE 'slave_open_temp_tables'; checksum table mysql.columns_priv, mysql.db, mysql.func, mysql.help_category, mysql.help_keyword, mysql.help_relation, mysql.plugin, mysql.procs_priv, mysql.roles_mapping, mysql.tables_priv, mysql.time_zone, mysql.time_zone_leap_second, mysql.time_zone_name, mysql.time_zone_transition, mysql.time_zone_transition_type, mysql.global_priv; SELECT * FROM INFORMATION_SCHEMA.PLUGINS WHERE PLUGIN_STATUS != 'INACTIVE'; select * from information_schema.session_variables where variable_name = 'debug_sync'; END NULL NULL SQL NO CONTAINS SQL NULL DEFINER YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss root@localhost utf8mb3 utf8mb3_general_ci utf8mb4_uca1400_ai_ci +check_testcase def mtr check_testcase PROCEDURE NULL NULL NULL NULL NULL NULL NULL NULL SQL BEGIN SELECT * FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES WHERE variable_name NOT IN ('timestamp') AND variable_name not like "Last_IO_Err*" AND variable_name != 'INNODB_IBUF_MAX_SIZE' AND variable_name != 'INNODB_LOG_FILE_BUFFERING' AND variable_name != 'INNODB_USE_NATIVE_AIO' AND variable_name != 'INNODB_BUFFER_POOL_LOAD_AT_STARTUP' AND variable_name not like 'GTID%POS' AND variable_name != 'GTID_BINLOG_STATE' AND variable_name != 'THREAD_POOL_SIZE' ORDER BY variable_name; SELECT * FROM INFORMATION_SCHEMA.SCHEMATA ORDER BY BINARY SCHEMA_NAME; SELECT * FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME NOT IN ('mtr_wsrep_notify', 'wsrep_schema') ORDER BY BINARY SCHEMA_NAME; SELECT table_name AS tables_in_test FROM INFORMATION_SCHEMA.TABLES WHERE table_schema='test'; SELECT CONCAT(table_schema, '.', table_name) AS tables_in_mysql FROM INFORMATION_SCHEMA.TABLES WHERE table_schema='mysql' ORDER BY tables_in_mysql; SELECT CONCAT(table_schema, '.', table_name) AS columns_in_mysql, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_octet_length, numeric_precision, numeric_scale, character_set_name, collation_name, column_type, column_key, extra, column_comment FROM INFORMATION_SCHEMA.COLUMNS WHERE table_schema='mysql' ORDER BY columns_in_mysql, ordinal_position; SELECT * FROM INFORMATION_SCHEMA.EVENTS; SELECT * FROM INFORMATION_SCHEMA.TRIGGERS WHERE TRIGGER_NAME NOT IN ('gs_insert', 'ts_insert') AND TRIGGER_SCHEMA != 'sys'; SELECT * FROM INFORMATION_SCHEMA.ROUTINES WHERE ROUTINE_SCHEMA != 'sys'; SHOW STATUS LIKE 'slave_open_temp_tables'; checksum table mysql.columns_priv, mysql.db, mysql.func, mysql.help_category, mysql.help_keyword, mysql.help_relation, mysql.plugin, mysql.procs_priv, mysql.roles_mapping, mysql.tables_priv, mysql.time_zone, mysql.time_zone_leap_second, mysql.time_zone_name, mysql.time_zone_transition, mysql.time_zone_transition_type, mysql.global_priv; SELECT * FROM INFORMATION_SCHEMA.PLUGINS WHERE PLUGIN_STATUS != 'INACTIVE'; select * from information_schema.session_variables where variable_name = 'debug_sync'; END NULL NULL SQL NO CONTAINS SQL NULL DEFINER YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss root@localhost utf8mb3 utf8mb3_general_ci utf8mb4_uca1400_ai_ci check_warnings def mtr check_warnings PROCEDURE NULL NULL NULL NULL NULL NULL NULL NULL SQL BEGIN DECLARE `pos` bigint unsigned; SET SQL_LOG_BIN=0, SQL_SAFE_UPDATES=0; UPDATE error_log el, global_suppressions gs SET suspicious=0 WHERE el.suspicious=1 AND el.line REGEXP gs.pattern; UPDATE error_log el, test_suppressions ts SET suspicious=0 WHERE el.suspicious=1 AND el.line REGEXP ts.pattern; SELECT COUNT(*) INTO @num_warnings FROM error_log WHERE suspicious=1; IF @num_warnings > 0 THEN SELECT line FROM error_log WHERE suspicious=1; SELECT 2 INTO result; ELSE SELECT 0 INTO RESULT; END IF; TRUNCATE test_suppressions; DROP TABLE error_log; END NULL NULL SQL NO CONTAINS SQL NULL DEFINER YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss root@localhost utf8mb3 utf8mb3_general_ci utf8mb4_uca1400_ai_ci AddGeometryColumn def mysql AddGeometryColumn PROCEDURE NULL NULL NULL NULL NULL NULL NULL NULL SQL begin set @qwe= concat('ALTER TABLE ', t_schema, '.', t_name, ' ADD ', geometry_column,' GEOMETRY REF_SYSTEM_ID=', t_srid); PREPARE ls from @qwe; execute ls; deallocate prepare ls; end NULL NULL SQL NO CONTAINS SQL NULL INVOKER YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss mariadb.sys@localhost latin1 latin1_swedish_ci utf8mb4_uca1400_ai_ci @@ -213,7 +213,7 @@ sp_6_408002_2 def db_datadict_2 sp_6_408002_2 PROCEDURE NULL NULL NULL NULL NUL SELECT * FROM db_datadict_2.res_6_408002_2; END NULL NULL SQL NO CONTAINS SQL NULL DEFINER YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss root@localhost latin1 latin1_swedish_ci utf8mb4_uca1400_ai_ci add_suppression def mtr add_suppression PROCEDURE NULL NULL NULL NULL NULL NULL NULL NULL SQL BEGIN INSERT INTO test_suppressions (pattern) VALUES (pattern); FLUSH NO_WRITE_TO_BINLOG TABLE test_suppressions; END NULL NULL SQL NO CONTAINS SQL NULL DEFINER YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss root@localhost utf8mb3 utf8mb3_general_ci utf8mb4_uca1400_ai_ci -check_testcase def mtr check_testcase PROCEDURE NULL NULL NULL NULL NULL NULL NULL NULL SQL BEGIN SELECT * FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES WHERE variable_name NOT IN ('timestamp') AND variable_name not like "Last_IO_Err*" AND variable_name != 'INNODB_IBUF_MAX_SIZE' AND variable_name != 'INNODB_LOG_FILE_BUFFERING' AND variable_name != 'INNODB_USE_NATIVE_AIO' AND variable_name != 'INNODB_BUFFER_POOL_LOAD_AT_STARTUP' AND variable_name not like 'GTID%POS' AND variable_name != 'GTID_BINLOG_STATE' AND variable_name != 'THREAD_POOL_SIZE' ORDER BY variable_name; SELECT * FROM INFORMATION_SCHEMA.SCHEMATA ORDER BY BINARY SCHEMA_NAME; SELECT * FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME NOT IN ('mtr_wsrep_notify', 'wsrep_schema') ORDER BY BINARY SCHEMA_NAME; SELECT table_name AS tables_in_test FROM INFORMATION_SCHEMA.TABLES WHERE table_schema='test'; SELECT CONCAT(table_schema, '.', table_name) AS tables_in_mysql FROM INFORMATION_SCHEMA.TABLES WHERE table_schema='mysql' ORDER BY tables_in_mysql; SELECT CONCAT(table_schema, '.', table_name) AS columns_in_mysql, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_octet_length, numeric_precision, numeric_scale, character_set_name, collation_name, column_type, column_key, extra, column_comment FROM INFORMATION_SCHEMA.COLUMNS WHERE table_schema='mysql' ORDER BY columns_in_mysql; SELECT * FROM INFORMATION_SCHEMA.EVENTS; SELECT * FROM INFORMATION_SCHEMA.TRIGGERS WHERE TRIGGER_NAME NOT IN ('gs_insert', 'ts_insert') AND TRIGGER_SCHEMA != 'sys'; SELECT * FROM INFORMATION_SCHEMA.ROUTINES WHERE ROUTINE_SCHEMA != 'sys'; SHOW STATUS LIKE 'slave_open_temp_tables'; checksum table mysql.columns_priv, mysql.db, mysql.func, mysql.help_category, mysql.help_keyword, mysql.help_relation, mysql.plugin, mysql.procs_priv, mysql.roles_mapping, mysql.tables_priv, mysql.time_zone, mysql.time_zone_leap_second, mysql.time_zone_name, mysql.time_zone_transition, mysql.time_zone_transition_type, mysql.global_priv; SELECT * FROM INFORMATION_SCHEMA.PLUGINS WHERE PLUGIN_STATUS != 'INACTIVE'; select * from information_schema.session_variables where variable_name = 'debug_sync'; END NULL NULL SQL NO CONTAINS SQL NULL DEFINER YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss root@localhost utf8mb3 utf8mb3_general_ci utf8mb4_uca1400_ai_ci +check_testcase def mtr check_testcase PROCEDURE NULL NULL NULL NULL NULL NULL NULL NULL SQL BEGIN SELECT * FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES WHERE variable_name NOT IN ('timestamp') AND variable_name not like "Last_IO_Err*" AND variable_name != 'INNODB_IBUF_MAX_SIZE' AND variable_name != 'INNODB_LOG_FILE_BUFFERING' AND variable_name != 'INNODB_USE_NATIVE_AIO' AND variable_name != 'INNODB_BUFFER_POOL_LOAD_AT_STARTUP' AND variable_name not like 'GTID%POS' AND variable_name != 'GTID_BINLOG_STATE' AND variable_name != 'THREAD_POOL_SIZE' ORDER BY variable_name; SELECT * FROM INFORMATION_SCHEMA.SCHEMATA ORDER BY BINARY SCHEMA_NAME; SELECT * FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME NOT IN ('mtr_wsrep_notify', 'wsrep_schema') ORDER BY BINARY SCHEMA_NAME; SELECT table_name AS tables_in_test FROM INFORMATION_SCHEMA.TABLES WHERE table_schema='test'; SELECT CONCAT(table_schema, '.', table_name) AS tables_in_mysql FROM INFORMATION_SCHEMA.TABLES WHERE table_schema='mysql' ORDER BY tables_in_mysql; SELECT CONCAT(table_schema, '.', table_name) AS columns_in_mysql, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_octet_length, numeric_precision, numeric_scale, character_set_name, collation_name, column_type, column_key, extra, column_comment FROM INFORMATION_SCHEMA.COLUMNS WHERE table_schema='mysql' ORDER BY columns_in_mysql, ordinal_position; SELECT * FROM INFORMATION_SCHEMA.EVENTS; SELECT * FROM INFORMATION_SCHEMA.TRIGGERS WHERE TRIGGER_NAME NOT IN ('gs_insert', 'ts_insert') AND TRIGGER_SCHEMA != 'sys'; SELECT * FROM INFORMATION_SCHEMA.ROUTINES WHERE ROUTINE_SCHEMA != 'sys'; SHOW STATUS LIKE 'slave_open_temp_tables'; checksum table mysql.columns_priv, mysql.db, mysql.func, mysql.help_category, mysql.help_keyword, mysql.help_relation, mysql.plugin, mysql.procs_priv, mysql.roles_mapping, mysql.tables_priv, mysql.time_zone, mysql.time_zone_leap_second, mysql.time_zone_name, mysql.time_zone_transition, mysql.time_zone_transition_type, mysql.global_priv; SELECT * FROM INFORMATION_SCHEMA.PLUGINS WHERE PLUGIN_STATUS != 'INACTIVE'; select * from information_schema.session_variables where variable_name = 'debug_sync'; END NULL NULL SQL NO CONTAINS SQL NULL DEFINER YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss root@localhost utf8mb3 utf8mb3_general_ci utf8mb4_uca1400_ai_ci check_warnings def mtr check_warnings PROCEDURE NULL NULL NULL NULL NULL NULL NULL NULL SQL BEGIN DECLARE `pos` bigint unsigned; SET SQL_LOG_BIN=0, SQL_SAFE_UPDATES=0; UPDATE error_log el, global_suppressions gs SET suspicious=0 WHERE el.suspicious=1 AND el.line REGEXP gs.pattern; UPDATE error_log el, test_suppressions ts SET suspicious=0 WHERE el.suspicious=1 AND el.line REGEXP ts.pattern; SELECT COUNT(*) INTO @num_warnings FROM error_log WHERE suspicious=1; IF @num_warnings > 0 THEN SELECT line FROM error_log WHERE suspicious=1; SELECT 2 INTO result; ELSE SELECT 0 INTO RESULT; END IF; TRUNCATE test_suppressions; DROP TABLE error_log; END NULL NULL SQL NO CONTAINS SQL NULL DEFINER YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss root@localhost utf8mb3 utf8mb3_general_ci utf8mb4_uca1400_ai_ci AddGeometryColumn def mysql AddGeometryColumn PROCEDURE NULL NULL NULL NULL NULL NULL NULL NULL SQL begin set @qwe= concat('ALTER TABLE ', t_schema, '.', t_name, ' ADD ', geometry_column,' GEOMETRY REF_SYSTEM_ID=', t_srid); PREPARE ls from @qwe; execute ls; deallocate prepare ls; end NULL NULL SQL NO CONTAINS SQL NULL INVOKER YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss mariadb.sys@localhost latin1 latin1_swedish_ci utf8mb4_uca1400_ai_ci @@ -229,7 +229,7 @@ sp_6_408002_2 def db_datadict_2 sp_6_408002_2 PROCEDURE NULL NULL NULL NULL NUL SELECT * FROM db_datadict_2.res_6_408002_2; END NULL NULL SQL NO CONTAINS SQL NULL DEFINER YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss root@localhost latin1 latin1_swedish_ci utf8mb4_uca1400_ai_ci add_suppression def mtr add_suppression PROCEDURE NULL NULL NULL NULL NULL NULL NULL NULL SQL BEGIN INSERT INTO test_suppressions (pattern) VALUES (pattern); FLUSH NO_WRITE_TO_BINLOG TABLE test_suppressions; END NULL NULL SQL NO CONTAINS SQL NULL DEFINER YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss root@localhost utf8mb3 utf8mb3_general_ci utf8mb4_uca1400_ai_ci -check_testcase def mtr check_testcase PROCEDURE NULL NULL NULL NULL NULL NULL NULL NULL SQL BEGIN SELECT * FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES WHERE variable_name NOT IN ('timestamp') AND variable_name not like "Last_IO_Err*" AND variable_name != 'INNODB_IBUF_MAX_SIZE' AND variable_name != 'INNODB_LOG_FILE_BUFFERING' AND variable_name != 'INNODB_USE_NATIVE_AIO' AND variable_name != 'INNODB_BUFFER_POOL_LOAD_AT_STARTUP' AND variable_name not like 'GTID%POS' AND variable_name != 'GTID_BINLOG_STATE' AND variable_name != 'THREAD_POOL_SIZE' ORDER BY variable_name; SELECT * FROM INFORMATION_SCHEMA.SCHEMATA ORDER BY BINARY SCHEMA_NAME; SELECT * FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME NOT IN ('mtr_wsrep_notify', 'wsrep_schema') ORDER BY BINARY SCHEMA_NAME; SELECT table_name AS tables_in_test FROM INFORMATION_SCHEMA.TABLES WHERE table_schema='test'; SELECT CONCAT(table_schema, '.', table_name) AS tables_in_mysql FROM INFORMATION_SCHEMA.TABLES WHERE table_schema='mysql' ORDER BY tables_in_mysql; SELECT CONCAT(table_schema, '.', table_name) AS columns_in_mysql, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_octet_length, numeric_precision, numeric_scale, character_set_name, collation_name, column_type, column_key, extra, column_comment FROM INFORMATION_SCHEMA.COLUMNS WHERE table_schema='mysql' ORDER BY columns_in_mysql; SELECT * FROM INFORMATION_SCHEMA.EVENTS; SELECT * FROM INFORMATION_SCHEMA.TRIGGERS WHERE TRIGGER_NAME NOT IN ('gs_insert', 'ts_insert') AND TRIGGER_SCHEMA != 'sys'; SELECT * FROM INFORMATION_SCHEMA.ROUTINES WHERE ROUTINE_SCHEMA != 'sys'; SHOW STATUS LIKE 'slave_open_temp_tables'; checksum table mysql.columns_priv, mysql.db, mysql.func, mysql.help_category, mysql.help_keyword, mysql.help_relation, mysql.plugin, mysql.procs_priv, mysql.roles_mapping, mysql.tables_priv, mysql.time_zone, mysql.time_zone_leap_second, mysql.time_zone_name, mysql.time_zone_transition, mysql.time_zone_transition_type, mysql.global_priv; SELECT * FROM INFORMATION_SCHEMA.PLUGINS WHERE PLUGIN_STATUS != 'INACTIVE'; select * from information_schema.session_variables where variable_name = 'debug_sync'; END NULL NULL SQL NO CONTAINS SQL NULL DEFINER YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss root@localhost utf8mb3 utf8mb3_general_ci utf8mb4_uca1400_ai_ci +check_testcase def mtr check_testcase PROCEDURE NULL NULL NULL NULL NULL NULL NULL NULL SQL BEGIN SELECT * FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES WHERE variable_name NOT IN ('timestamp') AND variable_name not like "Last_IO_Err*" AND variable_name != 'INNODB_IBUF_MAX_SIZE' AND variable_name != 'INNODB_LOG_FILE_BUFFERING' AND variable_name != 'INNODB_USE_NATIVE_AIO' AND variable_name != 'INNODB_BUFFER_POOL_LOAD_AT_STARTUP' AND variable_name not like 'GTID%POS' AND variable_name != 'GTID_BINLOG_STATE' AND variable_name != 'THREAD_POOL_SIZE' ORDER BY variable_name; SELECT * FROM INFORMATION_SCHEMA.SCHEMATA ORDER BY BINARY SCHEMA_NAME; SELECT * FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME NOT IN ('mtr_wsrep_notify', 'wsrep_schema') ORDER BY BINARY SCHEMA_NAME; SELECT table_name AS tables_in_test FROM INFORMATION_SCHEMA.TABLES WHERE table_schema='test'; SELECT CONCAT(table_schema, '.', table_name) AS tables_in_mysql FROM INFORMATION_SCHEMA.TABLES WHERE table_schema='mysql' ORDER BY tables_in_mysql; SELECT CONCAT(table_schema, '.', table_name) AS columns_in_mysql, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_octet_length, numeric_precision, numeric_scale, character_set_name, collation_name, column_type, column_key, extra, column_comment FROM INFORMATION_SCHEMA.COLUMNS WHERE table_schema='mysql' ORDER BY columns_in_mysql, ordinal_position; SELECT * FROM INFORMATION_SCHEMA.EVENTS; SELECT * FROM INFORMATION_SCHEMA.TRIGGERS WHERE TRIGGER_NAME NOT IN ('gs_insert', 'ts_insert') AND TRIGGER_SCHEMA != 'sys'; SELECT * FROM INFORMATION_SCHEMA.ROUTINES WHERE ROUTINE_SCHEMA != 'sys'; SHOW STATUS LIKE 'slave_open_temp_tables'; checksum table mysql.columns_priv, mysql.db, mysql.func, mysql.help_category, mysql.help_keyword, mysql.help_relation, mysql.plugin, mysql.procs_priv, mysql.roles_mapping, mysql.tables_priv, mysql.time_zone, mysql.time_zone_leap_second, mysql.time_zone_name, mysql.time_zone_transition, mysql.time_zone_transition_type, mysql.global_priv; SELECT * FROM INFORMATION_SCHEMA.PLUGINS WHERE PLUGIN_STATUS != 'INACTIVE'; select * from information_schema.session_variables where variable_name = 'debug_sync'; END NULL NULL SQL NO CONTAINS SQL NULL DEFINER YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss root@localhost utf8mb3 utf8mb3_general_ci utf8mb4_uca1400_ai_ci check_warnings def mtr check_warnings PROCEDURE NULL NULL NULL NULL NULL NULL NULL NULL SQL BEGIN DECLARE `pos` bigint unsigned; SET SQL_LOG_BIN=0, SQL_SAFE_UPDATES=0; UPDATE error_log el, global_suppressions gs SET suspicious=0 WHERE el.suspicious=1 AND el.line REGEXP gs.pattern; UPDATE error_log el, test_suppressions ts SET suspicious=0 WHERE el.suspicious=1 AND el.line REGEXP ts.pattern; SELECT COUNT(*) INTO @num_warnings FROM error_log WHERE suspicious=1; IF @num_warnings > 0 THEN SELECT line FROM error_log WHERE suspicious=1; SELECT 2 INTO result; ELSE SELECT 0 INTO RESULT; END IF; TRUNCATE test_suppressions; DROP TABLE error_log; END NULL NULL SQL NO CONTAINS SQL NULL DEFINER YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss root@localhost utf8mb3 utf8mb3_general_ci utf8mb4_uca1400_ai_ci AddGeometryColumn def mysql AddGeometryColumn PROCEDURE NULL NULL NULL NULL NULL NULL NULL NULL SQL begin set @qwe= concat('ALTER TABLE ', t_schema, '.', t_name, ' ADD ', geometry_column,' GEOMETRY REF_SYSTEM_ID=', t_srid); PREPARE ls from @qwe; execute ls; deallocate prepare ls; end NULL NULL SQL NO CONTAINS SQL NULL INVOKER YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss mariadb.sys@localhost latin1 latin1_swedish_ci utf8mb4_uca1400_ai_ci diff --git a/mysql-test/suite/funcs_1/r/is_tables_is.result b/mysql-test/suite/funcs_1/r/is_tables_is.result index 8c55fadc0a6d6..5fc3ec87f26f0 100644 --- a/mysql-test/suite/funcs_1/r/is_tables_is.result +++ b/mysql-test/suite/funcs_1/r/is_tables_is.result @@ -16,9 +16,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME ALL_PLUGINS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -91,9 +91,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME CHECK_CONSTRAINTS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -191,9 +191,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME COLUMNS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -291,9 +291,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME EVENTS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -566,9 +566,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME OPTIMIZER_TRACE TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -591,9 +591,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME PARAMETERS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -616,9 +616,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME PARTITIONS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -666,9 +666,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME PLUGINS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -691,9 +691,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME PROCESSLIST TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -741,9 +741,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME ROUTINES TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -991,9 +991,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME SYSTEM_VARIABLES TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -1166,9 +1166,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME TRIGGERS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -1266,9 +1266,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME VIEWS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -1307,9 +1307,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME ALL_PLUGINS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -1382,9 +1382,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME CHECK_CONSTRAINTS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -1482,9 +1482,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME COLUMNS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -1582,9 +1582,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME EVENTS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -1857,9 +1857,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME OPTIMIZER_TRACE TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -1882,9 +1882,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME PARAMETERS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -1907,9 +1907,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME PARTITIONS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -1957,9 +1957,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME PLUGINS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -1982,9 +1982,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME PROCESSLIST TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -2032,9 +2032,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME ROUTINES TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -2282,9 +2282,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME SYSTEM_VARIABLES TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -2457,9 +2457,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME TRIGGERS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -2557,9 +2557,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME VIEWS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# diff --git a/mysql-test/suite/funcs_1/r/is_tables_is_embedded.result b/mysql-test/suite/funcs_1/r/is_tables_is_embedded.result index a9cc694a867ce..6db8e5ce878f3 100644 --- a/mysql-test/suite/funcs_1/r/is_tables_is_embedded.result +++ b/mysql-test/suite/funcs_1/r/is_tables_is_embedded.result @@ -16,9 +16,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME ALL_PLUGINS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -91,9 +91,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME CHECK_CONSTRAINTS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -191,9 +191,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME COLUMNS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -291,9 +291,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME EVENTS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -566,9 +566,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME OPTIMIZER_TRACE TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -591,9 +591,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME PARAMETERS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -616,9 +616,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME PARTITIONS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -666,9 +666,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME PLUGINS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -691,9 +691,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME PROCESSLIST TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -741,9 +741,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME ROUTINES TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -966,9 +966,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME SYSTEM_VARIABLES TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -1141,9 +1141,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME TRIGGERS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -1241,9 +1241,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME VIEWS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -1282,9 +1282,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME ALL_PLUGINS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -1357,9 +1357,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME CHECK_CONSTRAINTS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -1457,9 +1457,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME COLUMNS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -1557,9 +1557,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME EVENTS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -1832,9 +1832,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME OPTIMIZER_TRACE TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -1857,9 +1857,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME PARAMETERS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -1882,9 +1882,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME PARTITIONS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -1932,9 +1932,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME PLUGINS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -1957,9 +1957,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME PROCESSLIST TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -2007,9 +2007,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME ROUTINES TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -2232,9 +2232,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME SYSTEM_VARIABLES TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -2407,9 +2407,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME TRIGGERS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# @@ -2507,9 +2507,9 @@ TABLE_CATALOG def TABLE_SCHEMA information_schema TABLE_NAME VIEWS TABLE_TYPE SYSTEM VIEW -ENGINE MYISAM_OR_MARIA +ENGINE MEMORY VERSION 11 -ROW_FORMAT DYNAMIC_OR_PAGE +ROW_FORMAT Fixed TABLE_ROWS #TBLR# AVG_ROW_LENGTH #ARL# DATA_LENGTH #DL# diff --git a/mysql-test/suite/funcs_1/r/is_triggers.result b/mysql-test/suite/funcs_1/r/is_triggers.result index 347e96999cd5a..1111014d93cd2 100644 --- a/mysql-test/suite/funcs_1/r/is_triggers.result +++ b/mysql-test/suite/funcs_1/r/is_triggers.result @@ -77,7 +77,7 @@ TRIGGERS CREATE TEMPORARY TABLE `TRIGGERS` ( `CHARACTER_SET_CLIENT` varchar(32) NOT NULL, `COLLATION_CONNECTION` varchar(64) NOT NULL, `DATABASE_COLLATION` varchar(64) NOT NULL -) DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci +) ENGINE=MEMORY DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci SHOW COLUMNS FROM information_schema.TRIGGERS; Field Type Null Key Default Extra TRIGGER_CATALOG varchar(512) NO NULL diff --git a/mysql-test/suite/funcs_1/r/is_triggers_embedded.result b/mysql-test/suite/funcs_1/r/is_triggers_embedded.result index a1cfee2f243ef..1322d50038fca 100644 --- a/mysql-test/suite/funcs_1/r/is_triggers_embedded.result +++ b/mysql-test/suite/funcs_1/r/is_triggers_embedded.result @@ -77,7 +77,7 @@ TRIGGERS CREATE TEMPORARY TABLE `TRIGGERS` ( `CHARACTER_SET_CLIENT` varchar(32) NOT NULL, `COLLATION_CONNECTION` varchar(64) NOT NULL, `DATABASE_COLLATION` varchar(64) NOT NULL -) DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci +) ENGINE=MEMORY DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci SHOW COLUMNS FROM information_schema.TRIGGERS; Field Type Null Key Default Extra TRIGGER_CATALOG varchar(512) NO NULL diff --git a/mysql-test/suite/funcs_1/r/is_views.result b/mysql-test/suite/funcs_1/r/is_views.result index 6a86e7464a0fb..c67b372937f9b 100644 --- a/mysql-test/suite/funcs_1/r/is_views.result +++ b/mysql-test/suite/funcs_1/r/is_views.result @@ -53,7 +53,7 @@ VIEWS CREATE TEMPORARY TABLE `VIEWS` ( `CHARACTER_SET_CLIENT` varchar(32) NOT NULL, `COLLATION_CONNECTION` varchar(64) NOT NULL, `ALGORITHM` varchar(10) NOT NULL -) DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci +) ENGINE=MEMORY DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci SHOW COLUMNS FROM information_schema.VIEWS; Field Type Null Key Default Extra TABLE_CATALOG varchar(512) NO NULL diff --git a/mysql-test/suite/funcs_1/r/is_views_embedded.result b/mysql-test/suite/funcs_1/r/is_views_embedded.result index f64562aadd164..67faf6b30ccfa 100644 --- a/mysql-test/suite/funcs_1/r/is_views_embedded.result +++ b/mysql-test/suite/funcs_1/r/is_views_embedded.result @@ -53,7 +53,7 @@ VIEWS CREATE TEMPORARY TABLE `VIEWS` ( `CHARACTER_SET_CLIENT` varchar(32) NOT NULL, `COLLATION_CONNECTION` varchar(64) NOT NULL, `ALGORITHM` varchar(10) NOT NULL -) DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci +) ENGINE=MEMORY DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci SHOW COLUMNS FROM information_schema.VIEWS; Field Type Null Key Default Extra TABLE_CATALOG varchar(512) NO NULL diff --git a/mysql-test/suite/funcs_1/r/processlist_priv_no_prot.result b/mysql-test/suite/funcs_1/r/processlist_priv_no_prot.result index f56f2bab463e5..4d07c7804a0a3 100644 --- a/mysql-test/suite/funcs_1/r/processlist_priv_no_prot.result +++ b/mysql-test/suite/funcs_1/r/processlist_priv_no_prot.result @@ -46,7 +46,7 @@ PROCESSLIST CREATE TEMPORARY TABLE `PROCESSLIST` ( `INFO_BINARY` blob, `TID` bigint(10) NOT NULL, `TMP_SPACE_USED` bigint(10) NOT NULL -) DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci +) ENGINE=MEMORY DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci SHOW processlist; Id User Host db Command Time State Info Progress ID root HOST_NAME information_schema Query TIME starting SHOW processlist TIME_MS @@ -128,7 +128,7 @@ PROCESSLIST CREATE TEMPORARY TABLE `PROCESSLIST` ( `INFO_BINARY` blob, `TID` bigint(10) NOT NULL, `TMP_SPACE_USED` bigint(10) NOT NULL -) DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci +) ENGINE=MEMORY DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci SHOW processlist; Id User Host db Command Time State Info Progress ID ddicttestuser1 HOST_NAME information_schema Query TIME starting SHOW processlist TIME_MS diff --git a/mysql-test/suite/funcs_1/r/processlist_priv_ps.result b/mysql-test/suite/funcs_1/r/processlist_priv_ps.result index 8df870ad76930..77e746fd59c91 100644 --- a/mysql-test/suite/funcs_1/r/processlist_priv_ps.result +++ b/mysql-test/suite/funcs_1/r/processlist_priv_ps.result @@ -46,7 +46,7 @@ PROCESSLIST CREATE TEMPORARY TABLE `PROCESSLIST` ( `INFO_BINARY` blob, `TID` bigint(10) NOT NULL, `TMP_SPACE_USED` bigint(10) NOT NULL -) DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci +) ENGINE=MEMORY DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci SHOW processlist; Id User Host db Command Time State Info Progress ID root HOST_NAME information_schema Query TIME starting SHOW processlist TIME_MS @@ -128,7 +128,7 @@ PROCESSLIST CREATE TEMPORARY TABLE `PROCESSLIST` ( `INFO_BINARY` blob, `TID` bigint(10) NOT NULL, `TMP_SPACE_USED` bigint(10) NOT NULL -) DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci +) ENGINE=MEMORY DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci SHOW processlist; Id User Host db Command Time State Info Progress ID ddicttestuser1 HOST_NAME information_schema Query TIME starting SHOW processlist TIME_MS diff --git a/mysql-test/suite/funcs_1/r/processlist_val_no_prot.result b/mysql-test/suite/funcs_1/r/processlist_val_no_prot.result index 0182f40199373..2348ed276a817 100644 --- a/mysql-test/suite/funcs_1/r/processlist_val_no_prot.result +++ b/mysql-test/suite/funcs_1/r/processlist_val_no_prot.result @@ -32,7 +32,7 @@ PROCESSLIST CREATE TEMPORARY TABLE `PROCESSLIST` ( `INFO_BINARY` blob, `TID` bigint(10) NOT NULL, `TMP_SPACE_USED` bigint(10) NOT NULL -) DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci +) ENGINE=MEMORY DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci # Ensure that the information about the own connection is correct. #-------------------------------------------------------------------------- diff --git a/mysql-test/suite/funcs_1/r/processlist_val_ps.result b/mysql-test/suite/funcs_1/r/processlist_val_ps.result index a634fb485cfc0..996f41316314b 100644 --- a/mysql-test/suite/funcs_1/r/processlist_val_ps.result +++ b/mysql-test/suite/funcs_1/r/processlist_val_ps.result @@ -32,7 +32,7 @@ PROCESSLIST CREATE TEMPORARY TABLE `PROCESSLIST` ( `INFO_BINARY` blob, `TID` bigint(10) NOT NULL, `TMP_SPACE_USED` bigint(10) NOT NULL -) DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci +) ENGINE=MEMORY DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci # Ensure that the information about the own connection is correct. #-------------------------------------------------------------------------- diff --git a/mysql-test/suite/heap/blob.result b/mysql-test/suite/heap/blob.result new file mode 100644 index 0000000000000..b1c04665d8639 --- /dev/null +++ b/mysql-test/suite/heap/blob.result @@ -0,0 +1,812 @@ +drop table if exists t1,t2; +# +# Basic CRUD with BLOB column +# +create table t1 (a int not null, b blob, primary key(a)) engine=memory; +insert into t1 values (1, 'hello'), (2, 'world'); +select * from t1 order by a; +a b +1 hello +2 world +select * from t1 where a=1; +a b +1 hello +select * from t1 where a=2; +a b +2 world +update t1 set b='updated' where a=1; +select * from t1 order by a; +a b +1 updated +2 world +delete from t1 where a=2; +select * from t1 order by a; +a b +1 updated +insert into t1 values (3, 'new row'); +select * from t1 order by a; +a b +1 updated +3 new row +drop table t1; +# +# Multiple BLOB/TEXT columns of different types +# +create table t1 ( +id int not null auto_increment, +t tinyblob, +b blob, +m mediumblob, +tx text, +primary key(id) +) engine=memory; +insert into t1 (t, b, m, tx) values +('tiny1', 'blob1', 'medium1', 'text1'), +('tiny2', 'blob2', 'medium2', 'text2'); +select * from t1 order by id; +id t b m tx +1 tiny1 blob1 medium1 text1 +2 tiny2 blob2 medium2 text2 +update t1 set b='blob_updated', tx='text_updated' where id=1; +select * from t1 order by id; +id t b m tx +1 tiny1 blob_updated medium1 text_updated +2 tiny2 blob2 medium2 text2 +delete from t1 where id=2; +select * from t1 order by id; +id t b m tx +1 tiny1 blob_updated medium1 text_updated +drop table t1; +# +# NULL and empty blob values +# +create table t1 (a int not null, b blob, c text, primary key(a)) engine=memory; +insert into t1 values (1, NULL, NULL); +insert into t1 values (2, '', ''); +insert into t1 values (3, 'data', 'text'); +select a, b, c, length(b), length(c) from t1 order by a; +a b c length(b) length(c) +1 NULL NULL NULL NULL +2 0 0 +3 data text 4 4 +update t1 set b=NULL where a=3; +select a, b, c, length(b), length(c) from t1 order by a; +a b c length(b) length(c) +1 NULL NULL NULL NULL +2 0 0 +3 NULL text NULL 4 +update t1 set b='restored' where a=3; +select a, b, c, length(b), length(c) from t1 order by a; +a b c length(b) length(c) +1 NULL NULL NULL NULL +2 0 0 +3 restored text 8 4 +drop table t1; +# +# Large BLOBs spanning multiple continuation runs +# For (int, blob): recbuffer=16, visible=15, leaf block ~1021 slots. +# Max run payload ~15305 bytes. Sizes chosen to span multiple runs +# and not align to visible (15 bytes). +# +create table t1 (a int not null, b blob, primary key(a)) engine=memory; +insert into t1 values (1, repeat('A', 1000)); +insert into t1 values (2, repeat('B', 50000)); +insert into t1 values (3, repeat('C', 63001)); +select a, length(b), left(b, 5), right(b, 5) from t1 order by a; +a length(b) left(b, 5) right(b, 5) +1 1000 AAAAA AAAAA +2 50000 BBBBB BBBBB +3 63001 CCCCC CCCCC +select a from t1 where b = repeat('A', 1000); +a +1 +select a from t1 where b = repeat('B', 50000); +a +2 +select a from t1 where b = repeat('C', 63001); +a +3 +update t1 set b=repeat('D', 63001) where a=1; +select a, length(b), left(b, 5), right(b, 5) from t1 order by a; +a length(b) left(b, 5) right(b, 5) +1 63001 DDDDD DDDDD +2 50000 BBBBB BBBBB +3 63001 CCCCC CCCCC +select a from t1 where b = repeat('D', 63001); +a +1 +update t1 set b=repeat('E', 100) where a=2; +select a, length(b), left(b, 5), right(b, 5) from t1 order by a; +a length(b) left(b, 5) right(b, 5) +1 63001 DDDDD DDDDD +2 100 EEEEE EEEEE +3 63001 CCCCC CCCCC +drop table t1; +# +# Mixed operations: insert, delete, insert (free list reuse) +# +create table t1 (a int not null, b blob, primary key(a)) engine=memory; +insert into t1 values (1, repeat('X', 20000)); +insert into t1 values (2, repeat('Y', 50000)); +insert into t1 values (3, repeat('Z', 10000)); +delete from t1 where a=2; +insert into t1 values (4, repeat('W', 40000)); +select a, length(b) from t1 order by a; +a length(b) +1 20000 +3 10000 +4 40000 +select a from t1 where b = repeat('X', 20000); +a +1 +select a from t1 where b = repeat('Z', 10000); +a +3 +select a from t1 where b = repeat('W', 40000); +a +4 +delete from t1; +insert into t1 values (10, repeat('R', 50000)); +insert into t1 values (20, repeat('S', 50000)); +select a, length(b) from t1 order by a; +a length(b) +10 50000 +20 50000 +drop table t1; +# +# Free list fragmentation: NULL-blob rows interleaved with large-blob rows +# +# When rows with NULL blobs and rows with large blobs are deleted, the +# free list gets primary-record slots (from NULL rows) interleaved between +# continuation slots (from large-blob rows). The peek-then-unlink +# algorithm must find contiguous continuation groups despite these +# interleaving primary slots breaking address contiguity. +# +# After deleting all rows and reinserting, the new blob data must be +# correct -- verifying no free list corruption from the fragmented state. +# +create table t1 (a int not null, b blob, primary key(a)) engine=memory; +insert into t1 values (1, NULL); +insert into t1 values (2, repeat('A', 20000)); +insert into t1 values (3, NULL); +insert into t1 values (4, repeat('B', 30000)); +insert into t1 values (5, NULL); +insert into t1 values (6, repeat('C', 25000)); +select a, length(b) from t1 order by a; +a length(b) +1 NULL +2 20000 +3 NULL +4 30000 +5 NULL +6 25000 +delete from t1 where a=2; +delete from t1 where a=4; +delete from t1 where a=6; +delete from t1 where a=1; +delete from t1 where a=3; +delete from t1 where a=5; +insert into t1 values (10, repeat('D', 35000)); +insert into t1 values (20, repeat('E', 20000)); +insert into t1 values (30, repeat('F', 15000)); +select a, length(b) from t1 order by a; +a length(b) +10 35000 +20 20000 +30 15000 +select a from t1 where b = repeat('D', 35000); +a +10 +select a from t1 where b = repeat('E', 20000); +a +20 +select a from t1 where b = repeat('F', 15000); +a +30 +delete from t1 where a=10; +insert into t1 values (40, repeat('G', 40000)); +select a, length(b) from t1 order by a; +a length(b) +20 20000 +30 15000 +40 40000 +select a from t1 where b = repeat('E', 20000); +a +20 +select a from t1 where b = repeat('F', 15000); +a +30 +select a from t1 where b = repeat('G', 40000); +a +40 +drop table t1; +# +# Free list scavenging with mixed NULL and non-NULL blob columns +# +# Multiple blob columns where some are NULL and others are large. +# This creates rows with partial continuation chains -- the NULL +# columns have no chain while the non-NULL columns do. +# +create table t1 ( +a int not null, +b blob, +c blob, +primary key(a) +) engine=memory; +insert into t1 values (1, repeat('X', 15000), NULL); +insert into t1 values (2, NULL, repeat('Y', 25000)); +insert into t1 values (3, repeat('Z', 10000), repeat('W', 20000)); +insert into t1 values (4, NULL, NULL); +select a, length(b), length(c) from t1 order by a; +a length(b) length(c) +1 15000 NULL +2 NULL 25000 +3 10000 20000 +4 NULL NULL +delete from t1 where a=1; +delete from t1 where a=3; +insert into t1 values (5, repeat('P', 18000), repeat('Q', 22000)); +select a, length(b), length(c) from t1 order by a; +a length(b) length(c) +2 NULL 25000 +4 NULL NULL +5 18000 22000 +select a from t1 where b is null and c = repeat('Y', 25000); +a +2 +select a from t1 where b = repeat('P', 18000) and c = repeat('Q', 22000); +a +5 +delete from t1; +insert into t1 values (6, repeat('R', 30000), repeat('S', 30000)); +select a, length(b), length(c) from t1 order by a; +a length(b) length(c) +6 30000 30000 +select a from t1 where b = repeat('R', 30000) and c = repeat('S', 30000); +a +6 +drop table t1; +# +# TRUNCATE with BLOB data +# +create table t1 (a int not null, b blob, primary key(a)) engine=memory; +insert into t1 values (1, repeat('T', 30000)), (2, repeat('U', 30000)); +select count(*) from t1; +count(*) +2 +truncate table t1; +select count(*) from t1; +count(*) +0 +insert into t1 values (1, 'after truncate'); +select * from t1; +a b +1 after truncate +drop table t1; +# +# Full table scan correctness +# +create table t1 (a int not null, b blob) engine=memory; +insert into t1 values (1, repeat('a', 500)); +insert into t1 values (2, repeat('b', 23000)); +insert into t1 values (3, repeat('c', 51000)); +insert into t1 values (4, NULL); +insert into t1 values (5, ''); +select a, length(b), left(b, 5) from t1 order by a; +a length(b) left(b, 5) +1 500 aaaaa +2 23000 bbbbb +3 51000 ccccc +4 NULL NULL +5 0 +select count(*) from t1; +count(*) +5 +drop table t1; +# +# Hash index on non-blob column with blob data present +# +create table t1 ( +a int not null, +b varchar(20) not null, +c blob, +primary key(a), +key(b) +) engine=memory; +insert into t1 values (1, 'key1', repeat('h', 20000)); +insert into t1 values (2, 'key2', repeat('i', 33000)); +insert into t1 values (3, 'key1', repeat('j', 10000)); +select a, b, length(c) from t1 where b='key1' order by a; +a b length(c) +1 key1 20000 +3 key1 10000 +select a, b, length(c) from t1 where b='key2'; +a b length(c) +2 key2 33000 +select a, b, length(c) from t1 where a=2; +a b length(c) +2 key2 33000 +drop table t1; +# +# BTREE index on non-blob column with blob data present +# +create table t1 ( +a int not null, +b int not null, +c blob, +key b_idx using btree (b) +) engine=memory; +insert into t1 values (1, 10, repeat('p', 17000)); +insert into t1 values (2, 20, repeat('q', 25000)); +insert into t1 values (3, 30, repeat('r', 41000)); +insert into t1 values (4, 20, repeat('s', 19000)); +select a, b, length(c) from t1 where b=20 order by a; +a b length(c) +2 20 25000 +4 20 19000 +select a, b, length(c) from t1 where b>=20 order by b, a; +a b length(c) +2 20 25000 +4 20 19000 +3 30 41000 +drop table t1; +# +# REPLACE with BLOB column +# +create table t1 (a int not null, b blob, primary key(a)) engine=memory; +insert into t1 values (1, 'original'); +insert into t1 values (2, repeat('x', 30000)); +replace into t1 values (1, repeat('replaced', 5000)); +select a, length(b), left(b, 20) from t1 order by a; +a length(b) left(b, 20) +1 40000 replacedreplacedrepl +2 30000 xxxxxxxxxxxxxxxxxxxx +replace into t1 values (2, 'short'); +select a, length(b), left(b, 20) from t1 order by a; +a length(b) left(b, 20) +1 40000 replacedreplacedrepl +2 5 short +drop table t1; +# +# INSERT ... SELECT with BLOB data +# +create table t1 (a int not null, b blob, primary key(a)) engine=memory; +create table t2 (a int not null, b blob, primary key(a)) engine=memory; +insert into t1 values (1, repeat('m', 22000)), (2, repeat('n', 37000)); +insert into t2 select * from t1; +select a, length(b) from t2 order by a; +a length(b) +1 22000 +2 37000 +select a from t2 where b=repeat('m', 22000); +a +1 +select a from t2 where b=repeat('n', 37000); +a +2 +drop table t1, t2; +# +# TINYBLOB NOT NULL edge case (reclength=9, minimal visible_offset) +# +CREATE TABLE t_tiny (b TINYBLOB NOT NULL) ENGINE=MEMORY; +INSERT INTO t_tiny VALUES ('hello'), ('world'); +SELECT * FROM t_tiny; +b +hello +world +DROP TABLE t_tiny; +# +# TINYBLOB NULL edge case (reclength=10) +# +CREATE TABLE t_tiny2 (b TINYBLOB) ENGINE=MEMORY; +INSERT INTO t_tiny2 VALUES ('foo'), ('bar'); +SELECT * FROM t_tiny2; +b +foo +bar +DROP TABLE t_tiny2; +# +# Blob-only table with no primary key +# +create table t1 (b blob) engine=memory; +insert into t1 values (repeat('A', 5000)), (repeat('B', 10000)); +select length(b), left(b, 3) from t1 order by b; +length(b) left(b, 3) +5000 AAA +10000 BBB +Warnings: +Warning 4202 2 values were longer than max_sort_length. Sorting used only the first 1024 bytes +delete from t1; +insert into t1 values ('short1'), ('short2'); +select b from t1 order by b; +b +short1 +short2 +drop table t1; +# +# Table-full error with blob data (no partial rows from failed inserts) +# +set @save_max= @@max_heap_table_size; +set @@max_heap_table_size= 65536; +create table t1 (a int not null, b blob, primary key(a)) engine=memory; +insert into t1 values (1, repeat('x', 30000)); +insert into t1 values (2, repeat('y', 30000)); +ERROR HY000: The table 't1' is full +insert into t1 values (3, repeat('z', 30000)); +ERROR HY000: The table 't1' is full +select count(*) as row_count from t1; +row_count +1 +select a, length(b) from t1 where a=1; +a length(b) +1 30000 +select count(*) as corrupted from t1 +where b is not null and length(b) > 0 +and b != repeat(left(b, 1), length(b)); +corrupted +0 +select count(*) as scan_count from t1; +scan_count +1 +set @@max_heap_table_size= @save_max; +drop table t1; +# +# Multiple blob columns with different sizes in same row +# +create table t1 ( +a int not null, +b tinyblob, +c blob, +d mediumblob, +primary key(a) +) engine=memory; +insert into t1 values (1, repeat('p', 200), repeat('q', 30000), repeat('r', 60000)); +insert into t1 values (2, 'small', repeat('s', 15000), repeat('t', 45000)); +select a, length(b), length(c), length(d), left(b,3), left(c,3), left(d,3) from t1 order by a; +a length(b) length(c) length(d) left(b,3) left(c,3) left(d,3) +1 200 30000 60000 ppp qqq rrr +2 5 15000 45000 sma sss ttt +select a from t1 where b=repeat('p', 200) and c=repeat('q', 30000) and d=repeat('r', 60000); +a +1 +select a from t1 where b='small' and c=repeat('s', 15000) and d=repeat('t', 45000); +a +2 +drop table t1; +# +# UPDATE failure preserves old blob data (table-full during blob grow) +# +set @save_max= @@max_heap_table_size; +set @@max_heap_table_size= 65536; +create table t1 (a int not null, b longblob, primary key(a)) engine=memory; +insert into t1 values (1, repeat('A', 5000)); +insert into t1 values (2, repeat('B', 5000)); +update t1 set b=repeat('X', 200000) where a=1; +ERROR HY000: The table 't1' is full +select a, length(b), left(b, 5), right(b, 5) from t1 order by a; +a length(b) left(b, 5) right(b, 5) +1 5000 AAAAA AAAAA +2 5000 BBBBB BBBBB +select a from t1 where b=repeat('A', 5000); +a +1 +select a from t1 where b=repeat('B', 5000); +a +2 +set @@max_heap_table_size= @save_max; +drop table t1; +# +# UPDATE of non-blob column preserves unchanged blob chains +# +create table t1 (a int not null, b blob, c blob, d int, primary key(a)) engine=memory; +insert into t1 values (1, repeat('X', 5000), repeat('Y', 3000), 10); +insert into t1 values (2, repeat('A', 200), repeat('B', 400), 20); +update t1 set d=99 where a=1; +select a, length(b), left(b,3), length(c), left(c,3), d from t1 order by a; +a length(b) left(b,3) length(c) left(c,3) d +1 5000 XXX 3000 YYY 99 +2 200 AAA 400 BBB 20 +select a from t1 where b=repeat('X', 5000) and c=repeat('Y', 3000); +a +1 +update t1 set b='changed' where a=1; +select a, length(b), b, length(c), left(c,3), d from t1 order by a; +a length(b) b length(c) left(c,3) d +1 7 changed 3000 YYY 99 +2 200 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA 400 BBB 20 +select a from t1 where c=repeat('Y', 3000); +a +1 +update t1 set b=repeat('A', 200) where a=2; +select a, length(b), left(b,3), length(c), left(c,3), d from t1 order by a; +a length(b) left(b,3) length(c) left(c,3) d +1 7 cha 3000 YYY 99 +2 200 AAA 400 BBB 20 +select a from t1 where b=repeat('A', 200) and c=repeat('B', 400); +a +2 +check table t1; +Table Op Msg_type Msg_text +test.t1 check note The storage engine for the table doesn't support check +drop table t1; +# +# Large blob exceeding uint16 run_rec_count cap (65535 records) +# +# With recbuffer=16, visible=15, a 1MB blob needs ~69906 records, +# exceeding the uint16 max of 65535. The free list scavenging must +# split into multiple runs at the cap boundary. +# Delete-then-reinsert exercises scavenging of the freed chain. +# +set @save_max= @@max_heap_table_size; +set @@max_heap_table_size= 64 * 1024 * 1024; +create table t1 (a int not null, b longblob, primary key(a)) engine=memory; +insert into t1 values (1, repeat('A', 1048576)); +insert into t1 values (2, repeat('B', 1048576)); +select a, length(b), left(b, 5), right(b, 5) from t1 order by a; +a length(b) left(b, 5) right(b, 5) +1 1048576 AAAAA AAAAA +2 1048576 BBBBB BBBBB +select a from t1 where b = repeat('A', 1048576); +a +1 +select a from t1 where b = repeat('B', 1048576); +a +2 +delete from t1 where a=1; +delete from t1 where a=2; +insert into t1 values (3, repeat('C', 1048576)); +insert into t1 values (4, repeat('D', 1048576)); +select a, length(b), left(b, 5), right(b, 5) from t1 order by a; +a length(b) left(b, 5) right(b, 5) +3 1048576 CCCCC CCCCC +4 1048576 DDDDD DDDDD +select a from t1 where b = repeat('C', 1048576); +a +3 +select a from t1 where b = repeat('D', 1048576); +a +4 +set @@max_heap_table_size= @save_max; +drop table t1; +# +# Zero-copy Case A: tiny blobs fitting in rec 0 payload (wide table) +# +create table t_casea (a int not null, b varchar(480), c blob, primary key(a)) engine=memory; +insert into t_casea values (1, repeat('v', 480), repeat('A', 400)); +insert into t_casea values (2, repeat('w', 480), repeat('B', 100)); +select a, length(c), left(c,3) from t_casea order by a; +a length(c) left(c,3) +1 400 AAA +2 100 BBB +select a from t_casea where c = repeat('A', 400); +a +1 +select a from t_casea where c = repeat('B', 100); +a +2 +drop table t_casea; +# +# Zero-copy Case B: medium blobs (single run, multiple records) +# +create table t_caseb (a int not null, b blob, primary key(a)) engine=memory; +insert into t_caseb values (1, repeat('M', 8000)); +insert into t_caseb values (2, repeat('N', 15000)); +select a, length(b), left(b,3), right(b,3) from t_caseb order by a; +a length(b) left(b,3) right(b,3) +1 8000 MMM MMM +2 15000 NNN NNN +select a from t_caseb where b = repeat('M', 8000); +a +1 +select a from t_caseb where b = repeat('N', 15000); +a +2 +delete from t_caseb where a=1; +insert into t_caseb values (3, repeat('O', 12000)); +select a, length(b) from t_caseb order by a; +a length(b) +2 15000 +3 12000 +select a from t_caseb where b = repeat('O', 12000); +a +3 +drop table t_caseb; +# +# Zero-copy Case B->C boundary (large blobs forcing multi-run) +# +create table t_boundary (a int not null, b blob, primary key(a)) engine=memory; +insert into t_boundary values (1, repeat('X', 15000)); +insert into t_boundary values (2, repeat('Y', 50000)); +select a, length(b), left(b,3) from t_boundary order by a; +a length(b) left(b,3) +1 15000 XXX +2 50000 YYY +select a from t_boundary where b = repeat('X', 15000); +a +1 +select a from t_boundary where b = repeat('Y', 50000); +a +2 +drop table t_boundary; +# +# Non-blob table regression: ensure no behavioral change +# +create table t1 (a int not null, b varchar(100), primary key(a)) engine=memory; +insert into t1 values (1, 'no blob here'), (2, 'still no blob'); +select * from t1 order by a; +a b +1 no blob here +2 still no blob +update t1 set b='changed' where a=1; +select * from t1 order by a; +a b +1 changed +2 still no blob +delete from t1 where a=2; +select * from t1 order by a; +a b +1 changed +drop table t1; +# +# INSERT ON DUPLICATE KEY UPDATE with blobs +# +create table t1 (a int not null, b blob, c blob, primary key(a)) engine=memory; +insert into t1 values (1, repeat('A', 5000), repeat('B', 3000)); +insert into t1 values (2, repeat('C', 8000), repeat('D', 12000)); +insert into t1 values (1, repeat('X', 10000), repeat('Y', 7000)) +on duplicate key update b=values(b), c=values(c); +select a, length(b), left(b,3), length(c), left(c,3) from t1 order by a; +a length(b) left(b,3) length(c) left(c,3) +1 10000 XXX 7000 YYY +2 8000 CCC 12000 DDD +select a from t1 where b=repeat('X', 10000) and c=repeat('Y', 7000); +a +1 +insert into t1 values (3, repeat('M', 6000), repeat('N', 4000)) +on duplicate key update b=values(b); +select a, length(b), left(b,3) from t1 where a=3; +a length(b) left(b,3) +3 6000 MMM +insert into t1 values (2, NULL, NULL) +on duplicate key update b=NULL, c=NULL; +select a, b, c from t1 where a=2; +a b c +2 NULL NULL +insert into t1 values (2, repeat('R', 15000), repeat('S', 20000)) +on duplicate key update b=values(b), c=values(c); +select a, length(b), length(c) from t1 where a=2; +a length(b) length(c) +2 15000 20000 +select a from t1 where b=repeat('R', 15000) and c=repeat('S', 20000); +a +2 +drop table t1; +# +# JSON column on MEMORY table +# +create table t1 (a int not null, j json, primary key(a)) engine=memory; +insert into t1 values (1, '{"key": "value", "num": 42}'); +insert into t1 values (2, json_array(1, 2, 3, repeat('x', 5000))); +select a, json_value(j, '$.key') as k, json_value(j, '$.num') as n from t1 where a=1; +a k n +1 value 42 +select a, json_value(j, '$[0]') as first_elem, length(j) from t1 where a=2; +a first_elem length(j) +2 1 5013 +update t1 set j=json_object('updated', true, 'data', repeat('y', 8000)) where a=1; +select a, json_value(j, '$.updated') as upd, length(j) from t1 where a=1; +a upd length(j) +1 1 8029 +delete from t1 where a=2; +select count(*) from t1; +count(*) +1 +drop table t1; +# +# LONGBLOB at uint16 run_rec_count split boundary +# +# With recbuffer=16, visible=15, a single run's max payload is +# (15-10) + 65534*16 = 1,048,549 bytes. Blobs near this boundary +# exercise the run-splitting logic at the uint16 cap. +# +set @save_max= @@max_heap_table_size; +set @@max_heap_table_size= 128*1024*1024; +create table t1 (a int not null, b longblob, primary key(a)) engine=memory; +insert into t1 values (1, repeat('A', 1048549)); +select a, length(b), left(b,3), right(b,3) from t1 where a=1; +a length(b) left(b,3) right(b,3) +1 1048549 AAA AAA +select a from t1 where b=repeat('A', 1048549); +a +1 +insert into t1 values (2, repeat('B', 1048550)); +select a, length(b), left(b,3), right(b,3) from t1 where a=2; +a length(b) left(b,3) right(b,3) +2 1048550 BBB BBB +select a from t1 where b=repeat('B', 1048550); +a +2 +insert into t1 values (3, repeat('C', 2097152)); +select a, length(b), left(b,3), right(b,3) from t1 where a=3; +a length(b) left(b,3) right(b,3) +3 2097152 CCC CCC +select a from t1 where b=repeat('C', 2097152); +a +3 +delete from t1; +insert into t1 values (4, repeat('D', 1048550)); +select a from t1 where b=repeat('D', 1048550); +a +4 +set @@max_heap_table_size= @save_max; +drop table t1; +# +# Case A/B/C exact boundary blob sizes +# +# (int, blob) table: recbuffer=16, visible=15, header=10 +# Case A max: 5 bytes (fits in rec 0 payload area) +# Case B approx max: ~16325 bytes (single run, zero-copy) +# Case C: anything larger than one run +# +create table t1 (a int not null, b blob, primary key(a)) engine=memory; +insert into t1 values (1, repeat('A', 5)); +select a, length(b), b from t1 where a=1; +a length(b) b +1 5 AAAAA +select a from t1 where b=repeat('A', 5); +a +1 +insert into t1 values (2, repeat('B', 6)); +select a, length(b), b from t1 where a=2; +a length(b) b +2 6 BBBBBB +select a from t1 where b=repeat('B', 6); +a +2 +insert into t1 values (3, repeat('C', 10000)); +select a, length(b), left(b,3) from t1 where a=3; +a length(b) left(b,3) +3 10000 CCC +select a from t1 where b=repeat('C', 10000); +a +3 +insert into t1 values (4, repeat('D', 50000)); +select a, length(b), left(b,3) from t1 where a=4; +a length(b) left(b,3) +4 50000 DDD +select a from t1 where b=repeat('D', 50000); +a +4 +select a, length(b) from t1 order by a; +a length(b) +1 5 +2 6 +3 10000 +4 50000 +update t1 set b=repeat('E', 30000) where a=1; +select a, length(b), left(b,3) from t1 where a=1; +a length(b) left(b,3) +1 30000 EEE +select a from t1 where b=repeat('E', 30000); +a +1 +update t1 set b='tiny' where a=4; +select a, length(b), b from t1 where a=4; +a length(b) b +4 4 tiny +drop table t1; +# +# BTREE index on blob column is rejected +# +create table t1 (a int, b blob, key b_idx using btree (b)) engine=memory; +ERROR 42000: BLOB column `b` can't be used in key specification in the MEMORY table +# HASH index on explicit blob key is also rejected (user tables) +create table t1 (a int, b blob, key b_idx using hash (b)) engine=memory; +ERROR 42000: BLOB column `b` can't be used in key specification in the MEMORY table +# But BTREE on non-blob column with blob data column works +create table t1 (a int not null, b blob, key a_idx using btree (a)) engine=memory; +insert into t1 values (1, repeat('x', 5000)), (2, repeat('y', 8000)); +select a, length(b) from t1 where a=2; +a length(b) +2 8000 +drop table t1; diff --git a/mysql-test/suite/heap/blob.test b/mysql-test/suite/heap/blob.test new file mode 100644 index 0000000000000..8d0e2308dea58 --- /dev/null +++ b/mysql-test/suite/heap/blob.test @@ -0,0 +1,592 @@ +# +# MDEV-38975: HEAP engine BLOB/TEXT/JSON/GEOMETRY column support +# +# Tests basic CRUD, multiple blob types, large blobs, free list reuse, +# fragmentation, max_heap_table_size overflow, zero-copy edge cases +# (Case A single-rec, Case B zerocopy, Case B->C transition), and +# blob preservation during non-blob column updates. +# + +--disable_warnings +drop table if exists t1,t2; +--enable_warnings + +--echo # +--echo # Basic CRUD with BLOB column +--echo # +create table t1 (a int not null, b blob, primary key(a)) engine=memory; +insert into t1 values (1, 'hello'), (2, 'world'); +select * from t1 order by a; +select * from t1 where a=1; +select * from t1 where a=2; +update t1 set b='updated' where a=1; +select * from t1 order by a; +delete from t1 where a=2; +select * from t1 order by a; +insert into t1 values (3, 'new row'); +select * from t1 order by a; +drop table t1; + +--echo # +--echo # Multiple BLOB/TEXT columns of different types +--echo # +create table t1 ( + id int not null auto_increment, + t tinyblob, + b blob, + m mediumblob, + tx text, + primary key(id) +) engine=memory; +insert into t1 (t, b, m, tx) values + ('tiny1', 'blob1', 'medium1', 'text1'), + ('tiny2', 'blob2', 'medium2', 'text2'); +select * from t1 order by id; +update t1 set b='blob_updated', tx='text_updated' where id=1; +select * from t1 order by id; +delete from t1 where id=2; +select * from t1 order by id; +drop table t1; + +--echo # +--echo # NULL and empty blob values +--echo # +create table t1 (a int not null, b blob, c text, primary key(a)) engine=memory; +insert into t1 values (1, NULL, NULL); +insert into t1 values (2, '', ''); +insert into t1 values (3, 'data', 'text'); +select a, b, c, length(b), length(c) from t1 order by a; +update t1 set b=NULL where a=3; +select a, b, c, length(b), length(c) from t1 order by a; +update t1 set b='restored' where a=3; +select a, b, c, length(b), length(c) from t1 order by a; +drop table t1; + +--echo # +--echo # Large BLOBs spanning multiple continuation runs +--echo # For (int, blob): recbuffer=16, visible=15, leaf block ~1021 slots. +--echo # Max run payload ~15305 bytes. Sizes chosen to span multiple runs +--echo # and not align to visible (15 bytes). +--echo # +create table t1 (a int not null, b blob, primary key(a)) engine=memory; +insert into t1 values (1, repeat('A', 1000)); +insert into t1 values (2, repeat('B', 50000)); +insert into t1 values (3, repeat('C', 63001)); +select a, length(b), left(b, 5), right(b, 5) from t1 order by a; +# Verify data integrity +select a from t1 where b = repeat('A', 1000); +select a from t1 where b = repeat('B', 50000); +select a from t1 where b = repeat('C', 63001); +# Update small to large (multi-run) +update t1 set b=repeat('D', 63001) where a=1; +select a, length(b), left(b, 5), right(b, 5) from t1 order by a; +select a from t1 where b = repeat('D', 63001); +# Update large to small +update t1 set b=repeat('E', 100) where a=2; +select a, length(b), left(b, 5), right(b, 5) from t1 order by a; +drop table t1; + +--echo # +--echo # Mixed operations: insert, delete, insert (free list reuse) +--echo # +create table t1 (a int not null, b blob, primary key(a)) engine=memory; +insert into t1 values (1, repeat('X', 20000)); +insert into t1 values (2, repeat('Y', 50000)); +insert into t1 values (3, repeat('Z', 10000)); +delete from t1 where a=2; +# This insert should reuse freed continuation records +insert into t1 values (4, repeat('W', 40000)); +select a, length(b) from t1 order by a; +select a from t1 where b = repeat('X', 20000); +select a from t1 where b = repeat('Z', 10000); +select a from t1 where b = repeat('W', 40000); +# Delete all and reinsert +delete from t1; +insert into t1 values (10, repeat('R', 50000)); +insert into t1 values (20, repeat('S', 50000)); +select a, length(b) from t1 order by a; +drop table t1; + +--echo # +--echo # Free list fragmentation: NULL-blob rows interleaved with large-blob rows +--echo # +--echo # When rows with NULL blobs and rows with large blobs are deleted, the +--echo # free list gets primary-record slots (from NULL rows) interleaved between +--echo # continuation slots (from large-blob rows). The peek-then-unlink +--echo # algorithm must find contiguous continuation groups despite these +--echo # interleaving primary slots breaking address contiguity. +--echo # +--echo # After deleting all rows and reinserting, the new blob data must be +--echo # correct -- verifying no free list corruption from the fragmented state. +--echo # +create table t1 (a int not null, b blob, primary key(a)) engine=memory; +# Insert alternating: NULL blob, large blob, NULL blob, large blob +# The primary slots for NULL rows will sit between continuation runs +# on the free list after deletion. +insert into t1 values (1, NULL); +insert into t1 values (2, repeat('A', 20000)); +insert into t1 values (3, NULL); +insert into t1 values (4, repeat('B', 30000)); +insert into t1 values (5, NULL); +insert into t1 values (6, repeat('C', 25000)); +select a, length(b) from t1 order by a; +# Delete in an order that creates maximum free list interleaving: +# large blob rows first (their continuation slots go to free list), +# then NULL rows (their primary slots go to free list head, +# interleaving with the continuation slots). +delete from t1 where a=2; +delete from t1 where a=4; +delete from t1 where a=6; +delete from t1 where a=1; +delete from t1 where a=3; +delete from t1 where a=5; +# Reinsert large blobs -- these should either scavenge contiguous groups +# from the fragmented free list or fall through to tail allocation. +# Either way, data must be correct. +insert into t1 values (10, repeat('D', 35000)); +insert into t1 values (20, repeat('E', 20000)); +insert into t1 values (30, repeat('F', 15000)); +select a, length(b) from t1 order by a; +select a from t1 where b = repeat('D', 35000); +select a from t1 where b = repeat('E', 20000); +select a from t1 where b = repeat('F', 15000); +# Second cycle: delete and reinsert again to exercise scavenging of +# the runs we just created (which are now interleaved differently). +delete from t1 where a=10; +insert into t1 values (40, repeat('G', 40000)); +select a, length(b) from t1 order by a; +select a from t1 where b = repeat('E', 20000); +select a from t1 where b = repeat('F', 15000); +select a from t1 where b = repeat('G', 40000); +drop table t1; + +--echo # +--echo # Free list scavenging with mixed NULL and non-NULL blob columns +--echo # +--echo # Multiple blob columns where some are NULL and others are large. +--echo # This creates rows with partial continuation chains -- the NULL +--echo # columns have no chain while the non-NULL columns do. +--echo # +create table t1 ( + a int not null, + b blob, + c blob, + primary key(a) +) engine=memory; +insert into t1 values (1, repeat('X', 15000), NULL); +insert into t1 values (2, NULL, repeat('Y', 25000)); +insert into t1 values (3, repeat('Z', 10000), repeat('W', 20000)); +insert into t1 values (4, NULL, NULL); +select a, length(b), length(c) from t1 order by a; +# Delete rows with different blob patterns to create varied free list state +delete from t1 where a=1; +delete from t1 where a=3; +# Insert new rows that should scavenge from the freed continuation slots +insert into t1 values (5, repeat('P', 18000), repeat('Q', 22000)); +select a, length(b), length(c) from t1 order by a; +select a from t1 where b is null and c = repeat('Y', 25000); +select a from t1 where b = repeat('P', 18000) and c = repeat('Q', 22000); +# Delete everything, reinsert to verify full cleanup +delete from t1; +insert into t1 values (6, repeat('R', 30000), repeat('S', 30000)); +select a, length(b), length(c) from t1 order by a; +select a from t1 where b = repeat('R', 30000) and c = repeat('S', 30000); +drop table t1; + +--echo # +--echo # TRUNCATE with BLOB data +--echo # +create table t1 (a int not null, b blob, primary key(a)) engine=memory; +insert into t1 values (1, repeat('T', 30000)), (2, repeat('U', 30000)); +select count(*) from t1; +truncate table t1; +select count(*) from t1; +insert into t1 values (1, 'after truncate'); +select * from t1; +drop table t1; + +--echo # +--echo # Full table scan correctness +--echo # +create table t1 (a int not null, b blob) engine=memory; +insert into t1 values (1, repeat('a', 500)); +insert into t1 values (2, repeat('b', 23000)); +insert into t1 values (3, repeat('c', 51000)); +insert into t1 values (4, NULL); +insert into t1 values (5, ''); +# Full scan should return exactly 5 rows, no continuation record leaks +select a, length(b), left(b, 5) from t1 order by a; +select count(*) from t1; +drop table t1; + +--echo # +--echo # Hash index on non-blob column with blob data present +--echo # +create table t1 ( + a int not null, + b varchar(20) not null, + c blob, + primary key(a), + key(b) +) engine=memory; +insert into t1 values (1, 'key1', repeat('h', 20000)); +insert into t1 values (2, 'key2', repeat('i', 33000)); +insert into t1 values (3, 'key1', repeat('j', 10000)); +select a, b, length(c) from t1 where b='key1' order by a; +select a, b, length(c) from t1 where b='key2'; +select a, b, length(c) from t1 where a=2; +drop table t1; + +--echo # +--echo # BTREE index on non-blob column with blob data present +--echo # +create table t1 ( + a int not null, + b int not null, + c blob, + key b_idx using btree (b) +) engine=memory; +insert into t1 values (1, 10, repeat('p', 17000)); +insert into t1 values (2, 20, repeat('q', 25000)); +insert into t1 values (3, 30, repeat('r', 41000)); +insert into t1 values (4, 20, repeat('s', 19000)); +select a, b, length(c) from t1 where b=20 order by a; +select a, b, length(c) from t1 where b>=20 order by b, a; +drop table t1; + +--echo # +--echo # REPLACE with BLOB column +--echo # +create table t1 (a int not null, b blob, primary key(a)) engine=memory; +insert into t1 values (1, 'original'); +insert into t1 values (2, repeat('x', 30000)); +replace into t1 values (1, repeat('replaced', 5000)); +select a, length(b), left(b, 20) from t1 order by a; +replace into t1 values (2, 'short'); +select a, length(b), left(b, 20) from t1 order by a; +drop table t1; + +--echo # +--echo # INSERT ... SELECT with BLOB data +--echo # +create table t1 (a int not null, b blob, primary key(a)) engine=memory; +create table t2 (a int not null, b blob, primary key(a)) engine=memory; +insert into t1 values (1, repeat('m', 22000)), (2, repeat('n', 37000)); +insert into t2 select * from t1; +select a, length(b) from t2 order by a; +select a from t2 where b=repeat('m', 22000); +select a from t2 where b=repeat('n', 37000); +drop table t1, t2; + +--echo # +--echo # TINYBLOB NOT NULL edge case (reclength=9, minimal visible_offset) +--echo # +CREATE TABLE t_tiny (b TINYBLOB NOT NULL) ENGINE=MEMORY; +INSERT INTO t_tiny VALUES ('hello'), ('world'); +SELECT * FROM t_tiny; +DROP TABLE t_tiny; + +--echo # +--echo # TINYBLOB NULL edge case (reclength=10) +--echo # +CREATE TABLE t_tiny2 (b TINYBLOB) ENGINE=MEMORY; +INSERT INTO t_tiny2 VALUES ('foo'), ('bar'); +SELECT * FROM t_tiny2; +DROP TABLE t_tiny2; + +--echo # +--echo # Blob-only table with no primary key +--echo # +create table t1 (b blob) engine=memory; +insert into t1 values (repeat('A', 5000)), (repeat('B', 10000)); +select length(b), left(b, 3) from t1 order by b; +delete from t1; +insert into t1 values ('short1'), ('short2'); +select b from t1 order by b; +drop table t1; + +--echo # +--echo # Table-full error with blob data (no partial rows from failed inserts) +--echo # +set @save_max= @@max_heap_table_size; +# Variable must be set before CREATE TABLE (limit is captured at creation). +set @@max_heap_table_size= 65536; +create table t1 (a int not null, b blob, primary key(a)) engine=memory; +# Insert until table is full; blob data consumes continuation slots +--disable_abort_on_error +insert into t1 values (1, repeat('x', 30000)); +insert into t1 values (2, repeat('y', 30000)); +insert into t1 values (3, repeat('z', 30000)); +--enable_abort_on_error +# Verify row count: only successfully inserted rows exist +select count(*) as row_count from t1; +# At least the first row should be readable +select a, length(b) from t1 where a=1; +# Verify no corrupted blob data in any row +select count(*) as corrupted from t1 + where b is not null and length(b) > 0 + and b != repeat(left(b, 1), length(b)); +# Full scan returns correct count (no phantom partial rows) +select count(*) as scan_count from t1; +set @@max_heap_table_size= @save_max; +drop table t1; + +--echo # +--echo # Multiple blob columns with different sizes in same row +--echo # +create table t1 ( + a int not null, + b tinyblob, + c blob, + d mediumblob, + primary key(a) +) engine=memory; +insert into t1 values (1, repeat('p', 200), repeat('q', 30000), repeat('r', 60000)); +insert into t1 values (2, 'small', repeat('s', 15000), repeat('t', 45000)); +select a, length(b), length(c), length(d), left(b,3), left(c,3), left(d,3) from t1 order by a; +# Verify data integrity +select a from t1 where b=repeat('p', 200) and c=repeat('q', 30000) and d=repeat('r', 60000); +select a from t1 where b='small' and c=repeat('s', 15000) and d=repeat('t', 45000); +drop table t1; + +--echo # +--echo # UPDATE failure preserves old blob data (table-full during blob grow) +--echo # +set @save_max= @@max_heap_table_size; +# Size chosen so two rows with small blobs fit, but updating one to a +# large blob exhausts the table before the new chain is fully written. +# Variable must be set before CREATE TABLE (limit is captured at creation). +# Use LONGBLOB so the 200KB value is accepted by the column type. +set @@max_heap_table_size= 65536; +create table t1 (a int not null, b longblob, primary key(a)) engine=memory; +insert into t1 values (1, repeat('A', 5000)); +insert into t1 values (2, repeat('B', 5000)); +# This update should fail: the new blob is too large for the table +--error ER_RECORD_FILE_FULL +update t1 set b=repeat('X', 200000) where a=1; +# Old data must survive intact after the failed update +select a, length(b), left(b, 5), right(b, 5) from t1 order by a; +select a from t1 where b=repeat('A', 5000); +select a from t1 where b=repeat('B', 5000); +set @@max_heap_table_size= @save_max; +drop table t1; + +--echo # +--echo # UPDATE of non-blob column preserves unchanged blob chains +--echo # +create table t1 (a int not null, b blob, c blob, d int, primary key(a)) engine=memory; +insert into t1 values (1, repeat('X', 5000), repeat('Y', 3000), 10); +insert into t1 values (2, repeat('A', 200), repeat('B', 400), 20); +# Update only the non-blob column -- blobs must not be rewritten +update t1 set d=99 where a=1; +select a, length(b), left(b,3), length(c), left(c,3), d from t1 order by a; +select a from t1 where b=repeat('X', 5000) and c=repeat('Y', 3000); +# Update one blob, leave the other unchanged +update t1 set b='changed' where a=1; +select a, length(b), b, length(c), left(c,3), d from t1 order by a; +select a from t1 where c=repeat('Y', 3000); +# SET blob_col = same_value (different pointer, same data -- INSERT ON DUP KEY pattern) +update t1 set b=repeat('A', 200) where a=2; +select a, length(b), left(b,3), length(c), left(c,3), d from t1 order by a; +select a from t1 where b=repeat('A', 200) and c=repeat('B', 400); +check table t1; +drop table t1; + +--echo # +--echo # Large blob exceeding uint16 run_rec_count cap (65535 records) +--echo # +--echo # With recbuffer=16, visible=15, a 1MB blob needs ~69906 records, +--echo # exceeding the uint16 max of 65535. The free list scavenging must +--echo # split into multiple runs at the cap boundary. +--echo # Delete-then-reinsert exercises scavenging of the freed chain. +--echo # +set @save_max= @@max_heap_table_size; +set @@max_heap_table_size= 64 * 1024 * 1024; +create table t1 (a int not null, b longblob, primary key(a)) engine=memory; +insert into t1 values (1, repeat('A', 1048576)); +insert into t1 values (2, repeat('B', 1048576)); +select a, length(b), left(b, 5), right(b, 5) from t1 order by a; +select a from t1 where b = repeat('A', 1048576); +select a from t1 where b = repeat('B', 1048576); +# Delete both rows -- puts ~140K contiguous records on free list +delete from t1 where a=1; +delete from t1 where a=2; +# Reinsert -- scavenges from free list, must split at uint16 boundary +insert into t1 values (3, repeat('C', 1048576)); +insert into t1 values (4, repeat('D', 1048576)); +select a, length(b), left(b, 5), right(b, 5) from t1 order by a; +select a from t1 where b = repeat('C', 1048576); +select a from t1 where b = repeat('D', 1048576); +set @@max_heap_table_size= @save_max; +drop table t1; + +--echo # +--echo # Zero-copy Case A: tiny blobs fitting in rec 0 payload (wide table) +--echo # +create table t_casea (a int not null, b varchar(480), c blob, primary key(a)) engine=memory; +insert into t_casea values (1, repeat('v', 480), repeat('A', 400)); +insert into t_casea values (2, repeat('w', 480), repeat('B', 100)); +select a, length(c), left(c,3) from t_casea order by a; +select a from t_casea where c = repeat('A', 400); +select a from t_casea where c = repeat('B', 100); +drop table t_casea; + +--echo # +--echo # Zero-copy Case B: medium blobs (single run, multiple records) +--echo # +create table t_caseb (a int not null, b blob, primary key(a)) engine=memory; +insert into t_caseb values (1, repeat('M', 8000)); +insert into t_caseb values (2, repeat('N', 15000)); +select a, length(b), left(b,3), right(b,3) from t_caseb order by a; +select a from t_caseb where b = repeat('M', 8000); +select a from t_caseb where b = repeat('N', 15000); +# Delete and reinsert to exercise free list -> tail fallback +delete from t_caseb where a=1; +insert into t_caseb values (3, repeat('O', 12000)); +select a, length(b) from t_caseb order by a; +select a from t_caseb where b = repeat('O', 12000); +drop table t_caseb; + +--echo # +--echo # Zero-copy Case B->C boundary (large blobs forcing multi-run) +--echo # +create table t_boundary (a int not null, b blob, primary key(a)) engine=memory; +# Case B: single run, zero-copy +insert into t_boundary values (1, repeat('X', 15000)); +# Case C: large enough to span multiple leaf blocks +insert into t_boundary values (2, repeat('Y', 50000)); +select a, length(b), left(b,3) from t_boundary order by a; +select a from t_boundary where b = repeat('X', 15000); +select a from t_boundary where b = repeat('Y', 50000); +drop table t_boundary; + +--echo # +--echo # Non-blob table regression: ensure no behavioral change +--echo # +create table t1 (a int not null, b varchar(100), primary key(a)) engine=memory; +insert into t1 values (1, 'no blob here'), (2, 'still no blob'); +select * from t1 order by a; +update t1 set b='changed' where a=1; +select * from t1 order by a; +delete from t1 where a=2; +select * from t1 order by a; +drop table t1; + +--echo # +--echo # INSERT ON DUPLICATE KEY UPDATE with blobs +--echo # +create table t1 (a int not null, b blob, c blob, primary key(a)) engine=memory; +insert into t1 values (1, repeat('A', 5000), repeat('B', 3000)); +insert into t1 values (2, repeat('C', 8000), repeat('D', 12000)); +# IODKU: update both blobs on conflict +insert into t1 values (1, repeat('X', 10000), repeat('Y', 7000)) + on duplicate key update b=values(b), c=values(c); +select a, length(b), left(b,3), length(c), left(c,3) from t1 order by a; +select a from t1 where b=repeat('X', 10000) and c=repeat('Y', 7000); +# IODKU: no conflict (new row) +insert into t1 values (3, repeat('M', 6000), repeat('N', 4000)) + on duplicate key update b=values(b); +select a, length(b), left(b,3) from t1 where a=3; +# IODKU: update blob to NULL +insert into t1 values (2, NULL, NULL) + on duplicate key update b=NULL, c=NULL; +select a, b, c from t1 where a=2; +# IODKU: update NULL to non-NULL blob +insert into t1 values (2, repeat('R', 15000), repeat('S', 20000)) + on duplicate key update b=values(b), c=values(c); +select a, length(b), length(c) from t1 where a=2; +select a from t1 where b=repeat('R', 15000) and c=repeat('S', 20000); +drop table t1; + +--echo # +--echo # JSON column on MEMORY table +--echo # +create table t1 (a int not null, j json, primary key(a)) engine=memory; +insert into t1 values (1, '{"key": "value", "num": 42}'); +insert into t1 values (2, json_array(1, 2, 3, repeat('x', 5000))); +select a, json_value(j, '$.key') as k, json_value(j, '$.num') as n from t1 where a=1; +select a, json_value(j, '$[0]') as first_elem, length(j) from t1 where a=2; +update t1 set j=json_object('updated', true, 'data', repeat('y', 8000)) where a=1; +select a, json_value(j, '$.updated') as upd, length(j) from t1 where a=1; +delete from t1 where a=2; +select count(*) from t1; +drop table t1; + +--echo # +--echo # LONGBLOB at uint16 run_rec_count split boundary +--echo # +--echo # With recbuffer=16, visible=15, a single run's max payload is +--echo # (15-10) + 65534*16 = 1,048,549 bytes. Blobs near this boundary +--echo # exercise the run-splitting logic at the uint16 cap. +--echo # +set @save_max= @@max_heap_table_size; +set @@max_heap_table_size= 128*1024*1024; +create table t1 (a int not null, b longblob, primary key(a)) engine=memory; +# Exactly at single-run max +insert into t1 values (1, repeat('A', 1048549)); +select a, length(b), left(b,3), right(b,3) from t1 where a=1; +select a from t1 where b=repeat('A', 1048549); +# One byte over: forces split into two runs +insert into t1 values (2, repeat('B', 1048550)); +select a, length(b), left(b,3), right(b,3) from t1 where a=2; +select a from t1 where b=repeat('B', 1048550); +# Well over: 2MB blob = ~2 full runs +insert into t1 values (3, repeat('C', 2097152)); +select a, length(b), left(b,3), right(b,3) from t1 where a=3; +select a from t1 where b=repeat('C', 2097152); +delete from t1; +# Delete-and-reinsert at boundary to exercise scavenging split +insert into t1 values (4, repeat('D', 1048550)); +select a from t1 where b=repeat('D', 1048550); +set @@max_heap_table_size= @save_max; +drop table t1; + +--echo # +--echo # Case A/B/C exact boundary blob sizes +--echo # +--echo # (int, blob) table: recbuffer=16, visible=15, header=10 +--echo # Case A max: 5 bytes (fits in rec 0 payload area) +--echo # Case B approx max: ~16325 bytes (single run, zero-copy) +--echo # Case C: anything larger than one run +--echo # +create table t1 (a int not null, b blob, primary key(a)) engine=memory; +# Case A: exactly 5 bytes (max that fits in primary record) +insert into t1 values (1, repeat('A', 5)); +select a, length(b), b from t1 where a=1; +select a from t1 where b=repeat('A', 5); +# Case A: 6 bytes (one over, must use continuation) +insert into t1 values (2, repeat('B', 6)); +select a, length(b), b from t1 where a=2; +select a from t1 where b=repeat('B', 6); +# Case B: medium blob, single run +insert into t1 values (3, repeat('C', 10000)); +select a, length(b), left(b,3) from t1 where a=3; +select a from t1 where b=repeat('C', 10000); +# Case C: large blob, forces multiple runs +insert into t1 values (4, repeat('D', 50000)); +select a, length(b), left(b,3) from t1 where a=4; +select a from t1 where b=repeat('D', 50000); +# Verify all coexist and are intact +select a, length(b) from t1 order by a; +# Update from Case A to Case C +update t1 set b=repeat('E', 30000) where a=1; +select a, length(b), left(b,3) from t1 where a=1; +select a from t1 where b=repeat('E', 30000); +# Update from Case C to Case A +update t1 set b='tiny' where a=4; +select a, length(b), b from t1 where a=4; +drop table t1; + +--echo # +--echo # BTREE index on blob column is rejected +--echo # +--error ER_BLOB_USED_AS_KEY +create table t1 (a int, b blob, key b_idx using btree (b)) engine=memory; +--echo # HASH index on explicit blob key is also rejected (user tables) +--error ER_BLOB_USED_AS_KEY +create table t1 (a int, b blob, key b_idx using hash (b)) engine=memory; +--echo # But BTREE on non-blob column with blob data column works +create table t1 (a int not null, b blob, key a_idx using btree (a)) engine=memory; +insert into t1 values (1, repeat('x', 5000)), (2, repeat('y', 8000)); +select a, length(b) from t1 where a=2; +drop table t1; diff --git a/mysql-test/suite/heap/blob_big.inc b/mysql-test/suite/heap/blob_big.inc new file mode 100644 index 0000000000000..4ea9f4f35420d --- /dev/null +++ b/mysql-test/suite/heap/blob_big.inc @@ -0,0 +1,235 @@ +--source include/have_sequence.inc + +--echo # +--echo # Test create_internal_tmp_table_from_heap() with blob/text columns. +--echo # +--echo # Field_blob_key is used for blob/text columns that appear as key columns +--echo # (GROUP BY, SELECT DISTINCT, UNION DISTINCT) in internal tmp tables +--echo # created by class Create_tmp_table. +--echo # + +# SHOW STATUS output differs across protocol variants; disable them globally. +--disable_cursor_protocol +--disable_ps_protocol +--disable_ps2_protocol + +--echo # +--echo # Setup: 50 distinct text/blob values of ~1500 bytes each. +--echo # Row count: 100 (50 original + 50 duplicate rows for aggregation). +--echo # Blob data in the tmp table: ~75 KB (50 * 1500 bytes), +--echo # which is between 64 K and 1 M. +--echo # + +CREATE TABLE t1 ( + pk INT NOT NULL AUTO_INCREMENT PRIMARY KEY, + a TEXT NOT NULL, + b BLOB, + v INT NOT NULL +); + +# Each value is a 1500-char string uniquely identifying the row. +# LPAD(seq,4,'0') gives '0001'..'0050' so all 50 values are distinct. +INSERT INTO t1 (a, b, v) + SELECT REPEAT(LPAD(seq, 4, '0'), 375), + REPEAT(LPAD(seq, 4, '0'), 375), + seq + FROM seq_1_to_50; + +# Duplicate rows: same a/b values, v offset by 100 (v=101..150). +INSERT INTO t1 (a, b, v) + SELECT a, b, v + 100 FROM t1 ORDER BY pk; + +# Sanity: 100 rows, 50 distinct a values, total v = 1+...+50 + 101+...+150 = 7550 +SELECT COUNT(*), COUNT(DISTINCT a), SUM(v) FROM t1; + +set @save_max_sort_length=@@max_sort_length; +set @save_sort_buffer_size=@@sort_buffer_size; +set @@max_sort_length=65536; +set @@sort_buffer_size=1024*1024*128; + +--echo # +--echo # ================================================================ +--echo # Run 1: tmp_memory_table_size=1M -- tmp table stays in HEAP +--echo # ================================================================ +--echo # + +--echo # --- GROUP BY on TEXT column (end_update path) --- +--echo # Tmp table: 50 groups x ~1500-byte TEXT key = ~75 KB. Fits in 1 M HEAP. +FLUSH STATUS; +SELECT COUNT(*), SUM(gsum) FROM ( + SELECT a, SUM(v) AS gsum FROM t1 GROUP BY a +) dt; +--sorted_result +SELECT VARIABLE_NAME, + IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); + +--echo # --- SELECT DISTINCT on TEXT column (end_write path) --- +FLUSH STATUS; +SELECT COUNT(*) AS cnt_distinct FROM (SELECT DISTINCT a FROM t1) dt; +--sorted_result +SELECT VARIABLE_NAME, + IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); + +--echo # --- SELECT DISTINCT on BLOB column --- +FLUSH STATUS; +SELECT COUNT(*) AS cnt_distinct FROM (SELECT DISTINCT b FROM t1) dt; +--sorted_result +SELECT VARIABLE_NAME, + IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); + +--echo # --- SELECT DISTINCT on TEXT + INT columns --- +FLUSH STATUS; +SELECT COUNT(*) AS cnt_distinct FROM (SELECT DISTINCT a, v FROM t1) dt; +--sorted_result +SELECT VARIABLE_NAME, + IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); + +--echo # --- GROUP BY TEXT column + HAVING --- +FLUSH STATUS; +SELECT COUNT(*) AS groups_with_2rows FROM ( + SELECT a FROM t1 GROUP BY a HAVING COUNT(*) = 2 +) dt; +--sorted_result +SELECT VARIABLE_NAME, + IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); + +--echo # --- GROUP BY on TEXT + BLOB columns (two blob key columns) --- +FLUSH STATUS; +SELECT COUNT(*) AS groups FROM ( + SELECT a, b, COUNT(*) AS cnt FROM t1 GROUP BY a, b +) dt; +--sorted_result +SELECT VARIABLE_NAME, + IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); + +--echo # --- GROUP BY TEXT WITH ROLLUP --- +--echo # 50 groups + 1 NULL summary row = 51 +FLUSH STATUS; +SELECT COUNT(*) AS rows_incl_rollup FROM ( + SELECT a, SUM(v) AS sv FROM t1 GROUP BY a WITH ROLLUP +) dt; +--sorted_result +SELECT VARIABLE_NAME, + IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); + +--echo # --- UNION DISTINCT on TEXT column (end_write path) --- +--echo # Both halves share the same 50 distinct 'a' values -> 50 rows after UNION. +FLUSH STATUS; +SELECT COUNT(*) AS union_rows FROM ( + SELECT a FROM t1 WHERE v <= 50 + UNION + SELECT a FROM t1 WHERE v > 100 +) dt; +--sorted_result +SELECT VARIABLE_NAME, + IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); + +--echo # --- UNION DISTINCT: same table both sides (all 50 distinct values) --- +FLUSH STATUS; +SELECT COUNT(*) AS distinct_rows FROM ( + SELECT a FROM t1 UNION SELECT a FROM t1 +) dt; +--sorted_result +SELECT VARIABLE_NAME, + IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); + +--echo # --- GROUP_CONCAT(DISTINCT text_col) --- +--echo # GROUP_CONCAT uses blob_storage for blob values; the HEAP key part is +--echo # tiny (12 bytes/row). Expect Created_tmp_disk_tables=0 when HEAP is used. +FLUSH STATUS; +SELECT LENGTH(GROUP_CONCAT(DISTINCT LEFT(a, 4) ORDER BY a)) AS gc_len FROM t1; +--sorted_result +SELECT VARIABLE_NAME, + IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); + +--echo # --- GROUP_CONCAT(text ORDER BY ...) -- blob_storage path, no DISTINCT +--echo # No HEAP key table is needed; blob data goes to blob_storage (MEM_ROOT). + +FLUSH STATUS; +SELECT LENGTH(GROUP_CONCAT(LEFT(a, 4) ORDER BY v)) AS gc_len FROM t1; +--sorted_result +SELECT VARIABLE_NAME, + IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); + +--echo # --- ORDER BY on TEXT column (DISTINCT subquery + ORDER BY) --- +FLUSH STATUS; +SELECT COUNT(*) AS cnt FROM ( + SELECT DISTINCT a FROM t1 ORDER BY a +) dt; +--sorted_result +SELECT VARIABLE_NAME, + IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); + +--echo # --- GROUP BY with aggregate ORDER BY (two-pass: group then sort) --- +FLUSH STATUS; +SELECT COUNT(*), SUM(gv) FROM ( + SELECT a, SUM(v) AS gv FROM t1 GROUP BY a ORDER BY SUM(v) +) dt; +--sorted_result +SELECT VARIABLE_NAME, + IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); + +DROP TABLE t1; + +--echo # +--echo # Test with geometry fields +--echo # + +CREATE TABLE t1 ( + pk INT NOT NULL AUTO_INCREMENT PRIMARY KEY, + g GEOMETRY, + v INT NOT NULL +); + +INSERT INTO t1 (g, v) + SELECT PolygonFromText(concat('POLYGON((10 10,20 10,20 20,10 20,10 10, 10 10,20 10,20 20,10 20,10 10, 10 10,20 10,20 20,10 20,10 10,', seq, " ", seq, ", ", seq+10, " ", seq+10, ", 10 10))")),seq from seq_1_to_200; + +# Duplicate rows: same g values, v offset by 400 (v=401..800250). +INSERT INTO t1 (g, v) + SELECT g, v + 200 FROM t1 ORDER BY pk; + +FLUSH STATUS; +SELECT COUNT(*), COUNT(DISTINCT g), SUM(v) FROM t1; +SELECT LENGTH(GROUP_CONCAT(g ORDER BY v)) AS gc_len FROM t1; +--sorted_result +SELECT VARIABLE_NAME, + IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); + +DROP TABLE t1; + +--echo # Clean up + +set @@max_sort_length=@save_max_sort_length; +set @@sort_buffer_size=@save_sort_buffer_size; + +--enable_ps2_protocol +--enable_ps_protocol +--enable_cursor_protocol diff --git a/mysql-test/suite/heap/blob_big1.result b/mysql-test/suite/heap/blob_big1.result new file mode 100644 index 0000000000000..72c7174d53021 --- /dev/null +++ b/mysql-test/suite/heap/blob_big1.result @@ -0,0 +1,253 @@ +# +# Test create_internal_tmp_table_from_heap() with blob/text columns. +# +SET @@tmp_memory_table_size = 1024*1024; +# +# Test create_internal_tmp_table_from_heap() with blob/text columns. +# +# Field_blob_key is used for blob/text columns that appear as key columns +# (GROUP BY, SELECT DISTINCT, UNION DISTINCT) in internal tmp tables +# created by class Create_tmp_table. +# +# +# Setup: 50 distinct text/blob values of ~1500 bytes each. +# Row count: 100 (50 original + 50 duplicate rows for aggregation). +# Blob data in the tmp table: ~75 KB (50 * 1500 bytes), +# which is between 64 K and 1 M. +# +CREATE TABLE t1 ( +pk INT NOT NULL AUTO_INCREMENT PRIMARY KEY, +a TEXT NOT NULL, +b BLOB, +v INT NOT NULL +); +INSERT INTO t1 (a, b, v) +SELECT REPEAT(LPAD(seq, 4, '0'), 375), +REPEAT(LPAD(seq, 4, '0'), 375), +seq +FROM seq_1_to_50; +INSERT INTO t1 (a, b, v) +SELECT a, b, v + 100 FROM t1 ORDER BY pk; +SELECT COUNT(*), COUNT(DISTINCT a), SUM(v) FROM t1; +COUNT(*) COUNT(DISTINCT a) SUM(v) +100 50 7550 +set @save_max_sort_length=@@max_sort_length; +set @save_sort_buffer_size=@@sort_buffer_size; +set @@max_sort_length=65536; +set @@sort_buffer_size=1024*1024*128; +# +# ================================================================ +# Run 1: tmp_memory_table_size=1M -- tmp table stays in HEAP +# ================================================================ +# +# --- GROUP BY on TEXT column (end_update path) --- +# Tmp table: 50 groups x ~1500-byte TEXT key = ~75 KB. Fits in 1 M HEAP. +FLUSH STATUS; +SELECT COUNT(*), SUM(gsum) FROM ( +SELECT a, SUM(v) AS gsum FROM t1 GROUP BY a +) dt; +COUNT(*) SUM(gsum) +50 7550 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES OFF +CREATED_TMP_TABLES ON +# --- SELECT DISTINCT on TEXT column (end_write path) --- +FLUSH STATUS; +SELECT COUNT(*) AS cnt_distinct FROM (SELECT DISTINCT a FROM t1) dt; +cnt_distinct +50 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES OFF +CREATED_TMP_TABLES ON +# --- SELECT DISTINCT on BLOB column --- +FLUSH STATUS; +SELECT COUNT(*) AS cnt_distinct FROM (SELECT DISTINCT b FROM t1) dt; +cnt_distinct +50 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES OFF +CREATED_TMP_TABLES ON +# --- SELECT DISTINCT on TEXT + INT columns --- +FLUSH STATUS; +SELECT COUNT(*) AS cnt_distinct FROM (SELECT DISTINCT a, v FROM t1) dt; +cnt_distinct +100 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES OFF +CREATED_TMP_TABLES ON +# --- GROUP BY TEXT column + HAVING --- +FLUSH STATUS; +SELECT COUNT(*) AS groups_with_2rows FROM ( +SELECT a FROM t1 GROUP BY a HAVING COUNT(*) = 2 +) dt; +groups_with_2rows +50 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES OFF +CREATED_TMP_TABLES ON +# --- GROUP BY on TEXT + BLOB columns (two blob key columns) --- +FLUSH STATUS; +SELECT COUNT(*) AS groups FROM ( +SELECT a, b, COUNT(*) AS cnt FROM t1 GROUP BY a, b +) dt; +groups +50 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES OFF +CREATED_TMP_TABLES ON +# --- GROUP BY TEXT WITH ROLLUP --- +# 50 groups + 1 NULL summary row = 51 +FLUSH STATUS; +SELECT COUNT(*) AS rows_incl_rollup FROM ( +SELECT a, SUM(v) AS sv FROM t1 GROUP BY a WITH ROLLUP +) dt; +rows_incl_rollup +51 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES OFF +CREATED_TMP_TABLES ON +# --- UNION DISTINCT on TEXT column (end_write path) --- +# Both halves share the same 50 distinct 'a' values -> 50 rows after UNION. +FLUSH STATUS; +SELECT COUNT(*) AS union_rows FROM ( +SELECT a FROM t1 WHERE v <= 50 +UNION +SELECT a FROM t1 WHERE v > 100 +) dt; +union_rows +50 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES OFF +CREATED_TMP_TABLES ON +# --- UNION DISTINCT: same table both sides (all 50 distinct values) --- +FLUSH STATUS; +SELECT COUNT(*) AS distinct_rows FROM ( +SELECT a FROM t1 UNION SELECT a FROM t1 +) dt; +distinct_rows +50 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES OFF +CREATED_TMP_TABLES ON +# --- GROUP_CONCAT(DISTINCT text_col) --- +# GROUP_CONCAT uses blob_storage for blob values; the HEAP key part is +# tiny (12 bytes/row). Expect Created_tmp_disk_tables=0 when HEAP is used. +FLUSH STATUS; +SELECT LENGTH(GROUP_CONCAT(DISTINCT LEFT(a, 4) ORDER BY a)) AS gc_len FROM t1; +gc_len +249 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES OFF +CREATED_TMP_TABLES ON +# --- GROUP_CONCAT(text ORDER BY ...) -- blob_storage path, no DISTINCT +# No HEAP key table is needed; blob data goes to blob_storage (MEM_ROOT). +FLUSH STATUS; +SELECT LENGTH(GROUP_CONCAT(LEFT(a, 4) ORDER BY v)) AS gc_len FROM t1; +gc_len +499 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES OFF +CREATED_TMP_TABLES ON +# --- ORDER BY on TEXT column (DISTINCT subquery + ORDER BY) --- +FLUSH STATUS; +SELECT COUNT(*) AS cnt FROM ( +SELECT DISTINCT a FROM t1 ORDER BY a +) dt; +cnt +50 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES OFF +CREATED_TMP_TABLES ON +# --- GROUP BY with aggregate ORDER BY (two-pass: group then sort) --- +FLUSH STATUS; +SELECT COUNT(*), SUM(gv) FROM ( +SELECT a, SUM(v) AS gv FROM t1 GROUP BY a ORDER BY SUM(v) +) dt; +COUNT(*) SUM(gv) +50 7550 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES OFF +CREATED_TMP_TABLES ON +DROP TABLE t1; +# +# Test with geometry fields +# +CREATE TABLE t1 ( +pk INT NOT NULL AUTO_INCREMENT PRIMARY KEY, +g GEOMETRY, +v INT NOT NULL +); +INSERT INTO t1 (g, v) +SELECT PolygonFromText(concat('POLYGON((10 10,20 10,20 20,10 20,10 10, 10 10,20 10,20 20,10 20,10 10, 10 10,20 10,20 20,10 20,10 10,', seq, " ", seq, ", ", seq+10, " ", seq+10, ", 10 10))")),seq from seq_1_to_200; +INSERT INTO t1 (g, v) +SELECT g, v + 200 FROM t1 ORDER BY pk; +FLUSH STATUS; +SELECT COUNT(*), COUNT(DISTINCT g), SUM(v) FROM t1; +COUNT(*) COUNT(DISTINCT g) SUM(v) +400 200 80200 +SELECT LENGTH(GROUP_CONCAT(g ORDER BY v)) AS gc_len FROM t1; +gc_len +122399 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES OFF +CREATED_TMP_TABLES ON +DROP TABLE t1; +# Clean up +set @@max_sort_length=@save_max_sort_length; +set @@sort_buffer_size=@save_sort_buffer_size; diff --git a/mysql-test/suite/heap/blob_big1.test b/mysql-test/suite/heap/blob_big1.test new file mode 100644 index 0000000000000..2ca2815c900a8 --- /dev/null +++ b/mysql-test/suite/heap/blob_big1.test @@ -0,0 +1,8 @@ +--echo # +--echo # Test create_internal_tmp_table_from_heap() with blob/text columns. +--echo # + +# Test with all data fitting in memory tables + +SET @@tmp_memory_table_size = 1024*1024; +--source blob_big.inc diff --git a/mysql-test/suite/heap/blob_big2.result b/mysql-test/suite/heap/blob_big2.result new file mode 100644 index 0000000000000..9297ae2ed1f00 --- /dev/null +++ b/mysql-test/suite/heap/blob_big2.result @@ -0,0 +1,253 @@ +# +# Test create_internal_tmp_table_from_heap() with blob/text columns. +# +SET @@tmp_memory_table_size = 32768; +# +# Test create_internal_tmp_table_from_heap() with blob/text columns. +# +# Field_blob_key is used for blob/text columns that appear as key columns +# (GROUP BY, SELECT DISTINCT, UNION DISTINCT) in internal tmp tables +# created by class Create_tmp_table. +# +# +# Setup: 50 distinct text/blob values of ~1500 bytes each. +# Row count: 100 (50 original + 50 duplicate rows for aggregation). +# Blob data in the tmp table: ~75 KB (50 * 1500 bytes), +# which is between 64 K and 1 M. +# +CREATE TABLE t1 ( +pk INT NOT NULL AUTO_INCREMENT PRIMARY KEY, +a TEXT NOT NULL, +b BLOB, +v INT NOT NULL +); +INSERT INTO t1 (a, b, v) +SELECT REPEAT(LPAD(seq, 4, '0'), 375), +REPEAT(LPAD(seq, 4, '0'), 375), +seq +FROM seq_1_to_50; +INSERT INTO t1 (a, b, v) +SELECT a, b, v + 100 FROM t1 ORDER BY pk; +SELECT COUNT(*), COUNT(DISTINCT a), SUM(v) FROM t1; +COUNT(*) COUNT(DISTINCT a) SUM(v) +100 50 7550 +set @save_max_sort_length=@@max_sort_length; +set @save_sort_buffer_size=@@sort_buffer_size; +set @@max_sort_length=65536; +set @@sort_buffer_size=1024*1024*128; +# +# ================================================================ +# Run 1: tmp_memory_table_size=1M -- tmp table stays in HEAP +# ================================================================ +# +# --- GROUP BY on TEXT column (end_update path) --- +# Tmp table: 50 groups x ~1500-byte TEXT key = ~75 KB. Fits in 1 M HEAP. +FLUSH STATUS; +SELECT COUNT(*), SUM(gsum) FROM ( +SELECT a, SUM(v) AS gsum FROM t1 GROUP BY a +) dt; +COUNT(*) SUM(gsum) +50 7550 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES ON +CREATED_TMP_TABLES ON +# --- SELECT DISTINCT on TEXT column (end_write path) --- +FLUSH STATUS; +SELECT COUNT(*) AS cnt_distinct FROM (SELECT DISTINCT a FROM t1) dt; +cnt_distinct +50 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES ON +CREATED_TMP_TABLES ON +# --- SELECT DISTINCT on BLOB column --- +FLUSH STATUS; +SELECT COUNT(*) AS cnt_distinct FROM (SELECT DISTINCT b FROM t1) dt; +cnt_distinct +50 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES ON +CREATED_TMP_TABLES ON +# --- SELECT DISTINCT on TEXT + INT columns --- +FLUSH STATUS; +SELECT COUNT(*) AS cnt_distinct FROM (SELECT DISTINCT a, v FROM t1) dt; +cnt_distinct +100 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES ON +CREATED_TMP_TABLES ON +# --- GROUP BY TEXT column + HAVING --- +FLUSH STATUS; +SELECT COUNT(*) AS groups_with_2rows FROM ( +SELECT a FROM t1 GROUP BY a HAVING COUNT(*) = 2 +) dt; +groups_with_2rows +50 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES ON +CREATED_TMP_TABLES ON +# --- GROUP BY on TEXT + BLOB columns (two blob key columns) --- +FLUSH STATUS; +SELECT COUNT(*) AS groups FROM ( +SELECT a, b, COUNT(*) AS cnt FROM t1 GROUP BY a, b +) dt; +groups +50 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES ON +CREATED_TMP_TABLES ON +# --- GROUP BY TEXT WITH ROLLUP --- +# 50 groups + 1 NULL summary row = 51 +FLUSH STATUS; +SELECT COUNT(*) AS rows_incl_rollup FROM ( +SELECT a, SUM(v) AS sv FROM t1 GROUP BY a WITH ROLLUP +) dt; +rows_incl_rollup +51 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES ON +CREATED_TMP_TABLES ON +# --- UNION DISTINCT on TEXT column (end_write path) --- +# Both halves share the same 50 distinct 'a' values -> 50 rows after UNION. +FLUSH STATUS; +SELECT COUNT(*) AS union_rows FROM ( +SELECT a FROM t1 WHERE v <= 50 +UNION +SELECT a FROM t1 WHERE v > 100 +) dt; +union_rows +50 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES ON +CREATED_TMP_TABLES ON +# --- UNION DISTINCT: same table both sides (all 50 distinct values) --- +FLUSH STATUS; +SELECT COUNT(*) AS distinct_rows FROM ( +SELECT a FROM t1 UNION SELECT a FROM t1 +) dt; +distinct_rows +50 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES ON +CREATED_TMP_TABLES ON +# --- GROUP_CONCAT(DISTINCT text_col) --- +# GROUP_CONCAT uses blob_storage for blob values; the HEAP key part is +# tiny (12 bytes/row). Expect Created_tmp_disk_tables=0 when HEAP is used. +FLUSH STATUS; +SELECT LENGTH(GROUP_CONCAT(DISTINCT LEFT(a, 4) ORDER BY a)) AS gc_len FROM t1; +gc_len +249 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES OFF +CREATED_TMP_TABLES ON +# --- GROUP_CONCAT(text ORDER BY ...) -- blob_storage path, no DISTINCT +# No HEAP key table is needed; blob data goes to blob_storage (MEM_ROOT). +FLUSH STATUS; +SELECT LENGTH(GROUP_CONCAT(LEFT(a, 4) ORDER BY v)) AS gc_len FROM t1; +gc_len +499 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES OFF +CREATED_TMP_TABLES ON +# --- ORDER BY on TEXT column (DISTINCT subquery + ORDER BY) --- +FLUSH STATUS; +SELECT COUNT(*) AS cnt FROM ( +SELECT DISTINCT a FROM t1 ORDER BY a +) dt; +cnt +50 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES ON +CREATED_TMP_TABLES ON +# --- GROUP BY with aggregate ORDER BY (two-pass: group then sort) --- +FLUSH STATUS; +SELECT COUNT(*), SUM(gv) FROM ( +SELECT a, SUM(v) AS gv FROM t1 GROUP BY a ORDER BY SUM(v) +) dt; +COUNT(*) SUM(gv) +50 7550 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES ON +CREATED_TMP_TABLES ON +DROP TABLE t1; +# +# Test with geometry fields +# +CREATE TABLE t1 ( +pk INT NOT NULL AUTO_INCREMENT PRIMARY KEY, +g GEOMETRY, +v INT NOT NULL +); +INSERT INTO t1 (g, v) +SELECT PolygonFromText(concat('POLYGON((10 10,20 10,20 20,10 20,10 10, 10 10,20 10,20 20,10 20,10 10, 10 10,20 10,20 20,10 20,10 10,', seq, " ", seq, ", ", seq+10, " ", seq+10, ", 10 10))")),seq from seq_1_to_200; +INSERT INTO t1 (g, v) +SELECT g, v + 200 FROM t1 ORDER BY pk; +FLUSH STATUS; +SELECT COUNT(*), COUNT(DISTINCT g), SUM(v) FROM t1; +COUNT(*) COUNT(DISTINCT g) SUM(v) +400 200 80200 +SELECT LENGTH(GROUP_CONCAT(g ORDER BY v)) AS gc_len FROM t1; +gc_len +122399 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES ON +CREATED_TMP_TABLES ON +DROP TABLE t1; +# Clean up +set @@max_sort_length=@save_max_sort_length; +set @@sort_buffer_size=@save_sort_buffer_size; diff --git a/mysql-test/suite/heap/blob_big2.test b/mysql-test/suite/heap/blob_big2.test new file mode 100644 index 0000000000000..ebed9e8c04682 --- /dev/null +++ b/mysql-test/suite/heap/blob_big2.test @@ -0,0 +1,9 @@ +--echo # +--echo # Test create_internal_tmp_table_from_heap() with blob/text columns. +--echo # + +# Test with forcing conversion of temporary table to Aria + +SET @@tmp_memory_table_size = 32768; +--source blob_big.inc + diff --git a/mysql-test/suite/heap/blob_big3.result b/mysql-test/suite/heap/blob_big3.result new file mode 100644 index 0000000000000..1a06e2f78c5bc --- /dev/null +++ b/mysql-test/suite/heap/blob_big3.result @@ -0,0 +1,253 @@ +# +# Test create_internal_tmp_table_from_heap() with blob/text columns. +# +SET @@tmp_memory_table_size= 0; +# +# Test create_internal_tmp_table_from_heap() with blob/text columns. +# +# Field_blob_key is used for blob/text columns that appear as key columns +# (GROUP BY, SELECT DISTINCT, UNION DISTINCT) in internal tmp tables +# created by class Create_tmp_table. +# +# +# Setup: 50 distinct text/blob values of ~1500 bytes each. +# Row count: 100 (50 original + 50 duplicate rows for aggregation). +# Blob data in the tmp table: ~75 KB (50 * 1500 bytes), +# which is between 64 K and 1 M. +# +CREATE TABLE t1 ( +pk INT NOT NULL AUTO_INCREMENT PRIMARY KEY, +a TEXT NOT NULL, +b BLOB, +v INT NOT NULL +); +INSERT INTO t1 (a, b, v) +SELECT REPEAT(LPAD(seq, 4, '0'), 375), +REPEAT(LPAD(seq, 4, '0'), 375), +seq +FROM seq_1_to_50; +INSERT INTO t1 (a, b, v) +SELECT a, b, v + 100 FROM t1 ORDER BY pk; +SELECT COUNT(*), COUNT(DISTINCT a), SUM(v) FROM t1; +COUNT(*) COUNT(DISTINCT a) SUM(v) +100 50 7550 +set @save_max_sort_length=@@max_sort_length; +set @save_sort_buffer_size=@@sort_buffer_size; +set @@max_sort_length=65536; +set @@sort_buffer_size=1024*1024*128; +# +# ================================================================ +# Run 1: tmp_memory_table_size=1M -- tmp table stays in HEAP +# ================================================================ +# +# --- GROUP BY on TEXT column (end_update path) --- +# Tmp table: 50 groups x ~1500-byte TEXT key = ~75 KB. Fits in 1 M HEAP. +FLUSH STATUS; +SELECT COUNT(*), SUM(gsum) FROM ( +SELECT a, SUM(v) AS gsum FROM t1 GROUP BY a +) dt; +COUNT(*) SUM(gsum) +50 7550 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES ON +CREATED_TMP_TABLES ON +# --- SELECT DISTINCT on TEXT column (end_write path) --- +FLUSH STATUS; +SELECT COUNT(*) AS cnt_distinct FROM (SELECT DISTINCT a FROM t1) dt; +cnt_distinct +50 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES ON +CREATED_TMP_TABLES ON +# --- SELECT DISTINCT on BLOB column --- +FLUSH STATUS; +SELECT COUNT(*) AS cnt_distinct FROM (SELECT DISTINCT b FROM t1) dt; +cnt_distinct +50 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES ON +CREATED_TMP_TABLES ON +# --- SELECT DISTINCT on TEXT + INT columns --- +FLUSH STATUS; +SELECT COUNT(*) AS cnt_distinct FROM (SELECT DISTINCT a, v FROM t1) dt; +cnt_distinct +100 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES ON +CREATED_TMP_TABLES ON +# --- GROUP BY TEXT column + HAVING --- +FLUSH STATUS; +SELECT COUNT(*) AS groups_with_2rows FROM ( +SELECT a FROM t1 GROUP BY a HAVING COUNT(*) = 2 +) dt; +groups_with_2rows +50 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES ON +CREATED_TMP_TABLES ON +# --- GROUP BY on TEXT + BLOB columns (two blob key columns) --- +FLUSH STATUS; +SELECT COUNT(*) AS groups FROM ( +SELECT a, b, COUNT(*) AS cnt FROM t1 GROUP BY a, b +) dt; +groups +50 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES ON +CREATED_TMP_TABLES ON +# --- GROUP BY TEXT WITH ROLLUP --- +# 50 groups + 1 NULL summary row = 51 +FLUSH STATUS; +SELECT COUNT(*) AS rows_incl_rollup FROM ( +SELECT a, SUM(v) AS sv FROM t1 GROUP BY a WITH ROLLUP +) dt; +rows_incl_rollup +51 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES ON +CREATED_TMP_TABLES ON +# --- UNION DISTINCT on TEXT column (end_write path) --- +# Both halves share the same 50 distinct 'a' values -> 50 rows after UNION. +FLUSH STATUS; +SELECT COUNT(*) AS union_rows FROM ( +SELECT a FROM t1 WHERE v <= 50 +UNION +SELECT a FROM t1 WHERE v > 100 +) dt; +union_rows +50 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES ON +CREATED_TMP_TABLES ON +# --- UNION DISTINCT: same table both sides (all 50 distinct values) --- +FLUSH STATUS; +SELECT COUNT(*) AS distinct_rows FROM ( +SELECT a FROM t1 UNION SELECT a FROM t1 +) dt; +distinct_rows +50 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES ON +CREATED_TMP_TABLES ON +# --- GROUP_CONCAT(DISTINCT text_col) --- +# GROUP_CONCAT uses blob_storage for blob values; the HEAP key part is +# tiny (12 bytes/row). Expect Created_tmp_disk_tables=0 when HEAP is used. +FLUSH STATUS; +SELECT LENGTH(GROUP_CONCAT(DISTINCT LEFT(a, 4) ORDER BY a)) AS gc_len FROM t1; +gc_len +499 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES ON +CREATED_TMP_TABLES ON +# --- GROUP_CONCAT(text ORDER BY ...) -- blob_storage path, no DISTINCT +# No HEAP key table is needed; blob data goes to blob_storage (MEM_ROOT). +FLUSH STATUS; +SELECT LENGTH(GROUP_CONCAT(LEFT(a, 4) ORDER BY v)) AS gc_len FROM t1; +gc_len +499 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES ON +CREATED_TMP_TABLES ON +# --- ORDER BY on TEXT column (DISTINCT subquery + ORDER BY) --- +FLUSH STATUS; +SELECT COUNT(*) AS cnt FROM ( +SELECT DISTINCT a FROM t1 ORDER BY a +) dt; +cnt +50 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES ON +CREATED_TMP_TABLES ON +# --- GROUP BY with aggregate ORDER BY (two-pass: group then sort) --- +FLUSH STATUS; +SELECT COUNT(*), SUM(gv) FROM ( +SELECT a, SUM(v) AS gv FROM t1 GROUP BY a ORDER BY SUM(v) +) dt; +COUNT(*) SUM(gv) +50 7550 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES ON +CREATED_TMP_TABLES ON +DROP TABLE t1; +# +# Test with geometry fields +# +CREATE TABLE t1 ( +pk INT NOT NULL AUTO_INCREMENT PRIMARY KEY, +g GEOMETRY, +v INT NOT NULL +); +INSERT INTO t1 (g, v) +SELECT PolygonFromText(concat('POLYGON((10 10,20 10,20 20,10 20,10 10, 10 10,20 10,20 20,10 20,10 10, 10 10,20 10,20 20,10 20,10 10,', seq, " ", seq, ", ", seq+10, " ", seq+10, ", 10 10))")),seq from seq_1_to_200; +INSERT INTO t1 (g, v) +SELECT g, v + 200 FROM t1 ORDER BY pk; +FLUSH STATUS; +SELECT COUNT(*), COUNT(DISTINCT g), SUM(v) FROM t1; +COUNT(*) COUNT(DISTINCT g) SUM(v) +400 200 80200 +SELECT LENGTH(GROUP_CONCAT(g ORDER BY v)) AS gc_len FROM t1; +gc_len +122399 +SELECT VARIABLE_NAME, +IF(VARIABLE_VALUE > 0, 'ON', 'OFF') AS USED +FROM INFORMATION_SCHEMA.SESSION_STATUS +WHERE VARIABLE_NAME IN ('CREATED_TMP_TABLES', 'CREATED_TMP_DISK_TABLES'); +VARIABLE_NAME USED +CREATED_TMP_DISK_TABLES ON +CREATED_TMP_TABLES ON +DROP TABLE t1; +# Clean up +set @@max_sort_length=@save_max_sort_length; +set @@sort_buffer_size=@save_sort_buffer_size; diff --git a/mysql-test/suite/heap/blob_big3.test b/mysql-test/suite/heap/blob_big3.test new file mode 100644 index 0000000000000..9e0fc072ac30d --- /dev/null +++ b/mysql-test/suite/heap/blob_big3.test @@ -0,0 +1,10 @@ +--echo # +--echo # Test create_internal_tmp_table_from_heap() with blob/text columns. +--echo # + +# Test with all temporary tables being directly created in Aria + +SET @@tmp_memory_table_size= 0; +--source blob_big.inc + + diff --git a/mysql-test/suite/heap/blob_dedup.result b/mysql-test/suite/heap/blob_dedup.result new file mode 100644 index 0000000000000..293b050b5e710 --- /dev/null +++ b/mysql-test/suite/heap/blob_dedup.result @@ -0,0 +1,19 @@ +CREATE TABLE t1 (a mediumtext) ENGINE=HEAP; +INSERT INTO t1 VALUES ('abc'),('def'),('abc'),('abc'),('def'),('abcd'); +SELECT DISTINCT a FROM t1; +a +abc +abcd +def +DROP TABLE t1; +CREATE TABLE t1 (a mediumtext); +CREATE TABLE t2 (b varchar(20)); +INSERT INTO t1 VALUES ('a'),('b'),('a'),('c'); +INSERT INTO t2 VALUES ('b'),('d'); +SELECT left(a,100000000) FROM t1 UNION SELECT b FROM t2; +left(a,100000000) +a +b +c +d +DROP TABLE t1, t2; diff --git a/mysql-test/suite/heap/blob_dedup.test b/mysql-test/suite/heap/blob_dedup.test new file mode 100644 index 0000000000000..810501db3430a --- /dev/null +++ b/mysql-test/suite/heap/blob_dedup.test @@ -0,0 +1,20 @@ +# +# MDEV-38975: Test DISTINCT and UNION deduplication with BLOB columns +# in HEAP temporary tables. Verifies that hash-based deduplication +# correctly compares blob data (not pointer values) by using duplicate +# values that must be deduplicated. +# + +CREATE TABLE t1 (a mediumtext) ENGINE=HEAP; +INSERT INTO t1 VALUES ('abc'),('def'),('abc'),('abc'),('def'),('abcd'); +--sorted_result +SELECT DISTINCT a FROM t1; +DROP TABLE t1; + +CREATE TABLE t1 (a mediumtext); +CREATE TABLE t2 (b varchar(20)); +INSERT INTO t1 VALUES ('a'),('b'),('a'),('c'); +INSERT INTO t2 VALUES ('b'),('d'); +--sorted_result +SELECT left(a,100000000) FROM t1 UNION SELECT b FROM t2; +DROP TABLE t1, t2; diff --git a/mysql-test/suite/heap/blob_fallback.result b/mysql-test/suite/heap/blob_fallback.result new file mode 100644 index 0000000000000..d71cad18ec99f --- /dev/null +++ b/mysql-test/suite/heap/blob_fallback.result @@ -0,0 +1,57 @@ +drop table if exists t1; +# +# Setup: constrained MEMORY table with blob column +# +set @save_max_heap_table_size=@@max_heap_table_size; +set max_heap_table_size=1*1024*1024; +create table t1 (a int not null, b blob, hash char(64), primary key(a)) engine=memory; +# +# Fill table with short-blob rows to consume capacity +# +select count(*) as total_rows from t1; +total_rows +500 +# +# Delete every other row to create fragmentation +# +select count(*) as rows_after_delete from t1; +rows_after_delete +250 +# +# Insert new rows with blobs -- exercises free-list scavenge +# +set @b1= concat(repeat('ABCD', 50)); +set @b2= concat(repeat('EFGH', 125)); +set @b3= concat(repeat('IJ', 25)); +insert into t1 values (10000, @b1, sha2(@b1, 256)); +insert into t1 values (10001, @b2, sha2(@b2, 256)); +insert into t1 values (10002, @b3, sha2(@b3, 256)); +# +# Verify data integrity +# +select a, length(b) as blob_len, left(b, 5) as prefix from t1 +where a >= 10000 order by a; +a blob_len prefix +10000 200 ABCDA +10001 500 EFGHE +10002 50 IJIJI +select a, sha2(b, 256) = hash as intact from t1 where a=10000; +a intact +10000 1 +select a, sha2(b, 256) = hash as intact from t1 where a=10001; +a intact +10001 1 +select a, sha2(b, 256) = hash as intact from t1 where a=10002; +a intact +10002 1 +# +# Verify scan still works correctly +# +select count(*) as scan_count from t1; +scan_count +253 +# +# Cleanup +# +drop table t1; +set max_heap_table_size=@save_max_heap_table_size; diff --git a/mysql-test/suite/heap/blob_fallback.test b/mysql-test/suite/heap/blob_fallback.test new file mode 100644 index 0000000000000..71ff8011282c1 --- /dev/null +++ b/mysql-test/suite/heap/blob_fallback.test @@ -0,0 +1,85 @@ +# +# MDEV-38975: Free-list scavenge fallback for blob allocation +# +# Tests that blob inserts succeed via the Step 3 free-list scavenge +# fallback when the table is near max_heap_table_size and has a +# heavily fragmented free list (alternating used/deleted rows). +# +# Without the fallback, blob allocation fails with table-full even +# though free slots exist -- they're just too scattered for normal +# free-list reuse (Step 1) and the tail is at capacity (Step 2). +# + +--disable_warnings +drop table if exists t1; +--enable_warnings + +--echo # +--echo # Setup: constrained MEMORY table with blob column +--echo # +set @save_max_heap_table_size=@@max_heap_table_size; +set max_heap_table_size=1*1024*1024; + +create table t1 (a int not null, b blob, hash char(64), primary key(a)) engine=memory; + +--echo # +--echo # Fill table with short-blob rows to consume capacity +--echo # + +--disable_query_log +--let $i=0 +while ($i < 500) +{ + eval insert into t1 values ($i, repeat('x', 100), sha2(repeat('x', 100), 256)); + --inc $i +} +--enable_query_log + +select count(*) as total_rows from t1; + +--echo # +--echo # Delete every other row to create fragmentation +--echo # + +--disable_query_log +--let $i=0 +while ($i < 500) +{ + eval delete from t1 where a=$i; + --inc $i + --inc $i +} +--enable_query_log + +select count(*) as rows_after_delete from t1; + +--echo # +--echo # Insert new rows with blobs -- exercises free-list scavenge +--echo # +set @b1= concat(repeat('ABCD', 50)); +set @b2= concat(repeat('EFGH', 125)); +set @b3= concat(repeat('IJ', 25)); +insert into t1 values (10000, @b1, sha2(@b1, 256)); +insert into t1 values (10001, @b2, sha2(@b2, 256)); +insert into t1 values (10002, @b3, sha2(@b3, 256)); + +--echo # +--echo # Verify data integrity +--echo # +select a, length(b) as blob_len, left(b, 5) as prefix from t1 + where a >= 10000 order by a; + +select a, sha2(b, 256) = hash as intact from t1 where a=10000; +select a, sha2(b, 256) = hash as intact from t1 where a=10001; +select a, sha2(b, 256) = hash as intact from t1 where a=10002; + +--echo # +--echo # Verify scan still works correctly +--echo # +select count(*) as scan_count from t1; + +--echo # +--echo # Cleanup +--echo # +drop table t1; +set max_heap_table_size=@save_max_heap_table_size; diff --git a/mysql-test/suite/heap/blob_find_unique.result b/mysql-test/suite/heap/blob_find_unique.result new file mode 100644 index 0000000000000..6adb24890599a --- /dev/null +++ b/mysql-test/suite/heap/blob_find_unique.result @@ -0,0 +1,178 @@ +SET @save_max_heap_table_size= @@max_heap_table_size; +SET @@max_heap_table_size= 32*1024*1024; +# +# SELECT ... EXCEPT SELECT with large blob columns stored in multiple +# segments (>16KB) +# +CREATE TABLE t1 (a INT, b TEXT) ENGINE=HEAP; +INSERT INTO t1 VALUES (1, REPEAT('a', 20000)), (2, REPEAT('b', 20000)), +(3, REPEAT('c', 20000)); +CREATE TABLE t2 (a INT, b TEXT) ENGINE=HEAP; +INSERT INTO t2 VALUES (2, REPEAT('b', 20000)), (3, REPEAT('c', 20000)); +SELECT a, LENGTH(b), LEFT(b, 1) FROM ( +SELECT a, b FROM t1 EXCEPT SELECT a, b FROM t2 +) dt; +a LENGTH(b) LEFT(b, 1) +1 20000 a +DROP TABLE t1, t2; +# +# SELECT ... INTERSECT SELECT with large blobs (multiple segments) +# +CREATE TABLE t1 (a INT, b TEXT) ENGINE=HEAP; +INSERT INTO t1 VALUES (1, REPEAT('x', 25000)), (2, REPEAT('y', 25000)), +(3, REPEAT('z', 25000)); +CREATE TABLE t2 (a INT, b TEXT) ENGINE=HEAP; +INSERT INTO t2 VALUES (2, REPEAT('y', 25000)), (4, REPEAT('w', 25000)); +SELECT a, LENGTH(b), LEFT(b, 1) FROM ( +SELECT a, b FROM t1 INTERSECT SELECT a, b FROM t2 +) dt; +a LENGTH(b) LEFT(b, 1) +2 25000 y +DROP TABLE t1, t2; +# +# INTERSECT ALL with large blobs -- duplicates preserved +# +CREATE TABLE t1 (a INT, b TEXT) ENGINE=HEAP; +INSERT INTO t1 VALUES (1, REPEAT('p', 20000)), (1, REPEAT('p', 20000)), +(2, REPEAT('q', 20000)); +CREATE TABLE t2 (a INT, b TEXT) ENGINE=HEAP; +INSERT INTO t2 VALUES (1, REPEAT('p', 20000)), (2, REPEAT('q', 20000)), +(2, REPEAT('q', 20000)); +SELECT a, LENGTH(b), LEFT(b, 1) FROM ( +SELECT a, b FROM t1 INTERSECT ALL SELECT a, b FROM t2 +) dt; +a LENGTH(b) LEFT(b, 1) +1 20000 p +2 20000 q +DROP TABLE t1, t2; +# +# EXCEPT ALL with large blobs -- duplicates tracked +# +CREATE TABLE t1 (a INT, b TEXT) ENGINE=HEAP; +INSERT INTO t1 VALUES (1, REPEAT('m', 20000)), (1, REPEAT('m', 20000)), +(1, REPEAT('m', 20000)), (2, REPEAT('n', 20000)); +CREATE TABLE t2 (a INT, b TEXT) ENGINE=HEAP; +INSERT INTO t2 VALUES (1, REPEAT('m', 20000)), (2, REPEAT('n', 20000)); +SELECT a, LENGTH(b), LEFT(b, 1) FROM ( +SELECT a, b FROM t1 EXCEPT ALL SELECT a, b FROM t2 +) dt; +a LENGTH(b) LEFT(b, 1) +1 20000 m +1 20000 m +DROP TABLE t1, t2; +# +# Multiple blob columns -- verify both are materialized for comparison +# +CREATE TABLE t1 (a TEXT, b TEXT) ENGINE=HEAP; +INSERT INTO t1 VALUES (REPEAT('a', 20000), REPEAT('1', 20000)), +(REPEAT('b', 20000), REPEAT('2', 20000)), +(REPEAT('c', 20000), REPEAT('3', 20000)); +CREATE TABLE t2 (a TEXT, b TEXT) ENGINE=HEAP; +INSERT INTO t2 VALUES (REPEAT('b', 20000), REPEAT('2', 20000)); +SELECT LENGTH(a), LEFT(a, 1), LENGTH(b), LEFT(b, 1) FROM ( +SELECT a, b FROM t1 EXCEPT SELECT a, b FROM t2 +) dt; +LENGTH(a) LEFT(a, 1) LENGTH(b) LEFT(b, 1) +20000 a 20000 1 +20000 c 20000 3 +DROP TABLE t1, t2; +# +# Mixed blob sizes: tiny (single record), medium (single run), large (multiple segments) +# +CREATE TABLE t1 (a INT, b TEXT) ENGINE=HEAP; +INSERT INTO t1 VALUES (1, 'hi'), (2, REPEAT('m', 5000)), +(3, REPEAT('L', 25000)); +CREATE TABLE t2 (a INT, b TEXT) ENGINE=HEAP; +INSERT INTO t2 VALUES (2, REPEAT('m', 5000)); +SELECT a, LENGTH(b) FROM ( +SELECT a, b FROM t1 EXCEPT SELECT a, b FROM t2 +) dt; +a LENGTH(b) +1 2 +3 25000 +DROP TABLE t1, t2; +# +# PAD SPACE: trailing spaces should compare equal +# +CREATE TABLE t1 (a TEXT) ENGINE=HEAP; +INSERT INTO t1 VALUES ('hello'); +CREATE TABLE t2 (a TEXT) ENGINE=HEAP; +INSERT INTO t2 VALUES ('hello '); +SELECT COUNT(*) FROM ( +SELECT a FROM t1 INTERSECT SELECT a FROM t2 +) dt; +COUNT(*) +1 +DROP TABLE t1, t2; +# +# Mismatch path: hash collision forcing multiple comparisons. +# Many rows with same integer but different blobs to exercise +# the restore-on-mismatch loop. +# +CREATE TABLE t1 (a TEXT) ENGINE=HEAP; +INSERT INTO t1 VALUES (REPEAT('a', 20000)), (REPEAT('b', 20000)), +(REPEAT('c', 20000)), (REPEAT('d', 20000)), +(REPEAT('e', 20000)); +CREATE TABLE t2 (a TEXT) ENGINE=HEAP; +INSERT INTO t2 VALUES (REPEAT('c', 20000)), (REPEAT('e', 20000)); +SELECT LEFT(a, 1), LENGTH(a) FROM ( +SELECT a FROM t1 EXCEPT SELECT a FROM t2 +) dt; +LEFT(a, 1) LENGTH(a) +a 20000 +b 20000 +d 20000 +DROP TABLE t1, t2; +# +# Content integrity: verify blob data survives intact +# +CREATE TABLE t1 (a TEXT) ENGINE=HEAP; +INSERT INTO t1 VALUES (REPEAT('X', 20000)); +CREATE TABLE t2 (a TEXT) ENGINE=HEAP; +INSERT INTO t2 VALUES (REPEAT('Y', 20000)); +SELECT LENGTH(a), a = REPEAT('X', 20000) AS intact FROM ( +SELECT a FROM t1 EXCEPT SELECT a FROM t2 +) dt; +LENGTH(a) intact +20000 1 +DROP TABLE t1, t2; +# +# EXCEPT with expressions producing temp table blobs +# +SELECT LENGTH(a), LEFT(a, 3) FROM ( +SELECT a FROM ( +SELECT REPEAT('abc', 7000) AS a +UNION ALL SELECT REPEAT('def', 7000) +UNION ALL SELECT REPEAT('abc', 7000) +) t1 +EXCEPT +SELECT a FROM ( +SELECT REPEAT('abc', 7000) AS a +) t2 +) dt; +LENGTH(a) LEFT(a, 3) +21000 def +# +# EXCEPT with hash-colliding blob values +# +# Binary blobs 0x0069 and 0x0107 collide under my_hash_sort_bin() +# (both hash to 66889). In find_unique_row(), the hash chain walk +# encounters a non-matching entry with the same hash, triggering the +# record restore (memcpy from input_copy) before continuing to the +# next chain entry. +# +CREATE TABLE t1 (val BLOB NOT NULL); +INSERT INTO t1 VALUES (UNHEX('0069')), (UNHEX('0107')), (UNHEX('FF')); +CREATE TABLE t2 (val BLOB NOT NULL); +INSERT INTO t2 VALUES (UNHEX('0069')); +# Should return 0107 and FF (everything in t1 except 0069) +SELECT HEX(val) FROM (SELECT val FROM t1 EXCEPT SELECT val FROM t2) dt; +HEX(val) +0107 +FF +# INTERSECT: should return only 0069 +SELECT HEX(val) FROM (SELECT val FROM t1 INTERSECT SELECT val FROM t2) dt; +HEX(val) +0069 +DROP TABLE t1, t2; +SET @@max_heap_table_size= @save_max_heap_table_size; diff --git a/mysql-test/suite/heap/blob_find_unique.test b/mysql-test/suite/heap/blob_find_unique.test new file mode 100644 index 0000000000000..571838c8dcf5f --- /dev/null +++ b/mysql-test/suite/heap/blob_find_unique.test @@ -0,0 +1,202 @@ +# +# MDEV-38975: Test find_unique_row() optimization for blob columns. +# Exercises the materialize-then-compare path via EXCEPT, INTERSECT, +# INTERSECT ALL, and EXCEPT ALL, which call find_unique_row() on the +# HEAP temp table. +# +# Blob storage layouts tested: +# Tiny -- fits in a single record without a run header +# Medium -- fits in a single contiguous run of records +# Large -- requires multiple runs linked via next_cont pointers +# + +SET @save_max_heap_table_size= @@max_heap_table_size; +SET @@max_heap_table_size= 32*1024*1024; + +--echo # +--echo # SELECT ... EXCEPT SELECT with large blob columns stored in multiple +--echo # segments (>16KB) +--echo # +CREATE TABLE t1 (a INT, b TEXT) ENGINE=HEAP; +INSERT INTO t1 VALUES (1, REPEAT('a', 20000)), (2, REPEAT('b', 20000)), + (3, REPEAT('c', 20000)); +CREATE TABLE t2 (a INT, b TEXT) ENGINE=HEAP; +INSERT INTO t2 VALUES (2, REPEAT('b', 20000)), (3, REPEAT('c', 20000)); + +--sorted_result +SELECT a, LENGTH(b), LEFT(b, 1) FROM ( + SELECT a, b FROM t1 EXCEPT SELECT a, b FROM t2 +) dt; + +DROP TABLE t1, t2; + +--echo # +--echo # SELECT ... INTERSECT SELECT with large blobs (multiple segments) +--echo # +CREATE TABLE t1 (a INT, b TEXT) ENGINE=HEAP; +INSERT INTO t1 VALUES (1, REPEAT('x', 25000)), (2, REPEAT('y', 25000)), + (3, REPEAT('z', 25000)); +CREATE TABLE t2 (a INT, b TEXT) ENGINE=HEAP; +INSERT INTO t2 VALUES (2, REPEAT('y', 25000)), (4, REPEAT('w', 25000)); + +--sorted_result +SELECT a, LENGTH(b), LEFT(b, 1) FROM ( + SELECT a, b FROM t1 INTERSECT SELECT a, b FROM t2 +) dt; + +DROP TABLE t1, t2; + +--echo # +--echo # INTERSECT ALL with large blobs -- duplicates preserved +--echo # +CREATE TABLE t1 (a INT, b TEXT) ENGINE=HEAP; +INSERT INTO t1 VALUES (1, REPEAT('p', 20000)), (1, REPEAT('p', 20000)), + (2, REPEAT('q', 20000)); +CREATE TABLE t2 (a INT, b TEXT) ENGINE=HEAP; +INSERT INTO t2 VALUES (1, REPEAT('p', 20000)), (2, REPEAT('q', 20000)), + (2, REPEAT('q', 20000)); + +--sorted_result +SELECT a, LENGTH(b), LEFT(b, 1) FROM ( + SELECT a, b FROM t1 INTERSECT ALL SELECT a, b FROM t2 +) dt; + +DROP TABLE t1, t2; + +--echo # +--echo # EXCEPT ALL with large blobs -- duplicates tracked +--echo # +CREATE TABLE t1 (a INT, b TEXT) ENGINE=HEAP; +INSERT INTO t1 VALUES (1, REPEAT('m', 20000)), (1, REPEAT('m', 20000)), + (1, REPEAT('m', 20000)), (2, REPEAT('n', 20000)); +CREATE TABLE t2 (a INT, b TEXT) ENGINE=HEAP; +INSERT INTO t2 VALUES (1, REPEAT('m', 20000)), (2, REPEAT('n', 20000)); + +--sorted_result +SELECT a, LENGTH(b), LEFT(b, 1) FROM ( + SELECT a, b FROM t1 EXCEPT ALL SELECT a, b FROM t2 +) dt; + +DROP TABLE t1, t2; + +--echo # +--echo # Multiple blob columns -- verify both are materialized for comparison +--echo # +CREATE TABLE t1 (a TEXT, b TEXT) ENGINE=HEAP; +INSERT INTO t1 VALUES (REPEAT('a', 20000), REPEAT('1', 20000)), + (REPEAT('b', 20000), REPEAT('2', 20000)), + (REPEAT('c', 20000), REPEAT('3', 20000)); +CREATE TABLE t2 (a TEXT, b TEXT) ENGINE=HEAP; +INSERT INTO t2 VALUES (REPEAT('b', 20000), REPEAT('2', 20000)); + +--sorted_result +SELECT LENGTH(a), LEFT(a, 1), LENGTH(b), LEFT(b, 1) FROM ( + SELECT a, b FROM t1 EXCEPT SELECT a, b FROM t2 +) dt; + +DROP TABLE t1, t2; + +--echo # +--echo # Mixed blob sizes: tiny (single record), medium (single run), large (multiple segments) +--echo # +CREATE TABLE t1 (a INT, b TEXT) ENGINE=HEAP; +INSERT INTO t1 VALUES (1, 'hi'), (2, REPEAT('m', 5000)), + (3, REPEAT('L', 25000)); +CREATE TABLE t2 (a INT, b TEXT) ENGINE=HEAP; +INSERT INTO t2 VALUES (2, REPEAT('m', 5000)); + +--sorted_result +SELECT a, LENGTH(b) FROM ( + SELECT a, b FROM t1 EXCEPT SELECT a, b FROM t2 +) dt; + +DROP TABLE t1, t2; + +--echo # +--echo # PAD SPACE: trailing spaces should compare equal +--echo # +CREATE TABLE t1 (a TEXT) ENGINE=HEAP; +INSERT INTO t1 VALUES ('hello'); +CREATE TABLE t2 (a TEXT) ENGINE=HEAP; +INSERT INTO t2 VALUES ('hello '); + +SELECT COUNT(*) FROM ( + SELECT a FROM t1 INTERSECT SELECT a FROM t2 +) dt; + +DROP TABLE t1, t2; + +--echo # +--echo # Mismatch path: hash collision forcing multiple comparisons. +--echo # Many rows with same integer but different blobs to exercise +--echo # the restore-on-mismatch loop. +--echo # +CREATE TABLE t1 (a TEXT) ENGINE=HEAP; +INSERT INTO t1 VALUES (REPEAT('a', 20000)), (REPEAT('b', 20000)), + (REPEAT('c', 20000)), (REPEAT('d', 20000)), + (REPEAT('e', 20000)); +CREATE TABLE t2 (a TEXT) ENGINE=HEAP; +INSERT INTO t2 VALUES (REPEAT('c', 20000)), (REPEAT('e', 20000)); + +--sorted_result +SELECT LEFT(a, 1), LENGTH(a) FROM ( + SELECT a FROM t1 EXCEPT SELECT a FROM t2 +) dt; + +DROP TABLE t1, t2; + +--echo # +--echo # Content integrity: verify blob data survives intact +--echo # +CREATE TABLE t1 (a TEXT) ENGINE=HEAP; +INSERT INTO t1 VALUES (REPEAT('X', 20000)); +CREATE TABLE t2 (a TEXT) ENGINE=HEAP; +INSERT INTO t2 VALUES (REPEAT('Y', 20000)); + +SELECT LENGTH(a), a = REPEAT('X', 20000) AS intact FROM ( + SELECT a FROM t1 EXCEPT SELECT a FROM t2 +) dt; + +DROP TABLE t1, t2; + +--echo # +--echo # EXCEPT with expressions producing temp table blobs +--echo # +SELECT LENGTH(a), LEFT(a, 3) FROM ( + SELECT a FROM ( + SELECT REPEAT('abc', 7000) AS a + UNION ALL SELECT REPEAT('def', 7000) + UNION ALL SELECT REPEAT('abc', 7000) + ) t1 + EXCEPT + SELECT a FROM ( + SELECT REPEAT('abc', 7000) AS a + ) t2 +) dt; + +--echo # +--echo # EXCEPT with hash-colliding blob values +--echo # +--echo # Binary blobs 0x0069 and 0x0107 collide under my_hash_sort_bin() +--echo # (both hash to 66889). In find_unique_row(), the hash chain walk +--echo # encounters a non-matching entry with the same hash, triggering the +--echo # record restore (memcpy from input_copy) before continuing to the +--echo # next chain entry. +--echo # + +CREATE TABLE t1 (val BLOB NOT NULL); +INSERT INTO t1 VALUES (UNHEX('0069')), (UNHEX('0107')), (UNHEX('FF')); + +CREATE TABLE t2 (val BLOB NOT NULL); +INSERT INTO t2 VALUES (UNHEX('0069')); + +--echo # Should return 0107 and FF (everything in t1 except 0069) +--sorted_result +SELECT HEX(val) FROM (SELECT val FROM t1 EXCEPT SELECT val FROM t2) dt; + +--echo # INTERSECT: should return only 0069 +SELECT HEX(val) FROM (SELECT val FROM t1 INTERSECT SELECT val FROM t2) dt; + +DROP TABLE t1, t2; + +SET @@max_heap_table_size= @save_max_heap_table_size; diff --git a/mysql-test/suite/heap/blob_group_by.result b/mysql-test/suite/heap/blob_group_by.result new file mode 100644 index 0000000000000..d46a22976aecf --- /dev/null +++ b/mysql-test/suite/heap/blob_group_by.result @@ -0,0 +1,62 @@ +# +# MDEV-38975: GROUP BY on blob columns in HEAP internal temp tables +# Verify that GROUP BY correctly groups by blob/text content, +# not by internal pointer representation. +# +CREATE TABLE t1 (a TEXT, b INT); +INSERT INTO t1 VALUES ('foo', 1), ('foo', 2), ('bar', 3), ('bar', 4), ('baz', 5); +# GROUP BY on TEXT column should group by content +SELECT a, COUNT(*), SUM(b) FROM t1 GROUP BY a ORDER BY a; +a COUNT(*) SUM(b) +bar 2 7 +baz 1 5 +foo 2 3 +# GROUP BY with HAVING +SELECT a, COUNT(*) AS cnt FROM t1 GROUP BY a HAVING cnt > 1 ORDER BY a; +a cnt +bar 2 +foo 2 +DROP TABLE t1; +# +# GROUP BY on multiple blob columns +# +CREATE TABLE t1 (a TEXT, b TEXT, c INT); +INSERT INTO t1 VALUES ('x', 'p', 1), ('x', 'p', 2), +('x', 'q', 3), ('y', 'p', 4); +SELECT a, b, COUNT(*), SUM(c) FROM t1 GROUP BY a, b ORDER BY a, b; +a b COUNT(*) SUM(c) +x p 2 3 +x q 1 3 +y p 1 4 +DROP TABLE t1; +# +# GROUP BY blob with NULL values +# +CREATE TABLE t1 (a TEXT, b INT); +INSERT INTO t1 VALUES (NULL, 1), (NULL, 2), ('foo', 3), ('foo', 4), ('bar', 5); +SELECT a, COUNT(*), SUM(b) FROM t1 GROUP BY a ORDER BY a; +a COUNT(*) SUM(b) +NULL 2 3 +bar 1 5 +foo 2 7 +DROP TABLE t1; +# +# GROUP BY on blob values that produce identical hash +# +# Binary blobs 0x0069 and 0x0107 collide under my_hash_sort_bin() +# (both hash to 66889 with initial state nr=1, nr2=4). +# When GROUP BY creates an internal HEAP temp table with a hash key +# on the blob column, both groups land in the same hash bucket. +# hp_key_cmp() must correctly distinguish them via strnncollsp +# comparison; otherwise the groups merge. +# +CREATE TABLE t1 (val BLOB NOT NULL); +INSERT INTO t1 VALUES +(UNHEX('0069')), (UNHEX('0069')), (UNHEX('0069')), +(UNHEX('0107')), (UNHEX('0107')); +SELECT HEX(val), COUNT(*) AS cnt FROM t1 GROUP BY val ORDER BY val; +HEX(val) cnt +0069 3 +0107 2 +DROP TABLE t1; +# End of MDEV-38975 GROUP BY blob tests diff --git a/mysql-test/suite/heap/blob_group_by.test b/mysql-test/suite/heap/blob_group_by.test new file mode 100644 index 0000000000000..c536e7adab7cf --- /dev/null +++ b/mysql-test/suite/heap/blob_group_by.test @@ -0,0 +1,61 @@ +--source include/have_sequence.inc + +--echo # +--echo # MDEV-38975: GROUP BY on blob columns in HEAP internal temp tables +--echo # Verify that GROUP BY correctly groups by blob/text content, +--echo # not by internal pointer representation. +--echo # + +CREATE TABLE t1 (a TEXT, b INT); +INSERT INTO t1 VALUES ('foo', 1), ('foo', 2), ('bar', 3), ('bar', 4), ('baz', 5); + +--echo # GROUP BY on TEXT column should group by content +SELECT a, COUNT(*), SUM(b) FROM t1 GROUP BY a ORDER BY a; + +--echo # GROUP BY with HAVING +SELECT a, COUNT(*) AS cnt FROM t1 GROUP BY a HAVING cnt > 1 ORDER BY a; + +DROP TABLE t1; + +--echo # +--echo # GROUP BY on multiple blob columns +--echo # +CREATE TABLE t1 (a TEXT, b TEXT, c INT); +INSERT INTO t1 VALUES ('x', 'p', 1), ('x', 'p', 2), + ('x', 'q', 3), ('y', 'p', 4); + +SELECT a, b, COUNT(*), SUM(c) FROM t1 GROUP BY a, b ORDER BY a, b; + +DROP TABLE t1; + +--echo # +--echo # GROUP BY blob with NULL values +--echo # +CREATE TABLE t1 (a TEXT, b INT); +INSERT INTO t1 VALUES (NULL, 1), (NULL, 2), ('foo', 3), ('foo', 4), ('bar', 5); + +SELECT a, COUNT(*), SUM(b) FROM t1 GROUP BY a ORDER BY a; + +DROP TABLE t1; + +--echo # +--echo # GROUP BY on blob values that produce identical hash +--echo # +--echo # Binary blobs 0x0069 and 0x0107 collide under my_hash_sort_bin() +--echo # (both hash to 66889 with initial state nr=1, nr2=4). +--echo # When GROUP BY creates an internal HEAP temp table with a hash key +--echo # on the blob column, both groups land in the same hash bucket. +--echo # hp_key_cmp() must correctly distinguish them via strnncollsp +--echo # comparison; otherwise the groups merge. +--echo # + +CREATE TABLE t1 (val BLOB NOT NULL); +INSERT INTO t1 VALUES + (UNHEX('0069')), (UNHEX('0069')), (UNHEX('0069')), + (UNHEX('0107')), (UNHEX('0107')); + +SELECT HEX(val), COUNT(*) AS cnt FROM t1 GROUP BY val ORDER BY val; + +DROP TABLE t1; + +--echo # End of MDEV-38975 GROUP BY blob tests diff --git a/mysql-test/suite/heap/blob_group_concat_geom.result b/mysql-test/suite/heap/blob_group_concat_geom.result new file mode 100644 index 0000000000000..452c3d371886c --- /dev/null +++ b/mysql-test/suite/heap/blob_group_concat_geom.result @@ -0,0 +1,66 @@ +# +# Geometry expression in GROUP_CONCAT with ORDER BY +# +CREATE TABLE t (f GEOMETRY); +INSERT INTO t VALUES (POINT(1,1)), (POINT(2,2)); +SELECT LENGTH(GROUP_CONCAT(IF(TRUE, f, POINT(0,0)) ORDER BY 1)) > 0 AS ok FROM t; +ok +1 +DROP TABLE t; +# +# Geometry expression in GROUP_CONCAT with DISTINCT +# +CREATE TABLE t (f GEOMETRY); +INSERT INTO t VALUES (POINT(1,1)), (POINT(1,1)), (POINT(2,2)); +SELECT LENGTH(GROUP_CONCAT(DISTINCT COALESCE(f, POINT(0,0)))) > 0 AS ok FROM t; +ok +1 +DROP TABLE t; +# +# CASE expression returning geometry in GROUP_CONCAT +# +CREATE TABLE t (id INT, f GEOMETRY); +INSERT INTO t VALUES (1, POINT(1,1)), (2, POINT(2,2)); +SELECT LENGTH(GROUP_CONCAT(CASE WHEN id > 0 THEN f ELSE POINT(0,0) END ORDER BY id)) > 0 AS ok FROM t; +ok +1 +DROP TABLE t; +# +# Multiple geometry expressions in GROUP_CONCAT +# +CREATE TABLE t (f1 GEOMETRY, f2 GEOMETRY); +INSERT INTO t VALUES (POINT(1,1), POINT(3,3)), (POINT(2,2), POINT(4,4)); +SELECT LENGTH(GROUP_CONCAT(IF(TRUE, f1, POINT(0,0)), IF(TRUE, f2, POINT(0,0)) ORDER BY 1)) > 0 AS ok FROM t; +ok +1 +DROP TABLE t; +# +# User variable assignment with geometry in GROUP_CONCAT +# Item_func_set_user_var::create_tmp_field_ex() calls +# create_tmp_field_ex_from_handler() directly with type_handler(), +# bypassing type_handler_for_tmp_table(). +# +CREATE TABLE t (f GEOMETRY); +INSERT INTO t VALUES (POINT(1,1)), (POINT(2,2)); +SELECT LENGTH(GROUP_CONCAT(@g := f ORDER BY f)) > 0 AS ok FROM t; +ok +1 +DROP TABLE t; +# +# DEFAULT(geom) with expression default in GROUP_CONCAT +# Item_default_value::create_tmp_field_ex() has its own blob path +# via tmp_table_field_from_field_type() that also needs the downgrade. +# +CREATE TABLE t ( +id INT NOT NULL, +x DOUBLE NOT NULL DEFAULT 0, +g GEOMETRY DEFAULT (ST_GeomFromText(CONCAT('POINT(', x, ' 0)'))) +) ENGINE=MyISAM; +INSERT INTO t VALUES (1, 10, ST_GeomFromText('POINT(1 1)')); +INSERT INTO t VALUES (1, 20, ST_GeomFromText('POINT(2 2)')); +INSERT INTO t VALUES (2, 30, ST_GeomFromText('POINT(3 3)')); +SELECT id, LENGTH(GROUP_CONCAT(DEFAULT(g) ORDER BY id)) > 0 AS ok FROM t GROUP BY id; +id ok +1 1 +2 1 +DROP TABLE t; diff --git a/mysql-test/suite/heap/blob_group_concat_geom.test b/mysql-test/suite/heap/blob_group_concat_geom.test new file mode 100644 index 0000000000000..cf8e95b97618b --- /dev/null +++ b/mysql-test/suite/heap/blob_group_concat_geom.test @@ -0,0 +1,71 @@ +--source include/have_geometry.inc + +# +# GROUP_CONCAT with geometry expressions on HEAP temp tables. +# +# Field_geom::store() asserts !table->blob_storage, but GROUP_CONCAT +# with ORDER BY or DISTINCT sets blob_storage on its internal temp table. +# Geometry expressions (IF, CASE, COALESCE, etc.) go through the type +# handler field creation path which must downgrade Field_geom to plain +# Field_blob, matching the downgrade already done for direct field +# references in Field_blob::make_new_field(). +# + +--echo # +--echo # Geometry expression in GROUP_CONCAT with ORDER BY +--echo # +CREATE TABLE t (f GEOMETRY); +INSERT INTO t VALUES (POINT(1,1)), (POINT(2,2)); +SELECT LENGTH(GROUP_CONCAT(IF(TRUE, f, POINT(0,0)) ORDER BY 1)) > 0 AS ok FROM t; +DROP TABLE t; + +--echo # +--echo # Geometry expression in GROUP_CONCAT with DISTINCT +--echo # +CREATE TABLE t (f GEOMETRY); +INSERT INTO t VALUES (POINT(1,1)), (POINT(1,1)), (POINT(2,2)); +SELECT LENGTH(GROUP_CONCAT(DISTINCT COALESCE(f, POINT(0,0)))) > 0 AS ok FROM t; +DROP TABLE t; + +--echo # +--echo # CASE expression returning geometry in GROUP_CONCAT +--echo # +CREATE TABLE t (id INT, f GEOMETRY); +INSERT INTO t VALUES (1, POINT(1,1)), (2, POINT(2,2)); +SELECT LENGTH(GROUP_CONCAT(CASE WHEN id > 0 THEN f ELSE POINT(0,0) END ORDER BY id)) > 0 AS ok FROM t; +DROP TABLE t; + +--echo # +--echo # Multiple geometry expressions in GROUP_CONCAT +--echo # +CREATE TABLE t (f1 GEOMETRY, f2 GEOMETRY); +INSERT INTO t VALUES (POINT(1,1), POINT(3,3)), (POINT(2,2), POINT(4,4)); +SELECT LENGTH(GROUP_CONCAT(IF(TRUE, f1, POINT(0,0)), IF(TRUE, f2, POINT(0,0)) ORDER BY 1)) > 0 AS ok FROM t; +DROP TABLE t; + +--echo # +--echo # User variable assignment with geometry in GROUP_CONCAT +--echo # Item_func_set_user_var::create_tmp_field_ex() calls +--echo # create_tmp_field_ex_from_handler() directly with type_handler(), +--echo # bypassing type_handler_for_tmp_table(). +--echo # +CREATE TABLE t (f GEOMETRY); +INSERT INTO t VALUES (POINT(1,1)), (POINT(2,2)); +SELECT LENGTH(GROUP_CONCAT(@g := f ORDER BY f)) > 0 AS ok FROM t; +DROP TABLE t; + +--echo # +--echo # DEFAULT(geom) with expression default in GROUP_CONCAT +--echo # Item_default_value::create_tmp_field_ex() has its own blob path +--echo # via tmp_table_field_from_field_type() that also needs the downgrade. +--echo # +CREATE TABLE t ( + id INT NOT NULL, + x DOUBLE NOT NULL DEFAULT 0, + g GEOMETRY DEFAULT (ST_GeomFromText(CONCAT('POINT(', x, ' 0)'))) +) ENGINE=MyISAM; +INSERT INTO t VALUES (1, 10, ST_GeomFromText('POINT(1 1)')); +INSERT INTO t VALUES (1, 20, ST_GeomFromText('POINT(2 2)')); +INSERT INTO t VALUES (2, 30, ST_GeomFromText('POINT(3 3)')); +SELECT id, LENGTH(GROUP_CONCAT(DEFAULT(g) ORDER BY id)) > 0 AS ok FROM t GROUP BY id; +DROP TABLE t; diff --git a/mysql-test/suite/heap/blob_ops.result b/mysql-test/suite/heap/blob_ops.result new file mode 100644 index 0000000000000..a137d27861d1b --- /dev/null +++ b/mysql-test/suite/heap/blob_ops.result @@ -0,0 +1,191 @@ +# +# MDEV-38975: Blob/text operations exercising HEAP internal temp tables +# +CREATE TABLE t1 (id INT AUTO_INCREMENT PRIMARY KEY, k INT, text_col TEXT); +INSERT INTO t1 (k, text_col) VALUES +(1, 'alpha'), (1, 'alpha '), (1, 'beta'), +(2, 'gamma'), (2, 'gamma '), (2, 'delta'), +(3, 'alpha'), (3, 'epsilon'), (3, NULL), +(4, NULL), (4, NULL), (4, 'beta'); +# +# COUNT(DISTINCT text_col) +# +SELECT COUNT(DISTINCT text_col) FROM t1; +COUNT(DISTINCT text_col) +5 +# +# COUNT(DISTINCT text_col) with GROUP BY +# +SELECT k, COUNT(DISTINCT text_col) FROM t1 GROUP BY k ORDER BY k; +k COUNT(DISTINCT text_col) +1 2 +2 2 +3 2 +4 1 +# +# IN-subquery with blob/text +# +CREATE TABLE t2 (text_col TEXT); +INSERT INTO t2 VALUES ('alpha'), ('delta'); +SELECT id, text_col FROM t1 WHERE text_col IN (SELECT text_col FROM t2) ORDER BY id; +id text_col +1 alpha +2 alpha +6 delta +7 alpha +# +# GROUP BY text_col ORDER BY aggregate LIMIT +# +SELECT text_col, COUNT(*) AS cnt FROM t1 GROUP BY text_col ORDER BY cnt DESC, text_col LIMIT 3; +text_col cnt +NULL 3 +alpha 3 +beta 2 +# +# GROUP BY text_col WITH ROLLUP +# +SELECT text_col, COUNT(*) FROM t1 GROUP BY text_col WITH ROLLUP; +text_col COUNT(*) +NULL 12 +NULL 3 +alpha 3 +beta 2 +delta 1 +epsilon 1 +gamma 2 +# +# GROUP BY multiple columns WITH ROLLUP +# +SELECT k, text_col, COUNT(*) FROM t1 GROUP BY k, text_col WITH ROLLUP; +k text_col COUNT(*) +1 NULL 3 +1 alpha 2 +1 beta 1 +2 NULL 3 +2 delta 1 +2 gamma 2 +3 NULL 1 +3 NULL 3 +3 alpha 1 +3 epsilon 1 +4 NULL 2 +4 NULL 3 +4 beta 1 +NULL NULL 12 +# +# Window function with blob in SELECT list +# +SELECT id, text_col, ROW_NUMBER() OVER (PARTITION BY k ORDER BY id) AS rn +FROM t1 WHERE k <= 2 ORDER BY id; +id text_col rn +1 alpha 1 +2 alpha 2 +3 beta 3 +4 gamma 1 +5 gamma 2 +6 delta 3 +# +# RANK with blob column +# +SELECT text_col, k, RANK() OVER (ORDER BY text_col) AS rnk +FROM t1 WHERE text_col IS NOT NULL ORDER BY text_col, k, id; +text_col k rnk +alpha 1 1 +alpha 1 1 +alpha 3 1 +beta 1 4 +beta 4 4 +delta 2 6 +epsilon 3 7 +gamma 2 8 +gamma 2 8 +# +# Non-recursive CTE materializing blob column +# +WITH cte AS (SELECT text_col, k FROM t1 WHERE k <= 2) +SELECT DISTINCT text_col FROM cte ORDER BY text_col; +text_col +alpha +beta +delta +gamma +# +# CTE with self-join on blob column +# +WITH cte AS (SELECT text_col, k FROM t1 WHERE text_col IS NOT NULL) +SELECT COUNT(*) FROM cte a JOIN cte b ON a.k = b.k AND a.text_col = b.text_col; +COUNT(*) +13 +# +# CTE referenced twice (forces materialization) +# +WITH cte AS (SELECT DISTINCT text_col FROM t1 WHERE text_col IS NOT NULL) +SELECT a.text_col, b.text_col +FROM cte a JOIN cte b ON a.text_col > b.text_col ORDER BY a.text_col, b.text_col; +text_col text_col +beta alpha +delta alpha +delta beta +epsilon alpha +epsilon beta +epsilon delta +gamma alpha +gamma beta +gamma delta +gamma epsilon +DROP TABLE t1, t2; +# +# PAD SPACE: COUNT(DISTINCT) on TEXT with trailing-space variants +# PAD SPACE collations (latin1 default) treat trailing spaces as +# insignificant, so 'abc' and 'abc ' are the same value. +# +CREATE TABLE t_pad (id INT AUTO_INCREMENT PRIMARY KEY, t TEXT); +INSERT INTO t_pad (t) VALUES ('abc'), ('abc '), ('abc '), ('def'), ('def '); +# Must return 2 (not 5): 'abc' variants = 1 group, 'def' variants = 1 group +SELECT COUNT(DISTINCT t) FROM t_pad; +COUNT(DISTINCT t) +2 +DROP TABLE t_pad; +# +# GROUP BY text_col must use HEAP (not Aria) for internal tmp tables +# +CREATE TABLE t_grp (id INT AUTO_INCREMENT PRIMARY KEY, t TEXT); +INSERT INTO t_grp (t) VALUES ('alpha'), ('alpha'), ('beta'), ('gamma'), ('gamma'); +FLUSH STATUS; +SELECT t, COUNT(*) AS cnt FROM t_grp GROUP BY t ORDER BY t; +t cnt +alpha 2 +beta 1 +gamma 2 +SHOW STATUS LIKE 'Created_tmp_disk_tables'; +Variable_name Value +Created_tmp_disk_tables 0 +# +# DISTINCT text_col must use HEAP (not Aria) for internal tmp tables +# +FLUSH STATUS; +SELECT DISTINCT t FROM t_grp ORDER BY t; +t +alpha +beta +gamma +SHOW STATUS LIKE 'Created_tmp_disk_tables'; +Variable_name Value +Created_tmp_disk_tables 0 +# +# GROUP BY text_col with PAD SPACE trailing-space data +# PAD SPACE collations collapse 'abc' and 'abc ' into one group +# +CREATE TABLE t_grp_pad (id INT AUTO_INCREMENT PRIMARY KEY, t TEXT); +INSERT INTO t_grp_pad (t) VALUES ('abc'), ('abc '), ('abc '), ('def'), ('def '); +FLUSH STATUS; +# Must return 2 groups, not 5 +SELECT t, COUNT(*) AS cnt FROM t_grp_pad GROUP BY t ORDER BY t; +t cnt +abc 3 +def 2 +SHOW STATUS LIKE 'Created_tmp_disk_tables'; +Variable_name Value +Created_tmp_disk_tables 0 +DROP TABLE t_grp, t_grp_pad; +# End of MDEV-38975 blob operations tests diff --git a/mysql-test/suite/heap/blob_ops.test b/mysql-test/suite/heap/blob_ops.test new file mode 100644 index 0000000000000..3877da2c3bd6b --- /dev/null +++ b/mysql-test/suite/heap/blob_ops.test @@ -0,0 +1,121 @@ +--source include/have_sequence.inc +--source include/not_embedded.inc + +--echo # +--echo # MDEV-38975: Blob/text operations exercising HEAP internal temp tables +--echo # + +CREATE TABLE t1 (id INT AUTO_INCREMENT PRIMARY KEY, k INT, text_col TEXT); +INSERT INTO t1 (k, text_col) VALUES + (1, 'alpha'), (1, 'alpha '), (1, 'beta'), + (2, 'gamma'), (2, 'gamma '), (2, 'delta'), + (3, 'alpha'), (3, 'epsilon'), (3, NULL), + (4, NULL), (4, NULL), (4, 'beta'); + +--echo # +--echo # COUNT(DISTINCT text_col) +--echo # +SELECT COUNT(DISTINCT text_col) FROM t1; + +--echo # +--echo # COUNT(DISTINCT text_col) with GROUP BY +--echo # +SELECT k, COUNT(DISTINCT text_col) FROM t1 GROUP BY k ORDER BY k; + +--echo # +--echo # IN-subquery with blob/text +--echo # +CREATE TABLE t2 (text_col TEXT); +INSERT INTO t2 VALUES ('alpha'), ('delta'); +SELECT id, text_col FROM t1 WHERE text_col IN (SELECT text_col FROM t2) ORDER BY id; + +--echo # +--echo # GROUP BY text_col ORDER BY aggregate LIMIT +--echo # +SELECT text_col, COUNT(*) AS cnt FROM t1 GROUP BY text_col ORDER BY cnt DESC, text_col LIMIT 3; + +--echo # +--echo # GROUP BY text_col WITH ROLLUP +--echo # +--sorted_result +SELECT text_col, COUNT(*) FROM t1 GROUP BY text_col WITH ROLLUP; + +--echo # +--echo # GROUP BY multiple columns WITH ROLLUP +--echo # +--sorted_result +SELECT k, text_col, COUNT(*) FROM t1 GROUP BY k, text_col WITH ROLLUP; + +--echo # +--echo # Window function with blob in SELECT list +--echo # +SELECT id, text_col, ROW_NUMBER() OVER (PARTITION BY k ORDER BY id) AS rn +FROM t1 WHERE k <= 2 ORDER BY id; + +--echo # +--echo # RANK with blob column +--echo # +SELECT text_col, k, RANK() OVER (ORDER BY text_col) AS rnk +FROM t1 WHERE text_col IS NOT NULL ORDER BY text_col, k, id; + +--echo # +--echo # Non-recursive CTE materializing blob column +--echo # +WITH cte AS (SELECT text_col, k FROM t1 WHERE k <= 2) +SELECT DISTINCT text_col FROM cte ORDER BY text_col; + +--echo # +--echo # CTE with self-join on blob column +--echo # +WITH cte AS (SELECT text_col, k FROM t1 WHERE text_col IS NOT NULL) +SELECT COUNT(*) FROM cte a JOIN cte b ON a.k = b.k AND a.text_col = b.text_col; + +--echo # +--echo # CTE referenced twice (forces materialization) +--echo # +WITH cte AS (SELECT DISTINCT text_col FROM t1 WHERE text_col IS NOT NULL) +SELECT a.text_col, b.text_col +FROM cte a JOIN cte b ON a.text_col > b.text_col ORDER BY a.text_col, b.text_col; + +DROP TABLE t1, t2; + +--echo # +--echo # PAD SPACE: COUNT(DISTINCT) on TEXT with trailing-space variants +--echo # PAD SPACE collations (latin1 default) treat trailing spaces as +--echo # insignificant, so 'abc' and 'abc ' are the same value. +--echo # +CREATE TABLE t_pad (id INT AUTO_INCREMENT PRIMARY KEY, t TEXT); +INSERT INTO t_pad (t) VALUES ('abc'), ('abc '), ('abc '), ('def'), ('def '); +--echo # Must return 2 (not 5): 'abc' variants = 1 group, 'def' variants = 1 group +SELECT COUNT(DISTINCT t) FROM t_pad; +DROP TABLE t_pad; + +--echo # +--echo # GROUP BY text_col must use HEAP (not Aria) for internal tmp tables +--echo # +CREATE TABLE t_grp (id INT AUTO_INCREMENT PRIMARY KEY, t TEXT); +INSERT INTO t_grp (t) VALUES ('alpha'), ('alpha'), ('beta'), ('gamma'), ('gamma'); +FLUSH STATUS; +SELECT t, COUNT(*) AS cnt FROM t_grp GROUP BY t ORDER BY t; +SHOW STATUS LIKE 'Created_tmp_disk_tables'; + +--echo # +--echo # DISTINCT text_col must use HEAP (not Aria) for internal tmp tables +--echo # +FLUSH STATUS; +SELECT DISTINCT t FROM t_grp ORDER BY t; +SHOW STATUS LIKE 'Created_tmp_disk_tables'; + +--echo # +--echo # GROUP BY text_col with PAD SPACE trailing-space data +--echo # PAD SPACE collations collapse 'abc' and 'abc ' into one group +--echo # +CREATE TABLE t_grp_pad (id INT AUTO_INCREMENT PRIMARY KEY, t TEXT); +INSERT INTO t_grp_pad (t) VALUES ('abc'), ('abc '), ('abc '), ('def'), ('def '); +FLUSH STATUS; +--echo # Must return 2 groups, not 5 +SELECT t, COUNT(*) AS cnt FROM t_grp_pad GROUP BY t ORDER BY t; +SHOW STATUS LIKE 'Created_tmp_disk_tables'; +DROP TABLE t_grp, t_grp_pad; + +--echo # End of MDEV-38975 blob operations tests diff --git a/mysql-test/suite/heap/blob_replace_overlap.result b/mysql-test/suite/heap/blob_replace_overlap.result new file mode 100644 index 0000000000000..a2fdccbe72d51 --- /dev/null +++ b/mysql-test/suite/heap/blob_replace_overlap.result @@ -0,0 +1,87 @@ +# +# REPLACE into HEAP table with blobs must preserve blob data integrity. +# +# When REPLACE deletes a row whose blob is stored in HP_BLOCK (zero-copy), +# the freed continuation records may be reused by the subsequent insert. +# The blob source data and write destination then overlap, so the copy +# must handle overlapping ranges correctly. +# +# +# Single blob column, self-referencing REPLACE +# +CREATE TABLE t ( +pk INT PRIMARY KEY, +b TEXT, +c VARCHAR(8), +UNIQUE(c) +) ENGINE=HEAP CHARACTER SET latin1; +INSERT INTO t VALUES (1, REPEAT('x', 561), 'foo'); +REPLACE INTO t SELECT * FROM t; +SELECT pk, LENGTH(b), b = REPEAT('x', 561) AS blob_ok, c FROM t; +pk LENGTH(b) blob_ok c +1 561 1 foo +DROP TABLE t; +# +# Multiple blob columns, different sizes +# +CREATE TABLE t2 ( +pk INT PRIMARY KEY, +b1 TEXT, +b2 MEDIUMTEXT, +c VARCHAR(8), +UNIQUE(c) +) ENGINE=HEAP CHARACTER SET latin1; +INSERT INTO t2 VALUES (1, REPEAT('a', 400), REPEAT('b', 800), 'bar'); +REPLACE INTO t2 SELECT * FROM t2; +SELECT pk, LENGTH(b1), b1 = REPEAT('a', 400) AS b1_ok, +LENGTH(b2), b2 = REPEAT('b', 800) AS b2_ok, c FROM t2; +pk LENGTH(b1) b1_ok LENGTH(b2) b2_ok c +1 400 1 800 1 bar +DROP TABLE t2; +# +# Blob sizes exercising different storage paths: +# single-record run, multi-record run, bulk inner-record copy +# +CREATE TABLE t3 ( +pk INT PRIMARY KEY, +b TEXT, +c VARCHAR(8), +UNIQUE(c) +) ENGINE=HEAP CHARACTER SET latin1; +# Small blob (single-record run) +INSERT INTO t3 VALUES (1, REPEAT('s', 10), 'k1'); +REPLACE INTO t3 SELECT * FROM t3; +SELECT pk, LENGTH(b), b = REPEAT('s', 10) AS blob_ok, c FROM t3; +pk LENGTH(b) blob_ok c +1 10 1 k1 +DELETE FROM t3; +# Medium blob (few continuation records) +INSERT INTO t3 VALUES (1, REPEAT('m', 200), 'k2'); +REPLACE INTO t3 SELECT * FROM t3; +SELECT pk, LENGTH(b), b = REPEAT('m', 200) AS blob_ok, c FROM t3; +pk LENGTH(b) blob_ok c +1 200 1 k2 +DELETE FROM t3; +# Large blob (many continuation records with bulk copy) +INSERT INTO t3 VALUES (1, REPEAT('L', 2000), 'k3'); +REPLACE INTO t3 SELECT * FROM t3; +SELECT pk, LENGTH(b), b = REPEAT('L', 2000) AS blob_ok, c FROM t3; +pk LENGTH(b) blob_ok c +1 2000 1 k3 +DROP TABLE t3; +# +# INSERT ON DUPLICATE KEY UPDATE with blob (update path, no delete+insert) +# +CREATE TABLE t4 ( +pk INT PRIMARY KEY, +b TEXT, +c VARCHAR(8), +UNIQUE(c) +) ENGINE=HEAP CHARACTER SET latin1; +INSERT INTO t4 VALUES (1, REPEAT('z', 561), 'dup'); +INSERT INTO t4 VALUES (1, REPEAT('z', 561), 'dup') +ON DUPLICATE KEY UPDATE b = VALUES(b); +SELECT pk, LENGTH(b), b = REPEAT('z', 561) AS blob_ok, c FROM t4; +pk LENGTH(b) blob_ok c +1 561 1 dup +DROP TABLE t4; diff --git a/mysql-test/suite/heap/blob_replace_overlap.test b/mysql-test/suite/heap/blob_replace_overlap.test new file mode 100644 index 0000000000000..db9b824b4ab5c --- /dev/null +++ b/mysql-test/suite/heap/blob_replace_overlap.test @@ -0,0 +1,94 @@ +--echo # +--echo # REPLACE into HEAP table with blobs must preserve blob data integrity. +--echo # +--echo # When REPLACE deletes a row whose blob is stored in HP_BLOCK (zero-copy), +--echo # the freed continuation records may be reused by the subsequent insert. +--echo # The blob source data and write destination then overlap, so the copy +--echo # must handle overlapping ranges correctly. +--echo # + +--echo # +--echo # Single blob column, self-referencing REPLACE +--echo # +CREATE TABLE t ( + pk INT PRIMARY KEY, + b TEXT, + c VARCHAR(8), + UNIQUE(c) +) ENGINE=HEAP CHARACTER SET latin1; + +INSERT INTO t VALUES (1, REPEAT('x', 561), 'foo'); + +REPLACE INTO t SELECT * FROM t; + +SELECT pk, LENGTH(b), b = REPEAT('x', 561) AS blob_ok, c FROM t; + +DROP TABLE t; + +--echo # +--echo # Multiple blob columns, different sizes +--echo # +CREATE TABLE t2 ( + pk INT PRIMARY KEY, + b1 TEXT, + b2 MEDIUMTEXT, + c VARCHAR(8), + UNIQUE(c) +) ENGINE=HEAP CHARACTER SET latin1; + +INSERT INTO t2 VALUES (1, REPEAT('a', 400), REPEAT('b', 800), 'bar'); +REPLACE INTO t2 SELECT * FROM t2; + +SELECT pk, LENGTH(b1), b1 = REPEAT('a', 400) AS b1_ok, + LENGTH(b2), b2 = REPEAT('b', 800) AS b2_ok, c FROM t2; + +DROP TABLE t2; + +--echo # +--echo # Blob sizes exercising different storage paths: +--echo # single-record run, multi-record run, bulk inner-record copy +--echo # +CREATE TABLE t3 ( + pk INT PRIMARY KEY, + b TEXT, + c VARCHAR(8), + UNIQUE(c) +) ENGINE=HEAP CHARACTER SET latin1; + +--echo # Small blob (single-record run) +INSERT INTO t3 VALUES (1, REPEAT('s', 10), 'k1'); +REPLACE INTO t3 SELECT * FROM t3; +SELECT pk, LENGTH(b), b = REPEAT('s', 10) AS blob_ok, c FROM t3; + +DELETE FROM t3; + +--echo # Medium blob (few continuation records) +INSERT INTO t3 VALUES (1, REPEAT('m', 200), 'k2'); +REPLACE INTO t3 SELECT * FROM t3; +SELECT pk, LENGTH(b), b = REPEAT('m', 200) AS blob_ok, c FROM t3; + +DELETE FROM t3; + +--echo # Large blob (many continuation records with bulk copy) +INSERT INTO t3 VALUES (1, REPEAT('L', 2000), 'k3'); +REPLACE INTO t3 SELECT * FROM t3; +SELECT pk, LENGTH(b), b = REPEAT('L', 2000) AS blob_ok, c FROM t3; + +DROP TABLE t3; + +--echo # +--echo # INSERT ON DUPLICATE KEY UPDATE with blob (update path, no delete+insert) +--echo # +CREATE TABLE t4 ( + pk INT PRIMARY KEY, + b TEXT, + c VARCHAR(8), + UNIQUE(c) +) ENGINE=HEAP CHARACTER SET latin1; + +INSERT INTO t4 VALUES (1, REPEAT('z', 561), 'dup'); +INSERT INTO t4 VALUES (1, REPEAT('z', 561), 'dup') + ON DUPLICATE KEY UPDATE b = VALUES(b); +SELECT pk, LENGTH(b), b = REPEAT('z', 561) AS blob_ok, c FROM t4; + +DROP TABLE t4; diff --git a/mysql-test/suite/heap/blob_replication.result b/mysql-test/suite/heap/blob_replication.result new file mode 100644 index 0000000000000..100f5fbbe02c9 --- /dev/null +++ b/mysql-test/suite/heap/blob_replication.result @@ -0,0 +1,155 @@ +include/master-slave.inc +[connection master] +# +# Test 1: HEAP blob replication with PRIMARY KEY +# +CREATE TABLE t ( +pk INT AUTO_INCREMENT PRIMARY KEY, +a TEXT, +b TEXT, +c INT +) ENGINE=HEAP; +INSERT INTO t VALUES (1,'','',0); +UPDATE t SET a = 'foo'; +DELETE FROM t; +connection slave; +# Verify slave is consistent +SELECT * FROM t; +pk a b c +connection master; +DROP TABLE t; +connection slave; +# +# Test 2: HEAP blob replication without PRIMARY KEY +# +connection master; +CREATE TABLE t2 ( +a TEXT, +b TEXT, +c INT +) ENGINE=HEAP; +INSERT INTO t2 VALUES ('','',0); +UPDATE t2 SET a = 'foo'; +DELETE FROM t2; +connection slave; +# Verify slave is consistent +SELECT * FROM t2; +a b c +connection master; +DROP TABLE t2; +# +# Test 3: MDEV-39782 -- UPDATE on HEAP blob with non-empty data +# +# heap_update() was freeing old blob chains immediately via +# hp_free_run_chain(), but binlog_log_row() reads the before-image +# after update_row() returns. The zero-copy pointers in the old +# record dangled into freed HP_BLOCK records, corrupting the +# binlog event. The slave then failed to find the row. +# +connection master; +CREATE TABLE t3 (a BLOB) ENGINE=HEAP; +INSERT INTO t3 VALUES (0); +UPDATE t3 SET a = 1; +INSERT INTO t3 VALUES ('hello world'); +UPDATE t3 SET a = 'goodbye' WHERE a = 'hello world'; +INSERT INTO t3 VALUES (REPEAT('x', 1000)); +UPDATE t3 SET a = REPEAT('y', 2000); +connection slave; +# Verify slave is consistent +SELECT LENGTH(a), LEFT(a, 20) FROM t3; +LENGTH(a) LEFT(a, 20) +2000 yyyyyyyyyyyyyyyyyyyy +2000 yyyyyyyyyyyyyyyyyyyy +2000 yyyyyyyyyyyyyyyyyyyy +connection master; +SELECT LENGTH(a), LEFT(a, 20) FROM t3; +LENGTH(a) LEFT(a, 20) +2000 yyyyyyyyyyyyyyyyyyyy +2000 yyyyyyyyyyyyyyyyyyyy +2000 yyyyyyyyyyyyyyyyyyyy +DROP TABLE t3; +connection slave; +# +# Test 4: UPDATE with multiple blob columns changing independently +# +connection master; +CREATE TABLE t4 (a TEXT, b BLOB, c INT) ENGINE=HEAP; +INSERT INTO t4 VALUES ('alpha', 'beta', 1); +UPDATE t4 SET a = 'gamma'; +UPDATE t4 SET b = 'delta'; +UPDATE t4 SET a = 'epsilon', b = 'zeta', c = 2; +connection slave; +# Verify slave is consistent +SELECT * FROM t4; +a b c +epsilon zeta 2 +connection master; +DROP TABLE t4; +connection slave; +# +# Test 5: UPDATE blob to/from NULL +# +connection master; +CREATE TABLE t5 (a BLOB) ENGINE=HEAP; +INSERT INTO t5 VALUES ('data'); +UPDATE t5 SET a = NULL; +UPDATE t5 SET a = 'restored'; +connection slave; +# Verify slave is consistent +SELECT * FROM t5; +a +restored +connection master; +DROP TABLE t5; +# +# Test 6: REPLACE with blob -- self-referencing and cross-table +# +# REPLACE deletes the conflicting row (deferred blob free) then +# inserts the new row. The insert must materialize the blob data +# before flushing the deferred free, and the materialized buffer +# must stay valid for binlog_log_row() after heap_write() returns. +# +connection master; +CREATE TABLE t6 ( +pk INT PRIMARY KEY, +b TEXT, +c VARCHAR(8), +UNIQUE(c) +) ENGINE=HEAP; +INSERT INTO t6 VALUES (1, REPEAT('x', 561), 'foo'); +REPLACE INTO t6 SELECT * FROM t6; +connection slave; +# Verify slave has correct blob data after self-REPLACE +SELECT pk, LENGTH(b), b = REPEAT('x', 561) AS blob_ok, c FROM t6; +pk LENGTH(b) blob_ok c +1 561 1 foo +connection master; +SELECT pk, LENGTH(b), b = REPEAT('x', 561) AS blob_ok, c FROM t6; +pk LENGTH(b) blob_ok c +1 561 1 foo +# REPLACE with new value via VALUES +REPLACE INTO t6 VALUES (1, REPEAT('y', 800), 'foo'); +connection slave; +SELECT pk, LENGTH(b), b = REPEAT('y', 800) AS blob_ok, c FROM t6; +pk LENGTH(b) blob_ok c +1 800 1 foo +connection master; +# REPLACE with multiple blob columns +CREATE TABLE t7 ( +pk INT PRIMARY KEY, +b1 TEXT, +b2 MEDIUMTEXT, +c VARCHAR(8), +UNIQUE(c) +) ENGINE=HEAP; +INSERT INTO t7 VALUES (1, REPEAT('a', 400), REPEAT('b', 800), 'bar'); +REPLACE INTO t7 SELECT * FROM t7; +connection slave; +SELECT pk, LENGTH(b1), b1 = REPEAT('a', 400) AS b1_ok, +LENGTH(b2), b2 = REPEAT('b', 800) AS b2_ok, c FROM t7; +pk LENGTH(b1) b1_ok LENGTH(b2) b2_ok c +1 400 1 800 1 bar +connection master; +DROP TABLE t6, t7; +connection slave; +include/rpl_end.inc diff --git a/mysql-test/suite/heap/blob_replication.test b/mysql-test/suite/heap/blob_replication.test new file mode 100644 index 0000000000000..7cddd4565ced8 --- /dev/null +++ b/mysql-test/suite/heap/blob_replication.test @@ -0,0 +1,201 @@ +--source include/have_binlog_format_row.inc +--source include/master-slave.inc + +# +# Row-based replication of DML on HEAP tables with BLOB/TEXT columns. +# +# Tests that INSERT, UPDATE, and DELETE on HEAP blob tables replicate +# correctly to the slave, covering two scenarios: +# +# 1. Table with a PRIMARY KEY -- the slave uses rnd_pos_by_record() +# to locate rows. Validates that hp_free_blobs() does not chase +# stale pointers for zero-length blob columns. +# +# 2. Table without a PRIMARY KEY -- the slave uses a full table scan +# with record_compare() to locate rows. Validates that the +# master's binlog before-image contains valid blob data (not +# dangling zero-copy pointers freed by hp_free_blobs()). +# + +--echo # +--echo # Test 1: HEAP blob replication with PRIMARY KEY +--echo # + +CREATE TABLE t ( + pk INT AUTO_INCREMENT PRIMARY KEY, + a TEXT, + b TEXT, + c INT +) ENGINE=HEAP; + +INSERT INTO t VALUES (1,'','',0); +UPDATE t SET a = 'foo'; +DELETE FROM t; + +--sync_slave_with_master + +--echo # Verify slave is consistent +SELECT * FROM t; + +--connection master +DROP TABLE t; +--sync_slave_with_master + +--echo # +--echo # Test 2: HEAP blob replication without PRIMARY KEY +--echo # + +--connection master + +CREATE TABLE t2 ( + a TEXT, + b TEXT, + c INT +) ENGINE=HEAP; + +INSERT INTO t2 VALUES ('','',0); +UPDATE t2 SET a = 'foo'; +DELETE FROM t2; + +--sync_slave_with_master + +--echo # Verify slave is consistent +SELECT * FROM t2; + +--connection master +DROP TABLE t2; + +--echo # +--echo # Test 3: MDEV-39782 -- UPDATE on HEAP blob with non-empty data +--echo # +--echo # heap_update() was freeing old blob chains immediately via +--echo # hp_free_run_chain(), but binlog_log_row() reads the before-image +--echo # after update_row() returns. The zero-copy pointers in the old +--echo # record dangled into freed HP_BLOCK records, corrupting the +--echo # binlog event. The slave then failed to find the row. +--echo # + +--connection master + +CREATE TABLE t3 (a BLOB) ENGINE=HEAP; + +INSERT INTO t3 VALUES (0); +UPDATE t3 SET a = 1; + +INSERT INTO t3 VALUES ('hello world'); +UPDATE t3 SET a = 'goodbye' WHERE a = 'hello world'; + +INSERT INTO t3 VALUES (REPEAT('x', 1000)); +UPDATE t3 SET a = REPEAT('y', 2000); + +--sync_slave_with_master + +--echo # Verify slave is consistent +--sorted_result +SELECT LENGTH(a), LEFT(a, 20) FROM t3; + +--connection master +--sorted_result +SELECT LENGTH(a), LEFT(a, 20) FROM t3; +DROP TABLE t3; +--sync_slave_with_master + +--echo # +--echo # Test 4: UPDATE with multiple blob columns changing independently +--echo # + +--connection master + +CREATE TABLE t4 (a TEXT, b BLOB, c INT) ENGINE=HEAP; + +INSERT INTO t4 VALUES ('alpha', 'beta', 1); +UPDATE t4 SET a = 'gamma'; +UPDATE t4 SET b = 'delta'; +UPDATE t4 SET a = 'epsilon', b = 'zeta', c = 2; + +--sync_slave_with_master + +--echo # Verify slave is consistent +SELECT * FROM t4; + +--connection master +DROP TABLE t4; +--sync_slave_with_master + +--echo # +--echo # Test 5: UPDATE blob to/from NULL +--echo # + +--connection master + +CREATE TABLE t5 (a BLOB) ENGINE=HEAP; + +INSERT INTO t5 VALUES ('data'); +UPDATE t5 SET a = NULL; +UPDATE t5 SET a = 'restored'; + +--sync_slave_with_master + +--echo # Verify slave is consistent +SELECT * FROM t5; + +--connection master +DROP TABLE t5; + +--echo # +--echo # Test 6: REPLACE with blob -- self-referencing and cross-table +--echo # +--echo # REPLACE deletes the conflicting row (deferred blob free) then +--echo # inserts the new row. The insert must materialize the blob data +--echo # before flushing the deferred free, and the materialized buffer +--echo # must stay valid for binlog_log_row() after heap_write() returns. +--echo # + +--connection master + +CREATE TABLE t6 ( + pk INT PRIMARY KEY, + b TEXT, + c VARCHAR(8), + UNIQUE(c) +) ENGINE=HEAP; + +INSERT INTO t6 VALUES (1, REPEAT('x', 561), 'foo'); +REPLACE INTO t6 SELECT * FROM t6; + +--sync_slave_with_master +--echo # Verify slave has correct blob data after self-REPLACE +SELECT pk, LENGTH(b), b = REPEAT('x', 561) AS blob_ok, c FROM t6; + +--connection master +SELECT pk, LENGTH(b), b = REPEAT('x', 561) AS blob_ok, c FROM t6; + +--echo # REPLACE with new value via VALUES +REPLACE INTO t6 VALUES (1, REPEAT('y', 800), 'foo'); + +--sync_slave_with_master +SELECT pk, LENGTH(b), b = REPEAT('y', 800) AS blob_ok, c FROM t6; + +--connection master + +--echo # REPLACE with multiple blob columns +CREATE TABLE t7 ( + pk INT PRIMARY KEY, + b1 TEXT, + b2 MEDIUMTEXT, + c VARCHAR(8), + UNIQUE(c) +) ENGINE=HEAP; + +INSERT INTO t7 VALUES (1, REPEAT('a', 400), REPEAT('b', 800), 'bar'); +REPLACE INTO t7 SELECT * FROM t7; + +--sync_slave_with_master +SELECT pk, LENGTH(b1), b1 = REPEAT('a', 400) AS b1_ok, + LENGTH(b2), b2 = REPEAT('b', 800) AS b2_ok, c FROM t7; + +--connection master +DROP TABLE t6, t7; +--sync_slave_with_master + +--source include/rpl_end.inc diff --git a/mysql-test/suite/heap/blob_setvar_groupby.result b/mysql-test/suite/heap/blob_setvar_groupby.result new file mode 100644 index 0000000000000..0b6bec169d0ee --- /dev/null +++ b/mysql-test/suite/heap/blob_setvar_groupby.result @@ -0,0 +1,73 @@ +# +# Test GROUP BY on blob/text columns wrapped in user variable +# assignment expressions (@x := col). +# +# This exercises the hp_key_cmp() code path with Field_blob_key +# having packlength < 4. When the GROUP BY item is an expression +# like @x := col (Item_func_set_user_var), the temp table field +# is created via create_tmp_field_ex_from_handler(), which calls +# set_pack_length(blob_handler->length_bytes()) to preserve the +# original blob subtype. This produces Field_blob_key with +# packlength=1 (TINYBLOB), 2 (BLOB/TEXT), or 3 (MEDIUMBLOB) +# instead of the default 4. +# +# In contrast, blob_group_by.test uses plain Item_field column +# references, which go through create_tmp_field_from_item_field() +# -> Field_blob::make_new_field() -> Field_blob_key constructor +# that always sets packlength=4 regardless of the source type. +# +# hp_key_cmp() reads the stored record's blob length prefix using +# packlength, then reads the chain pointer at offset packlength. +# With a wrong packlength, both values are garbage: +# - PAD SPACE collation (TEXT): proceeds to materialize via the +# garbage chain pointer, crashing in hp_is_single_rec() +# - NO PAD collation (BLOB): garbage length != key length causes +# early mismatch, so the lookup misses and the subsequent write +# hits ER_DUP_ENTRY +# +# TEXT: PAD SPACE collation, packlength=2 +CREATE TABLE t1 (a TEXT); +INSERT INTO t1 VALUES (0),(0); +SELECT @x := a, count(*) FROM t1 GROUP BY 1; +@x := a count(*) +0 2 +DROP TABLE t1; +# BLOB: NO PAD (binary) collation, packlength=2 +CREATE TABLE t2 (a BLOB); +INSERT INTO t2 VALUES (0),(0); +SELECT @x := a, count(*) FROM t2 GROUP BY 1; +@x := a count(*) +0 2 +DROP TABLE t2; +# TINYBLOB (packlength=1) +CREATE TABLE t3 (a TINYBLOB); +INSERT INTO t3 VALUES ('abc'),('abc'),('def'); +SELECT @x := a, count(*) FROM t3 GROUP BY 1; +@x := a count(*) +abc 2 +def 1 +DROP TABLE t3; +# MEDIUMTEXT (packlength=3) +CREATE TABLE t4 (a MEDIUMTEXT); +INSERT INTO t4 VALUES ('hello'),('hello'),('world'); +SELECT @x := a, count(*) FROM t4 GROUP BY 1; +@x := a count(*) +hello 2 +world 1 +DROP TABLE t4; +# Multiple blob columns with variable assignment +CREATE TABLE t5 (a TEXT, b BLOB); +INSERT INTO t5 VALUES ('x','y'),('x','y'),('a','b'); +SELECT @x := a, @y := b, count(*) FROM t5 GROUP BY 1, 2; +@x := a @y := b count(*) +a b 1 +x y 2 +DROP TABLE t5; +# Verify non-expression blob GROUP BY still works +CREATE TABLE t6 (a TEXT); +INSERT INTO t6 VALUES ('foo'),('foo'),('bar'); +SELECT a, count(*) FROM t6 GROUP BY a; +a count(*) +bar 1 +foo 2 +DROP TABLE t6; diff --git a/mysql-test/suite/heap/blob_setvar_groupby.test b/mysql-test/suite/heap/blob_setvar_groupby.test new file mode 100644 index 0000000000000..db2d414cc2e55 --- /dev/null +++ b/mysql-test/suite/heap/blob_setvar_groupby.test @@ -0,0 +1,63 @@ +--echo # +--echo # Test GROUP BY on blob/text columns wrapped in user variable +--echo # assignment expressions (@x := col). +--echo # +--echo # This exercises the hp_key_cmp() code path with Field_blob_key +--echo # having packlength < 4. When the GROUP BY item is an expression +--echo # like @x := col (Item_func_set_user_var), the temp table field +--echo # is created via create_tmp_field_ex_from_handler(), which calls +--echo # set_pack_length(blob_handler->length_bytes()) to preserve the +--echo # original blob subtype. This produces Field_blob_key with +--echo # packlength=1 (TINYBLOB), 2 (BLOB/TEXT), or 3 (MEDIUMBLOB) +--echo # instead of the default 4. +--echo # +--echo # In contrast, blob_group_by.test uses plain Item_field column +--echo # references, which go through create_tmp_field_from_item_field() +--echo # -> Field_blob::make_new_field() -> Field_blob_key constructor +--echo # that always sets packlength=4 regardless of the source type. +--echo # +--echo # hp_key_cmp() reads the stored record's blob length prefix using +--echo # packlength, then reads the chain pointer at offset packlength. +--echo # With a wrong packlength, both values are garbage: +--echo # - PAD SPACE collation (TEXT): proceeds to materialize via the +--echo # garbage chain pointer, crashing in hp_is_single_rec() +--echo # - NO PAD collation (BLOB): garbage length != key length causes +--echo # early mismatch, so the lookup misses and the subsequent write +--echo # hits ER_DUP_ENTRY +--echo # + +--echo # TEXT: PAD SPACE collation, packlength=2 +CREATE TABLE t1 (a TEXT); +INSERT INTO t1 VALUES (0),(0); +SELECT @x := a, count(*) FROM t1 GROUP BY 1; +DROP TABLE t1; + +--echo # BLOB: NO PAD (binary) collation, packlength=2 +CREATE TABLE t2 (a BLOB); +INSERT INTO t2 VALUES (0),(0); +SELECT @x := a, count(*) FROM t2 GROUP BY 1; +DROP TABLE t2; + +--echo # TINYBLOB (packlength=1) +CREATE TABLE t3 (a TINYBLOB); +INSERT INTO t3 VALUES ('abc'),('abc'),('def'); +SELECT @x := a, count(*) FROM t3 GROUP BY 1; +DROP TABLE t3; + +--echo # MEDIUMTEXT (packlength=3) +CREATE TABLE t4 (a MEDIUMTEXT); +INSERT INTO t4 VALUES ('hello'),('hello'),('world'); +SELECT @x := a, count(*) FROM t4 GROUP BY 1; +DROP TABLE t4; + +--echo # Multiple blob columns with variable assignment +CREATE TABLE t5 (a TEXT, b BLOB); +INSERT INTO t5 VALUES ('x','y'),('x','y'),('a','b'); +SELECT @x := a, @y := b, count(*) FROM t5 GROUP BY 1, 2; +DROP TABLE t5; + +--echo # Verify non-expression blob GROUP BY still works +CREATE TABLE t6 (a TEXT); +INSERT INTO t6 VALUES ('foo'),('foo'),('bar'); +SELECT a, count(*) FROM t6 GROUP BY a; +DROP TABLE t6; diff --git a/mysql-test/suite/heap/blob_stress.result b/mysql-test/suite/heap/blob_stress.result new file mode 100644 index 0000000000000..28cd2e614dede --- /dev/null +++ b/mysql-test/suite/heap/blob_stress.result @@ -0,0 +1,181 @@ +drop table if exists t1, t_verify; +# +# Phase 1: Sustained insert/delete/update churn (200 cycles) +# +set @save_max_heap= @@max_heap_table_size; +set max_heap_table_size= 64*1024*1024; +create table t1 ( +id int not null primary key, +b1 blob, +b2 blob, +v int not null default 0 +) engine=memory; +create table t_verify ( +id int not null primary key +) engine=memory; +create procedure blob_stress(in cycles int) +begin +declare i int default 0; +declare base_id int; +declare sz1 int; +declare sz2 int; +declare del_id int; +declare heap_cnt int; +declare verify_cnt int; +while i < cycles do +set base_id = i * 5; +# Vary blob sizes: cycle through Case A (tiny), B (medium), C (large) +set sz1 = case i % 6 +when 0 then 5 + (i % 15) +when 1 then 2000 + (i % 10) * 500 +when 2 then 15000 + (i % 7) * 1000 +when 3 then 4000 + (i % 8) * 500 +when 4 then 18000 + (i % 5) * 1000 +else 10 + (i % 10) +end; +set sz2 = case (i + 3) % 6 +when 0 then 8 + (i % 12) +when 1 then 3000 + (i % 9) * 400 +when 2 then 14000 + (i % 6) * 800 +when 3 then 5000 + (i % 11) * 300 +when 4 then 16000 + (i % 4) * 1000 +else 15 + (i % 8) +end; +# INSERT 5 rows per iteration with different blob patterns +insert into t1 values +(base_id, repeat(char(65 + (i % 26)), sz1), +repeat(char(97 + (i % 26)), sz2), i); +insert into t_verify values (base_id); +insert into t1 values +(base_id + 1, repeat(char(66 + (i % 26)), sz2), null, i); +insert into t_verify values (base_id + 1); +insert into t1 values +(base_id + 2, null, repeat(char(67 + (i % 26)), sz1), i); +insert into t_verify values (base_id + 2); +insert into t1 values +(base_id + 3, repeat(char(68 + (i % 26)), greatest(sz1, sz2)), +repeat(char(69 + (i % 26)), least(sz1, sz2)), i); +insert into t_verify values (base_id + 3); +insert into t1 values +(base_id + 4, repeat(char(70 + (i % 26)), sz1 div 2), +repeat(char(71 + (i % 26)), sz2 div 2), i); +insert into t_verify values (base_id + 4); +# DELETE rows from earlier iterations (free-list churn) +if i >= 3 then +set del_id = (i - 3) * 5 + (i % 3); +delete from t1 where id = del_id; +delete from t_verify where id = del_id; +set del_id = (i - 2) * 5 + ((i + 1) % 5); +delete from t1 where id = del_id; +delete from t_verify where id = del_id; +end if; +# UPDATE: grow small blobs to large, shrink large to small, +# update non-blob column (blob chains must survive) +if i >= 2 then +update t1 set b1 = repeat(char(72 + (i % 26)), 20000 + (i % 3) * 2000) +where id = (i - 2) * 5 and b1 is not null; +update t1 set b2 = repeat(char(73 + (i % 26)), 8 + (i % 10)) +where id = (i - 2) * 5 + 3; +update t1 set v = v + 100 where id = (i - 1) * 5 + 1; +# NULL -> non-NULL: update rows that had NULL b2 to have data +update t1 set b2 = repeat(char(74 + (i % 26)), 3000 + (i % 5) * 500) +where id = (i - 2) * 5 + 2 and b2 is null; +# non-NULL -> NULL: set b1 to NULL for some rows +update t1 set b1 = null +where id = (i - 1) * 5 + 3 and b1 is not null; +end if; +# Inline integrity check every 50 iterations +if i % 50 = 49 then +set heap_cnt = (select count(*) from t1); +set verify_cnt = (select count(*) from t_verify); +if heap_cnt != verify_cnt then +signal sqlstate '45000' + set message_text = 'Row count mismatch during stress'; +end if; +end if; +set i = i + 1; +end while; +end | +# Phase 1 verification +# (HEAP does not support CHECK TABLE) +select (select count(*) from t1) = (select count(*) from t_verify) +as row_counts_match; +row_counts_match +1 +select count(*) as missing_from_heap +from t_verify where id not in (select id from t1); +missing_from_heap +0 +select count(*) as extra_in_heap +from t1 where id not in (select id from t_verify); +extra_in_heap +0 +select count(*) as corrupted_b1 from t1 +where b1 is not null and length(b1) > 0 +and b1 != repeat(left(b1, 1), length(b1)); +corrupted_b1 +0 +select count(*) as corrupted_b2 from t1 +where b2 is not null and length(b2) > 0 +and b2 != repeat(left(b2, 1), length(b2)); +corrupted_b2 +0 +drop procedure blob_stress; +drop table t1, t_verify; +set max_heap_table_size= @save_max_heap; +# +# Phase 2: Near-capacity stress with fragmentation recovery +# +set @save_max_heap= @@max_heap_table_size; +set max_heap_table_size= 2*1024*1024; +create table t1 ( +id int not null primary key, +b blob +) engine=memory; +select count(*) as rows_after_fill from t1; +rows_after_fill +100 +select count(*) as rows_after_fragment from t1; +rows_after_fragment +66 +# Phase 2 verification after fragmented reinsert +# (HEAP does not support CHECK TABLE) +select count(*) as corrupted from t1 +where b is not null and length(b) > 0 +and b != repeat(left(b, 1), length(b)); +corrupted +0 +delete from t1; +# Phase 2 verification after full-delete reinsert +# (HEAP does not support CHECK TABLE) +select count(*) as final_rows from t1; +final_rows +80 +select count(*) as corrupted from t1 +where b is not null and length(b) > 0 +and b != repeat(left(b, 1), length(b)); +corrupted +0 +drop table t1; +set max_heap_table_size= @save_max_heap; +# +# Phase 3: Repeated UPDATE grow/shrink cycles +# +set @save_max_heap= @@max_heap_table_size; +set max_heap_table_size= 2*1024*1024; +create table t1 ( +id int not null primary key, +b longblob +) engine=memory; +# Phase 3 verification +# (HEAP does not support CHECK TABLE) +select count(*) as row_count from t1; +row_count +50 +select count(*) as corrupted from t1 +where b is not null and length(b) > 0 +and b != repeat(left(b, 1), length(b)); +corrupted +0 +drop table t1; +set max_heap_table_size= @save_max_heap; diff --git a/mysql-test/suite/heap/blob_stress.test b/mysql-test/suite/heap/blob_stress.test new file mode 100644 index 0000000000000..eba2c15bf7741 --- /dev/null +++ b/mysql-test/suite/heap/blob_stress.test @@ -0,0 +1,305 @@ +# +# MDEV-38975: Stress test for HEAP blob insert/delete/update cycles +# +# Exercises free-list fragmentation, continuation chain reuse, +# allocation/deallocation balance, and data integrity under +# sustained mixed DML workload with varying blob sizes. +# +# Phase 1: 200-cycle stored procedure with mixed INSERT/DELETE/UPDATE, +# varying blob sizes (Case A/B/C), shadow table verification. +# Phase 2: Near-capacity fill/fragment/refill with free-list scavenging. +# Phase 3: Repeated UPDATE grow/shrink cycles. +# + +--disable_warnings +drop table if exists t1, t_verify; +--enable_warnings + +--echo # +--echo # Phase 1: Sustained insert/delete/update churn (200 cycles) +--echo # + +set @save_max_heap= @@max_heap_table_size; +set max_heap_table_size= 64*1024*1024; + +create table t1 ( + id int not null primary key, + b1 blob, + b2 blob, + v int not null default 0 +) engine=memory; + +create table t_verify ( + id int not null primary key +) engine=memory; + +delimiter |; + +create procedure blob_stress(in cycles int) +begin + declare i int default 0; + declare base_id int; + declare sz1 int; + declare sz2 int; + declare del_id int; + declare heap_cnt int; + declare verify_cnt int; + + while i < cycles do + set base_id = i * 5; + + # Vary blob sizes: cycle through Case A (tiny), B (medium), C (large) + set sz1 = case i % 6 + when 0 then 5 + (i % 15) + when 1 then 2000 + (i % 10) * 500 + when 2 then 15000 + (i % 7) * 1000 + when 3 then 4000 + (i % 8) * 500 + when 4 then 18000 + (i % 5) * 1000 + else 10 + (i % 10) + end; + + set sz2 = case (i + 3) % 6 + when 0 then 8 + (i % 12) + when 1 then 3000 + (i % 9) * 400 + when 2 then 14000 + (i % 6) * 800 + when 3 then 5000 + (i % 11) * 300 + when 4 then 16000 + (i % 4) * 1000 + else 15 + (i % 8) + end; + + # INSERT 5 rows per iteration with different blob patterns + insert into t1 values + (base_id, repeat(char(65 + (i % 26)), sz1), + repeat(char(97 + (i % 26)), sz2), i); + insert into t_verify values (base_id); + + insert into t1 values + (base_id + 1, repeat(char(66 + (i % 26)), sz2), null, i); + insert into t_verify values (base_id + 1); + + insert into t1 values + (base_id + 2, null, repeat(char(67 + (i % 26)), sz1), i); + insert into t_verify values (base_id + 2); + + insert into t1 values + (base_id + 3, repeat(char(68 + (i % 26)), greatest(sz1, sz2)), + repeat(char(69 + (i % 26)), least(sz1, sz2)), i); + insert into t_verify values (base_id + 3); + + insert into t1 values + (base_id + 4, repeat(char(70 + (i % 26)), sz1 div 2), + repeat(char(71 + (i % 26)), sz2 div 2), i); + insert into t_verify values (base_id + 4); + + # DELETE rows from earlier iterations (free-list churn) + if i >= 3 then + set del_id = (i - 3) * 5 + (i % 3); + delete from t1 where id = del_id; + delete from t_verify where id = del_id; + + set del_id = (i - 2) * 5 + ((i + 1) % 5); + delete from t1 where id = del_id; + delete from t_verify where id = del_id; + end if; + + # UPDATE: grow small blobs to large, shrink large to small, + # update non-blob column (blob chains must survive) + if i >= 2 then + update t1 set b1 = repeat(char(72 + (i % 26)), 20000 + (i % 3) * 2000) + where id = (i - 2) * 5 and b1 is not null; + + update t1 set b2 = repeat(char(73 + (i % 26)), 8 + (i % 10)) + where id = (i - 2) * 5 + 3; + + update t1 set v = v + 100 where id = (i - 1) * 5 + 1; + + # NULL -> non-NULL: update rows that had NULL b2 to have data + update t1 set b2 = repeat(char(74 + (i % 26)), 3000 + (i % 5) * 500) + where id = (i - 2) * 5 + 2 and b2 is null; + + # non-NULL -> NULL: set b1 to NULL for some rows + update t1 set b1 = null + where id = (i - 1) * 5 + 3 and b1 is not null; + end if; + + # Inline integrity check every 50 iterations + if i % 50 = 49 then + set heap_cnt = (select count(*) from t1); + set verify_cnt = (select count(*) from t_verify); + if heap_cnt != verify_cnt then + signal sqlstate '45000' + set message_text = 'Row count mismatch during stress'; + end if; + end if; + + set i = i + 1; + end while; +end | + +delimiter ;| + +--disable_query_log +call blob_stress(200); +--enable_query_log + +--echo # Phase 1 verification +--echo # (HEAP does not support CHECK TABLE) + +select (select count(*) from t1) = (select count(*) from t_verify) + as row_counts_match; + +select count(*) as missing_from_heap + from t_verify where id not in (select id from t1); + +select count(*) as extra_in_heap + from t1 where id not in (select id from t_verify); + +# Every non-NULL blob must be a repeat() of a single character. +# Corruption would produce mixed characters. +select count(*) as corrupted_b1 from t1 + where b1 is not null and length(b1) > 0 + and b1 != repeat(left(b1, 1), length(b1)); + +select count(*) as corrupted_b2 from t1 + where b2 is not null and length(b2) > 0 + and b2 != repeat(left(b2, 1), length(b2)); + +drop procedure blob_stress; +drop table t1, t_verify; +set max_heap_table_size= @save_max_heap; + +--echo # +--echo # Phase 2: Near-capacity stress with fragmentation recovery +--echo # + +set @save_max_heap= @@max_heap_table_size; +set max_heap_table_size= 2*1024*1024; + +create table t1 ( + id int not null primary key, + b blob +) engine=memory; + +# Fill table with medium blobs to approach capacity +--disable_query_log +--let $i= 0 +while ($i < 100) +{ + eval insert into t1 values ($i, repeat(char(65 + ($i % 26)), 5000 + ($i % 10) * 500)); + --inc $i +} +--enable_query_log + +select count(*) as rows_after_fill from t1; + +# Delete every third row to fragment the free list +--disable_query_log +--let $i= 0 +while ($i < 100) +{ + eval delete from t1 where id= $i; + --inc $i + --inc $i + --inc $i +} +--enable_query_log + +select count(*) as rows_after_fragment from t1; + +# Reinsert with different sizes -- exercises free-list scavenging +--disable_query_log +--let $i= 200 +while ($i < 260) +{ + eval insert into t1 values ($i, repeat(char(65 + ($i % 26)), 4000 + ($i % 7) * 500)); + --inc $i +} +--enable_query_log + +--echo # Phase 2 verification after fragmented reinsert +--echo # (HEAP does not support CHECK TABLE) + +select count(*) as corrupted from t1 + where b is not null and length(b) > 0 + and b != repeat(left(b, 1), length(b)); + +# Delete all and reinsert -- full free-list reuse +delete from t1; + +--disable_query_log +--let $i= 300 +while ($i < 380) +{ + eval insert into t1 values ($i, repeat(char(65 + ($i % 26)), 3000 + ($i % 12) * 500)); + --inc $i +} +--enable_query_log + +--echo # Phase 2 verification after full-delete reinsert +--echo # (HEAP does not support CHECK TABLE) + +select count(*) as final_rows from t1; + +select count(*) as corrupted from t1 + where b is not null and length(b) > 0 + and b != repeat(left(b, 1), length(b)); + +drop table t1; +set max_heap_table_size= @save_max_heap; + +--echo # +--echo # Phase 3: Repeated UPDATE grow/shrink cycles +--echo # + +set @save_max_heap= @@max_heap_table_size; +set max_heap_table_size= 2*1024*1024; + +create table t1 ( + id int not null primary key, + b longblob +) engine=memory; + +# Seed with small blobs +--disable_query_log +--let $i= 0 +while ($i < 50) +{ + eval insert into t1 values ($i, repeat(char(65 + ($i % 26)), 100)); + --inc $i +} +--enable_query_log + +# 20 grow/shrink cycles: even rows grow large, odd rows shrink to tiny +--disable_query_log +--let $cycle= 0 +while ($cycle < 20) +{ + --let $i= 0 + while ($i < 50) + { + eval update t1 set b= repeat(char(65 + (($i + $cycle) % 26)), 10000 + ($cycle % 5) * 2000) where id= $i; + --inc $i + --inc $i + } + --let $i= 1 + while ($i < 50) + { + eval update t1 set b= repeat(char(97 + (($i + $cycle) % 26)), 5 + ($cycle % 15)) where id= $i; + --inc $i + --inc $i + } + --inc $cycle +} +--enable_query_log + +--echo # Phase 3 verification +--echo # (HEAP does not support CHECK TABLE) + +select count(*) as row_count from t1; + +select count(*) as corrupted from t1 + where b is not null and length(b) > 0 + and b != repeat(left(b, 1), length(b)); + +drop table t1; +set max_heap_table_size= @save_max_heap; diff --git a/mysql-test/suite/heap/blob_truncate.result b/mysql-test/suite/heap/blob_truncate.result new file mode 100644 index 0000000000000..c4f64864ce211 --- /dev/null +++ b/mysql-test/suite/heap/blob_truncate.result @@ -0,0 +1,31 @@ +# +# DELETE + TRUNCATE + INSERT: deferred blob chains must not +# survive heap_clear() +# +CREATE TABLE t ( +id INT, +a TEXT, +b TEXT +) ENGINE=HEAP; +INSERT INTO t VALUES (1, REPEAT('x', 500), REPEAT('y', 500)); +INSERT INTO t VALUES (2, REPEAT('a', 1000), REPEAT('b', 1000)); +DELETE FROM t WHERE id = 1; +TRUNCATE TABLE t; +INSERT INTO t VALUES (3, REPEAT('z', 200), REPEAT('w', 200)); +SELECT id, LENGTH(a), LENGTH(b) FROM t; +id LENGTH(a) LENGTH(b) +3 200 200 +# +# Multiple rounds: DELETE all + TRUNCATE + refill +# +INSERT INTO t VALUES (4, REPEAT('m', 800), REPEAT('n', 800)); +DELETE FROM t WHERE id = 3; +DELETE FROM t WHERE id = 4; +TRUNCATE TABLE t; +INSERT INTO t VALUES (5, REPEAT('p', 300), REPEAT('q', 300)); +INSERT INTO t VALUES (6, REPEAT('r', 1500), REPEAT('s', 1500)); +SELECT id, LENGTH(a), LENGTH(b) FROM t ORDER BY id; +id LENGTH(a) LENGTH(b) +5 300 300 +6 1500 1500 +DROP TABLE t; diff --git a/mysql-test/suite/heap/blob_truncate.test b/mysql-test/suite/heap/blob_truncate.test new file mode 100644 index 0000000000000..e513c3685ae85 --- /dev/null +++ b/mysql-test/suite/heap/blob_truncate.test @@ -0,0 +1,44 @@ +# +# HEAP blob deferred free + TRUNCATE interaction. +# +# Validates that TRUNCATE TABLE (heap_clear) correctly invalidates +# pending deferred blob chain pointers. Without the fix, DELETE +# saves chain pointers into pending_blob_chains[], then TRUNCATE +# frees the HP_BLOCK tree. The next INSERT triggers a flush that +# dereferences the now-dangling chain pointers -- use-after-free. +# + +--echo # +--echo # DELETE + TRUNCATE + INSERT: deferred blob chains must not +--echo # survive heap_clear() +--echo # + +CREATE TABLE t ( + id INT, + a TEXT, + b TEXT +) ENGINE=HEAP; + +INSERT INTO t VALUES (1, REPEAT('x', 500), REPEAT('y', 500)); +INSERT INTO t VALUES (2, REPEAT('a', 1000), REPEAT('b', 1000)); + +DELETE FROM t WHERE id = 1; +TRUNCATE TABLE t; +INSERT INTO t VALUES (3, REPEAT('z', 200), REPEAT('w', 200)); + +SELECT id, LENGTH(a), LENGTH(b) FROM t; + +--echo # +--echo # Multiple rounds: DELETE all + TRUNCATE + refill +--echo # + +INSERT INTO t VALUES (4, REPEAT('m', 800), REPEAT('n', 800)); +DELETE FROM t WHERE id = 3; +DELETE FROM t WHERE id = 4; +TRUNCATE TABLE t; +INSERT INTO t VALUES (5, REPEAT('p', 300), REPEAT('q', 300)); +INSERT INTO t VALUES (6, REPEAT('r', 1500), REPEAT('s', 1500)); + +SELECT id, LENGTH(a), LENGTH(b) FROM t ORDER BY id; + +DROP TABLE t; diff --git a/mysql-test/suite/heap/blob_update_overflow.result b/mysql-test/suite/heap/blob_update_overflow.result new file mode 100644 index 0000000000000..4eb640d311173 --- /dev/null +++ b/mysql-test/suite/heap/blob_update_overflow.result @@ -0,0 +1,131 @@ +# +# MDEV-38975: Overflow-to-Aria on UPDATE path for GROUP BY temp tables +# +# When ha_update_tmp_row() fails with HA_ERR_RECORD_FILE_FULL on a +# HEAP temp table (blob aggregate grows during UPDATE), the server +# should transparently convert to Aria and retry the update, just +# as the INSERT path already does. +# +# MAX(TEXT) with monotonically growing values triggers this: each +# update stores a longer blob via result_field->store(), and +# heap_update()'s write-before-free strategy requires old + new +# continuation chains to coexist briefly, exhausting capacity. +# +SET @save_max_heap= @@max_heap_table_size; +SET @save_tmp= @@tmp_table_size; +# +# Test 1: MAX(TEXT) overflow during end_update() +# +# 3 groups x 50 interleaved rows. val = REPEAT('a', seq * 200). +# Since seq grows, each group's MAX increases on every update. +# The HEAP temp table (32KB, recbuffer=16, ~2048 records) overflows +# when per-group blob chains reach ~500 records (~8KB each). +# +SET max_heap_table_size = 32768; +SET tmp_table_size = 32768; +CREATE TABLE t1 (grp INT, val TEXT); +INSERT INTO t1 SELECT (seq % 3) + 1, REPEAT('a', seq * 200) +FROM seq_1_to_150; +FLUSH STATUS; +# Should overflow during UPDATE and convert HEAP->Aria transparently +SELECT grp, LENGTH(MAX(val)) AS max_len +FROM t1 GROUP BY grp ORDER BY grp; +grp max_len +1 30000 +2 29600 +3 29800 +# Verify HEAP->Aria conversion happened +SHOW STATUS LIKE 'Created_tmp_%'; +Variable_name Value +Created_tmp_disk_tables 1 +Created_tmp_files 0 +Created_tmp_tables 2 +DROP TABLE t1; +# +# Test 2: Verify result correctness after overflow +# +# Same data but also check the actual MAX content, not just length. +# +CREATE TABLE t1 (grp INT, val TEXT); +INSERT INTO t1 SELECT (seq % 3) + 1, REPEAT('a', seq * 200) +FROM seq_1_to_150; +# Group 1 gets seq=3,6,...,150 -> MAX is REPEAT('a', 30000) +# Group 2 gets seq=1,4,...,148 -> MAX is REPEAT('a', 29600) +# Group 3 gets seq=2,5,...,149 -> MAX is REPEAT('a', 29800) +FLUSH STATUS; +SELECT grp, +LENGTH(MAX(val)) AS max_len, +LEFT(MAX(val), 5) AS prefix, +RIGHT(MAX(val), 5) AS suffix +FROM t1 GROUP BY grp ORDER BY grp; +grp max_len prefix suffix +1 30000 aaaaa aaaaa +2 29600 aaaaa aaaaa +3 29800 aaaaa aaaaa +SHOW STATUS LIKE 'Created_tmp_%'; +Variable_name Value +Created_tmp_disk_tables 1 +Created_tmp_files 0 +Created_tmp_tables 2 +DROP TABLE t1; +# +# Test 3: Multiple blob aggregates (two MAX columns) +# +CREATE TABLE t1 (grp INT, v1 TEXT, v2 TEXT); +INSERT INTO t1 SELECT (seq % 3) + 1, +REPEAT('x', seq * 200), +REPEAT('y', seq * 150) +FROM seq_1_to_150; +FLUSH STATUS; +SELECT grp, +LENGTH(MAX(v1)) AS max1_len, +LENGTH(MAX(v2)) AS max2_len +FROM t1 GROUP BY grp ORDER BY grp; +grp max1_len max2_len +1 30000 22500 +2 29600 22200 +3 29800 22350 +SHOW STATUS LIKE 'Created_tmp_%'; +Variable_name Value +Created_tmp_disk_tables 1 +Created_tmp_files 0 +Created_tmp_tables 2 +DROP TABLE t1; +# +# Test 4: MIN(TEXT) with monotonically shrinking minimum +# +# Rows with decreasing-length values for MIN. Since shorter +# all-'a' strings are "less than" longer ones under PAD SPACE, +# MIN keeps getting replaced with shorter values. The blob +# doesn't grow, but the churn of rewriting chains can cause +# overflow when combined with other groups' large blobs. +# (MIN shrinks blobs, so this primarily tests that the conversion +# path works even when some groups have small blobs.) +# +CREATE TABLE t1 (grp INT, val TEXT); +# Mix growing and fixed-size values +INSERT INTO t1 SELECT (seq % 3) + 1, +CASE WHEN seq % 2 = 0 THEN REPEAT('b', seq * 200) +ELSE REPEAT('a', 50) +END +FROM seq_1_to_150; +FLUSH STATUS; +SELECT grp, +LENGTH(MAX(val)) AS max_len, +LENGTH(MIN(val)) AS min_len +FROM t1 GROUP BY grp ORDER BY grp; +grp max_len min_len +1 30000 50 +2 29600 50 +3 29200 50 +SHOW STATUS LIKE 'Created_tmp_%'; +Variable_name Value +Created_tmp_disk_tables 1 +Created_tmp_files 0 +Created_tmp_tables 2 +DROP TABLE t1; +# +# Cleanup +# +SET max_heap_table_size = @save_max_heap; +SET tmp_table_size = @save_tmp; diff --git a/mysql-test/suite/heap/blob_update_overflow.test b/mysql-test/suite/heap/blob_update_overflow.test new file mode 100644 index 0000000000000..dd24e14690e47 --- /dev/null +++ b/mysql-test/suite/heap/blob_update_overflow.test @@ -0,0 +1,154 @@ +--source include/have_sequence.inc + +--echo # +--echo # MDEV-38975: Overflow-to-Aria on UPDATE path for GROUP BY temp tables +--echo # +--echo # When ha_update_tmp_row() fails with HA_ERR_RECORD_FILE_FULL on a +--echo # HEAP temp table (blob aggregate grows during UPDATE), the server +--echo # should transparently convert to Aria and retry the update, just +--echo # as the INSERT path already does. +--echo # +--echo # MAX(TEXT) with monotonically growing values triggers this: each +--echo # update stores a longer blob via result_field->store(), and +--echo # heap_update()'s write-before-free strategy requires old + new +--echo # continuation chains to coexist briefly, exhausting capacity. +--echo # + +SET @save_max_heap= @@max_heap_table_size; +SET @save_tmp= @@tmp_table_size; + +--echo # +--echo # Test 1: MAX(TEXT) overflow during end_update() +--echo # +--echo # 3 groups x 50 interleaved rows. val = REPEAT('a', seq * 200). +--echo # Since seq grows, each group's MAX increases on every update. +--echo # The HEAP temp table (32KB, recbuffer=16, ~2048 records) overflows +--echo # when per-group blob chains reach ~500 records (~8KB each). +--echo # + +SET max_heap_table_size = 32768; +SET tmp_table_size = 32768; + +CREATE TABLE t1 (grp INT, val TEXT); +INSERT INTO t1 SELECT (seq % 3) + 1, REPEAT('a', seq * 200) +FROM seq_1_to_150; + +--disable_ps_protocol +--disable_ps2_protocol +--disable_view_protocol +--disable_cursor_protocol +FLUSH STATUS; + +--echo # Should overflow during UPDATE and convert HEAP->Aria transparently +SELECT grp, LENGTH(MAX(val)) AS max_len +FROM t1 GROUP BY grp ORDER BY grp; + +--echo # Verify HEAP->Aria conversion happened +SHOW STATUS LIKE 'Created_tmp_%'; +--enable_cursor_protocol +--enable_view_protocol +--enable_ps2_protocol +--enable_ps_protocol + +DROP TABLE t1; + +--echo # +--echo # Test 2: Verify result correctness after overflow +--echo # +--echo # Same data but also check the actual MAX content, not just length. +--echo # + +CREATE TABLE t1 (grp INT, val TEXT); +INSERT INTO t1 SELECT (seq % 3) + 1, REPEAT('a', seq * 200) +FROM seq_1_to_150; + +--echo # Group 1 gets seq=3,6,...,150 -> MAX is REPEAT('a', 30000) +--echo # Group 2 gets seq=1,4,...,148 -> MAX is REPEAT('a', 29600) +--echo # Group 3 gets seq=2,5,...,149 -> MAX is REPEAT('a', 29800) +--disable_ps_protocol +--disable_ps2_protocol +--disable_view_protocol +--disable_cursor_protocol +FLUSH STATUS; +SELECT grp, + LENGTH(MAX(val)) AS max_len, + LEFT(MAX(val), 5) AS prefix, + RIGHT(MAX(val), 5) AS suffix +FROM t1 GROUP BY grp ORDER BY grp; + +SHOW STATUS LIKE 'Created_tmp_%'; +--enable_cursor_protocol +--enable_view_protocol +--enable_ps2_protocol +--enable_ps_protocol +DROP TABLE t1; + +--echo # +--echo # Test 3: Multiple blob aggregates (two MAX columns) +--echo # + +CREATE TABLE t1 (grp INT, v1 TEXT, v2 TEXT); +INSERT INTO t1 SELECT (seq % 3) + 1, + REPEAT('x', seq * 200), + REPEAT('y', seq * 150) +FROM seq_1_to_150; + +--disable_ps_protocol +--disable_ps2_protocol +--disable_view_protocol +--disable_cursor_protocol +FLUSH STATUS; +SELECT grp, + LENGTH(MAX(v1)) AS max1_len, + LENGTH(MAX(v2)) AS max2_len +FROM t1 GROUP BY grp ORDER BY grp; + +SHOW STATUS LIKE 'Created_tmp_%'; +--enable_cursor_protocol +--enable_view_protocol +--enable_ps2_protocol +--enable_ps_protocol +DROP TABLE t1; + +--echo # +--echo # Test 4: MIN(TEXT) with monotonically shrinking minimum +--echo # +--echo # Rows with decreasing-length values for MIN. Since shorter +--echo # all-'a' strings are "less than" longer ones under PAD SPACE, +--echo # MIN keeps getting replaced with shorter values. The blob +--echo # doesn't grow, but the churn of rewriting chains can cause +--echo # overflow when combined with other groups' large blobs. +--echo # (MIN shrinks blobs, so this primarily tests that the conversion +--echo # path works even when some groups have small blobs.) +--echo # + +CREATE TABLE t1 (grp INT, val TEXT); +--echo # Mix growing and fixed-size values +INSERT INTO t1 SELECT (seq % 3) + 1, + CASE WHEN seq % 2 = 0 THEN REPEAT('b', seq * 200) + ELSE REPEAT('a', 50) + END +FROM seq_1_to_150; + +--disable_ps_protocol +--disable_ps2_protocol +--disable_view_protocol +--disable_cursor_protocol +FLUSH STATUS; +SELECT grp, + LENGTH(MAX(val)) AS max_len, + LENGTH(MIN(val)) AS min_len +FROM t1 GROUP BY grp ORDER BY grp; + +SHOW STATUS LIKE 'Created_tmp_%'; +--enable_cursor_protocol +--enable_view_protocol +--enable_ps2_protocol +--enable_ps_protocol +DROP TABLE t1; + +--echo # +--echo # Cleanup +--echo # +SET max_heap_table_size = @save_max_heap; +SET tmp_table_size = @save_tmp; diff --git a/mysql-test/suite/heap/heap_geometry.result b/mysql-test/suite/heap/heap_geometry.result new file mode 100644 index 0000000000000000000000000000000000000000..9ae27b289623265511c1ad91f071200f632fc204 GIT binary patch literal 3323 zcmcInT~pgS5N#jvuh^MB#HPNCWJ;heWeS8z-1hS1aU3P_)Ui`ZE^vQ!f5JWT z7l8y`+GZxoNV{5{J-gaf-WYGl=Tgy`H@x)x(G5jek}p$9aY~oooFrUAal}_5 zo8}RhG)b}vO+ytzUx<9A;w+_UF5+}f=T~EIV*83m&e5!h9g(|9GE{kGCxMid|a-ePZt6E_2*YG4zZ`@KEs*Feyf6#NdarDRODf zvn5yJp3+QFnkR`xJe|iW@0V`&;NVv-;@N$*KbDkB?N3ERsSPX%k5sjnW+GeSh*>6< zq2?fv(Sk3-y^6Ej1s4^bO48S)O+v{{i^6nTo)x6wlJ{GRwR%Q%OnInLatTWPp?0WZ zRPrA-js4&ZK79lNpWi7n%F-xQX6phzjg^js*|N1}i>hC{Wt+=vYU)MiSRm~!a~!Ks z*j%?;R?D(V53wU)fy#oerm0IJyOlPDNkTuvB%ao!L|L9HbGO>07$Q=w7~Kt>P!?eb z3YGF0cvvz9-v+Yu%C!iQ&;jcidMxz~2RJ^|iAC(6!y^QCJsJ7k3UT6_*l=#E6cX#c zb6fS%puX5rtQuOh4RXEJY4*Jgxw7dz0!;6WIF(!|ot>EuY?W2J4`8a5nm}5E-QGX! zG{-lv^{P<}FKqoi*v$DY++B8XxQSb@8pZJUaI@dS%{uM&!6xux*(k?94ZW&F`i3=y z3s)^i>0Jpfd8+6(R*NFxbvY{1MAixW`1&Aov<+0j`g&p|f`y`ZDQ3(?mak}XU#L+g zrd-sso}styK-;)Dzw*ZZ==|!;WW;R79KzSyah=ZLy3i%pW-f&87Gay(HfuxJ-XiQ! z$7US}J6nYNv~RP02=})L59q*V2M``?5q7C-vo3_)TKI-8Pe$LsADSHMb@Uy?p)buD z^*73D>pDyB=UHl+4mq|%uH)GFHotn!u}~^?n<`qvI?Z*tc$`92$!I{BitRF5&zSLQa^7m;FeD-eqn`lw2@0jI%Hu-G8Mg+R@d zcwW_L9tzydqQRJMzvGct^RY=8#55dz@J9N+VA3-%JOHi&OUwc(LiRJ=kW>Q`IY vHYDDg50Ne!%j2i!i&?+z3RF4#*8 @@ -265,7 +265,7 @@ RPAD(IFNULL(NESTING_EVENT_TYPE, 'NULL'), 18, ' ') NESTING_EVENT_TYPE, SQL_TEXT FROM performance_schema.events_statements_history_long s WHERE ((s.thread_id = @con1_thread_id) OR (@all_threads = 1)) -ORDER BY thread_id, r_event_id; +ORDER BY thread_id, r_event_id, r_end_event_id; THREAD_ID R_EVENT_ID R_END_EVENT_ID EVENT_NAME R_NESTING_EVENT_ID NESTING_EVENT_TYPE SQL_TXT thread_id 1 2 statement/sql/begin NULL NULL START TRANSACTION thread_id 2 5 transaction 1 STATEMENT @@ -397,7 +397,7 @@ RPAD(IFNULL(NESTING_EVENT_TYPE, 'NULL'), 18, ' ') NESTING_EVENT_TYPE, SQL_TEXT FROM performance_schema.events_statements_history_long s WHERE ((s.thread_id = @con1_thread_id) OR (@all_threads = 1)) -ORDER BY thread_id, r_event_id; +ORDER BY thread_id, r_event_id, r_end_event_id; THREAD_ID R_EVENT_ID R_END_EVENT_ID EVENT_NAME R_NESTING_EVENT_ID NESTING_EVENT_TYPE SQL_TXT thread_id 1 2 statement/sql/create_proc NULL NULL CREATE PROCEDURE tp_update() UPDATE t1 SET s1 = s1 + 1 thread_id 2 2 transaction 1 STATEMENT @@ -537,7 +537,7 @@ RPAD(IFNULL(NESTING_EVENT_TYPE, 'NULL'), 18, ' ') NESTING_EVENT_TYPE, SQL_TEXT FROM performance_schema.events_statements_history_long s WHERE ((s.thread_id = @con1_thread_id) OR (@all_threads = 1)) -ORDER BY thread_id, r_event_id; +ORDER BY thread_id, r_event_id, r_end_event_id; THREAD_ID R_EVENT_ID R_END_EVENT_ID EVENT_NAME R_NESTING_EVENT_ID NESTING_EVENT_TYPE SQL_TXT thread_id 1 2 statement/sql/create_proc NULL NULL CREATE PROCEDURE tp_start() START TRANSACTION thread_id 2 2 transaction 1 STATEMENT @@ -697,7 +697,7 @@ RPAD(IFNULL(NESTING_EVENT_TYPE, 'NULL'), 18, ' ') NESTING_EVENT_TYPE, SQL_TEXT FROM performance_schema.events_statements_history_long s WHERE ((s.thread_id = @con1_thread_id) OR (@all_threads = 1)) -ORDER BY thread_id, r_event_id; +ORDER BY thread_id, r_event_id, r_end_event_id; THREAD_ID R_EVENT_ID R_END_EVENT_ID EVENT_NAME R_NESTING_EVENT_ID NESTING_EVENT_TYPE SQL_TXT thread_id 1 2 statement/sql/create_proc NULL NULL CREATE PROCEDURE tp_rollback() ROLLBACK thread_id 2 2 transaction 1 STATEMENT @@ -871,7 +871,7 @@ RPAD(IFNULL(NESTING_EVENT_TYPE, 'NULL'), 18, ' ') NESTING_EVENT_TYPE, SQL_TEXT FROM performance_schema.events_statements_history_long s WHERE ((s.thread_id = @con1_thread_id) OR (@all_threads = 1)) -ORDER BY thread_id, r_event_id; +ORDER BY thread_id, r_event_id, r_end_event_id; THREAD_ID R_EVENT_ID R_END_EVENT_ID EVENT_NAME R_NESTING_EVENT_ID NESTING_EVENT_TYPE SQL_TXT thread_id 1 2 statement/sql/begin NULL NULL START TRANSACTION thread_id 3 3 statement/sql/insert 2 TRANSACTION INSERT INTO t1 VALUES (410, "INSERT 410") @@ -1005,7 +1005,7 @@ RPAD(IFNULL(NESTING_EVENT_TYPE, 'NULL'), 18, ' ') NESTING_EVENT_TYPE, SQL_TEXT FROM performance_schema.events_statements_history_long s WHERE ((s.thread_id = @con1_thread_id) OR (@all_threads = 1)) -ORDER BY thread_id, r_event_id; +ORDER BY thread_id, r_event_id, r_end_event_id; THREAD_ID R_EVENT_ID R_END_EVENT_ID EVENT_NAME R_NESTING_EVENT_ID NESTING_EVENT_TYPE SQL_TXT thread_id 1 2 statement/sql/begin NULL NULL START TRANSACTION thread_id 2 6 transaction 1 STATEMENT @@ -1238,7 +1238,7 @@ RPAD(IFNULL(NESTING_EVENT_TYPE, 'NULL'), 18, ' ') NESTING_EVENT_TYPE, SQL_TEXT FROM performance_schema.events_statements_history_long s WHERE ((s.thread_id = @con1_thread_id) OR (@all_threads = 1)) -ORDER BY thread_id, r_event_id; +ORDER BY thread_id, r_event_id, r_end_event_id; THREAD_ID R_EVENT_ID R_END_EVENT_ID EVENT_NAME R_NESTING_EVENT_ID NESTING_EVENT_TYPE SQL_TXT thread_id 1 2 statement/sql/begin NULL NULL START TRANSACTION thread_id 2 19 transaction 1 STATEMENT diff --git a/mysql-test/suite/perfschema/t/misc_session_status.test b/mysql-test/suite/perfschema/t/misc_session_status.test index b64d161a12b02..6d7d0c0ac1ab6 100644 --- a/mysql-test/suite/perfschema/t/misc_session_status.test +++ b/mysql-test/suite/perfschema/t/misc_session_status.test @@ -1,7 +1,6 @@ --source include/not_embedded.inc --source include/have_perfschema.inc --source include/not_msan.inc -# We cannot use valgrind build as it uses more memory than normal build --source include/not_valgrind_build.inc # This does not crash on 32 bit because of less memory used --source include/have_64bit.inc diff --git a/mysql-test/suite/plugins/r/sql_error_log_withdbinfo.result b/mysql-test/suite/plugins/r/sql_error_log_withdbinfo.result index 732e74d851662..8f9de3e5e82e5 100644 --- a/mysql-test/suite/plugins/r/sql_error_log_withdbinfo.result +++ b/mysql-test/suite/plugins/r/sql_error_log_withdbinfo.result @@ -31,9 +31,9 @@ CREATE DATABASE `NULL`; USE `NULL`; DROP DATABASE db; ERROR HY000: Can't drop database 'db'; database doesn't exist -TIME THREAD_ID HOSTNAME `mtr` WARNING 1286: Unknown storage engine 'InnoDB' : SELECT CONCAT(table_schema, '.', table_name) AS columns_in_mysql, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_octet_length, numeric_precision, numeric_scale, character_set_name, collation_name, column_type, column_key, extra, column_comment FROM INFORMATION_SCHEMA.COLUMNS WHERE table_schema='mysql' ORDER BY columns_in_mysql -TIME THREAD_ID HOSTNAME `mtr` WARNING 1286: Unknown storage engine 'InnoDB' : SELECT CONCAT(table_schema, '.', table_name) AS columns_in_mysql, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_octet_length, numeric_precision, numeric_scale, character_set_name, collation_name, column_type, column_key, extra, column_comment FROM INFORMATION_SCHEMA.COLUMNS WHERE table_schema='mysql' ORDER BY columns_in_mysql -TIME THREAD_ID HOSTNAME `mtr` WARNING 1286: Unknown storage engine 'InnoDB' : SELECT CONCAT(table_schema, '.', table_name) AS columns_in_mysql, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_octet_length, numeric_precision, numeric_scale, character_set_name, collation_name, column_type, column_key, extra, column_comment FROM INFORMATION_SCHEMA.COLUMNS WHERE table_schema='mysql' ORDER BY columns_in_mysql +TIME THREAD_ID HOSTNAME `mtr` WARNING 1286: Unknown storage engine 'InnoDB' : SELECT CONCAT(table_schema, '.', table_name) AS columns_in_mysql, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_octet_length, numeric_precision, numeric_scale, character_set_name, collation_name, column_type, column_key, extra, column_comment FROM INFORMATION_SCHEMA.COLUMNS WHERE table_schema='mysql' ORDER BY columns_in_mysql, ordinal_position +TIME THREAD_ID HOSTNAME `mtr` WARNING 1286: Unknown storage engine 'InnoDB' : SELECT CONCAT(table_schema, '.', table_name) AS columns_in_mysql, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_octet_length, numeric_precision, numeric_scale, character_set_name, collation_name, column_type, column_key, extra, column_comment FROM INFORMATION_SCHEMA.COLUMNS WHERE table_schema='mysql' ORDER BY columns_in_mysql, ordinal_position +TIME THREAD_ID HOSTNAME `mtr` WARNING 1286: Unknown storage engine 'InnoDB' : SELECT CONCAT(table_schema, '.', table_name) AS columns_in_mysql, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, character_octet_length, numeric_precision, numeric_scale, character_set_name, collation_name, column_type, column_key, extra, column_comment FROM INFORMATION_SCHEMA.COLUMNS WHERE table_schema='mysql' ORDER BY columns_in_mysql, ordinal_position TIME THREAD_ID HOSTNAME `test` ERROR 1238: Variable 'sql_error_log_with_db_and_thread_info' is a read only variable : SET sql_error_log_with_db_and_thread_info=OFF TIME THREAD_ID HOSTNAME `test` ERROR 1008: Can't drop database 'db'; database doesn't exist : DROP DATABASE db TIME THREAD_ID HOSTNAME NULL ERROR 1008: Can't drop database 'dbnodb'; database doesn't exist : DROP DATABASE dbnodb diff --git a/mysql-test/suite/sys_vars/r/tmp_disk_table_size_basic.result b/mysql-test/suite/sys_vars/r/tmp_disk_table_size_basic.result index 96314c64de4f8..220c25ddae1e0 100644 --- a/mysql-test/suite/sys_vars/r/tmp_disk_table_size_basic.result +++ b/mysql-test/suite/sys_vars/r/tmp_disk_table_size_basic.result @@ -149,6 +149,7 @@ ERROR 42S22: Unknown column 'tmp_disk_table_size' in 'SELECT' # failed on SELECT after setting tmp_disk_table_size. # SET @@tmp_disk_table_size=16384; +set @@tmp_memory_table_size=16384; CREATE VIEW v AS SELECT 'a'; SELECT table_name FROM INFORMATION_SCHEMA.views; ERROR HY000: The table '(temporary)' is full diff --git a/mysql-test/suite/sys_vars/t/tmp_disk_table_size_basic.test b/mysql-test/suite/sys_vars/t/tmp_disk_table_size_basic.test index 099be3544865b..6a967ceef19fa 100644 --- a/mysql-test/suite/sys_vars/t/tmp_disk_table_size_basic.test +++ b/mysql-test/suite/sys_vars/t/tmp_disk_table_size_basic.test @@ -201,6 +201,7 @@ SELECT tmp_disk_table_size = @@session.tmp_disk_table_size; --echo # SET @@tmp_disk_table_size=16384; +set @@tmp_memory_table_size=16384; CREATE VIEW v AS SELECT 'a'; --error ER_RECORD_FILE_FULL diff --git a/mysys/my_compare.c b/mysys/my_compare.c index 0f6c02553b4b0..366f7c77f9a23 100644 --- a/mysys/my_compare.c +++ b/mysys/my_compare.c @@ -266,6 +266,52 @@ int ha_key_cmp(HA_KEYSEG *keyseg, const uchar *a, b+=b_length; } break; + case HA_KEYTYPE_VARTEXT4: + { +#ifdef NOT_YET_USED + /* Only used for internal temporary tables */ + int a_length,b_length; + uchar *a_key, *b_key; + DBUG_ASSERT(!(nextflag & SEARCH_PREFIX)); + + a_length= uint4korr(a); + b_length= uint4korr(b); + memcpy(&a_key, a, sizeof(char*)); + memcpy(&b_key, b, sizeof(char*)); + if (piks && + (flag= ha_compare_char_varying(keyseg->charset, + a_key, a_length, + b_key, b_length, 0))) + return ((keyseg->flag & HA_REVERSE_SORT) ? -flag : flag); + a+= 4 + portable_sizeof_char_ptr; + b+= 4 + portable_sizeof_char_ptr; + break; +#else + abort(); +#endif /* NOT_YET_USED */ + } + case HA_KEYTYPE_VARBINARY4: + { +#ifdef NOT_YET_USED + /* Only used for internal temporary tables */ + int a_length,b_length; + uchar *a_key, *b_key; + DBUG_ASSERT(!(nextflag & SEARCH_PREFIX)); + + a_length= uint4korr(a); + b_length= uint4korr(b); + memcpy(&a_key, a, sizeof(char*)); + memcpy(&b_key, b, sizeof(char*)); + if (piks && + (flag= compare_bin(a_key, a_length, b_key, b_length, 0, 0))) + return ((keyseg->flag & HA_REVERSE_SORT) ? -flag : flag); + a+= 4 + portable_sizeof_char_ptr; + b+= 4 + portable_sizeof_char_ptr; + break; +#else + abort(); +#endif /* NOT_YET_USED */ + } case HA_KEYTYPE_INT8: { int i_1= (int) *((signed char*) a); @@ -623,6 +669,8 @@ HA_KEYSEG *ha_find_null(HA_KEYSEG *keyseg, const uchar *a) a= end; break; case HA_KEYTYPE_END: /* purecov: inspected */ + case HA_KEYTYPE_VARTEXT4: /* Not used yet */ + case HA_KEYTYPE_VARBINARY4: /* Not used yet */ /* keep compiler happy */ DBUG_ASSERT(0); break; diff --git a/plugin/type_cursor/plugin.cc b/plugin/type_cursor/plugin.cc index 03ac994c70317..fb3870ad6ccf3 100644 --- a/plugin/type_cursor/plugin.cc +++ b/plugin/type_cursor/plugin.cc @@ -329,8 +329,8 @@ class Field_sys_refcursor final :public Field_short, DBUG_RETURN(rc); } - Field *make_new_field(MEM_ROOT *root, TABLE *new_table, bool keep_type) - override + Field *make_new_field(MEM_ROOT *root, TABLE *new_table, bool keep_type, + const Tmp_field_param *param= 0) override { my_error(ER_ILLEGAL_PARAMETER_DATA_TYPE_FOR_OPERATION, MYF(0), sys_refcursor_str.str, "CREATE TABLE"); diff --git a/plugin/type_xmltype/sql_type_xmltype.cc b/plugin/type_xmltype/sql_type_xmltype.cc index 89c94315444af..b285cd1009cf7 100644 --- a/plugin/type_xmltype/sql_type_xmltype.cc +++ b/plugin/type_xmltype/sql_type_xmltype.cc @@ -164,7 +164,8 @@ bool Type_handler_xmltype:: const Type_handler *Type_handler_xmltype:: - type_handler_for_tmp_table(const Item *item) const + type_handler_for_tmp_table(const Item *item, + const Tmp_field_param *param) const { return &type_handler_xmltype; } diff --git a/plugin/type_xmltype/sql_type_xmltype.h b/plugin/type_xmltype/sql_type_xmltype.h index c393c24caa90f..2e158d37a3adc 100644 --- a/plugin/type_xmltype/sql_type_xmltype.h +++ b/plugin/type_xmltype/sql_type_xmltype.h @@ -61,8 +61,9 @@ class Type_handler_xmltype: public Type_handler_long_blob Type_handler_hybrid_field_type *handler, Type_all_attributes *func, Item **items, uint nitems) const override; - const Type_handler *type_handler_for_tmp_table(const Item *item) const - override; + const Type_handler *type_handler_for_tmp_table(const Item *item, + const Tmp_field_param *param) + const override; bool Item_append_extended_type_info(Send_field_extended_metadata *to, const Item *item) const override diff --git a/sql/create_tmp_table.h b/sql/create_tmp_table.h index ce86c9456e460..6a9f7b07d7e72 100644 --- a/sql/create_tmp_table.h +++ b/sql/create_tmp_table.h @@ -31,6 +31,7 @@ class Create_tmp_table: public Data_type_statistics // The following members are initialized in ctor uint m_alloced_field_count; bool m_using_unique_constraint; + bool m_heap_expected; uint m_temp_pool_slot; ORDER *m_group; bool m_distinct; @@ -59,6 +60,7 @@ class Create_tmp_table: public Data_type_statistics Create_tmp_table(ORDER *group, bool distinct, bool save_sum_fields, ulonglong select_options, ha_rows rows_limit); virtual ~Create_tmp_table() {} + handlerton *pick_engine(THD *thd, uint reclength); virtual bool choose_engine(THD *thd, TABLE *table, TMP_TABLE_PARAM *param); void add_field(TABLE *table, Field *field, uint fieldnr, bool force_not_null_cols); diff --git a/sql/field.cc b/sql/field.cc index 56945f6b2fd3f..ecabbb899ead1 100644 --- a/sql/field.cc +++ b/sql/field.cc @@ -50,8 +50,8 @@ const char field_separator=','; #define DOUBLE_TO_STRING_CONVERSION_BUFFER_SIZE FLOATING_POINT_BUFFER #define LONGLONG_TO_STRING_CONVERSION_BUFFER_SIZE 128 #define DECIMAL_TO_STRING_CONVERSION_BUFFER_SIZE 128 -#define BLOB_PACK_LENGTH_TO_MAX_LENGH(arg) \ - ((ulong) ((1LL << MY_MIN(arg, 4) * 8) - 1)) +#define BLOB_PACK_LENGTH_TO_MAX_LENGTH(arg) \ + ((ulong) ((1ULL << MY_MIN(arg, 4) * 8) - 1)) // Column marked for read or the field set to read out of record[0] bool Field::marked_for_read() const @@ -2137,6 +2137,11 @@ int Field_blob::store_from_statistical_minmax_field(Field *stat_field, return 0; } +void Field_blob::set_pack_length(uint32 packlength_arg) +{ + packlength= packlength_arg; + field_length= BLOB_PACK_LENGTH_TO_MAX_LENGTH(packlength); +} /** Pack the field into a format suitable for storage and transfer. @@ -2616,7 +2621,9 @@ bool Field::optimize_range(uint idx, uint part) const Field *Field::make_new_field(MEM_ROOT *root, TABLE *new_table, - bool keep_type __attribute__((unused))) + bool keep_type __attribute__((unused)), + const Tmp_field_param *param + __attribute__((unused))) { Field *tmp; if (!(tmp= (Field*) memdup_root(root,(char*) this,size_of()))) @@ -2648,7 +2655,7 @@ Field *Field::new_key_field(MEM_ROOT *root, TABLE *new_table, uchar *new_null_ptr, uint new_null_bit) { Field *tmp; - if ((tmp= make_new_field(root, new_table, table == new_table))) + if ((tmp= make_new_field(root, new_table, table == new_table, 0))) { tmp->ptr= new_ptr; tmp->null_ptr= new_null_ptr; @@ -2673,11 +2680,13 @@ Field *Field::new_key_field(MEM_ROOT *root, TABLE *new_table, */ Field *Field::create_tmp_field(MEM_ROOT *mem_root, TABLE *new_table, - bool maybe_null_arg) + bool maybe_null_arg, + const Tmp_field_param *param) { Field *new_field; - if ((new_field= make_new_field(mem_root, new_table, new_table == table))) + if ((new_field= make_new_field(mem_root, new_table, new_table == table, + param))) { new_field->init_for_tmp_table(this, new_table); new_field->flags|= flags & NO_DEFAULT_VALUE_FLAG; @@ -3505,10 +3514,11 @@ void Field_decimal::sql_type(String &res) const Field *Field_decimal::make_new_field(MEM_ROOT *root, TABLE *new_table, - bool keep_type) + bool keep_type, + const Tmp_field_param *param) { if (keep_type) - return Field_real::make_new_field(root, new_table, keep_type); + return Field_real::make_new_field(root, new_table, keep_type, param); Field *field= new (root) Field_new_decimal(NULL, field_length, maybe_null() ? (uchar*) "" : 0, 0, @@ -8061,11 +8071,12 @@ uint Field_string::get_key_image(uchar *buff, uint length, const uchar *ptr_arg, Field *Field_string::make_new_field(MEM_ROOT *root, TABLE *new_table, - bool keep_type) + bool keep_type, + const Tmp_field_param *param) { Field *field; if (type() != MYSQL_TYPE_VAR_STRING || keep_type) - field= Field::make_new_field(root, new_table, keep_type); + field= Field::make_new_field(root, new_table, keep_type, param); else if ((field= new (root) Field_varstring(field_length, maybe_null(), &field_name, new_table->s, charset()))) @@ -8073,7 +8084,7 @@ Field *Field_string::make_new_field(MEM_ROOT *root, TABLE *new_table, /* Old VARCHAR field which should be modified to a VARCHAR on copy This is done to ensure that ALTER TABLE will convert old VARCHAR fields - to now VARCHAR fields. + to new VARCHAR fields. */ field->init_for_make_new_field(new_table, orig_table); } @@ -8566,26 +8577,30 @@ int Field_varstring::cmp_binary(const uchar *a_ptr, const uchar *b_ptr, Field *Field_varstring::make_new_field(MEM_ROOT *root, TABLE *new_table, - bool keep_type) + bool keep_type, + const Tmp_field_param *param) { Field_varstring *res= (Field_varstring*) Field::make_new_field(root, new_table, - keep_type); + keep_type, + param); if (res) res->length_bytes= length_bytes; return res; } -Field *Field_varstring_compressed::make_new_field(MEM_ROOT *root, TABLE *new_table, - bool keep_type) +Field *Field_varstring_compressed::make_new_field(MEM_ROOT *root, + TABLE *new_table, + bool keep_type, + const Tmp_field_param *param) { Field_varstring *res; - if (new_table->s->is_optimizer_tmp_table()) + if (param && param->part_of_unique_key()) { /* - Compressed field cannot be part of a key. For optimizer temporary - table we create uncompressed substitute. + A compressed field cannot be part of an unique key. + Create an uncompressed substitute instead. */ res= new (root) Field_varstring(ptr, field_length, length_bytes, null_ptr, null_bit, Field::NONE, &field_name, @@ -8598,24 +8613,28 @@ Field *Field_varstring_compressed::make_new_field(MEM_ROOT *root, TABLE *new_tab } } else - res= (Field_varstring*) Field::make_new_field(root, new_table, keep_type); + res= (Field_varstring*) Field::make_new_field(root, new_table, keep_type, + param); if (res) res->length_bytes= length_bytes; return res; } + Field *Field_blob_compressed::make_new_field(MEM_ROOT *root, TABLE *new_table, - bool keep_type) + bool keep_type, + const Tmp_field_param *param) { Field_blob *res; - if (new_table->s->is_optimizer_tmp_table()) + if (param && param->part_of_unique_key()) { /* Compressed field cannot be part of a key. For optimizer temporary - table we create uncompressed substitute. + table we create uncompressed Field_blob_key substitute. */ - res= new (root) Field_blob(ptr, null_ptr, null_bit, Field::NONE, &field_name, - new_table->s, packlength, charset()); + res= new (root) Field_blob_key(ptr, null_ptr, null_bit, Field::NONE, + &field_name, + new_table->s, packlength, charset()); if (res) { res->init_for_make_new_field(new_table, orig_table); @@ -8624,7 +8643,8 @@ Field *Field_blob_compressed::make_new_field(MEM_ROOT *root, TABLE *new_table, } } else - res= (Field_blob *) Field::make_new_field(root, new_table, keep_type); + res= (Field_blob *) Field_blob::make_new_field(root, new_table, keep_type, + param); return res; } @@ -8889,7 +8909,7 @@ Field_blob::Field_blob(uchar *ptr_arg, uchar *null_ptr_arg, uchar null_bit_arg, const LEX_CSTRING *field_name_arg, TABLE_SHARE *share, uint blob_pack_length, const DTCollation &collation) - :Field_longstr(ptr_arg, BLOB_PACK_LENGTH_TO_MAX_LENGH(blob_pack_length), + :Field_longstr(ptr_arg, BLOB_PACK_LENGTH_TO_MAX_LENGTH(blob_pack_length), null_ptr_arg, null_bit_arg, unireg_check_arg, field_name_arg, collation), packlength(blob_pack_length) @@ -8940,6 +8960,50 @@ int Field_blob::copy_value(Field_blob *from) } +/* + Store blob value into table->blob_storage + Used with GROUP_CONCAT with ORDER BY/DISTINCT + + If with_zero_prefix is true, prepend a 0x00 byte so that + Field_blob_compressed::val_str() treats the value as uncompressed. +*/ + +int Field_blob::handle_group_concat(const char *from, size_t length, + CHARSET_INFO *cs, bool with_zero_prefix) +{ + DBUG_ASSERT(!f_is_hex_escape(flags)); + DBUG_ASSERT(field_charset() == cs); + DBUG_ASSERT(length <= max_data_length()); + + size_t new_length= length; + size_t copy_length= table->in_use->variables.group_concat_max_len; + if (new_length > copy_length) + { + new_length= Well_formed_prefix(cs, from, copy_length, new_length).length(); + table->blob_storage->set_truncated_value(true); + } + + char *tmp; + if (with_zero_prefix) + { + tmp= table->blob_storage->store_with_zero_prefix(from, new_length); + new_length++; // Count the extra 0 + } + else + tmp= table->blob_storage->store(from, new_length); + + if (!tmp) + goto oom_error; + Field_blob::store_length(new_length); + bmove(ptr + packlength, (uchar*) &tmp, sizeof(char*)); + return 0; + +oom_error: + reset(); + return -1; +} + + int Field_blob::store(const char *from,size_t length,CHARSET_INFO *cs) { DBUG_ASSERT(marked_for_write_or_computed()); @@ -8963,26 +9027,7 @@ int Field_blob::store(const char *from,size_t length,CHARSET_INFO *cs) */ if (table && table->blob_storage) // GROUP_CONCAT with ORDER BY | DISTINCT - { - DBUG_ASSERT(!f_is_hex_escape(flags)); - DBUG_ASSERT(field_charset() == cs); - DBUG_ASSERT(length <= max_data_length()); - - new_length= length; - copy_length= table->in_use->gconcat_max_len(); - if (new_length > copy_length) - { - new_length= Well_formed_prefix(cs, - from, copy_length, new_length).length(); - table->blob_storage->set_truncated_value(true); - } - if (!(tmp= table->blob_storage->store(from, new_length))) - goto oom_error; - - Field_blob::store_length(new_length); - bmove(ptr + packlength, (uchar*) &tmp, sizeof(char*)); - return 0; - } + return handle_group_concat(from, length, cs, false); /* If the 'from' address is in the range of the temporary 'value'- @@ -9032,7 +9077,7 @@ int Field_blob::store(const char *from,size_t length,CHARSET_INFO *cs) oom_error: /* Fatal OOM error */ - bzero(ptr,Field_blob::pack_length()); + reset(); return -1; } @@ -9172,12 +9217,12 @@ int Field_blob::cmp_binary(const uchar *a_ptr, const uchar *b_ptr, uint Field_blob::get_key_image_itRAW(const uchar *ptr_arg, uchar *buff, uint length) const { - size_t blob_length= get_length(ptr_arg); + uint32 blob_length= get_length(ptr_arg); const uchar *blob= get_ptr(ptr_arg); size_t local_char_length= length / mbmaxlen(); local_char_length= field_charset()->charpos(blob, blob + blob_length, local_char_length); - set_if_smaller(blob_length, local_char_length); + set_if_smaller(blob_length, (uint32) local_char_length); if (length > blob_length) { @@ -9206,13 +9251,13 @@ void Field_blob::set_key_image(const uchar *buff,uint length) int Field_blob::key_cmp(const uchar *key_ptr, uint max_key_length) const { uchar *blob1; - size_t blob_length=get_length(ptr); + uint32 blob_length=get_length(ptr); memcpy(&blob1, ptr+packlength, sizeof(char*)); CHARSET_INFO *cs= charset(); size_t local_char_length= max_key_length / cs->mbmaxlen; local_char_length= cs->charpos(blob1, blob1+blob_length, local_char_length); - set_if_smaller(blob_length, local_char_length); + set_if_smaller(blob_length, (uint32) local_char_length); return Field_blob::cmp(blob1, (uint32)blob_length, key_ptr+HA_KEY_BLOB_LENGTH, uint2korr(key_ptr)); @@ -9226,20 +9271,50 @@ int Field_blob::key_cmp(const uchar *a,const uchar *b) const #ifndef DBUG_OFF -/* helper to assert that new_table->blob_storage is NULL */ +/* + Helper to assert that the union, defined in table.h, still holds + only a bool-sized value, no pointer has been stored +*/ static struct blob_storage_check { union { bool b; intptr p; } val; blob_storage_check() { val.p= -1; val.b= false; } } blob_storage_check; #endif -Field *Field_blob::make_new_field(MEM_ROOT *root, TABLE *newt, bool keep_type) + +Field *Field_blob::make_new_field(MEM_ROOT *root, TABLE *newt, bool keep_type, + const Tmp_field_param *param) { DBUG_ASSERT((intptr(newt->blob_storage) & blob_storage_check.val.p) == 0); - if (newt->group_concat) + if (param && param->part_of_unique_key()) + { + /* + This is an internal temporary table using a unique key on a blob + to identify rows. Use Field_blob_key for storing the key in + 'blob format' (length + pointer) instead of 'varchar format' + (length + string) + */ + Field_blob_key *res; + res= new (root) Field_blob_key(field_length, maybe_null(), &field_name, + charset()); + if (res) + res->init_for_make_new_field(newt, orig_table); + return res; + } + if (newt->group_concat && type() != MYSQL_TYPE_BLOB) + { + /* + We are creating an internal temporary table for storing group_concat + result. Field_geom::store() lacks Blob_mem_storage support, + so GROUP_CONCAT with ORDER BY/DISTINCT on geometry columns would + read freed memory. Downgrade to plain Field_blob whose store() + routes data through table->blob_storage. + */ return new (root) Field_blob(field_length, maybe_null(), &field_name, charset()); - return Field::make_new_field(root, newt, keep_type); + } + + return Field::make_new_field(root, newt, keep_type, param); } @@ -9270,7 +9345,7 @@ Field *Field_blob::new_key_field(MEM_ROOT *root, TABLE *new_table, Binlog_type_info Field_blob::binlog_type_info() const { DBUG_ASSERT(Field_blob::type() == binlog_type()); - return Binlog_type_info(Field_blob::type(), pack_length_no_ptr(), 1, + return Binlog_type_info(Field_blob::type(), length_size(), 1, charset()); } @@ -9486,6 +9561,10 @@ int Field_blob_compressed::store(const char *from, size_t length, CHARSET_INFO *cs) { DBUG_ASSERT(marked_for_write_or_computed()); + + if (table && table->blob_storage) // GROUP_CONCAT with ORDER BY | DISTINCT + return handle_group_concat(from, length, cs, true); + uint compressed_length; uint max_length= max_data_length(); uint to_length= (uint) MY_MIN(max_length, mbmaxlen() * length + 1); @@ -9541,9 +9620,72 @@ longlong Field_blob_compressed::val_int(void) Binlog_type_info Field_blob_compressed::binlog_type_info() const { return Binlog_type_info(Field_blob_compressed::binlog_type(), - pack_length_no_ptr(), 1, charset()); + length_size(), 1, charset()); +} + + +/**************************************************************************** +** Field_blob_key +** Used for blob keys in internal temporary tables +****************************************************************************/ + +void Field_blob_key::set_key_image(const uchar *data, uint length) +{ +/* HEAP uses hp_hash.c for key ops; Aria converts to VARTEXT2 on overflow */ +#ifdef NOT_YET_USED + store_length(length); + memcpy(ptr+packlength, &data, sizeof(char*)); +#else + abort(); +#endif /* NOT_YET_USED */ +} + + +int Field_blob_key::key_cmp(const uchar *key_ptr, uint max_key_length) const +{ +/* HEAP uses hp_hash.c for key ops; Aria converts to VARTEXT2 on overflow */ +#ifdef NOT_YET_USED + uint32 blob_length= get_length(ptr); + memcpy(&blob1, ptr + packlength, sizeof(char*)); + return Field_blob_key::cmp(blob1, (uint32) blob_length, + key_ptr + 4, uint4korr(key_ptr)); +#else + abort(); +#endif /* NOT_YET_USED */ +} + +int Field_blob_key::key_cmp(const uchar *a, const uchar *b) const +{ +/* HEAP uses hp_hash.c for key ops; Aria converts to VARTEXT2 on overflow */ +#ifdef NOT_YET_USED + return Field_blob_key::cmp(a + 4, uint4korr(a), b+ 4, uint4korr(b)); +#else + abort(); +#endif /* NOT_YET_USED */ } + +Field *Field_blob_key::new_key_field(MEM_ROOT *root, TABLE *new_table, + uchar *new_ptr, uint32 length, + uchar *new_null_ptr, uint new_null_bit) +{ + Field_blob_key *res; + /* + length comes from blob->key_length which includes portable_sizeof_char_ptr + */ + length-= portable_sizeof_char_ptr; + DBUG_ASSERT(length > 0 && length <= 4); + + res= new (root) Field_blob_key(new_ptr, new_null_ptr, new_null_bit, + Field::NONE, &field_name, + table->s, length, charset()); + res->init(new_table); + /* key_fields are not stored in the table. Don't count this one */ + table->s->blob_fields--; + return res; +} + + /**************************************************************************** ** enum type. ** This is a string which only can have a selection of different values. @@ -9742,11 +9884,11 @@ void Field_enum::sql_type(String &res) const Field *Field_enum::make_new_field(MEM_ROOT *root, TABLE *new_table, - bool keep_type) + bool keep_type, const Tmp_field_param *param) { DBUG_ASSERT(m_typelib_attr); Field_enum *res= (Field_enum*) Field::make_new_field(root, new_table, - keep_type); + keep_type, param); if (!res || !(res->m_typelib_attr= m_typelib_attr->deep_copy(root))) return nullptr; diff --git a/sql/field.h b/sql/field.h index a509c194bb101..d8c22cd026d16 100644 --- a/sql/field.h +++ b/sql/field.h @@ -524,6 +524,19 @@ inline bool is_temporal_type_with_date(enum_field_types type) } +/* + Only needed for calc_group_buffer(), where we have an + enum_field_types but no Field object. + In all other cases use field->flags & BLOB_FLAG. +*/ +static inline bool is_any_blob_field_type(enum_field_types type) +{ + return type == MYSQL_TYPE_BLOB || type == MYSQL_TYPE_TINY_BLOB || + type == MYSQL_TYPE_MEDIUM_BLOB || type == MYSQL_TYPE_LONG_BLOB || + type == MYSQL_TYPE_GEOMETRY; +} + + enum enum_vcol_info_type { VCOL_GENERATED_VIRTUAL, VCOL_GENERATED_STORED, @@ -1206,7 +1219,8 @@ class Field: public Value_source which is located in RAM). */ virtual uint32 pack_length() const { return (uint32) field_length; } - + /* length to store a key in a key buffer */ + virtual uint32 key_pack_length() const { return pack_length(); } /* pack_length_in_rec() returns size (in bytes) used to store field data on storage (i.e. it returns the maximal size of the field in a row of the @@ -1220,11 +1234,10 @@ class Field: public Value_source DBUG_ENTER("Field::pack_length_from_metadata"); DBUG_RETURN(field_metadata); } - virtual uint row_pack_length() const { return 0; } + /* Length of row data in a record, not including packed length prefix */ + virtual uint row_pack_length() const { return pack_length(); } - /* - data_length() return the "real size" of the data in memory. - */ + /* Return the current size of data stored in the record */ virtual uint32 data_length() { return pack_length(); } virtual uint32 sort_length() const { return pack_length(); } @@ -1622,15 +1635,20 @@ class Field: public Value_source defines how to store, retrieve, or compare the field. */ virtual Field *make_new_field(MEM_ROOT *root, TABLE *new_table, - bool keep_type); + bool keep_type, const Tmp_field_param *param); + /* + new_key_field is used to create a new key field for an internal temporay + table or for copying data between a key field and a table field + */ virtual Field *new_key_field(MEM_ROOT *root, TABLE *new_table, uchar *new_ptr, uint32 length, uchar *new_null_ptr, uint new_null_bit); Field *create_tmp_field(MEM_ROOT *root, TABLE *new_table, - bool maybe_null_arg); - Field *create_tmp_field(MEM_ROOT *root, TABLE *new_table) + bool maybe_null_arg, const Tmp_field_param *param); + Field *create_tmp_field(MEM_ROOT *root, TABLE *new_table, + const Tmp_field_param *param) { - return create_tmp_field(root, new_table, maybe_null()); + return create_tmp_field(root, new_table, maybe_null(), param); } Field *clone(MEM_ROOT *mem_root, TABLE *new_table); Field *clone(MEM_ROOT *mem_root, TABLE *new_table, my_ptrdiff_t diff); @@ -2240,7 +2258,6 @@ class Field_num :public Field { return to->store(val_int(), MY_TEST(flags & UNSIGNED_FLAG)); } bool is_equal(const Column_definition &new_field) const override; - uint row_pack_length() const override { return pack_length(); } uint32 pack_length_from_metadata(uint field_metadata) const override { uint32 length= pack_length(); @@ -2497,8 +2514,8 @@ class Field_decimal final :public Field_real { unireg_check_arg, field_name_arg, dec_arg, zero_arg, unsigned_arg) {} - Field *make_new_field(MEM_ROOT *root, TABLE *new_table, bool keep_type) - override; + Field *make_new_field(MEM_ROOT *root, TABLE *new_table, bool keep_type, + const Tmp_field_param *param) override; const Type_handler *type_handler() const override { return &type_handler_olddecimal; } enum ha_base_keytype key_type() const override @@ -2625,7 +2642,6 @@ class Field_new_decimal final :public Field_num { uint size_of() const override { return sizeof *this; } uint32 pack_length() const override { return bin_size; } uint pack_length_from_metadata(uint field_metadata) const override; - uint row_pack_length() const override { return pack_length(); } bool compatible_field_size(uint field_metadata, const Relay_log_info *rli, uint16 mflags, int *order_var) const override; bool is_equal(const Column_definition &new_field) const override; @@ -3059,7 +3075,6 @@ class Field_float final :public Field_real { int cmp(const uchar *,const uchar *) const override; void sort_string(uchar *buff, uint length) override; uint32 pack_length() const override { return sizeof(float); } - uint row_pack_length() const override { return pack_length(); } ulonglong get_max_int_value() const override { /* @@ -3115,7 +3130,6 @@ class Field_double :public Field_real { int cmp(const uchar *,const uchar *) const override final; void sort_string(uchar *buff, uint length) override final; uint32 pack_length() const override final { return sizeof(double); } - uint row_pack_length() const override final { return pack_length(); } ulonglong get_max_int_value() const override final { /* @@ -3542,7 +3556,6 @@ class Field_timestampf :public Field_timestamp_with_dec { { return my_timestamp_binary_length(dec); } - uint row_pack_length() const override { return pack_length(); } uint pack_length_from_metadata(uint field_metadata) const override { DBUG_ENTER("Field_timestampf::pack_length_from_metadata"); @@ -3901,7 +3914,6 @@ class Field_timef final :public Field_time_with_dec { { return my_time_binary_length(dec); } - uint row_pack_length() const override { return pack_length(); } uint pack_length_from_metadata(uint field_metadata) const override { DBUG_ENTER("Field_timef::pack_length_from_metadata"); @@ -4109,7 +4121,6 @@ class Field_datetimef final :public Field_datetime_with_dec { { return my_datetime_binary_length(dec); } - uint row_pack_length() const override { return pack_length(); } uint pack_length_from_metadata(uint field_metadata) const override { DBUG_ENTER("Field_datetimef::pack_length_from_metadata"); @@ -4259,8 +4270,8 @@ class Field_string final :public Field_longstr { uint max_packed_col_length(uint max_length) const override; uint size_of() const override { return sizeof *this; } bool has_charset() const override { return charset() != &my_charset_bin; } - Field *make_new_field(MEM_ROOT *root, TABLE *new_table, bool keep_type) - override; + Field *make_new_field(MEM_ROOT *root, TABLE *new_table, bool keep_type, + const Tmp_field_param *param) override; uint get_key_image(uchar *buff, uint length, const uchar *ptr_arg, imagetype type) const override; sql_mode_t value_depends_on_sql_mode() const override; @@ -4391,8 +4402,8 @@ class Field_varstring :public Field_longstr { uint size_of() const override { return sizeof *this; } bool has_charset() const override { return charset() == &my_charset_bin ? FALSE : TRUE; } - Field *make_new_field(MEM_ROOT *root, TABLE *new_table, bool keep_type) - override; + Field *make_new_field(MEM_ROOT *root, TABLE *new_table, bool keep_type, + const Tmp_field_param *param) override; Field *new_key_field(MEM_ROOT *root, TABLE *new_table, uchar *new_ptr, uint32 length, uchar *new_null_ptr, uint new_null_bit) override; @@ -4461,7 +4472,8 @@ class Field_varstring_compressed final :public Field_varstring { { DBUG_ASSERT(0); return 0; } using Field_varstring::key_cmp; Binlog_type_info binlog_type_info() const override; - Field *make_new_field(MEM_ROOT *root, TABLE *new_table, bool keep_type) override; + Field *make_new_field(MEM_ROOT *root, TABLE *new_table, bool keep_type, + const Tmp_field_param *param) override; }; @@ -4514,18 +4526,6 @@ static inline void store_lowendian(ulonglong num, uchar *to, uint bytes) } } -static inline longlong read_lowendian(const uchar *from, uint bytes) -{ - switch(bytes) { - case 1: return from[0]; - case 2: return uint2korr(from); - case 3: return uint3korr(from); - case 4: return uint4korr(from); - case 8: return sint8korr(from); - default: DBUG_ASSERT(0); return 0; - } -} - extern LEX_CSTRING temp_lex_str; @@ -4550,12 +4550,15 @@ class Field_blob :public Field_longstr { static void do_copy_blob(const Copy_field *copy); static void do_conv_blob(const Copy_field *copy); uint get_key_image_itRAW(const uchar *ptr_arg, uchar *buff, uint length) const; + int handle_group_concat(const char *from, size_t length, + CHARSET_INFO *cs, bool with_zero_prefix); public: Field_blob(uchar *ptr_arg, uchar *null_ptr_arg, uchar null_bit_arg, enum utype unireg_check_arg, const LEX_CSTRING *field_name_arg, TABLE_SHARE *share, uint blob_pack_length, const DTCollation &collation); - Field_blob(uint32 len_arg,bool maybe_null_arg, const LEX_CSTRING *field_name_arg, + Field_blob(uint32 len_arg,bool maybe_null_arg, + const LEX_CSTRING *field_name_arg, const DTCollation &collation) :Field_longstr((uchar*) 0, len_arg, maybe_null_arg ? (uchar*) "": 0, 0, NONE, field_name_arg, collation), @@ -4671,7 +4674,7 @@ class Field_blob :public Field_longstr { int cmp(const uchar *a, uint32 a_length, const uchar *b, uint32 b_length) const; int cmp_binary(const uchar *a,const uchar *b, uint32 max_length=~0U) const - override; + override; int key_cmp(const uchar *,const uchar*) const override; int key_cmp(const uchar *str, uint length) const override; /* Never update the value of min_val for a blob field */ @@ -4691,9 +4694,10 @@ class Field_blob :public Field_longstr { @returns The length of the raw data itself without the pointer. */ - uint32 pack_length_no_ptr() const + uint32 length_size() const override { return (uint32) (packlength); } - uint row_pack_length() const override { return pack_length_no_ptr(); } + void set_pack_length(uint32 packlength_arg); + uint row_pack_length() const override { return packlength; } uint32 sort_length() const override; uint32 sort_suffix_length() const override; uint32 value_length() override { return get_length(); } @@ -4715,10 +4719,10 @@ class Field_blob :public Field_longstr { store_length(ptr, packlength, (uint32)number); } inline uint32 get_length(my_ptrdiff_t row_offset= 0) const - { return get_length(ptr+row_offset, this->packlength); } + { return get_length(ptr+row_offset, packlength); } uint32 get_length(const uchar *ptr, uint packlength) const; uint32 get_length(const uchar *ptr_arg) const - { return get_length(ptr_arg, this->packlength); } + { return get_length(ptr_arg, packlength); } inline uchar *get_ptr() const { return get_ptr(ptr); } inline uchar *get_ptr(const uchar *ptr_arg) const { @@ -4749,7 +4753,8 @@ class Field_blob :public Field_longstr { return get_key_image_itRAW(ptr_arg, buff, length); } void set_key_image(const uchar *buff,uint length) override; - Field *make_new_field(MEM_ROOT *, TABLE *new_table, bool keep_type) override; + Field *make_new_field(MEM_ROOT *, TABLE *new_table, bool keep_type, + const Tmp_field_param *param) override; Field *new_key_field(MEM_ROOT *root, TABLE *new_table, uchar *new_ptr, uint32 length, uchar *new_null_ptr, uint new_null_bit) override; @@ -4903,10 +4908,59 @@ class Field_blob_compressed final :public Field_blob { override { DBUG_ASSERT(0); return 0; } Binlog_type_info binlog_type_info() const override; - Field *make_new_field(MEM_ROOT *root, TABLE *new_table, bool keep_type) override; + Field *make_new_field(MEM_ROOT *root, TABLE *new_table, bool keep_type, + const Tmp_field_param *param) override; +}; + + +/* + class for using Blob keys for internal temporary tables. + + The difference to Field_blob is that the blob key is stored as + [length (4 bytes) ] [pointer to data (8 bytes)] + This allows us to use the whole blob as a key and also avoids copying + the blob value to the key. +*/ + +class Field_blob_key final :public Field_blob { +public: + Field_blob_key(uchar *ptr_arg, uchar *null_ptr_arg, + uchar null_bit_arg, enum utype unireg_check_arg, + const LEX_CSTRING *field_name_arg, TABLE_SHARE *share, + uint blob_pack_length, const DTCollation &collation) : + Field_blob(ptr_arg, null_ptr_arg, null_bit_arg, unireg_check_arg, + field_name_arg, share, blob_pack_length, collation) + {} + Field_blob_key(uint32 len_arg, bool maybe_null_arg, + const LEX_CSTRING *field_name_arg, + const DTCollation &collation) + :Field_blob(len_arg, maybe_null_arg, field_name_arg, collation) + {} + + /* Field blob keys have always a 4 byte length and HA_KEYTYPE_XXX4 */ + uint32 key_pack_length() const override + { return (uint32) (4 + portable_sizeof_char_ptr); } + uint16 key_part_length_bytes() const override { return 4; } + int key_cmp(const uchar *,const uchar*) const override; + int key_cmp(const uchar *str, uint length) const override; + uint get_key_image(uchar *buff, uint length, + const uchar *ptr_arg, imagetype type) const override + { + /* Internal temporary tables doesn't use key-only-reads */ + DBUG_ASSERT(0); + return 0; + } + void set_key_image(const uchar *buff,uint length) override; + enum ha_base_keytype key_type() const override + { return binary() ? HA_KEYTYPE_VARBINARY4 : HA_KEYTYPE_VARTEXT4; } + uint32 key_length() const override { return 4 + portable_sizeof_char_ptr; } + Field *new_key_field(MEM_ROOT *root, TABLE *new_table, + uchar *new_ptr, uint32 length, + uchar *new_null_ptr, uint new_null_bit) override; }; + class Field_enum :public Field_str, public Type_typelib_ptr_attributes { @@ -4931,8 +4985,8 @@ class Field_enum :public Field_str, { flags|=ENUM_FLAG; } - Field *make_new_field(MEM_ROOT *root, TABLE *new_table, bool keep_type) - override; + Field *make_new_field(MEM_ROOT *root, TABLE *new_table, bool keep_type, + const Tmp_field_param *param) override; const Type_handler *type_handler() const override { return &type_handler_enum; } enum ha_base_keytype key_type() const override; @@ -4992,7 +5046,6 @@ class Field_enum :public Field_str, uint size_of() const override { return sizeof *this; } uint pack_length_from_metadata(uint field_metadata) const override { return (field_metadata & 0x00ff); } - uint row_pack_length() const override { return pack_length(); } bool zero_pack() const override { return false; } bool optimize_range(uint, uint) const override { return false; } bool eq_def(const Field *field) const override; diff --git a/sql/item.cc b/sql/item.cc index e7cca88d9e7d9..e20dfe0504642 100644 --- a/sql/item.cc +++ b/sql/item.cc @@ -3374,7 +3374,8 @@ void Item_field::set_field(Field *field_par) any_privileges= 0; if (field->table->s->tmp_table == SYSTEM_TMP_TABLE || - field->table->s->tmp_table == INTERNAL_TMP_TABLE) + field->table->s->tmp_table == INTERNAL_TMP_TABLE || + field->table->s->tmp_table == RESULT_TMP_TABLE) set_refers_to_temp_table(); } diff --git a/sql/item.h b/sql/item.h index d47087ff619d4..7672a948b7aa2 100644 --- a/sql/item.h +++ b/sql/item.h @@ -670,21 +670,29 @@ class Tmp_field_param bool m_modify_item; bool m_table_cant_handle_bit_fields; bool m_make_copy_field; + bool m_part_of_unique_key; + bool m_group_concat; public: Tmp_field_param(bool group, bool modify_item, bool table_cant_handle_bit_fields, - bool make_copy_field) + bool make_copy_field, + bool part_of_unique_key, + bool group_concat) :m_group(group), m_modify_item(modify_item), m_table_cant_handle_bit_fields(table_cant_handle_bit_fields), - m_make_copy_field(make_copy_field) + m_make_copy_field(make_copy_field), + m_part_of_unique_key(part_of_unique_key), + m_group_concat(group_concat) { } bool group() const { return m_group; } bool modify_item() const { return m_modify_item; } bool table_cant_handle_bit_fields() const { return m_table_cant_handle_bit_fields; } bool make_copy_field() const { return m_make_copy_field; } + bool part_of_unique_key() const { return m_part_of_unique_key; } + bool group_concat() const { return m_group_concat; } void set_modify_item(bool to) { m_modify_item= to; } }; @@ -882,10 +890,13 @@ class Item :public Value_source, @retval NULL error @retval !NULL on success */ - Field *tmp_table_field_from_field_type(MEM_ROOT *root, TABLE *table) + Field *tmp_table_field_from_field_type(MEM_ROOT *root, TABLE *table, + const Tmp_field_param *param) { DBUG_ASSERT(fixed()); - const Type_handler *h= type_handler()->type_handler_for_tmp_table(this); + + const Type_handler *h= type_handler()->type_handler_for_tmp_table(this, + param); return h->make_and_init_table_field(root, &name, Record_addr(maybe_null()), *this, table); @@ -907,7 +918,7 @@ class Item :public Value_source, DBUG_ASSERT(!param->make_copy_field()); DBUG_ASSERT(!is_result_field()); DBUG_ASSERT(type() != NULL_ITEM); - return tmp_table_field_from_field_type(root, table); + return tmp_table_field_from_field_type(root, table, param); } Field *create_tmp_field_int(MEM_ROOT *root, TABLE *table, uint convert_int_length); @@ -3691,7 +3702,8 @@ class Item_result_field :public Item_fixed_hybrid /* Item with result field */ const Tmp_field_param *param) override { DBUG_ASSERT(fixed()); - const Type_handler *h= type_handler()->type_handler_for_tmp_table(this); + const Type_handler *h= type_handler()->type_handler_for_tmp_table(this, + param); return create_tmp_field_ex_from_handler(root, table, src, param, h); } void get_tmp_field_src(Tmp_field_src *src, const Tmp_field_param *param); @@ -4830,7 +4842,10 @@ class Item_int :public Item_num const Type_handler *type_handler() const override { return type_handler_long_or_longlong(); } Field *create_field_for_create_select(MEM_ROOT *root, TABLE *table) override - { return tmp_table_field_from_field_type(root, table); } + { + const Tmp_field_param param(false, false, false, false, false, false); + return tmp_table_field_from_field_type(root, table, ¶m); + } const longlong *const_ptr_longlong() const override { return &value; } bool val_bool() override { return value != 0; } longlong val_int() override @@ -8563,12 +8578,7 @@ class Item_type_holder: public Item, String *val_str(String*) override; bool get_date(THD *thd, MYSQL_TIME *ltime, date_mode_t fuzzydate) override; Field *create_tmp_field_ex(MEM_ROOT *root, TABLE *table, Tmp_field_src *src, - const Tmp_field_param *param) override - { - return Item_type_holder::real_type_handler()-> - make_and_init_table_field(root, &name, Record_addr(maybe_null()), - *this, table); - } + const Tmp_field_param *param) override; protected: Item *shallow_copy(THD *) const override { return nullptr; } Item *deep_copy(THD *) const override { return nullptr; } diff --git a/sql/item_func.h b/sql/item_func.h index c482cdca44c44..eb9cd47811ebc 100644 --- a/sql/item_func.h +++ b/sql/item_func.h @@ -255,7 +255,10 @@ class Item_func :public Item_func_or_sum public: friend class udf_handler; Field *create_field_for_create_select(MEM_ROOT *root, TABLE *table) override - { return tmp_table_field_from_field_type(root, table); } + { + const Tmp_field_param param(false, false, false, false, false, false); + return tmp_table_field_from_field_type(root, table, ¶m); + } Item *get_tmp_table_item(THD *thd) override; void fix_char_length_ulonglong(ulonglong max_char_length_arg) @@ -638,7 +641,9 @@ class Item_handled_func: public Item_func const Type_handler * type_handler_for_create_select(const Item_handled_func *item) const override { - return return_type_handler(item)->type_handler_for_tmp_table(item); + const Tmp_field_param param(false, false, false, false, false, false); + return return_type_handler(item)->type_handler_for_tmp_table(item, + ¶m); } double val_real(Item_handled_func *item) const override { diff --git a/sql/item_strfunc.h b/sql/item_strfunc.h index de50eaaf6f390..2e1f3c2571e54 100644 --- a/sql/item_strfunc.h +++ b/sql/item_strfunc.h @@ -2559,7 +2559,8 @@ class Item_temptable_rowid :public Item_str_func Item_temptable_rowid(TABLE *table_arg); const Type_handler *type_handler() const override { return &type_handler_string; } - Field *create_tmp_field(MEM_ROOT *root, bool group, TABLE *table) + Field *create_tmp_field(MEM_ROOT *root, bool group, TABLE *table, + Tmp_field_param *param) { return create_table_field_from_handler(root, table); } String *val_str(String *str) override; enum Functype functype() const override { return TEMPTABLE_ROWID; } diff --git a/sql/item_sum.cc b/sql/item_sum.cc index efc5b9450c3f6..c82e102d6f91e 100644 --- a/sql/item_sum.cc +++ b/sql/item_sum.cc @@ -797,11 +797,14 @@ bool Aggregator_distinct::setup(THD *thd) table->file->extra(HA_EXTRA_NO_ROWS); // Don't update rows table->no_rows=1; - if (table->s->db_type() == heap_hton) + if (table->s->db_type() == heap_hton && !table->s->blob_fields) { /* - No blobs, otherwise it would have been MyISAM: set up a compare - function and its arguments to use with Unique. + Unique tree compares raw record bytes (simple_raw_key_cmp or + composite_key_cmp). Blob fields store only a pointer in the + record, so raw comparison would compare pointer values, not + actual blob data. Skip the Unique tree path for blob tables + and fall through to the ha_write_tmp_row path below. */ qsort_cmp2 compare_key; void* cmp_arg; @@ -998,11 +1001,22 @@ bool Aggregator_distinct::add() */ return tree->unique_add(table->record[0] + table->s->null_bytes); } - if (unlikely((error= table->file->ha_write_tmp_row(table->record[0]))) && - table->file->is_fatal_error(error, HA_CHECK_DUP)) + if (unlikely((error= table->file->ha_write_tmp_row(table->record[0])))) { - table->file->print_error(error, MYF(0)); - return TRUE; + if (!table->file->is_fatal_error(error, HA_CHECK_DUP)) + return FALSE; // duplicate, not an error + /* + Non-duplicate write error. If the table is HEAP and the error + is HA_ERR_RECORD_FILE_FULL, create_internal_tmp_table_from_heap() + converts it to an on-disk engine and copies all rows plus the + overflow row (record[0]). For any other error it reports a + fatal error and returns 1. + */ + if (create_internal_tmp_table_from_heap(table->in_use, table, + tmp_table_param->start_recinfo, + &tmp_table_param->recinfo, + error, 0, NULL)) + return TRUE; } return FALSE; } @@ -1302,21 +1316,22 @@ void Item_sum_min_max::setup_hybrid(THD *thd, Item *item, Item *value_arg) Field *Item_sum_min_max::create_tmp_field(MEM_ROOT *root, - bool group, TABLE *table) + bool group, TABLE *table, + const Tmp_field_param *param) { DBUG_ENTER("Item_sum_min_max::create_tmp_field"); if (args[0]->type() == Item::FIELD_ITEM) { Field *field= ((Item_field*) args[0])->field; - if ((field= field->create_tmp_field(root, table, true))) + if ((field= field->create_tmp_field(root, table, true, 0))) { DBUG_ASSERT((field->flags & NOT_NULL_FLAG) == 0); field->field_name= name; } DBUG_RETURN(field); } - DBUG_RETURN(tmp_table_field_from_field_type(root, table)); + DBUG_RETURN(tmp_table_field_from_field_type(root, table, param)); } /*********************************************************************** @@ -2019,7 +2034,8 @@ Item *Item_sum_avg::copy_or_same(THD* thd) } -Field *Item_sum_avg::create_tmp_field(MEM_ROOT *root, bool group, TABLE *table) +Field *Item_sum_avg::create_tmp_field(MEM_ROOT *root, bool group, TABLE *table, + const Tmp_field_param *param) { if (group) @@ -2037,7 +2053,7 @@ Field *Item_sum_avg::create_tmp_field(MEM_ROOT *root, bool group, TABLE *table) field->init(table); return field; } - return tmp_table_field_from_field_type(root, table); + return tmp_table_field_from_field_type(root, table, param); } @@ -2263,7 +2279,8 @@ Item *Item_sum_variance::copy_or_same(THD* thd) pass around. */ Field *Item_sum_variance::create_tmp_field(MEM_ROOT *root, - bool group, TABLE *table) + bool group, TABLE *table, + const Tmp_field_param *param) { Field *field; if (group) @@ -3706,12 +3723,6 @@ extern "C" int group_concat_key_cmp_with_order(void *arg, const void *key1, order_item++) { Item *item= *(*order_item)->item; - /* - If field_item is a const item then either get_tmp_table_field returns 0 - or it is an item over a const table. - */ - if (item->const_item()) - continue; /* If item is a const item then either get_tmp_table_field returns 0 or it is an item over a const table. @@ -3897,6 +3908,7 @@ int dump_leaf_key(void* key_arg, element_count count __attribute__((unused)), Field *field= (*arg)->get_tmp_table_field(); if (field) { + /* Note that field->table can be different table! */ uint offset= (field->offset(field->table->record[0]) - table->s->null_bytes); DBUG_ASSERT(offset < table->s->reclength); @@ -3918,7 +3930,7 @@ int dump_leaf_key(void* key_arg, element_count count __attribute__((unused)), /* stop if length of result more than max_length */ if (result->length() > max_length) { - THD *thd= current_thd; + THD *thd= table->in_use; item->cut_max_length(result, old_length, max_length); item->warning_for_row= TRUE; report_cut_value_error(thd, item->row_count, item->func_name()); diff --git a/sql/item_sum.h b/sql/item_sum.h index 39ed79e7c0203..4f5fe10888a59 100644 --- a/sql/item_sum.h +++ b/sql/item_sum.h @@ -524,11 +524,12 @@ class Item_sum :public Item_func_or_sum aggregator_clear(); } virtual void make_unique() { force_copy_fields= TRUE; } - virtual Field *create_tmp_field(MEM_ROOT *root, bool group, TABLE *table); + virtual Field *create_tmp_field(MEM_ROOT *root, bool group, TABLE *table, + const Tmp_field_param *param); Field *create_tmp_field_ex(MEM_ROOT *root, TABLE *table, Tmp_field_src *src, const Tmp_field_param *param) override { - return create_tmp_field(root, param->group(), table); + return create_tmp_field(root, param->group(), table, param); } bool collect_outer_ref_processor(void *param) override; bool init_sum_func_check(THD *thd); @@ -996,7 +997,8 @@ class Item_sum_avg :public Item_sum_sum return has_with_distinct() ? name_distinct : name_normal; } Item *copy_or_same(THD* thd) override; - Field *create_tmp_field(MEM_ROOT *root, bool group, TABLE *table) override; + Field *create_tmp_field(MEM_ROOT *root, bool group, TABLE *table, + const Tmp_field_param *param) override; void cleanup() override { count= 0; @@ -1085,7 +1087,8 @@ class Item_sum_variance :public Item_sum_double return sample ? name_sample : name_normal; } Item *copy_or_same(THD* thd) override; - Field *create_tmp_field(MEM_ROOT *root, bool group, TABLE *table) override + Field *create_tmp_field(MEM_ROOT *root, bool group, TABLE *table, + const Tmp_field_param *param) override final; void cleanup() override final { @@ -1207,7 +1210,8 @@ class Item_sum_min_max :public Item_sum_hybrid bool any_value() { return was_values; } void no_rows_in_result() override; void restore_to_before_no_rows_in_result() override; - Field *create_tmp_field(MEM_ROOT *root, bool group, TABLE *table) override; + Field *create_tmp_field(MEM_ROOT *root, bool group, TABLE *table, + const Tmp_field_param *param) override; void setup_caches(THD *thd) override { setup_hybrid(thd, arguments()[0], NULL); } }; diff --git a/sql/item_timefunc.cc b/sql/item_timefunc.cc index d504816296b59..d622763e32be7 100644 --- a/sql/item_timefunc.cc +++ b/sql/item_timefunc.cc @@ -4137,7 +4137,8 @@ class Item_char_typecast_func_handler: public Item_handled_func::Handler_str const Type_handler * type_handler_for_create_select(const Item_handled_func *item) const override { - return return_type_handler(item)->type_handler_for_tmp_table(item); + const Tmp_field_param param(false, false, false, false, false, false); + return return_type_handler(item)->type_handler_for_tmp_table(item, ¶m); } bool fix_length_and_dec(Item_handled_func *item) const override diff --git a/sql/item_windowfunc.cc b/sql/item_windowfunc.cc index b7cf5ed07a560..cbd2fe6041252 100644 --- a/sql/item_windowfunc.cc +++ b/sql/item_windowfunc.cc @@ -460,7 +460,8 @@ bool Item_sum_hybrid_simple::get_date(THD *thd, MYSQL_TIME *ltime, date_mode_t f } Field *Item_sum_hybrid_simple::create_tmp_field(MEM_ROOT *root, - bool group, TABLE *table) + bool group, TABLE *table, + const Tmp_field_param *param) { DBUG_ASSERT(0); return NULL; diff --git a/sql/item_windowfunc.h b/sql/item_windowfunc.h index bff614372e2fc..b8b3cdf61e8ef 100644 --- a/sql/item_windowfunc.h +++ b/sql/item_windowfunc.h @@ -334,7 +334,8 @@ class Item_sum_hybrid_simple : public Item_sum_hybrid const Type_handler *type_handler() const override { return Type_handler_hybrid_field_type::type_handler(); } void update_field() override; - Field *create_tmp_field(MEM_ROOT *root, bool group, TABLE *table) override; + Field *create_tmp_field(MEM_ROOT *root, bool group, TABLE *table, + const Tmp_field_param *param) override; void clear() override { value->clear(); diff --git a/sql/opt_subselect.cc b/sql/opt_subselect.cc index bbc9434c4cf02..220464cf0884d 100644 --- a/sql/opt_subselect.cc +++ b/sql/opt_subselect.cc @@ -5032,7 +5032,8 @@ SJ_TMP_TABLE::create_sj_weedout_tmp_table(THD *thd) key_part_info->offset= field->offset(table->record[0]); key_part_info->length= (uint16) field->key_length(); key_part_info->type= (uint8) field->key_type(); - key_part_info->key_type = FIELDFLAG_BINARY; + key_part_info->key_type= FIELDFLAG_BINARY; + key_part_info->key_part_flag= field->key_part_flag(); if (!using_unique_constraint) { if (!(key_field= field->new_key_field(thd->mem_root, table, diff --git a/sql/records.cc b/sql/records.cc index 5a7f5385f06c5..442ddcd075214 100644 --- a/sql/records.cc +++ b/sql/records.cc @@ -193,7 +193,7 @@ bool init_read_record(READ_RECORD *info,THD *thd, TABLE *table, info->table=table; info->sort_info= filesort; - if ((table->s->tmp_table == INTERNAL_TMP_TABLE) && !using_addon_fields) + if ((table->s->tmp_table == RESULT_TMP_TABLE) && !using_addon_fields) (void) table->file->extra(HA_EXTRA_MMAP); if (using_addon_fields) diff --git a/sql/sql_delete.cc b/sql/sql_delete.cc index a3bfbd6eae080..f8718d8c21660 100644 --- a/sql/sql_delete.cc +++ b/sql/sql_delete.cc @@ -132,7 +132,9 @@ bool Update_plan::save_explain_data_intern(THD *thd, explain->table_tracker.set_gap_tracker(&explain->extra_time_tracker); table->file->set_time_tracker(&explain->table_tracker); - if (table->file->handler_stats && table->s->tmp_table != INTERNAL_TMP_TABLE) + if (table->file->handler_stats && + table->s->tmp_table != RESULT_TMP_TABLE && + table->s->tmp_table != INTERNAL_TMP_TABLE) explain->handler_for_stats= table->file; } @@ -1326,10 +1328,12 @@ multi_delete::initialize_tables(JOIN *join) if (temp_fields.push_back(item, thd->mem_root)) DBUG_RETURN(1); /* Make an unique key over the first field to avoid duplicated updates */ - ORDER group; - bzero((char*) &group, sizeof(group)); - group.direction= ORDER::ORDER_ASC; - group.item= (Item**) temp_fields.head_ref(); + ORDER *group= (ORDER*) alloc_root(thd->mem_root, sizeof(ORDER)); + if (!group) + DBUG_RETURN(1); + bzero((char*) group, sizeof(*group)); + group->direction= ORDER::ORDER_ASC; + group->item= (Item**) temp_fields.head_ref(); TMP_TABLE_PARAM *tmp_param; prior->shared = index; tmp_param= &tmp_table_param[prior->shared]; @@ -1337,9 +1341,9 @@ multi_delete::initialize_tables(JOIN *join) tmp_param->tmp_name="update"; tmp_param->field_count= temp_fields.elements; tmp_param->func_count= temp_fields.elements; - calc_group_buffer(tmp_param, &group); + calc_group_buffer(tmp_param, group); tmp_tables[index]=create_tmp_table(thd, tmp_param, temp_fields, - (ORDER*) &group, 0, 0, + group, 0, 0, TMP_TABLE_ALL_COLUMNS, HA_POS_ERROR, &empty_clex_str); if (!tmp_tables[index]) DBUG_RETURN(1); diff --git a/sql/sql_derived.cc b/sql/sql_derived.cc index c4f60a133ef7f..352e4be7f3325 100644 --- a/sql/sql_derived.cc +++ b/sql/sql_derived.cc @@ -874,27 +874,37 @@ bool mysql_derived_prepare(THD *thd, LEX *lex, TABLE_LIST *derived) distinct= (unit->first_select()->next_select() ? unit->union_distinct && !unit->union_distinct->next_select() : unit->distinct); - - if (!(derived->table) && - derived->derived_result->create_result_table(thd, &unit->types, - distinct, - (first_select->options | - thd->variables.option_bits | - TMP_TABLE_ALL_COLUMNS), - &derived->alias, - FALSE, FALSE, keep_row_order, - 0)) - { - thd->create_tmp_table_for_derived= FALSE; - if (thd->is_error()) + { + ulonglong create_options= (first_select->options | + thd->variables.option_bits | + TMP_TABLE_ALL_COLUMNS); + /* + Force a disk-based engine when the outer query uses FULLTEXT + functions, since HEAP does not support FULLTEXT indexes. + */ + if (derived->select_lex && + derived->select_lex->ftfunc_list->elements) + create_options= create_options | TMP_TABLE_FORCE_MYISAM; + + if (!(derived->table) && + derived->derived_result->create_result_table(thd, &unit->types, + distinct, + create_options, + &derived->alias, + FALSE, FALSE, + keep_row_order, 0)) { - /* - EOM error, or attempted to a create a table with a Field - of a not allowed data type, e.g. SYS_REFCURSOR. - */ - res= true; + thd->create_tmp_table_for_derived= FALSE; + if (thd->is_error()) + { + /* + EOM error, or attempted to a create a table with a Field + of a not allowed data type, e.g. SYS_REFCURSOR. + */ + res= true; + } + goto exit; } - goto exit; } thd->create_tmp_table_for_derived= FALSE; diff --git a/sql/sql_expression_cache.cc b/sql/sql_expression_cache.cc index 002fa02cd21be..556d0c652c82e 100644 --- a/sql/sql_expression_cache.cc +++ b/sql/sql_expression_cache.cc @@ -138,6 +138,23 @@ void Expression_cache_tmptable::init() goto error; } + /* + HEAP hash indexes on blob columns use a pointer-based key format + (4-byte length + data pointer). This is incompatible with the SQL + layer's key format (2-byte length + inline data) because + Field_blob::new_key_field() returns a Field_varstring. + + This check is slightly conservative: a blob only in the result + value would not affect the key. However, it matches the pre-blob + behavior where blobs forced Aria, which failed the heap_hton check + above and disabled the cache anyway. + */ + if (cache_table->s->blob_fields) + { + DBUG_PRINT("error", ("blob fields not supported in heap expression cache")); + goto error; + } + field_counter= 1; if (cache_table->alloc_keys(1) || diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc index 86258e2de9bc9..6b38c877a494c 100644 --- a/sql/sql_insert.cc +++ b/sql/sql_insert.cc @@ -3125,7 +3125,8 @@ TABLE *Delayed_insert::get_local_table(THD* client_thd) found_next_number_field= table->found_next_number_field; for (org_field= table->field; *org_field; org_field++, field++) { - if (!(*field= (*org_field)->make_new_field(client_thd->mem_root, copy, 1))) + if (!(*field= (*org_field)->make_new_field(client_thd->mem_root, copy, 1, + 0))) goto error; (*field)->unireg_check= (*org_field)->unireg_check; (*field)->invisible= (*org_field)->invisible; @@ -4847,7 +4848,7 @@ void select_insert::abort_result_set() Field *Item::create_field_for_create_select(MEM_ROOT *root, TABLE *table) { - static Tmp_field_param param(false, false, false, false); + static const Tmp_field_param param(false, false, false, false, false, false); Tmp_field_src src; return create_tmp_field_ex(root, table, &src, ¶m); } diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc index bcd6de564f550..9fd1d6531be28 100644 --- a/sql/sql_parse.cc +++ b/sql/sql_parse.cc @@ -8054,6 +8054,7 @@ bool add_to_list(THD *thd, SQL_I_List &list, Item *item,bool asc) order->used=0; order->counter_used= 0; order->fast_field_copier_setup= 0; + order->field= 0; if (thd->lex->clause_winfuncs.is_empty()) order->window_funcs.empty(); else if (order->window_funcs.copy(&thd->lex->clause_winfuncs, thd->mem_root)) diff --git a/sql/sql_select.cc b/sql/sql_select.cc index 9c2a99676d798..f079f8994d576 100644 --- a/sql/sql_select.cc +++ b/sql/sql_select.cc @@ -21621,7 +21621,7 @@ Field *Item::tmp_table_field_from_field_type_maybe_null(MEM_ROOT *root, DBUG_ASSERT(!param->make_copy_field() || type() == CONST_ITEM); DBUG_ASSERT(!is_result_field()); Field *result; - if ((result= tmp_table_field_from_field_type(root, table))) + if ((result= tmp_table_field_from_field_type(root, table, param))) { if (result && is_explicit_null) result->is_created_from_null_item= true; @@ -21630,7 +21630,8 @@ Field *Item::tmp_table_field_from_field_type_maybe_null(MEM_ROOT *root, } -Field *Item_sum::create_tmp_field(MEM_ROOT *root, bool group, TABLE *table) +Field *Item_sum::create_tmp_field(MEM_ROOT *root, bool group, TABLE *table, + const Tmp_field_param *param) { Field *UNINIT_VAR(new_field); @@ -21645,7 +21646,7 @@ Field *Item_sum::create_tmp_field(MEM_ROOT *root, bool group, TABLE *table) case TIME_RESULT: case DECIMAL_RESULT: case STRING_RESULT: - new_field= tmp_table_field_from_field_type(root, table); + new_field= tmp_table_field_from_field_type(root, table, param); break; case ROW_RESULT: // This case should never be chosen @@ -21698,7 +21699,7 @@ Item_field::create_tmp_field_from_item_field(MEM_ROOT *root, TABLE *new_table, */ Record_addr rec(orig_item ? orig_item->maybe_null() : maybe_null()); const Type_handler *handler= type_handler()-> - type_handler_for_tmp_table(this); + type_handler_for_tmp_table(this, param); result= handler->make_and_init_table_field(root, new_name, rec, *this, new_table); } @@ -21715,7 +21716,7 @@ Item_field::create_tmp_field_from_item_field(MEM_ROOT *root, TABLE *new_table, { bool tmp_maybe_null= param->modify_item() ? maybe_null() : field->maybe_null(); - result= field->create_tmp_field(root, new_table, tmp_maybe_null); + result= field->create_tmp_field(root, new_table, tmp_maybe_null, param); if (result && ! param->modify_item()) result->field_name= *new_name; } @@ -21752,7 +21753,7 @@ Field *Item_default_value::create_tmp_field_ex(MEM_ROOT *root, TABLE *table, as the we have to calculate the default value before we can use it. */ get_tmp_field_src(src, param); - Field *result= tmp_table_field_from_field_type(root, table); + Field *result= tmp_table_field_from_field_type(root, table, param); if (result && param->modify_item()) result_field= result; return result; @@ -21790,6 +21791,33 @@ Field *Item_ref::create_tmp_field_ex(MEM_ROOT *root, TABLE *table, } + +Field *Item_type_holder::create_tmp_field_ex(MEM_ROOT *root, TABLE *table, + Tmp_field_src *src, + const Tmp_field_param *param) +{ + Type_handler const *type_handler= Item_type_holder::real_type_handler(); + Type_handler_blob_common const *blob_handler; + if (param->part_of_unique_key() && + (blob_handler= + dynamic_cast(type_handler))) + { + Field_blob *blob_field= (Field_blob*) type_handler_blob_key. + make_and_init_table_field(root, &name, Record_addr(maybe_null()), + *this, table); + if (blob_field) + { + /* Fix length of blob to be able to return the original blob type */ + blob_field->set_pack_length(blob_handler->length_bytes()); + } + return blob_field; + } + return type_handler-> + make_and_init_table_field(root, &name, Record_addr(maybe_null()), + *this, table); +} + + void Item_result_field::get_tmp_field_src(Tmp_field_src *src, const Tmp_field_param *param) { @@ -21811,7 +21839,7 @@ Item_result_field::create_tmp_field_ex_from_handler( TABLE *table, Tmp_field_src *src, const Tmp_field_param *param, - const Type_handler *h) + const Type_handler *type_handler) { /* Possible Item types: @@ -21824,10 +21852,12 @@ Item_result_field::create_tmp_field_ex_from_handler( DBUG_ASSERT(type() != NULL_ITEM); get_tmp_field_src(src, param); Field *result; - if ((result= h->make_and_init_table_field(root, &name, - Record_addr(maybe_null()), - *this, table)) && - param->modify_item()) + + result= type_handler->make_and_init_table_field_ex(root, &name, + Record_addr(maybe_null()), + *this, param, table); + + if (result && param->modify_item()) result_field= result; return result; } @@ -21839,7 +21869,7 @@ Field *Item_func_sp::create_tmp_field_ex(MEM_ROOT *root, TABLE *table, { Field *result; get_tmp_field_src(src, param); - if ((result= sp_result_field->create_tmp_field(root, table))) + if ((result= sp_result_field->create_tmp_field(root, table, param))) { result->field_name= name; if (param->modify_item()) @@ -21893,6 +21923,8 @@ static bool make_json_valid_expr(TABLE *table, Field *field) @param make_copy_field Set when using with rollup when we want to have an exact copy of the field. + @param part_of_unique_key + Field is part of unique key @retval 0 on error @retval @@ -21905,11 +21937,13 @@ Field *create_tmp_field(TABLE *table, Item *item, Field **default_field, bool group, bool modify_item, bool table_cant_handle_bit_fields, - bool make_copy_field) + bool make_copy_field, + bool part_of_unique_key) { Tmp_field_src src; Tmp_field_param prm(group, modify_item, table_cant_handle_bit_fields, - make_copy_field); + make_copy_field, part_of_unique_key || group, + table->group_concat); Field *result= item->create_tmp_field_ex(table->in_use->mem_root, table, &src, &prm); if (is_json_type(item) && make_json_valid_expr(table, result)) @@ -21919,6 +21953,10 @@ Field *create_tmp_field(TABLE *table, Item *item, *default_field= src.default_field(); if (src.item_result_field()) *((*copy_func)++)= src.item_result_field(); + if (part_of_unique_key) + result->flags|= FIELD_PART_OF_TMP_UNIQUE | UNIQUE_KEY_FLAG; + if (group) + result->flags|= UNIQUE_KEY_FLAG; return result; } @@ -21964,6 +22002,7 @@ Create_tmp_table::Create_tmp_table(ORDER *group, bool distinct, ha_rows rows_limit) :m_alloced_field_count(0), m_using_unique_constraint(false), + m_heap_expected(false), m_temp_pool_slot(MY_BIT_NONE), m_group(group), m_distinct(distinct), @@ -22090,6 +22129,22 @@ TABLE *Create_tmp_table::start(THD *thd, */ fn_format(path, path, mysql_tmpdir, "", MY_REPLACE_EXT|MY_UNPACK_FILENAME); + /* + Early engine prediction: reclength is not known yet (fields haven't been + added), so pass 0 -- this is safe because pick_engine()'s only reclength + check is "> HA_MAX_REC_LENGTH", which 0 never triggers. Returns + heap_hton unless session-level overrides (tmp_memory_table_size=0, + etc.) force a disk-based engine. We use this + to avoid the too_big_for_varchar() / group_length >= MAX_BLOB_WIDTH + bail-outs that would force m_using_unique_constraint for HEAP tables + that natively support blob keys. + + Note: pick_engine() also reads m_using_unique_constraint, which is + false at this point. The guards below that set it to true are gated + by !m_heap_expected, so there is no circular dependency in practice. + */ + m_heap_expected= (pick_engine(thd, 0) == heap_hton); + if (m_group) { ORDER **prev= &m_group; @@ -22104,19 +22159,18 @@ TABLE *Create_tmp_table::start(THD *thd, param->group_parts--; continue; } - else - prev= &(tmp->next); + prev= &(tmp->next); /* - marker == 4 means two things: + marker == MARKER_NULL_KEY means two things: - store NULLs in the key, and - convert BIT fields to 64-bit long, needed because MEMORY tables can't index BIT fields. */ (*tmp->item)->marker= MARKER_NULL_KEY; // Store null in key - if ((*tmp->item)->too_big_for_varchar()) + if (!m_heap_expected && (*tmp->item)->too_big_for_varchar()) m_using_unique_constraint= true; } - if (param->group_length >= MAX_BLOB_WIDTH) + if (!m_heap_expected && param->group_length >= MAX_BLOB_WIDTH) m_using_unique_constraint= true; if (m_group) m_distinct= 0; // Can't use distinct @@ -22199,6 +22253,7 @@ TABLE *Create_tmp_table::start(THD *thd, table->s= share; init_tmp_table_share(thd, share, "", 0, "(temporary)", tmpname, true); + share->tmp_table= RESULT_TMP_TABLE; share->blob_field= blob_field; share->table_charset= param->table_charset; share->primary_key= MAX_KEY; // Indicate no primary key @@ -22300,9 +22355,11 @@ bool Create_tmp_table::add_fields(THD *thd, create_tmp_field(table, arg, ©_func, tmp_from_field, &m_default_field[fieldnr], m_group != 0, not_all_columns, - distinct_record_structure , false); + distinct_record_structure, false, + (current_counter == distinct)); if (!new_field) goto err; // Should be OOM + tmp_from_field++; thd->mem_root= mem_root_save; @@ -22326,8 +22383,6 @@ bool Create_tmp_table::add_fields(THD *thd, */ arg->set_maybe_null(); } - if (current_counter == distinct) - new_field->flags|= FIELD_PART_OF_TMP_UNIQUE; } } } @@ -22364,13 +22419,15 @@ bool Create_tmp_table::add_fields(THD *thd, */ item->marker == MARKER_NULL_KEY || param->bit_fields_as_long, - param->force_copy_fields); + param->force_copy_fields, + (current_counter == distinct)); if (unlikely(!new_field)) { if (unlikely(thd->is_fatal_error || item->cols() == 1)) goto err; // Got OOM continue; // Some kind of const item } + if (type == Item::SUM_FUNC_ITEM) { Item_sum *agg_item= (Item_sum *) item; @@ -22408,8 +22465,6 @@ bool Create_tmp_table::add_fields(THD *thd, m_group_null_items++; new_field->flags|= GROUP_FLAG; } - if (current_counter == distinct) - new_field->flags|= FIELD_PART_OF_TMP_UNIQUE; } } @@ -22434,34 +22489,45 @@ bool Create_tmp_table::add_fields(THD *thd, } +/* + Predict which engine a temporary table will use, based on session + variables and the current m_using_unique_constraint / m_select_options + state. Called early (before fields are added) with reclength=0 to set + m_heap_expected, and again from choose_engine() with the real reclength. +*/ + +handlerton *Create_tmp_table::pick_engine(THD *thd, uint reclength) +{ + if (m_using_unique_constraint || + reclength > HA_MAX_REC_LENGTH || + (m_select_options & TMP_TABLE_FORCE_MYISAM) || + thd->variables.tmp_memory_table_size == 0) + return TMP_ENGINE_HTON; + return heap_hton; +} + + bool Create_tmp_table::choose_engine(THD *thd, TABLE *table, TMP_TABLE_PARAM *param) { TABLE_SHARE *share= table->s; DBUG_ENTER("Create_tmp_table::choose_engine"); + + handlerton *engine= pick_engine(thd, share->reclength); + share->db_plugin= ha_lock_engine(0, engine); + table->file= get_new_handler(share, &table->mem_root, share->db_type()); + /* - If result table is small; use a heap, otherwise TMP_TABLE_HTON (Aria) - In the future we should try making storage engine selection more dynamic + When a disk engine was chosen for reasons other than key limits + (e.g. TMP_TABLE_FORCE_MYISAM, tmp_memory_table_size=0), + the GROUP BY key may still exceed the disk engine's max_key_parts() + or max_key_length(). Fall back to unique constraint hash dedup. */ + if (engine == TMP_ENGINE_HTON && m_group && + (param->group_parts > table->file->max_key_parts() || + param->group_length > table->file->max_key_length())) + m_using_unique_constraint= true; - if (share->blob_fields || m_using_unique_constraint || - (m_select_options & TMP_TABLE_FORCE_MYISAM) || - thd->variables.tmp_memory_table_size == 0) - { - share->db_plugin= ha_lock_engine(0, TMP_ENGINE_HTON); - table->file= get_new_handler(share, &table->mem_root, - share->db_type()); - if (m_group && - (param->group_parts > table->file->max_key_parts() || - param->group_length > table->file->max_key_length())) - m_using_unique_constraint= true; - } - else - { - share->db_plugin= ha_lock_engine(0, heap_hton); - table->file= get_new_handler(share, &table->mem_root, - share->db_type()); - } DBUG_RETURN(!table->file); } @@ -22506,9 +22572,14 @@ bool Create_tmp_table::finalize(THD *thd, if (!m_using_unique_constraint) share->reclength+= m_group_null_items; // null flag is stored separately - if (share->blob_fields == 0) + if (share->blob_fields == 0 || share->db_type() == heap_hton) { - /* We need to ensure that first byte is not 0 for the delete link */ + /* + We need to ensure that first byte is not 0 for the delete link. + HEAP uses fixed-width rows even with blobs (blob data lives in + separate continuation records within the same HP_BLOCK, not + inline in the primary record), so it still needs this guard. + */ if (m_field_count[other]) m_null_count[other]++; else @@ -22527,11 +22598,14 @@ bool Create_tmp_table::finalize(THD *thd, if (!share->reclength) share->reclength= 1; // Dummy select share->stored_rec_length= share->reclength; - /* Use packed rows if there is blobs or a lot of space to gain */ - if (share->blob_fields || - (string_total_length() >= STRING_TOTAL_LENGTH_TO_PACK_ROWS && - (share->reclength / string_total_length() <= RATIO_TO_PACK_ROWS || - string_total_length() / string_count() >= AVG_STRING_LENGTH_TO_PACK_ROWS))) + /* + HEAP does not use recinfo->type, so packed format is harmless for HEAP. + It is needed for correct packing when converting to Aria. + */ + if ((share->blob_fields || + (string_total_length() >= STRING_TOTAL_LENGTH_TO_PACK_ROWS && + (share->reclength / string_total_length() <= RATIO_TO_PACK_ROWS || + string_total_length() / string_count() >= AVG_STRING_LENGTH_TO_PACK_ROWS)))) use_packed_rows= 1; { @@ -22562,8 +22636,13 @@ bool Create_tmp_table::finalize(THD *thd, share->null_bytes= share->null_bytes_for_compare= whole_null_pack_length; } - if (share->blob_fields == 0) + if (share->blob_fields == 0 || share->db_type() == heap_hton) { + /* + Same first-byte guard as above: HEAP with blobs still uses + fixed-width rows and needs a non-zero first byte for the + delete-link mechanism. + */ null_counter[(m_field_count[other] ? other : distinct)]++; } @@ -22716,12 +22795,8 @@ bool Create_tmp_table::finalize(THD *thd, m_key_part_info->offset= field->offset(table->record[0]); m_key_part_info->length= (uint16) field->key_length(); m_key_part_info->type= (uint8) field->key_type(); - m_key_part_info->key_type = - ((ha_base_keytype) m_key_part_info->type == HA_KEYTYPE_TEXT || - (ha_base_keytype) m_key_part_info->type == HA_KEYTYPE_VARTEXT1 || - (ha_base_keytype) m_key_part_info->type == HA_KEYTYPE_VARTEXT2) ? - 0 : FIELDFLAG_BINARY; - m_key_part_info->key_part_flag= 0; + m_key_part_info->key_type= field->binary() ? FIELDFLAG_BINARY : 0; + m_key_part_info->key_part_flag= field->key_part_flag(); if (!m_using_unique_constraint) { cur_group->buff=(char*) m_group_buff; @@ -22739,14 +22814,96 @@ bool Create_tmp_table::finalize(THD *thd, (*cur_group->item)->base_flags&= ~item_base_t::MAYBE_NULL; } + /* + For blob/geometry GROUP BY keys, + m_key_part_info->length, set from field->key_length(), + contains 0 (blobs) or packlength (geometry), both too + small to hold actual data. Use the item's max_length + capped to MAX_BLOB_WIDTH so new_key_field gets a + usable size. + */ + uint32 key_field_length= m_key_part_info->length; + if ((field->flags & BLOB_FLAG) && + key_field_length <= ((Field_blob*) field)->length_size()) + { + key_field_length= MY_MIN((*cur_group->item)->max_length, + (MAX_BLOB_WIDTH - 4 - 1)); + /* + Verify that the group buffer has room for this blob key + field. For native blob columns calc_group_buffer() sees + the blob type from the start and always allocates enough + space. This can only overflow when a varchar is promoted + to blob after calc_group_buffer() has already sized the + buffer (varchar-to-blob promotion path). + */ + uint32 need= key_field_length + 4 /* length_bytes */ + + MY_TEST(maybe_null); + if (m_group_buff + need > + param->group_buff + param->group_length) + { + m_using_unique_constraint= true; + DBUG_ASSERT(0); // Should be impossible + break; + } + } + /* + Set key_part_flag from the actual field type AFTER the overflow + check. This ensures that if we break out due to a promoted + blob overflowing the group buffer, key_part_flag retains the + original SQL-layer value (HA_VAR_LENGTH_PART for varchar), + not HA_BLOB_PART. This prevents rebuild_key_from_group_buff() + from being called on a key buffer that has varchar format. + */ + m_key_part_info->key_part_flag= field->key_part_flag(); + + /* Create a new field which value is stored in the group buffer */ if (!(cur_group->field= field->new_key_field(thd->mem_root,table, m_group_buff + MY_TEST(maybe_null), - m_key_part_info->length, + key_field_length, field->null_ptr, field->null_bit))) goto err; /* purecov: inspected */ + /* + Mark that a key segment is a blob. Needed for + create_internal_tmp_table_from_heap to convert keys with blobs + to unique. + */ + if (cur_group->field->flags & BLOB_FLAG) + { + /* We only come here for Field_blob_key's */ + DBUG_ASSERT(cur_group->field[0].key_length() == + 4 + portable_sizeof_char_ptr); + /* Tell engine to that this key includes a blob */ + keyinfo->flags|= HA_BLOB_PART_KEY; + } + + /* + Verify key_part_info consistency with the GROUP BY key field. + A blob key field must have VARTEXT4/VARBINARY4 type and the + matching key_length so heap_prepare_hp_create_info() takes the + direct blob path (with its own DBUG_ASSERTs) rather than the + VARTEXT2 promotion path that silently patches stale values. + */ + DBUG_ASSERT(!(cur_group->field->flags & BLOB_FLAG) || + (m_key_part_info->length == + 4 + portable_sizeof_char_ptr)); + DBUG_ASSERT(!(cur_group->field->flags & BLOB_FLAG) || + ((ha_base_keytype) m_key_part_info->type == + HA_KEYTYPE_VARBINARY4 || + (ha_base_keytype) m_key_part_info->type == + HA_KEYTYPE_VARTEXT4)); + DBUG_ASSERT(!(m_key_part_info->key_part_flag & HA_BLOB_PART) || + (cur_group->field->flags & BLOB_FLAG)); + /* + Set store_length for all GROUP BY key parts so + rebuild_key_from_group_buff() can advance through the key buffer. + store_length = key field pack_length + null flag byte. + */ + m_key_part_info->store_length= + cur_group->field->key_pack_length() + MY_TEST(maybe_null); + if (maybe_null) { /* @@ -22762,7 +22919,7 @@ bool Create_tmp_table::finalize(THD *thd, cur_group->buff++; // Pointer to field data m_group_buff++; // Skip null flag } - m_group_buff+= cur_group->field->pack_length(); + m_group_buff+= cur_group->field->key_pack_length(); } keyinfo->key_length+= m_key_part_info->length; } @@ -22797,9 +22954,17 @@ bool Create_tmp_table::finalize(THD *thd, */ keyinfo->flags|= HA_UNIQUE_HASH; } + /* + HEAP-specific: skip null-bits helper key part. + HEAP handles NULLs per-segment in its hash index, so it does not + need the extra null-key-part that MyISAM/Aria use for unique blob + constraints. + */ + bool need_null_key_part= share->have_unique_constraint() && + share->db_type() != heap_hton && + null_pack_length[distinct]; keyinfo->user_defined_key_parts= m_field_count[distinct] + - ((keyinfo->flags & HA_UNIQUE_HASH) ? - MY_TEST(null_pack_length[distinct]) : 0); + MY_TEST(need_null_key_part); keyinfo->ext_key_parts= keyinfo->user_defined_key_parts; keyinfo->usable_key_parts= keyinfo->user_defined_key_parts; table->distinct= 1; @@ -22846,23 +23011,31 @@ bool Create_tmp_table::finalize(THD *thd, blobs can distinguish NULL from 0. This extra field is not needed when we do not use UNIQUE indexes for blobs. */ - if (null_pack_length[distinct] && (keyinfo->flags & HA_UNIQUE_HASH)) + if (need_null_key_part) { m_key_part_info->null_bit=0; m_key_part_info->offset= null_pack_base[distinct]; m_key_part_info->length= null_pack_length[distinct]; + /* + Use empty_clex_str (not null_clex_str) for the field name: + HEAP keeps share->keys visible to EXPLAIN, which iterates + key parts and calls strlen() on the name. A NULL name from + null_clex_str causes SIGSEGV in Explain_index_use::set(). + Empty string keeps the implicit field invisible in output. + */ m_key_part_info->field= new Field_string(table->record[0], (uint32) m_key_part_info->length, (uchar*) 0, (uint) 0, Field::NONE, - &null_clex_str, &my_charset_bin); + &empty_clex_str, &my_charset_bin); if (!m_key_part_info->field) goto err; m_key_part_info->field->init(table); m_key_part_info->key_type=FIELDFLAG_BINARY; m_key_part_info->type= HA_KEYTYPE_BINARY; m_key_part_info->fieldnr= m_key_part_info->field->field_index + 1; + m_key_part_info->key_part_flag= m_key_part_info->field->key_part_flag(); m_key_part_info++; } /* Create a distinct key over the columns we are going to return */ @@ -22870,19 +23043,21 @@ bool Create_tmp_table::finalize(THD *thd, i < share->fields; i++, reg_field++) { - if (!((*reg_field)->flags & FIELD_PART_OF_TMP_UNIQUE)) + Field *field= *reg_field; + if (!(field->flags & FIELD_PART_OF_TMP_UNIQUE)) continue; - m_key_part_info->field= *reg_field; - (*reg_field)->flags |= PART_KEY_FLAG; + m_key_part_info->field= field; + field->flags |= PART_KEY_FLAG; if (m_key_part_info == keyinfo->key_part) - (*reg_field)->key_start.set_bit(0); - m_key_part_info->null_bit= (*reg_field)->null_bit; - m_key_part_info->null_offset= (uint) ((*reg_field)->null_ptr - + field->key_start.set_bit(0); + m_key_part_info->null_bit= field->null_bit; + m_key_part_info->null_offset= (uint) (field->null_ptr - (uchar*) table->record[0]); - m_key_part_info->offset= (*reg_field)->offset(table->record[0]); - m_key_part_info->length= (uint16) (*reg_field)->pack_length(); - m_key_part_info->fieldnr= (*reg_field)->field_index + 1; + m_key_part_info->offset= field->offset(table->record[0]); + m_key_part_info->length= (uint16) field->key_pack_length(); + m_key_part_info->fieldnr= field->field_index + 1; + m_key_part_info->key_part_flag= field->key_part_flag(); /* TODO: The below method of computing the key format length of the key part is a copy/paste from opt_range.cc, and table.cc. @@ -22893,21 +23068,41 @@ bool Create_tmp_table::finalize(THD *thd, */ m_key_part_info->store_length= m_key_part_info->length; - if ((*reg_field)->real_maybe_null()) + if (field->real_maybe_null()) { m_key_part_info->store_length+= HA_KEY_NULL_LENGTH; m_key_part_info->key_part_flag |= HA_NULL_PART; } - m_key_part_info->key_part_flag|= (*reg_field)->key_part_flag(); - m_key_part_info->store_length+= (*reg_field)->key_part_length_bytes(); + + m_key_part_info->store_length+= field->key_part_length_bytes(); keyinfo->key_length+= m_key_part_info->store_length; - m_key_part_info->type= (uint8) (*reg_field)->key_type(); - m_key_part_info->key_type = - ((ha_base_keytype) m_key_part_info->type == HA_KEYTYPE_TEXT || - (ha_base_keytype) m_key_part_info->type == HA_KEYTYPE_VARTEXT1 || - (ha_base_keytype) m_key_part_info->type == HA_KEYTYPE_VARTEXT2) ? - 0 : FIELDFLAG_BINARY; + m_key_part_info->type= (uint8) field->key_type(); + m_key_part_info->key_type= field->binary() ? FIELDFLAG_BINARY : 0; + + if (field->flags & BLOB_FLAG) + { + /* + Tell engine to that this key includes a blob + Heap engine will store blob key as length + pointer + This is safe as for unique detection we will never + try to read the data through a key. + */ + keyinfo->flags|= HA_BLOB_PART_KEY; + m_key_part_info->length= 4 + portable_sizeof_char_ptr; + m_key_part_info->store_length= m_key_part_info->length; + m_key_part_info->type= (field->binary() ? + HA_KEYTYPE_VARBINARY4 : + HA_KEYTYPE_VARTEXT4); + } + + DBUG_ASSERT(!(m_key_part_info->key_part_flag & HA_BLOB_PART) || + m_key_part_info->length == 4 + portable_sizeof_char_ptr); + DBUG_ASSERT(!(m_key_part_info->key_part_flag & HA_BLOB_PART) || + ((ha_base_keytype) m_key_part_info->type == + HA_KEYTYPE_VARBINARY4 || + (ha_base_keytype) m_key_part_info->type == + HA_KEYTYPE_VARTEXT4)); m_key_part_info++; } @@ -23468,7 +23663,7 @@ bool create_internal_tmp_table(TABLE *table, KEY *org_keyinfo, */ if (keyinfo->key_length > table->file->max_key_length() || keyinfo->user_defined_key_parts > table->file->max_key_parts() || - (keyinfo->flags & HA_UNIQUE_HASH)) + (keyinfo->flags & (HA_UNIQUE_HASH | HA_BLOB_PART_KEY))) { if (!(keyinfo->flags & (HA_NOSAME | HA_UNIQUE_HASH))) { @@ -23717,7 +23912,7 @@ bool create_internal_tmp_table(TABLE *table, KEY *org_keyinfo, seg->type= ((keyinfo->key_part[i].key_type & FIELDFLAG_BINARY) ? HA_KEYTYPE_VARBINARY2 : HA_KEYTYPE_VARTEXT2); - seg->bit_start= (uint8)(field->pack_length() - portable_sizeof_char_ptr); + seg->bit_start= (uint8) ((Field_blob*) field)->pack_length_no_ptr(); seg->flag= HA_BLOB_PART; seg->length=0; // Whole blob in unique constraint } @@ -23954,7 +24149,8 @@ free_tmp_table(THD *thd, TABLE *entry) as it will mark the transaction read/write */ DBUG_ASSERT(entry->s->tmp_table == SYSTEM_TMP_TABLE || - entry->s->tmp_table == INTERNAL_TMP_TABLE); + entry->s->tmp_table == INTERNAL_TMP_TABLE || + entry->s->tmp_table == RESULT_TMP_TABLE); entry->file->drop_table(entry->s->path.str); delete entry->file; entry->file= NULL; @@ -23964,6 +24160,19 @@ free_tmp_table(THD *thd, TABLE *entry) /* free blobs */ for (Field **ptr=entry->field ; *ptr ; ptr++) (*ptr)->free(); + /* + Free GROUP BY key fields. group->field is created by new_key_field() + and points into the group buffer, not into entry->field[], so the + loop above doesn't cover them. + TABLE::group is a union with TABLE::context (for hlindexes) but is + NULL-initialized by bzero in create_tmp_table and only set non-NULL + by finalize() for GROUP BY tables. Callers that pass a non-NULL + group to create_tmp_table must heap-allocate the ORDER (e.g. via + alloc_root) so table->group remains valid until free_tmp_table. + */ + for (ORDER *group= entry->group ; group ; group= group->next) + if (group->field) + group->field->free(); if (entry->temp_pool_slot != MY_BIT_NONE) temp_pool_clear_bit(entry->temp_pool_slot); @@ -26410,6 +26619,60 @@ end_write(JOIN *join, JOIN_TAB *join_tab __attribute__((unused)), @seealso end_unique_update() */ +/* + Convert a HEAP temp table to Aria after ha_update_tmp_row() fails with + HA_ERR_RECORD_FILE_FULL, then re-locate the duplicate row and retry the + update. + + After conversion the HEAP blocks are freed, so any blob pointers in + record[0] that referenced HEAP memory become dangling. We re-derive + record[0] by reading the pre-update row from Aria (ha_rnd_pos into + record[1], restore_record, update_tmptable_sum_func) before retrying. + update_field() is stateless (reads result_field + args[0], writes + result_field) so re-running it on the pre-update values is safe and + produces the correct post-update result. + + @return 0 success (table converted, update applied, index re-opened) + @return -1 error (already printed) +*/ + +static int +convert_heap_to_aria_update(JOIN *join, JOIN_TAB *join_tab, + TABLE *table, int error) +{ + bool is_duplicate; + if (create_internal_tmp_table_from_heap(join->thd, table, + join_tab->tmp_table_param->start_recinfo, + &join_tab->tmp_table_param->recinfo, + error, 1, &is_duplicate)) + return -1; + DBUG_ASSERT(is_duplicate); + table->file->get_dup_key(HA_ERR_FOUND_DUPP_KEY); + if (unlikely((error= table->file->ha_rnd_init(0)))) + { + table->file->print_error(error, MYF(0)); + return -1; + } + error= table->file->ha_rnd_pos(table->record[1], + table->file->dup_ref); + if (likely(!error)) + { + restore_record(table, record[1]); + update_tmptable_sum_func(join->sum_funcs, table); + error= table->file->ha_update_tmp_row(table->record[1], + table->record[0]); + } + if (unlikely(error) || + unlikely((error= table->file->ha_rnd_end())) || + unlikely((error= table->file->ha_index_init(0, 0)))) + { + table->file->print_error(error, MYF(0)); + return -1; + } + return 0; +} + + static enum_nested_loop_state end_update(JOIN *join, JOIN_TAB *join_tab __attribute__((unused)), bool end_of_records) @@ -26452,8 +26715,9 @@ end_update(JOIN *join, JOIN_TAB *join_tab __attribute__((unused)), if (unlikely((error= table->file->ha_update_tmp_row(table->record[1], table->record[0])))) { - table->file->print_error(error,MYF(0)); /* purecov: inspected */ - DBUG_RETURN(NESTED_LOOP_ERROR); /* purecov: inspected */ + if (convert_heap_to_aria_update(join, join_tab, table, error)) + DBUG_RETURN(NESTED_LOOP_ERROR); + join_tab->aggr->set_write_func(end_unique_update); } goto end; } @@ -26505,7 +26769,7 @@ end_unique_update(JOIN *join, JOIN_TAB *join_tab __attribute__((unused)), DBUG_RETURN(NESTED_LOOP_OK); init_tmptable_sum_functions(join->sum_funcs); - copy_fields(join_tab->tmp_table_param); // Groups are copied twice. + copy_fields(join_tab->tmp_table_param); // Groups are copied twice. if (copy_funcs(join_tab->tmp_table_param->items_to_copy, join->thd)) DBUG_RETURN(NESTED_LOOP_ERROR); /* purecov: inspected */ @@ -26538,8 +26802,9 @@ end_unique_update(JOIN *join, JOIN_TAB *join_tab __attribute__((unused)), if (unlikely((error= table->file->ha_update_tmp_row(table->record[1], table->record[0])))) { - table->file->print_error(error,MYF(0)); /* purecov: inspected */ - DBUG_RETURN(NESTED_LOOP_ERROR); /* purecov: inspected */ + if (convert_heap_to_aria_update(join, join_tab, table, error)) + DBUG_RETURN(NESTED_LOOP_ERROR); + rnd_inited= true; } if (!rnd_inited && ((error= table->file->ha_rnd_end()) || @@ -28443,8 +28708,15 @@ JOIN_TAB::remove_duplicates() table->file->info(HA_STATUS_VARIABLE); table->reginfo.lock_type=TL_WRITE; - if (table->s->db_type() == heap_hton || - (!table->s->blob_fields && + /* + remove_dup_with_hash_index() copies field data into a flat key buffer + via make_sort_key_part() and compares with memcmp. For LONGBLOB + fields, sort_length() returns UINT_MAX32, making the key buffer + impractically large. Fall back to the row-by-row compare path + for tables with blobs. + */ + if (!table->s->blob_fields && + (table->s->db_type() == heap_hton || ((ALIGN_SIZE(keylength) + HASH_OVERHEAD) * table->file->stats.records < thd->variables.sortbuff_size))) error= remove_dup_with_hash_index(join->thd, table, field_count, @@ -29463,7 +29735,9 @@ void calc_group_buffer(TMP_TABLE_PARAM *param, ORDER *group) if (field) { enum_field_types type; - if ((type= field->type()) == MYSQL_TYPE_BLOB) + DBUG_ASSERT((bool) (field->flags & BLOB_FLAG) == + is_any_blob_field_type(field->type())); + if (is_any_blob_field_type(type= field->type())) key_length+=MAX_BLOB_WIDTH; // Can't be used as a key else if (type == MYSQL_TYPE_VARCHAR || type == MYSQL_TYPE_VAR_STRING) key_length+= field->field_length + HA_KEY_BLOB_LENGTH; @@ -29473,7 +29747,7 @@ void calc_group_buffer(TMP_TABLE_PARAM *param, ORDER *group) key_length+= 8; // Big enough } else - key_length+= field->pack_length(); + key_length+= field->key_pack_length(); } else { @@ -29502,12 +29776,12 @@ void calc_group_buffer(TMP_TABLE_PARAM *param, ORDER *group) case STRING_RESULT: { enum enum_field_types type= group_item->field_type(); - if (type == MYSQL_TYPE_BLOB) + if (is_any_blob_field_type(type)) key_length+= MAX_BLOB_WIDTH; // Can't be used as a key else { /* - Group strings are taken as varstrings and require an length field. + Group strings are taken as varstrings and require a length field. A field is not yet created by create_tmp_field_ex() and the sizes should match up. */ @@ -29760,7 +30034,7 @@ setup_copy_fields(THD *thd, TMP_TABLE_PARAM *param, */ field= item->field; item->result_field=field->make_new_field(thd->mem_root, - field->table, 1); + field->table, 1, 0); /* We need to allocate one extra byte for null handling and another extra byte to not get warnings from purify in @@ -31103,7 +31377,7 @@ bool JOIN_TAB::save_explain_data(Explain_table_access *eta, be able to print the engine statistics. */ if (table->file->handler_stats && - table->s->tmp_table != INTERNAL_TMP_TABLE) + table->s->tmp_table != RESULT_TMP_TABLE) eta->handler_for_stats= table->file; if (likely(thd->lex->analyze_stmt)) diff --git a/sql/sql_show.cc b/sql/sql_show.cc index 882db8f3c1353..8283d45fc4bb7 100644 --- a/sql/sql_show.cc +++ b/sql/sql_show.cc @@ -9700,6 +9700,30 @@ static bool optimize_for_get_all_tables(THD *thd, TABLE_LIST *tables, COND *cond } +/* + Remove field form blob list if we replaced it with a varchar field +*/ + +static void remove_field_from_blob_list(TABLE *table, uint fieldnr) +{ + uint *blob_field= table->s->blob_field; + uint *end= blob_field+ table->s->blob_fields; + + for (; blob_field < end ; blob_field++) + { + if (*blob_field == fieldnr) + { + if (blob_field+1 < end) + bmove(blob_field, blob_field+1, + (char*) end - (char*) blob_field - sizeof(*blob_field)); + table->s->blob_fields--; + return; + } + } + DBUG_ASSERT(0); // Field not found +} + + bool optimize_schema_tables_memory_usage(List &tables) { DBUG_ENTER("optimize_schema_tables_memory_usage"); @@ -9741,6 +9765,7 @@ bool optimize_schema_tables_memory_usage(List &tables) } else { + bool was_blob= field->flags & BLOB_FLAG; field= new (thd->mem_root) Field_string(cur, 0, field->null_ptr, field->null_bit, Field::NONE, &field->field_name, field->dtcollation()); @@ -9748,6 +9773,8 @@ bool optimize_schema_tables_memory_usage(List &tables) field->field_index= i; DBUG_ASSERT(field->pack_length_in_rec() == 0); table->field[i]= field; + if (was_blob) + remove_field_from_blob_list(table, i); } } if ((table->s->reclength= (ulong)(cur - table->record[0])) == 0) diff --git a/sql/sql_table.cc b/sql/sql_table.cc index 4d2c1d6861647..f81987edf331b 100644 --- a/sql/sql_table.cc +++ b/sql/sql_table.cc @@ -4255,7 +4255,7 @@ bool Column_definition::prepare_blob_field(THD *thd) real_field_type() == FIELD_TYPE_MEDIUM_BLOB) { /* The user has given a length to the blob column */ - set_handler(Type_handler::blob_type_handler((uint) length)); + set_handler(Type_handler::blob_type_handler((uint) length, 0)); pack_length= type_handler()->calc_pack_length(0); } length= 0; diff --git a/sql/sql_trigger.cc b/sql/sql_trigger.cc index c580d74803278..8a2cb874d6b07 100644 --- a/sql/sql_trigger.cc +++ b/sql/sql_trigger.cc @@ -1580,7 +1580,7 @@ bool Table_triggers_list::prepare_record_accessors(TABLE *table) { Field *f; if (!(f= *trg_fld= (*fld)->make_new_field(&table->mem_root, table, - table == (*fld)->table))) + table == (*fld)->table,0))) return 1; f->flags= (*fld)->flags; @@ -1621,7 +1621,7 @@ bool Table_triggers_list::prepare_record_accessors(TABLE *table) for (fld= table->field, trg_fld= record1_field; *fld; fld++, trg_fld++) { if (!(*trg_fld= (*fld)->make_new_field(&table->mem_root, table, - table == (*fld)->table))) + table == (*fld)->table, 0))) return 1; (*trg_fld)->move_field_offset((my_ptrdiff_t)(table->record[1] - table->record[0])); diff --git a/sql/sql_type.cc b/sql/sql_type.cc index ee17b81500417..81f2dc7368234 100644 --- a/sql/sql_type.cc +++ b/sql/sql_type.cc @@ -168,6 +168,7 @@ Named_type_handler type_handler_varchar_compres Named_type_handler type_handler_tiny_blob("tinyblob"); Named_type_handler type_handler_medium_blob("mediumblob"); Named_type_handler type_handler_long_blob("longblob"); +Named_type_handler type_handler_blob_key("blob_key"); Named_type_handler type_handler_blob("blob"); Named_type_handler type_handler_blob_compressed("blob"); @@ -1408,19 +1409,23 @@ Type_handler::string_type_handler(uint max_octet_length) const Type_handler * -Type_handler::varstring_type_handler(const Item *item) +Type_handler::varstring_type_handler(const Item *item, + const Tmp_field_param *param) { if (!item->max_length) return &type_handler_string; if (item->too_big_for_varchar()) - return blob_type_handler(item->max_length); + return blob_type_handler(item->max_length, param); return &type_handler_varchar; } const Type_handler * -Type_handler::blob_type_handler(uint max_octet_length) +Type_handler::blob_type_handler(uint max_octet_length, + const Tmp_field_param *param) { + if (param && param->part_of_unique_key()) + return &type_handler_blob_key; if (max_octet_length <= 255) return &type_handler_tiny_blob; if (max_octet_length <= 65535) @@ -1432,9 +1437,15 @@ Type_handler::blob_type_handler(uint max_octet_length) const Type_handler * -Type_handler::blob_type_handler(const Item *item) +Type_handler::blob_type_handler(const Item *item, const Tmp_field_param *param) { - return blob_type_handler(item->max_length); + return blob_type_handler(item->max_length, param); +} + +const Type_handler * +Type_handler::blob_key_type_handler() +{ + return &type_handler_blob_key; } /** @@ -3725,6 +3736,20 @@ Field *Type_handler::make_and_init_table_field(MEM_ROOT *root, } +Field *Type_handler::make_and_init_table_field_ex(MEM_ROOT *root, + const LEX_CSTRING *name, + const Record_addr &addr, + const Type_all_attributes &attr, + const Tmp_field_param *param, + TABLE *table) const +{ + Field *field= make_table_field_ex(root, name, addr, attr, param, table->s); + if (field) + field->init(table); + return field; +} + + Field *Type_handler_int_result::make_table_field(MEM_ROOT *root, const LEX_CSTRING *name, const Record_addr &addr, @@ -4071,6 +4096,19 @@ Field *Type_handler_long_blob::make_table_field(MEM_ROOT *root, 4, attr.collation); } +Field *Type_handler_blob_key::make_table_field(MEM_ROOT *root, + const LEX_CSTRING *name, + const Record_addr &addr, + const Type_all_attributes &attr, + TABLE_SHARE *share) const + +{ + /* Note that the pack length (4) will be fixed by the caller */ + return new (root) + Field_blob_key(addr.ptr(), addr.null_ptr(), addr.null_bit(), + Field::NONE, name, share, 4, attr.collation); +} + Field *Type_handler_enum::make_table_field(MEM_ROOT *root, const LEX_CSTRING *name, @@ -4111,6 +4149,33 @@ Field *Type_handler_set::make_table_field(MEM_ROOT *root, } +/***************************************************************************/ + + +Field *Type_handler_blob_common::make_table_field_ex(MEM_ROOT *root, + const LEX_CSTRING *name, + const Record_addr &addr, + const Type_all_attributes &attr, + const Tmp_field_param *param, + TABLE_SHARE *share) const +{ + if (param->part_of_unique_key()) + { + Field *result= type_handler_blob_key. + make_table_field(root, name, addr, attr, share); + if (result) + { + DBUG_ASSERT(dynamic_cast(result)); + /* Fix length of blob to be able to return the original blob type */ + static_cast(result)->set_pack_length(length_bytes()); + } + return result; + } + + return make_table_field(root, name, addr, attr, share); +} + + /*************************************************************************/ Field *Type_handler_float::make_schema_field(MEM_ROOT *root, TABLE *table, @@ -4904,7 +4969,7 @@ bool Type_handler_blob_common:: { if (func->aggregate_attributes_string(func_name, items, nitems)) return true; - handler->set_handler(blob_type_handler(func->max_length)); + handler->set_handler(blob_type_handler(func->max_length, 0)); return false; } @@ -7481,32 +7546,31 @@ bool Type_handler_temporal_result:: const Type_handler * -Type_handler_null::type_handler_for_tmp_table(const Item *item) const -{ - return &type_handler_string; -} - - -const Type_handler * -Type_handler_null::type_handler_for_union(const Item *item) const +Type_handler_null::type_handler_for_tmp_table(const Item *item, + const Tmp_field_param *param) + const { return &type_handler_string; } -const Type_handler * -Type_handler_olddecimal::type_handler_for_tmp_table(const Item *item) const +const Type_handler *Type_handler_olddecimal:: +type_handler_for_tmp_table(const Item *item, + const Tmp_field_param *param) const { return &type_handler_newdecimal; } -const Type_handler * -Type_handler_olddecimal::type_handler_for_union(const Item *item) const + +const Type_handler *Type_handler_blob_common:: +type_handler_for_tmp_table(const Item *item, const Tmp_field_param *param) + const { - return &type_handler_newdecimal; + return (param->part_of_unique_key() ? + blob_key_type_handler() : + blob_type_handler(item, 0)); } - /***************************************************************************/ void Type_handler::set_null_if_needed(const Item *item, st_value *value) const diff --git a/sql/sql_type.h b/sql/sql_type.h index 15b0d22958b93..9d666312ac078 100644 --- a/sql/sql_type.h +++ b/sql/sql_type.h @@ -105,6 +105,7 @@ class sp_type_def; class sp_head; class sp_instr; class my_var; +class Tmp_field_param; #define my_charset_numeric my_charset_latin1 @@ -3983,7 +3984,8 @@ class Type_handler THD *thd, const Log_event_data_type &type); static const Type_handler *odbc_literal_type_handler(const LEX_CSTRING *str); - static const Type_handler *blob_type_handler(uint max_octet_length); + static const Type_handler *blob_type_handler(uint max_octet_length, + const Tmp_field_param *param); static const Type_handler *string_type_handler(uint max_octet_length); static const Type_handler *bit_and_int_mixture_handler(uint max_char_len); static const Type_handler *type_handler_long_or_longlong(uint max_char_len, @@ -3995,8 +3997,11 @@ class Type_handler If max_length == 0 create a CHAR(0) @param item - the Item to get the handler to. */ - static const Type_handler *varstring_type_handler(const Item *item); - static const Type_handler *blob_type_handler(const Item *item); + static const Type_handler + *varstring_type_handler(const Item *item, const Tmp_field_param *param); + static const Type_handler *blob_type_handler(const Item *item, + const Tmp_field_param *param); + static const Type_handler *blob_key_type_handler(); static const Type_handler *get_handler_by_field_type(enum_field_types type); static const Type_handler *get_handler_by_real_type(enum_field_types type); static const Type_collection * @@ -4174,11 +4179,9 @@ class Type_handler { return this; } - virtual const Type_handler *type_handler_for_tmp_table(const Item *) const - { - return this; - } - virtual const Type_handler *type_handler_for_union(const Item *) const + virtual const Type_handler + *type_handler_for_tmp_table(const Item *, + const Tmp_field_param *param) const { return this; } @@ -4456,11 +4459,26 @@ class Type_handler const Record_addr &addr, const Type_all_attributes &attr, TABLE_SHARE *share) const= 0; + virtual Field *make_table_field_ex(MEM_ROOT *root, + const LEX_CSTRING *name, + const Record_addr &addr, + const Type_all_attributes &attr, + const Tmp_field_param *param, + TABLE_SHARE *share) const + { + return make_table_field(root, name, addr, attr, share); + } Field *make_and_init_table_field(MEM_ROOT *root, const LEX_CSTRING *name, const Record_addr &addr, const Type_all_attributes &attr, TABLE *table) const; + Field *make_and_init_table_field_ex(MEM_ROOT *root, + const LEX_CSTRING *name, + const Record_addr &addr, + const Type_all_attributes &attr, + const Tmp_field_param *param, + TABLE *table) const; virtual Field *make_schema_field(MEM_ROOT *root, TABLE *table, const Record_addr &addr, @@ -7177,8 +7195,9 @@ class Type_handler_olddecimal: public Type_handler_decimal_result enum_field_types field_type() const override { return MYSQL_TYPE_DECIMAL; } uint32 max_display_length_for_field(const Conv_source &src) const override; uint32 calc_pack_length(uint32 length) const override { return length; } - const Type_handler *type_handler_for_tmp_table(const Item *item) const override; - const Type_handler *type_handler_for_union(const Item *item) const override; + const Type_handler + *type_handler_for_tmp_table(const Item *item, + const Tmp_field_param *param) const override; void show_binlog_type(const Conv_source &src, const Field &, String *str) const override; Field *make_conversion_table_field(MEM_ROOT *root, @@ -7256,8 +7275,9 @@ class Type_handler_null: public Type_handler_general_purpose_string return DYN_COL_NULL; } const Type_handler *type_handler_for_comparison() const override; - const Type_handler *type_handler_for_tmp_table(const Item *item) const override; - const Type_handler *type_handler_for_union(const Item *) const override; + const Type_handler *type_handler_for_tmp_table(const Item *item, + const Tmp_field_param *param) + const override; uint32 max_display_length(const Item *item) const override { return 0; } uint32 max_display_length_for_field(const Conv_source &src) const override { @@ -7364,10 +7384,11 @@ class Type_handler_string: public Type_handler_longstr bool is_param_long_data_type() const override { return true; } uint32 max_display_length_for_field(const Conv_source &src) const override; uint32 calc_pack_length(uint32 length) const override { return length; } - const Type_handler *type_handler_for_tmp_table(const Item *item) const - override + const Type_handler *type_handler_for_tmp_table(const Item *item, + const Tmp_field_param *param) + const override { - return varstring_type_handler(item); + return varstring_type_handler(item, param); } bool partition_field_check(const LEX_CSTRING &, Item *item_expr) const override @@ -7416,9 +7437,11 @@ class Type_handler_var_string: public Type_handler_string return MYSQL_TYPE_VARCHAR; } const Type_handler *type_handler_for_implicit_upgrade() const override; - const Type_handler *type_handler_for_tmp_table(const Item *item) const override + const Type_handler *type_handler_for_tmp_table(const Item *item, + const Tmp_field_param *param) + const override { - return varstring_type_handler(item); + return varstring_type_handler(item, param); } uint32 max_display_length_for_field(const Conv_source &src) const override; void show_binlog_type(const Conv_source &src, const Field &dst, String *str) @@ -7427,10 +7450,6 @@ class Type_handler_var_string: public Type_handler_string handler *file, ulonglong table_flags) const override { return Column_definition_prepare_stage2_legacy_num(c, MYSQL_TYPE_STRING); } - const Type_handler *type_handler_for_union(const Item *item) const override - { - return varstring_type_handler(item); - } }; @@ -7454,14 +7473,11 @@ class Type_handler_varchar: public Type_handler_longstr { return (length + (length < 256 ? 1: 2)); } - const Type_handler *type_handler_for_tmp_table(const Item *item) const - override - { - return varstring_type_handler(item); - } - const Type_handler *type_handler_for_union(const Item *item) const override + const Type_handler *type_handler_for_tmp_table(const Item *item, + const Tmp_field_param *param) + const override { - return varstring_type_handler(item); + return varstring_type_handler(item, param); } bool is_param_long_data_type() const override { return true; } bool partition_field_check(const LEX_CSTRING &, Item *item_expr) @@ -7562,15 +7578,9 @@ class Type_handler_blob_common: public Type_handler_longstr Field *make_conversion_table_field(MEM_ROOT *root, TABLE *table, uint metadata, const Field *target) const override; - const Type_handler *type_handler_for_tmp_table(const Item *item) const - override - { - return blob_type_handler(item); - } - const Type_handler *type_handler_for_union(const Item *item) const override - { - return blob_type_handler(item); - } + const Type_handler *type_handler_for_tmp_table(const Item *item, + const Tmp_field_param *param) + const override; bool subquery_type_allows_materialization(const Item *, const Item *, bool) const override { @@ -7625,6 +7635,12 @@ class Type_handler_blob_common: public Type_handler_longstr const Bit_addr &bit, const Column_definition_attributes *attr, uint32 flags) const override; + Field *make_table_field_ex(MEM_ROOT *root, + const LEX_CSTRING *name, + const Record_addr &addr, + const Type_all_attributes &attr, + const Tmp_field_param *param, + TABLE_SHARE *share) const override; const Vers_type_handler *vers() const override; }; @@ -7683,6 +7699,18 @@ class Type_handler_long_blob: public Type_handler_blob_common }; +class Type_handler_blob_key: public Type_handler_long_blob +{ +public: + virtual ~Type_handler_blob_key() = default; + Field *make_table_field(MEM_ROOT *root, + const LEX_CSTRING *name, + const Record_addr &addr, + const Type_all_attributes &attr, + TABLE_SHARE *share) const override; +}; + + class Type_handler_blob: public Type_handler_blob_common { public: @@ -8038,6 +8066,7 @@ extern Named_type_handler type_handler_hex_hybrid; extern Named_type_handler type_handler_tiny_blob; extern Named_type_handler type_handler_medium_blob; extern MYSQL_PLUGIN_IMPORT Named_type_handler type_handler_long_blob; +extern MYSQL_PLUGIN_IMPORT Named_type_handler type_handler_blob_key; extern Named_type_handler type_handler_blob; extern Named_type_handler type_handler_blob_compressed; diff --git a/sql/sql_type_fixedbin.h b/sql/sql_type_fixedbin.h index 66510e1f2816a..30be782eb4635 100644 --- a/sql/sql_type_fixedbin.h +++ b/sql/sql_type_fixedbin.h @@ -713,7 +713,6 @@ class Type_handler_fbt: public Type_handler return Data_type_compatibility::OK; } - uint row_pack_length() const override { return pack_length(); } Binlog_type_info binlog_type_info() const override { @@ -896,7 +895,7 @@ class Type_handler_fbt: public Type_handler const override { if (item->max_length > MAX_FIELD_VARCHARLENGTH) - return Type_handler::blob_type_handler(item->max_length); + return Type_handler::blob_type_handler(item->max_length, 0); if (item->max_length > 255) return &type_handler_varchar; return &type_handler_string; diff --git a/sql/sql_type_geom.cc b/sql/sql_type_geom.cc index be05614087082..d865a1fc2d0a8 100644 --- a/sql/sql_type_geom.cc +++ b/sql/sql_type_geom.cc @@ -256,6 +256,24 @@ const Type_handler *Type_handler_geometry::type_handler_for_comparison() const } +const Type_handler* +Type_handler_geometry::type_handler_for_tmp_table(const Item *, + const Tmp_field_param *param) + const +{ + if (param->group_concat()) + { + /* + Field_geom::store() asserts !table->blob_storage, but GROUP_CONCAT + with ORDER BY/DISTINCT sets blob_storage on its temp table. + Downgrade to plain Field_blob. + */ + return &type_handler_long_blob; + } + return this; +} + + Field *Type_handler_geometry::make_conversion_table_field(MEM_ROOT *root, TABLE *table, uint metadata, @@ -515,6 +533,33 @@ Field *Type_handler_geometry::make_table_field(MEM_ROOT *root, } +Field *Type_handler_geometry::make_table_field_ex(MEM_ROOT *root, + const LEX_CSTRING *name, + const Record_addr &addr, + const Type_all_attributes &attr, + const Tmp_field_param *param, + TABLE_SHARE *share) const +{ + if (param->part_of_unique_key()) + { + /* Unique keys parts must be of type blob_key for memory tables */ + return type_handler_blob_key. + make_table_field(root, name, addr, attr, share); + } + if (param->group_concat()) + { + /* + Field_geom::store() asserts !table->blob_storage, but GROUP_CONCAT + with ORDER BY/DISTINCT sets blob_storage on its temp table. + Downgrade to plain Field_blob. + */ + return type_handler_long_blob. + make_table_field(root, name, addr, attr, share); + } + return make_table_field(root, name, addr, attr, share); +} + + bool Type_handler_geometry:: Item_hybrid_func_fix_attributes(THD *thd, const LEX_CSTRING &op_name, @@ -886,6 +931,7 @@ int Field_geom::store_decimal(const my_decimal *) int Field_geom::store(const char *from, size_t length, CHARSET_INFO *cs) { + DBUG_ASSERT(!table->blob_storage); const char *dummy; Geometry_buffer buffer; Geometry *geom; @@ -1004,6 +1050,6 @@ uint Field_geom::get_key_image(uchar *buff,uint length, const uchar *ptr_arg, Binlog_type_info Field_geom::binlog_type_info() const { DBUG_ASSERT(Field_geom::type() == binlog_type()); - return Binlog_type_info(Field_geom::type(), pack_length_no_ptr(), 1, + return Binlog_type_info(Field_geom::type(), length_size(), 1, field_charset(), type_handler_geom()->geometry_type()); } diff --git a/sql/sql_type_geom.h b/sql/sql_type_geom.h index dc374b6d46b0d..1336cd3dd76aa 100644 --- a/sql/sql_type_geom.h +++ b/sql/sql_type_geom.h @@ -84,6 +84,9 @@ class Type_handler_geometry: public Type_handler_string_result uint32 max_display_length_for_field(const Conv_source &src) const override; uint32 calc_pack_length(uint32 length) const override; const Type_collection *type_collection() const override; + const Type_handler *type_handler_for_tmp_table(const Item *, + const Tmp_field_param *param) + const override; const Type_handler *type_handler_for_comparison() const override; virtual geometry_types geometry_type() const { return GEOM_GEOMETRY; } virtual Item *create_typecast_item(THD *thd, Item *item, @@ -177,6 +180,13 @@ class Type_handler_geometry: public Type_handler_string_result const Type_all_attributes &attr, TABLE_SHARE *share) const override; + Field *make_table_field_ex(MEM_ROOT *root, + const LEX_CSTRING *name, + const Record_addr &addr, + const Type_all_attributes &attr, + const Tmp_field_param *param, + TABLE_SHARE *share) const override; + Field *make_table_field_from_def(TABLE_SHARE *share, MEM_ROOT *mem_root, const LEX_CSTRING *name, diff --git a/sql/sql_update.cc b/sql/sql_update.cc index c7911d71fa6df..0aeda5cc50043 100644 --- a/sql/sql_update.cc +++ b/sql/sql_update.cc @@ -2202,8 +2202,10 @@ multi_update::initialize_tables(JOIN *join) TABLE *table=table_ref->table; uint index= table_ref->shared; List temp_fields; - ORDER group; TMP_TABLE_PARAM *tmp_param; + ORDER *group= (ORDER*) alloc_root(thd->mem_root, sizeof(*group)); + if (!group) + DBUG_RETURN(1); table->file->prepare_for_modify(true, true); @@ -2296,19 +2298,25 @@ multi_update::initialize_tables(JOIN *join) temp_fields.append(fields_for_table[index]); /* Make an unique key over the first field to avoid duplicated updates */ - bzero((char*) &group, sizeof(group)); - group.direction= ORDER::ORDER_ASC; - group.item= (Item**) temp_fields.head_ref(); + bzero((char*) group, sizeof(*group)); + group->direction= ORDER::ORDER_ASC; + group->item= (Item**) temp_fields.head_ref(); tmp_param= &tmp_table_param[index]; tmp_param->init(); tmp_param->tmp_name="update"; tmp_param->field_count= temp_fields.elements; tmp_param->func_count= temp_fields.elements - 1; - calc_group_buffer(tmp_param, &group); + calc_group_buffer(tmp_param, group); + /* small table, ignore @@tmp_memory_table_size=0 */ + ulonglong save_tmp_memory_table_size= thd->variables.tmp_memory_table_size; + if (!save_tmp_memory_table_size) + thd->variables.tmp_memory_table_size= SIZE_T_MAX; tmp_tables[index]=create_tmp_table(thd, tmp_param, temp_fields, - (ORDER*) &group, 0, 0, - TMP_TABLE_ALL_COLUMNS, HA_POS_ERROR, &empty_clex_str); + (ORDER*) group, 0, 0, + TMP_TABLE_ALL_COLUMNS, HA_POS_ERROR, + &empty_clex_str); + thd->variables.tmp_memory_table_size= save_tmp_memory_table_size; if (!tmp_tables[index]) DBUG_RETURN(1); tmp_tables[index]->file->extra(HA_EXTRA_WRITE_CACHE); diff --git a/sql/table.cc b/sql/table.cc index fb25f4ddaf1bf..3676393244380 100644 --- a/sql/table.cc +++ b/sql/table.cc @@ -4328,7 +4328,7 @@ bool copy_keys_from_share(TABLE *outparam, MEM_ROOT *root) We are using only a prefix of the column as a key: Create a new field for the key part that matches the index */ - field= key_part->field=field->make_new_field(root, outparam, 0); + field= key_part->field=field->make_new_field(root, outparam, 0, 0); field->field_length= key_part->length; } } @@ -8575,7 +8575,8 @@ bool TABLE::alloc_keys(uint key_count) { KEY *new_key_info; key_part_map *new_const_key_parts; - DBUG_ASSERT(s->tmp_table == INTERNAL_TMP_TABLE); + DBUG_ASSERT(s->tmp_table == INTERNAL_TMP_TABLE || + s->tmp_table == RESULT_TMP_TABLE); if (!multi_alloc_root(&mem_root, &new_key_info, sizeof(*key_info)*(s->total_keys+key_count), diff --git a/sql/table.h b/sql/table.h index ba15cf45a1257..f70dd083dfaa2 100644 --- a/sql/table.h +++ b/sql/table.h @@ -392,8 +392,12 @@ typedef struct st_grant_info enum tmp_table_type { - NO_TMP_TABLE= 0, NON_TRANSACTIONAL_TMP_TABLE, TRANSACTIONAL_TMP_TABLE, - INTERNAL_TMP_TABLE, SYSTEM_TMP_TABLE + NO_TMP_TABLE= 0, // Normal table + NON_TRANSACTIONAL_TMP_TABLE, // CREATE TEMPORARY ... TRANSACTIONAL=0 + TRANSACTIONAL_TMP_TABLE, // CREATE TEMPORARY ... TRANSACTIONAL=1 + INTERNAL_TMP_TABLE, // Table created for different purposes + RESULT_TMP_TABLE, // Holds intermediate SELECT results + SYSTEM_TMP_TABLE // Created by mysql_schema_table }; enum release_type { RELEASE_NORMAL, RELEASE_WAIT_FOR_DROP }; @@ -1192,7 +1196,7 @@ struct TABLE_SHARE bool is_optimizer_tmp_table() { - return tmp_table == INTERNAL_TMP_TABLE && !db.length && table_name.length; + return tmp_table == RESULT_TMP_TABLE; } bool visit_subgraph(Wait_for_flush *waiting_ticket, @@ -1325,6 +1329,20 @@ class Blob_mem_storage: public Sql_alloc { return (char*) memdup_root(&storage, from, length); } + /* + Store string with a 0 prefix. This is used for storing + Field_blob_compressed fields in a not compressed format. + */ + char *store_with_zero_prefix(const char *from, size_t length) + { + char *res= (char*) alloc_root(&storage, length+1); + if (res) + { + res[0]= 0; + memcpy(res+1, from, length); + } + return res; + } void set_truncated_value(bool is_truncated_value) { truncated_value= is_truncated_value; diff --git a/storage/csv/ha_tina.cc b/storage/csv/ha_tina.cc index 221fbe3f6fd1f..1bc208672cb4f 100644 --- a/storage/csv/ha_tina.cc +++ b/storage/csv/ha_tina.cc @@ -858,7 +858,7 @@ int ha_tina::find_current_row(uchar *buf) uchar *src, *tgt; uint length, packlength; - packlength= blob->pack_length_no_ptr(); + packlength= blob->length_size(); length= blob->get_length(blob->ptr); memcpy(&src, blob->ptr + packlength, sizeof(char*)); if (src) diff --git a/storage/heap/CMakeLists.txt b/storage/heap/CMakeLists.txt index a26124d0c1cae..d46669d5db757 100644 --- a/storage/heap/CMakeLists.txt +++ b/storage/heap/CMakeLists.txt @@ -1,23 +1,24 @@ # Copyright (c) 2006, 2010, Oracle and/or its affiliates. All rights reserved. -# +# # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; version 2 of the License. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1335 USA -SET(HEAP_SOURCES _check.c _rectest.c hp_block.c hp_clear.c hp_close.c hp_create.c - ha_heap.cc - hp_delete.c hp_extra.c hp_hash.c hp_info.c hp_open.c hp_panic.c - hp_rename.c hp_rfirst.c hp_rkey.c hp_rlast.c hp_rnext.c hp_rprev.c - hp_rrnd.c hp_rsame.c hp_scan.c hp_static.c hp_update.c hp_write.c) +SET(HEAP_SOURCES _check.c _rectest.c hp_block.c hp_blob.c hp_clear.c hp_close.c + hp_create.c ha_heap.cc hp_delete.c hp_extra.c + hp_hash.c hp_info.c hp_open.c hp_panic.c hp_rename.c + hp_rfirst.c hp_rkey.c hp_rlast.c hp_rnext.c + hp_rprev.c hp_rrnd.c hp_rsame.c hp_scan.c hp_static.c + hp_update.c hp_write.c) MYSQL_ADD_PLUGIN(heap ${HEAP_SOURCES} STORAGE_ENGINE MANDATORY RECOMPILE_FOR_EMBEDDED) @@ -29,7 +30,16 @@ ENDIF() IF(WITH_UNIT_TESTS) ADD_EXECUTABLE(hp_test1 hp_test1.c) TARGET_LINK_LIBRARIES(hp_test1 heap mysys dbug strings) - ADD_EXECUTABLE(hp_test2 hp_test2.c) TARGET_LINK_LIBRARIES(hp_test2 heap mysys dbug strings) -ENDIF() \ No newline at end of file + MY_ADD_TESTS(hp_test_hash hp_test_scan hp_test_freelist hp_test_concurrent LINK_LIBRARIES heap mysys dbug strings) + + INCLUDE_DIRECTORIES(${CMAKE_SOURCE_DIR}/sql + ${CMAKE_SOURCE_DIR}/include) + + ADD_EXECUTABLE(hp_test_key_setup-t hp_test_key_setup-t.cc + ${CMAKE_SOURCE_DIR}/unittest/sql/dummy_builtins.cc) + TARGET_COMPILE_DEFINITIONS(hp_test_key_setup-t PRIVATE MYSQL_SERVER) + TARGET_LINK_LIBRARIES(hp_test_key_setup-t heap sql mytap) + MY_ADD_TEST(hp_test_key_setup) +ENDIF() diff --git a/storage/heap/_check.c b/storage/heap/_check.c index 1a640fa13da86..5761318a40418 100644 --- a/storage/heap/_check.c +++ b/storage/heap/_check.c @@ -18,7 +18,7 @@ #include "heapdef.h" -static int check_one_key(HP_KEYDEF *, uint, ulong, ulong, my_bool); +static int check_one_key(HP_INFO *, HP_KEYDEF *, uint, ulong, ulong, my_bool); static int check_one_rb_key(const HP_INFO *, uint, ulong, my_bool); @@ -31,7 +31,8 @@ static int check_one_rb_key(const HP_INFO *, uint, ulong, my_bool); print_status Prints some extra status NOTES - Doesn't change the state of the table handler + May allocate/reallocate the key_blob_buff scratch buffer in info + for blob key materialization; the logical table state is unchanged. RETURN VALUES 0 ok @@ -42,7 +43,9 @@ int heap_check_heap(const HP_INFO *info, my_bool print_status) { int error; uint key; - ulong records=0, deleted=0, pos, next_block; + ulong records=0, deleted=0, cont_count=0, pos, next_block; + ulong del_link_count; + uchar *del_ptr; HP_SHARE *share=info->s; uchar *current_ptr= info->current_ptr; DBUG_ENTER("heap_check_heap"); @@ -52,9 +55,32 @@ int heap_check_heap(const HP_INFO *info, my_bool print_status) if (share->keydef[key].algorithm == HA_KEY_ALG_BTREE) error|= check_one_rb_key(info, key, share->records, print_status); else - error|= check_one_key(share->keydef + key, key, share->records, - share->blength, print_status); + error|= check_one_key((HP_INFO*) info, share->keydef + key, key, + share->records, share->blength, print_status); } + + /* Verify scan-boundary invariant */ + if (share->total_records + share->deleted != share->block.last_allocated) + { + DBUG_PRINT("error",("last_allocated mismatch: total %lu + deleted %lu " + "!= last_allocated %lu", + (ulong) share->total_records, + (ulong) share->deleted, + (ulong) share->block.last_allocated)); + error= 1; + } + + /* Verify free list length matches share->deleted */ + del_link_count= 0; + for (del_ptr= share->del_link; del_ptr; del_ptr= *((uchar**) del_ptr)) + del_link_count++; + if (del_link_count != share->deleted) + { + DBUG_PRINT("error",("free list length %lu != share->deleted %lu", + del_link_count, (ulong) share->deleted)); + error= 1; + } + /* This is basicly the same code as in hp_scan, but we repeat it here to get shorter DBUG log file. @@ -68,9 +94,9 @@ int heap_check_heap(const HP_INFO *info, my_bool print_status) else { next_block+= share->block.records_in_block; - if (next_block >= share->records+share->deleted) + if (next_block >= share->total_records+share->deleted) { - next_block= share->records+share->deleted; + next_block= share->total_records+share->deleted; if (pos >= next_block) break; /* End of file */ } @@ -79,8 +105,57 @@ int heap_check_heap(const HP_INFO *info, my_bool print_status) if (!current_ptr[share->visible]) deleted++; + else if (hp_is_cont(current_ptr, share->visible)) + { + /* + Case A (HP_ROW_SINGLE_REC): single record, no header. + Case B/C: read run_rec_count from header and skip the entire run. + */ + if (hp_is_single_rec(current_ptr, share->visible)) + { + cont_count++; + } + else + { + uint16 run_rec_count= hp_cont_rec_count(current_ptr); + cont_count+= run_rec_count; + pos+= run_rec_count - 1; /* -1 because for-loop does pos++ */ + } + } else + { records++; + + /* Verify HP_ROW_HAS_CONT consistency for blob tables */ + if (share->blob_count && hp_has_cont(current_ptr, share->visible)) + { + HP_BLOB_DESC *desc, *desc_end; + my_bool has_any_chain= FALSE; + for (desc= share->blob_descs, + desc_end= desc + share->blob_count; + desc < desc_end; desc++) + { + if (hp_blob_length(desc, current_ptr) > 0) + { + uchar *chain; + memcpy(&chain, current_ptr + desc->offset + desc->packlength, + sizeof(chain)); + if (chain) + { + has_any_chain= TRUE; + break; + } + } + } + if (!has_any_chain) + { + DBUG_PRINT("error", + ("Primary record at pos %lu has HP_ROW_HAS_CONT " + "but no blob chain", pos)); + error= 1; + } + } + } } if (records != share->records || deleted != share->deleted) @@ -90,12 +165,19 @@ int heap_check_heap(const HP_INFO *info, my_bool print_status) deleted, (ulong) share->deleted)); error= 1; } + if (records + cont_count != share->total_records) + { + DBUG_PRINT("error",("total_records mismatch: primary %lu + cont %lu != %lu", + records, cont_count, + (ulong) share->total_records)); + error= 1; + } DBUG_RETURN(error); } -static int check_one_key(HP_KEYDEF *keydef, uint keynr, ulong records, - ulong blength, my_bool print_status) +static int check_one_key(HP_INFO *info, HP_KEYDEF *keydef, uint keynr, + ulong records, ulong blength, my_bool print_status) { int error; ulong i,found,max_links,seek,links; @@ -107,8 +189,12 @@ static int check_one_key(HP_KEYDEF *keydef, uint keynr, ulong records, hash_buckets_found= 0; for (i=found=max_links=seek=0 ; i < records ; i++) { + ulong recomputed; hash_info=hp_find_hash(&keydef->block,i); - if (hash_info->hash_of_key != hp_rec_hashnr(keydef, hash_info->ptr_to_rec)) + + recomputed= hp_rec_hashnr(info, keydef, hash_info->ptr_to_rec); + + if (hash_info->hash_of_key != recomputed) { DBUG_PRINT("error", ("Found row with wrong hash_of_key at position %lu", i)); @@ -151,13 +237,13 @@ static int check_one_key(HP_KEYDEF *keydef, uint keynr, ulong records, ("key: %u records: %ld seeks: %lu max links: %lu " "hitrate: %.2f buckets: %lu", keynr, records,seek,max_links, - (float) seek / (float) (records ? records : 1), + (float) seek / (float) (records ? records : 1), hash_buckets_found)); if (print_status) printf("Key: %u records: %ld seeks: %lu max links: %lu " "hitrate: %.2f buckets: %lu\n", keynr, records, seek, max_links, - (float) seek / (float) (records ? records : 1), + (float) seek / (float) (records ? records : 1), hash_buckets_found); return error; } @@ -173,7 +259,7 @@ static int check_one_rb_key(const HP_INFO *info, uint keynr, ulong records, uint not_used[2]; TREE_ELEMENT **last_pos; TREE_ELEMENT *parents[MAX_TREE_HEIGHT+1]; - + if ((key= tree_search_edge(&keydef->rb_tree, parents, &last_pos, offsetof(TREE_ELEMENT, left)))) { @@ -185,13 +271,13 @@ static int check_one_rb_key(const HP_INFO *info, uint keynr, ulong records, key_length, SEARCH_FIND | SEARCH_SAME, not_used)) { error= 1; - DBUG_PRINT("error",("Record in wrong link: key: %u Record: %p\n", + DBUG_PRINT("error",("Record in wrong link: key: %u Record: %p\n", keynr, recpos)); } else found++; key= tree_search_next(&keydef->rb_tree, &last_pos, - offsetof(TREE_ELEMENT, left), + offsetof(TREE_ELEMENT, left), offsetof(TREE_ELEMENT, right)); } while (key); } diff --git a/storage/heap/ha_heap.cc b/storage/heap/ha_heap.cc index 520d024002bc5..065b7e6b97bbb 100644 --- a/storage/heap/ha_heap.cc +++ b/storage/heap/ha_heap.cc @@ -21,9 +21,10 @@ #include "sql_plugin.h" #include "ha_heap.h" #include "sql_base.h" +#include "field.h" static handler *heap_create_handler(handlerton *, TABLE_SHARE *, MEM_ROOT *); -static int heap_prepare_hp_create_info(TABLE *, bool, HP_CREATE_INFO *); +int heap_prepare_hp_create_info(TABLE *, bool, HP_CREATE_INFO *); static int heap_panic(handlerton *hton, ha_panic_function flag) @@ -123,6 +124,7 @@ int ha_heap::open(const char *name, int mode, uint test_if_locked) rc= heap_create(name, &create_info, &internal_share, &created_new_share); my_free(create_info.keydef); + my_free(create_info.blob_descs); if (rc) goto end; @@ -420,6 +422,47 @@ void ha_heap::position(const uchar *record) *(HEAP_PTR*) ref= heap_position(file); // Ref is aligned } + +int ha_heap::remember_rnd_pos() +{ + saved_current_record= file->current_record; + position((uchar*) 0); // Store position in ref + return 0; +} + + +int ha_heap::restart_rnd_next(uchar *buf) +{ + /* + Restore the scan position saved by remember_rnd_pos(). + + heap_scan() uses current_record as a sequential counter and next_block + as a cached upper bound for the current HP_BLOCK segment. Within one + segment, heap_scan() advances current_ptr by recbuffer without calling + hp_find_record(). heap_rrnd() (called via rnd_pos) doesn't update + these, so we restore them here. + + next_block is set to the next records_in_block-aligned boundary after + saved_current_record. We MUST then cap it at total_records + deleted + (== block.last_allocated), which is the number of actually allocated + slots in the HP_BLOCK. Without this cap, if the saved position falls + in the last block segment and rows have been deleted between + remember_rnd_pos() and restart_rnd_next() (e.g. by + remove_dup_with_compare), next_block can exceed the allocated range. + heap_scan() would then take the fast path (pos < next_block) and walk + current_ptr past the last allocated slot into unmapped memory, causing + a segfault. + */ + file->current_record= saved_current_record; + file->next_block= saved_current_record - + (saved_current_record % file->s->block.records_in_block) + + file->s->block.records_in_block; + ulong scan_end= file->s->total_records + file->s->deleted; + if (file->next_block > scan_end) + file->next_block= scan_end; + return rnd_pos(buf, ref); +} + int ha_heap::info(uint flag) { HEAPINFO hp_info; @@ -484,10 +527,12 @@ int ha_heap::reset_auto_increment(ulonglong value) int ha_heap::external_lock(THD *thd, int lock_type) { -#if !defined(DBUG_OFF) && defined(EXTRA_DEBUG) +#if !defined(DBUG_OFF) && defined(EXTRA_HEAP_DEBUG) if (lock_type == F_UNLCK && file->s->changed && heap_check_heap(file, 0)) return HA_ERR_CRASHED; #endif + if (lock_type == F_UNLCK) + hp_flush_pending_blob_free(file); return 0; // No external locking } @@ -531,7 +576,7 @@ int ha_heap::disable_indexes(key_map map, bool persist) enable_indexes() DESCRIPTION - Enable indexes taht might have been disabled by disable_index() before. + Enable indexes that might have been disabled by disable_index() before. The function works only if both data and indexes are empty, since the heap storage engine cannot repair the indexes. To be sure, call handler::delete_all_rows() before. @@ -637,8 +682,8 @@ ha_rows ha_heap::records_in_range(uint inx, const key_range *min_key, } -static int heap_prepare_hp_create_info(TABLE *table_arg, bool internal_table, - HP_CREATE_INFO *hp_create_info) +int heap_prepare_hp_create_info(TABLE *table_arg, bool internal_table, + HP_CREATE_INFO *hp_create_info) { TABLE_SHARE *share= table_arg->s; uint key, parts, mem_per_row= 0, keys= share->keys; @@ -697,18 +742,60 @@ static int heap_prepare_hp_create_info(TABLE *table_arg, bool internal_table, seg->type= field->key_type(); else { - if ((seg->type = field->key_type()) != (int) HA_KEYTYPE_TEXT && + if ((seg->type= key_part->type) != (int) HA_KEYTYPE_TEXT && seg->type != HA_KEYTYPE_VARTEXT1 && seg->type != HA_KEYTYPE_VARTEXT2 && + seg->type != HA_KEYTYPE_VARTEXT4 && seg->type != HA_KEYTYPE_VARBINARY1 && seg->type != HA_KEYTYPE_VARBINARY2 && + seg->type != HA_KEYTYPE_VARBINARY4 && seg->type != HA_KEYTYPE_BIT) seg->type= HA_KEYTYPE_BINARY; } seg->start= (uint) key_part->offset; seg->length= (uint) key_part->length; seg->flag= key_part->key_part_flag; + seg->bit_length= 0; + seg->bit_start= 0; + seg->bit_pos= 0; + + DBUG_ASSERT((seg->flag & HA_BLOB_PART) == + (field->key_part_flag() & HA_BLOB_PART)); + if (seg->flag & HA_BLOB_PART) + { + /* + Blob key segment: 4-byte length + pointer to data. + Field_blob_key (GROUP BY) returns VARTEXT4/VARBINARY4. + Field_blob (DISTINCT/UNION) returns VARTEXT2/VARBINARY2. + Promote the latter to VARTEXT4 format so hp_create.c and the + hash functions handle them identically. + */ + if (seg->type == HA_KEYTYPE_VARBINARY2 || + seg->type == HA_KEYTYPE_VARTEXT2) + { + /* + Old type blob key (length + data). Convert it to + a 4 byte length and pointer. + */ + seg->type= HA_KEYTYPE_VARTEXT4; // Safe, see heap_create() + seg->length= 4 + portable_sizeof_char_ptr; + /* One cannot use this key for index_read() anymore */ + seg->flag|= HA_NO_KEY_READ; + } + else + { + /* + Blob key with a 4 byte length and a pointer to data + HA_KEYTYPE_VARBINARY4 will be converted to HA_KEYTYPE_TEXT4 + in heap_create(). + */ + DBUG_ASSERT(seg->length == 4 + portable_sizeof_char_ptr); + DBUG_ASSERT(seg->type == HA_KEYTYPE_VARBINARY4 || + seg->type == HA_KEYTYPE_VARTEXT4); + } + seg->bit_start= ((Field_blob*) field)->length_size(); + } if (field->flags & (ENUM_FLAG | SET_FLAG)) seg->charset= &my_charset_bin; else @@ -741,21 +828,51 @@ static int heap_prepare_hp_create_info(TABLE *table_arg, bool internal_table, seg->bit_pos= (uint) (((Field_bit *) field)->bit_ptr - (uchar*) table_arg->record[0]); } - else - { - seg->bit_length= seg->bit_start= 0; - seg->bit_pos= 0; - } } } + if (table_arg->found_next_number_field) { keydef[share->next_number_index].flag|= HA_AUTO_KEY; found_real_auto_increment= share->next_number_key_offset == 0; } + + /* Populate blob column descriptors */ + if (share->blob_fields) + { + HP_BLOB_DESC *blob_descs; + blob_descs= (HP_BLOB_DESC*) my_malloc(hp_key_memory_HP_BLOB, + share->blob_fields * + sizeof(HP_BLOB_DESC), + MYF(MY_WME | MY_THREAD_SPECIFIC)); + if (!blob_descs) + { + my_free(keydef); + return my_errno; + } + for (uint blob_index= 0; blob_index < share->blob_fields; blob_index++) + { + Field *field= table_arg->field[share->blob_field[blob_index]]; + Field_blob *blob= (Field_blob*) field; + + DBUG_ASSERT(field->type() == MYSQL_TYPE_BLOB || + field->type() == MYSQL_TYPE_GEOMETRY); + + blob_descs[blob_index].offset= + (uint) blob->offset(table_arg->record[0]); + blob_descs[blob_index].packlength= blob->length_size(); + } + hp_create_info->blob_descs= blob_descs; + hp_create_info->blob_count= share->blob_fields; + } + hp_create_info->auto_key= auto_key; hp_create_info->auto_key_type= auto_key_type; - hp_create_info->max_table_size= MY_MAX(current_thd->variables.max_heap_table_size, sizeof(HP_PTRS)); + hp_create_info->max_table_size= + MY_MAX((internal_table ? + current_thd->variables.tmp_memory_table_size : + current_thd->variables.max_heap_table_size), + sizeof(HP_PTRS)); hp_create_info->with_auto_increment= found_real_auto_increment; hp_create_info->internal_table= internal_table; @@ -794,6 +911,7 @@ int ha_heap::create(const char *name, TABLE *table_arg, create_info->auto_increment_value - 1 : 0); error= heap_create(name, &hp_create_info, &internal_share, &created); my_free(hp_create_info.keydef); + my_free(hp_create_info.blob_descs); DBUG_ASSERT(file == 0); return (error); } @@ -855,27 +973,75 @@ int ha_heap::find_unique_row(uchar *record, uint unique_idx) DBUG_ASSERT(keyinfo->flag & HA_NOSAME); if (!share->records) DBUG_RETURN(1); // not found + ulong rec_hash= hp_rec_hashnr(0, keyinfo, record); HASH_INFO *pos= hp_find_hash(&keyinfo->block, - hp_mask(hp_rec_hashnr(keyinfo, record), + hp_mask(rec_hash, share->blength, share->records)); + + if (!share->blob_count) + { + /* Non-blob path: compare then copy */ + do + { + if (pos->hash_of_key == rec_hash && + !hp_rec_key_cmp(keyinfo, record, pos->ptr_to_rec, file)) + { + file->current_hash_ptr= pos; + file->current_ptr= pos->ptr_to_rec; + file->update= HA_STATE_AKTIV; + memcpy(record, file->current_ptr, (size_t) share->reclength); + DBUG_RETURN(0); + } + } while ((pos= pos->next_key)); + DBUG_RETURN(1); + } + + /* + Blob path: materialize-then-compare. + + Pre-materialize all blobs via hp_read_blobs() before comparison, + then compare with info=NULL (do not materialize blob in + ha_rec_key_cmp()) since both records have direct data + pointers. + */ + uchar *input_copy= (uchar*) my_safe_alloca(share->reclength); + if (!input_copy) + DBUG_RETURN(-1); + memcpy(input_copy, record, (size_t) share->reclength); + + int result= 1; do { - if (!hp_rec_key_cmp(keyinfo, pos->ptr_to_rec, record)) + if (pos->hash_of_key != rec_hash) + continue; + + memcpy(record, pos->ptr_to_rec, (size_t) share->reclength); + if (hp_read_blobs(file, record, pos->ptr_to_rec)) + { + result= -1; /* my_errno is set to HA_ERR_OUT_OF_MEM */ + break; + } + if (!hp_rec_key_cmp(keyinfo, input_copy, record, NULL)) { file->current_hash_ptr= pos; file->current_ptr= pos->ptr_to_rec; - file->update = HA_STATE_AKTIV; - /* - We compare it only by record in the index, so better to read all - records. - */ - memcpy(record, file->current_ptr, (size_t) share->reclength); - - DBUG_RETURN(0); // found and position set + file->update= HA_STATE_AKTIV; + result= 0; + break; } + memcpy(record, input_copy, (size_t) share->reclength); + } while ((pos= pos->next_key)); + + if (result) + { + /* + In case of error, restore orginal record, for possible + error messages by caller. + */ + memcpy(record, input_copy, (size_t) share->reclength); } - while ((pos= pos->next_key)); - DBUG_RETURN(1); // not found + my_safe_afree(input_copy, share->reclength); + DBUG_RETURN(result); } struct st_mysql_storage_engine heap_storage_engine= diff --git a/storage/heap/ha_heap.h b/storage/heap/ha_heap.h index 51bdb2d2e1a65..c1cf5b7e35440 100644 --- a/storage/heap/ha_heap.h +++ b/storage/heap/ha_heap.h @@ -27,6 +27,7 @@ class ha_heap final : public handler key_map btree_keys; /* number of records changed since last statistics update */ ulong records_changed; + ulong saved_current_record; /* for remember_rnd_pos() / restart_rnd_next() */ uint key_stat_version; my_bool internal_table; public: @@ -37,11 +38,12 @@ class ha_heap final : public handler enum row_type get_row_type() const override { return ROW_TYPE_FIXED; } ulonglong table_flags() const override { - return (HA_FAST_KEY_READ | HA_NO_BLOBS | HA_NULL_IN_KEY | + return (HA_FAST_KEY_READ | HA_NULL_IN_KEY | HA_BINLOG_ROW_CAPABLE | HA_BINLOG_STMT_CAPABLE | HA_CAN_SQL_HANDLER | HA_CAN_ONLINE_BACKUPS | HA_REC_NOT_IN_SEQ | HA_CAN_INSERT_DELAYED | HA_NO_TRANSACTIONS | - HA_HAS_RECORDS | HA_STATS_RECORDS_IS_EXACT | HA_CAN_HASH_KEYS); + HA_HAS_RECORDS | HA_STATS_RECORDS_IS_EXACT | HA_CAN_HASH_KEYS | + HA_CAN_GEOMETRY); } ulong index_flags(uint inx, uint part, bool all_parts) const override { @@ -82,6 +84,8 @@ class ha_heap final : public handler int rnd_next(uchar *buf) override; int rnd_pos(uchar * buf, uchar *pos) override; void position(const uchar *record) override; + int remember_rnd_pos() override; + int restart_rnd_next(uchar *buf) override; int can_continue_handler_scan() override; int info(uint) override; int extra(enum ha_extra_function operation) override; diff --git a/storage/heap/heapdef.h b/storage/heap/heapdef.h index e51fe88d8e2b7..b3b46d474de79 100644 --- a/storage/heap/heapdef.h +++ b/storage/heap/heapdef.h @@ -22,6 +22,7 @@ C_MODE_START #include "heap.h" /* Structs & some defines */ #include "my_tree.h" + /* When allocating keys /rows in the internal block structure, do it within the following boundaries. @@ -33,6 +34,123 @@ C_MODE_START #define HP_MIN_RECORDS_IN_BLOCK 16 #define HP_MAX_RECORDS_IN_BLOCK 8192 +/* Flags stored in the 'visible' byte at end of each record */ +#define HP_ROW_ACTIVE 1 /* Bit 0: record is active (not deleted) */ +#define HP_ROW_HAS_CONT 2 /* Bit 1: primary record has continuation chain(s) */ +#define HP_ROW_IS_CONT 4 /* Bit 2: this record IS a continuation record */ +#define HP_ROW_CONT_ZEROCOPY 8 /* Bit 3: zero-copy layout (data in rec 1..N-1) */ +#define HP_ROW_SINGLE_REC 16 /* Bit 4: single-record run, no header -- data at offset 0 */ +#define HP_ROW_MULTIPLE_REC 32 /* Bit 5: multi-run chain, data needs reassembly */ + +/* + Continuation run header: next_cont pointer + run_rec_count. + Stored at the beginning of the first blob segment in each run. +*/ +#define HP_CONT_REC_COUNT_SIZE sizeof(uint16) +#define HP_CONT_HEADER_SIZE (sizeof(uchar*) + HP_CONT_REC_COUNT_SIZE) + +/* + Row flags byte predicates. + The flags byte is at offset 'visible' in each primary or run-header record. +*/ + +/* Record is active (not deleted) */ +static inline my_bool hp_is_active(const uchar *rec, uint visible) +{ + return (rec[visible] & HP_ROW_ACTIVE) != 0; +} + +/* Primary record that owns blob continuation chain(s) */ +static inline my_bool hp_has_cont(const uchar *rec, uint visible) +{ + return (rec[visible] & HP_ROW_HAS_CONT) != 0; +} + +/* This record IS a continuation run header (rec 0 of a run) */ +static inline my_bool hp_is_cont(const uchar *rec, uint visible) +{ + return (rec[visible] & HP_ROW_IS_CONT) != 0; +} + +/* Case A: single-record run, no header -- data at offset 0 */ +static inline my_bool hp_is_single_rec(const uchar *rec, uint visible) +{ + return (rec[visible] & HP_ROW_SINGLE_REC) != 0; +} + +/* Case B: single run, data in rec 1..N-1 -- zero-copy read */ +static inline my_bool hp_is_zerocopy(const uchar *rec, uint visible) +{ + return (rec[visible] & HP_ROW_CONT_ZEROCOPY) != 0; +} + +/* Case C: multi-run chain -- data needs reassembly into blob_buff */ +static inline my_bool hp_is_multi_run(const uchar *rec, uint visible) +{ + return (rec[visible] & HP_ROW_MULTIPLE_REC) != 0; +} + +/* + Continuation run header accessors. + Read next_cont pointer and run_rec_count from the first record of a run. +*/ +static inline const uchar *hp_cont_next(const uchar *chain) +{ + const uchar *next; + memcpy(&next, chain, sizeof(uchar*)); + return next; +} + +static inline uint16 hp_cont_rec_count(const uchar *chain) +{ + return uint2korr(chain + sizeof(uchar*)); +} + +/* + Blob continuation run storage format. + + Case A (HP_ROW_SINGLE_REC): Single-record run, no header. + Data starts at offset 0, full `visible` bytes available for + payload. Detected by hp_is_single_rec(). + Zero-copy: blob pointer -> chain. + + Case B (HP_ROW_CONT_ZEROCOPY): Single run, multiple records. + Header in rec 0, data contiguous in rec 1..N-1. Detected by + hp_is_zerocopy(). + Zero-copy: blob pointer -> chain + recbuffer. + + Case C (HP_ROW_MULTIPLE_REC): One or more runs linked via + next_cont. Header in each run's rec 0, data in rec 0 (after + header) + rec 1..N-1. Detected by hp_is_multi_run(). + Requires reassembly into blob_buff. +*/ + +/* + Minimum contiguous run size parameters. + Runs smaller than this are not worth scavenging from the + delete list because the per-run header overhead (10 bytes) + becomes a significant fraction of payload. Skip them and + allocate from the tail instead. + + HP_CONT_MIN_RUN_BYTES: absolute floor for minimum run payload. + HP_CONT_RUN_FRACTION_NUM/DEN: minimum run size as a fraction + of blob size. + min_run_bytes = MAX(blob_length * NUM / DEN, + HP_CONT_MIN_RUN_BYTES) +*/ +#define HP_CONT_MIN_RUN_BYTES 128 +#define HP_CONT_RUN_FRACTION_NUM 1 +#define HP_CONT_RUN_FRACTION_DEN 10 + +static inline uint32 hp_blob_min_run_bytes(uint32 blob_length) +{ + uint32 length= (blob_length / HP_CONT_RUN_FRACTION_DEN * + HP_CONT_RUN_FRACTION_NUM); + set_if_bigger(length, HP_CONT_MIN_RUN_BYTES); + set_if_smaller(length, blob_length); + return length; +} + /* Some extern variables */ extern LIST *heap_open_list,*heap_share_list; @@ -81,13 +199,16 @@ extern uchar *hp_search(HP_INFO *info,HP_KEYDEF *keyinfo,const uchar *key, uint nextflag); extern uchar *hp_search_next(HP_INFO *info, HP_KEYDEF *keyinfo, const uchar *key, HASH_INFO *pos); -extern ulong hp_rec_hashnr(HP_KEYDEF *keyinfo,const uchar *rec); +extern ulong hp_rec_hashnr(HP_INFO *info, HP_KEYDEF *keyinfo,const uchar *rec); extern void hp_movelink(HASH_INFO *pos,HASH_INFO *next_link, HASH_INFO *newlink); extern int hp_rec_key_cmp(HP_KEYDEF *keydef,const uchar *rec1, - const uchar *rec2); + const uchar *rec2, HP_INFO *info); extern int hp_key_cmp(HP_KEYDEF *keydef,const uchar *rec, - const uchar *key); + const uchar *key, HP_INFO *info); +extern const uchar *hp_materialize_one_blob(HP_INFO *info, + const uchar *chain, + uint32 data_len); extern void hp_make_key(HP_KEYDEF *keydef,uchar *key,const uchar *rec); extern uint hp_rb_make_key(HP_KEYDEF *keydef, uchar *key, const uchar *rec, uchar *recpos); @@ -104,12 +225,35 @@ extern ha_rows hp_rows_in_memory(size_t reclength, size_t index_size, size_t memory_limit); extern size_t hp_memory_needed_per_row(size_t reclength); +extern uchar *hp_alloc_from_tail(HP_SHARE *info, uint *blocks); +extern uchar *next_free_record_pos(HP_SHARE *info); +static inline uint32 hp_blob_length(const HP_BLOB_DESC *desc, + const uchar *record) +{ + return (uint32) read_lowendian(record + desc->offset, desc->packlength); +} +extern int hp_write_one_blob(HP_SHARE *share, const uchar *data_ptr, + uint32 data_len, uchar **first_run_out); +extern int hp_write_blobs(HP_INFO *info, const uchar *record, uchar *pos); +extern int hp_read_blobs(HP_INFO *info, uchar *record, const uchar *pos); +extern void hp_free_blobs(HP_SHARE *share, uchar *pos); +extern void hp_free_run_chain(HP_SHARE *share, uchar *chain); +extern void hp_shrink_tail(HP_SHARE *share); +extern void hp_flush_pending_blob_free_impl(HP_INFO *info); + +static inline void hp_flush_pending_blob_free(HP_INFO *info) +{ + if (info->has_pending_blob_free) + hp_flush_pending_blob_free_impl(info); +} + extern mysql_mutex_t THR_LOCK_heap; extern PSI_memory_key hp_key_memory_HP_SHARE; extern PSI_memory_key hp_key_memory_HP_INFO; extern PSI_memory_key hp_key_memory_HP_PTRS; extern PSI_memory_key hp_key_memory_HP_KEYDEF; +extern PSI_memory_key hp_key_memory_HP_BLOB; #ifdef HAVE_PSI_INTERFACE void init_heap_psi_keys(); diff --git a/storage/heap/hp_blob.c b/storage/heap/hp_blob.c new file mode 100644 index 0000000000000..e0237ffc3199e --- /dev/null +++ b/storage/heap/hp_blob.c @@ -0,0 +1,930 @@ +/* Copyright (c) 2026, MariaDB Corporation. + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; version 2 of the License. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1335 USA */ + +/* + LOB (BLOB/TEXT) support for HEAP tables using variable-length + continuation runs. + + Each blob column's data is stored as a chain of continuation "runs". + A run is a contiguous sequence of recbuffer-sized records in the same + HP_BLOCK. The first record of each run stores a header (next_cont + pointer + run_rec_count); subsequent records carry pure blob payload. + Runs are linked together via the next_cont pointer. + + This design amortizes the per-run header overhead across many records, + giving near-100% space efficiency for typical blob sizes (150 KB and + above), even when recbuffer is very small (e.g. 16 bytes). +*/ + +#include "heapdef.h" +#include + +/* + Try to reclaim deleted records at the tail of the HP_BLOCK tree. + + After hp_free_run_chain() puts tail-allocated records onto the delete + list, this function checks if those records are at the current tail + (highest positions in the last leaf block). If so, it pops them from + the delete list and decrements last_allocated, effectively returning + them for future tail allocation. + + Crosses block boundaries by updating last_blocks to the previous + leaf via hp_find_block(). Empty blocks stay allocated in the tree + (freed at table drop) but their slots become available for reuse. + + Maintains the scan-boundary invariant: + total_records + deleted == block.last_allocated + Each reclaimed slot does deleted-- and last_allocated--, keeping the + difference (total_records) unchanged. + + @param share Table share +*/ + +void hp_shrink_tail(HP_SHARE *share) +{ + HP_BLOCK *block= &share->block; + uint recbuffer= block->recbuffer; + ulong records_in_block= block->records_in_block; + ulong block_pos; + uchar *last_blocks, *tail_pos; + + DBUG_ASSERT(share->total_records + share->deleted == + block->last_allocated); + + if (block->last_allocated > block->high_water_allocated) + block->high_water_allocated= block->last_allocated; + + if (!share->del_link) + return; + DBUG_ASSERT(block->last_allocated); + + if (!(block_pos= block->last_allocated % records_in_block)) + { + /* + We were about to go to a new leaf. Move to end of previous leaf + */ + block_pos= records_in_block; + } + last_blocks= (uchar*) block->level_info[0].last_blocks; + tail_pos= last_blocks + (block_pos - 1) * recbuffer; + + while (share->del_link == tail_pos) + { + share->del_link= *((uchar**) share->del_link); + share->deleted--; + block->last_allocated--; + tail_pos-= recbuffer; + block_pos--; + + /* + When the current leaf block becomes empty (block_pos has reached 0), + find the previous leaf block so subsequent iterations can + continue reclaiming there. The empty block stays allocated + in the tree (freed at table drop). + */ + if (block_pos == 0) + { + if (!block->last_allocated) + break; + tail_pos= hp_find_block(block, block->last_allocated - 1); + block_pos= records_in_block; + block->level_info[0].last_blocks= + (HP_PTRS*)(tail_pos - (records_in_block - 1) * recbuffer); + } + } + DBUG_ASSERT(share->total_records + share->deleted == + share->block.last_allocated); +} + + +/* + Flush deferred blob chain free from a previous heap_delete(). + + heap_delete() saves chain pointers instead of freeing them + immediately, so that callers (binlog writer, AFTER DELETE triggers) + can still read blob data from the record buffer after delete_row() + returns. This function does the actual free. + + @param info Table handle with pending_blob_chains to free +*/ + +void hp_flush_pending_blob_free_impl(HP_INFO *info) +{ + HP_SHARE *share= info->s; + uint i; + DBUG_ASSERT(info->has_pending_blob_free); + for (i= 0; i < share->blob_count; i++) + { + if (info->pending_blob_chains[i]) + hp_free_run_chain(share, info->pending_blob_chains[i]); + } + hp_shrink_tail(share); + info->has_pending_blob_free= FALSE; +} + + +/* + Free one continuation chain of variable-length runs. + + Walks from the first run, reads run_rec_count from each, frees all + records individually to the delete list, then follows next_cont to the + next run. + + Maintains the scan-boundary invariant: + total_records + deleted == block.last_allocated + Each freed slot does total_records-- and deleted++, keeping the sum + constant. heap_scan() relies on this sum to know when to stop. + + @param share Table share + @param chain Pointer to first record of first run (or NULL) +*/ + +void hp_free_run_chain(HP_SHARE *share, uchar *chain) +{ + uint recbuffer= share->block.recbuffer; + uint visible= share->visible; + + while (chain) + { + uchar *next_run; + uint16 run_rec_count; + uint16 j; + + if (hp_is_single_rec(chain, visible)) + { + /* Case A: single record, no header */ + next_run= NULL; + run_rec_count= 1; + } + else + { + /* Case B/C: header present with next_cont and run_rec_count */ + memcpy(&next_run, chain, sizeof(uchar*)); + run_rec_count= uint2korr(chain + sizeof(uchar*)); + } + + for (j= 0; j < run_rec_count; j++) + { + uchar *pos= chain + j * recbuffer; + *((uchar**) pos)= share->del_link; + share->del_link= pos; + pos[visible]= 0; + share->deleted++; + share->total_records--; + } + + chain= next_run; + } +} + + +/* + Write blob data into a contiguous run of records. + + Writes the run header (next_cont=NULL, run_rec_count) in the first + record, then copies blob data across all records in the run, + advancing *offset. + + @param share Table share + @param data Source blob data + @param data_len Total blob data length + @param run_start Pointer to first record of the run + @param run_rec_count Number of consecutive records in this run + @param format Storage format bit (HP_ROW_SINGLE_REC / HP_ROW_CONT_ZEROCOPY / HP_ROW_MULTIPLE_REC) + @param offset [in/out] Current offset into blob data + + @note Caller must link runs by overwriting next_cont in the previous run. +*/ + +static void hp_write_run_data(HP_SHARE *share, const uchar *data, + uint32 data_len, uchar *run_start, + uint16 run_rec_count, + uint8 format, + uint32 *offset) +{ + uint visible= share->visible; + uint recbuffer= share->block.recbuffer; + uint32 off= *offset; + uint32 remaining= data_len - off; + uint32 chunk; + + if (format & HP_ROW_SINGLE_REC) + { + /* + Case A: single-record run, no header. Data starts at offset 0, + full `visible` bytes available. HP_ROW_SINGLE_REC signals the + reader that there is no run header to parse. + */ + DBUG_ASSERT(run_rec_count == 1); + DBUG_ASSERT(remaining <= visible); + run_start[visible]= HP_ROW_ACTIVE | HP_ROW_IS_CONT | HP_ROW_SINGLE_REC; + memcpy(run_start, data + off, remaining); + *offset= off + remaining; + return; + } + + /* First record: run header + flags byte */ + *((uchar**) run_start)= NULL; + int2store(run_start + sizeof(uchar*), run_rec_count); + run_start[visible]= HP_ROW_ACTIVE | HP_ROW_IS_CONT | + (format & (HP_ROW_CONT_ZEROCOPY | HP_ROW_MULTIPLE_REC)); + + /* + We come here when we need data in the initial run block. + In other words, we are not writing a multi-row zerocopy block. + */ + if (format & HP_ROW_MULTIPLE_REC) + { + chunk= visible - HP_CONT_HEADER_SIZE; + if (chunk > remaining) + chunk= remaining; + memcpy(run_start + HP_CONT_HEADER_SIZE, data + off, chunk); + off+= chunk; + remaining-= chunk; + } + + /* + Inner records (rec 1..N-1): full recbuffer payload, no flags byte. + This makes data in inner records contiguous, enabling zero-copy reads + for single-run blobs (Case B). + */ + run_start+= recbuffer; + if (run_rec_count > 2) + { + /* Copy all remaining blocks except the last one */ + uint data_length= (run_rec_count - 2) * recbuffer; + DBUG_ASSERT(remaining > data_length); + memcpy(run_start, data + off, data_length); + run_start+= data_length; + off+= data_length; + remaining-= data_length; + } + if (run_rec_count > 1) + { + /* Copy last, likely part block */ + DBUG_ASSERT(remaining != 0); + chunk= remaining < recbuffer ? remaining : recbuffer; + memcpy(run_start, data + off, chunk); + off+= chunk; + remaining-= chunk; + } + + *offset= off; +} + + +/* + Unlink a contiguous group from the delete list and write blob data into it. + Does not support zerocopy (always uses HP_ROW_MULTIPLE_REC format). + + @param share Table share + @param data_ptr Blob data + @param data_len Total blob data length + @param run_start Lowest address of the contiguous group + @param run_count Number of contiguous records in the group + @param data_offset [in/out] Current offset into blob data + @param first_run [out] Pointer to first run; undefined until return + @param prev_run_start [in/out] Pointer to previous run's start +*/ + +static void hp_unlink_and_write_run(HP_SHARE *share, const uchar *data_ptr, + uint32 data_len, uchar *run_start, + uint16 run_count, uint32 *data_offset, + uchar **first_run, uchar **prev_run_start) +{ + uint recbuffer= share->block.recbuffer; + DBUG_ASSERT(share->del_link == run_start + (run_count-1) * recbuffer); + DBUG_ASSERT(share->del_link >= run_start && + share->del_link < run_start + run_count * recbuffer); + + share->del_link= *(uchar**) (share->del_link - + (run_count-1) * recbuffer); + share->deleted-= run_count; + share->total_records+= run_count; + + hp_write_run_data(share, data_ptr, data_len, run_start, + run_count, HP_ROW_MULTIPLE_REC, data_offset); + + if (*prev_run_start) + memcpy(*prev_run_start, &run_start, sizeof(run_start)); + else + *first_run= run_start; + *prev_run_start= run_start; +} + + +/* + Write one blob column's data into a chain of continuation runs. + + Allocates contiguous runs from the delete list and/or block tail, + copies blob data into them, and returns the first run pointer. + On failure, frees any partially allocated chain. + + @param share Table share + @param data_ptr Blob data to write + @param data_len Blob data length (must be > 0) + @param first_run_out [out] Pointer to first run's first record + + @return 0 on success, my_errno on failure +*/ + +int hp_write_one_blob(HP_SHARE *share, const uchar *data_ptr, + uint32 data_len, uchar **first_run_out) +{ + uint visible= share->visible; + uint recbuffer= share->block.recbuffer; + uint32 min_run_bytes; + uint32 min_run_records; + uchar *first_run= NULL; + uchar *prev_run_start= NULL; + uint32 data_offset= 0; + uint32 first_payload= visible - HP_CONT_HEADER_SIZE; + uint32 total_records_needed= + (data_len <= first_payload ? 1 : + 1 + (data_len - first_payload + recbuffer - 1) / recbuffer); + my_bool is_only_run; + + /* + Calculate minimum acceptable contiguous run size for delete-list reuse. + + The delete-list walk (Step 1 below) rejects contiguous groups smaller + than min_run_records, bailing to tail allocation instead. This + prevents excessive chain fragmentation for large blobs: accepting + tiny fragments would produce long chains of many short runs, each + with its own 10-byte header overhead and pointer dereference on read. + + The threshold is the larger of: + - 1/10 of the blob size (caps chain length at ~10 runs) + - 128 bytes absolute floor (HP_CONT_MIN_RUN_BYTES) + - 2 records minimum (a single-slot run is pure overhead) + + For small blobs whose total bytes or records needed is below this + threshold, the fragmentation concern doesn't apply - the entire blob + fits in one short run. Cap both min_run_bytes and min_run_records + so the delete list can satisfy the allocation without falling through + to the tail unnecessarily. + */ + min_run_bytes= hp_blob_min_run_bytes(data_len); + min_run_records= (min_run_bytes + recbuffer - 1) / recbuffer; + if (min_run_records < 2) + min_run_records= 2; + + if (total_records_needed < min_run_records) + min_run_records= total_records_needed; + + /* + Step 1: Try to allocate contiguous runs from the top of the delete list. + + Peek at delete list records by walking next pointers without unlinking. + Track contiguous groups (descending addresses - LIFO order from + hp_free_run_chain). On discontinuity: if the group qualifies + (>= min_run_records), unlink and use it; if it doesn't, the tail + of the delete_link is too small. Instead of continue searching + for a larger block, we stop searching. + */ + { + uchar *run_start; + uint16 run_count= 1; + uchar *pos; + uint32 max_run= MY_MIN(total_records_needed, UINT_MAX16); + + if ((run_start= share->del_link)) + { + pos= *((uchar**) run_start); + run_count= 1; + for (; pos ; pos= *((uchar**) pos)) + { + /* + Only check descending direction: hp_free_run_chain() frees + records in ascending address order (j=0..N), so LIFO pushes + them onto the delete list in reverse - consecutive delete + list entries have descending addresses. Ascending adjacency + from unrelated deletes is ignored intentionally; we only + recover runs that were freed together. + */ + if (run_count == total_records_needed) + break; /* Use this run */ + + if (pos == run_start - recbuffer) + { + run_start= pos; + run_count++; + if (run_count < max_run) + continue; + if (run_count == total_records_needed) + break; /* Use this run */ + /* run_count is now UINT_MAX16 */ + } + + /* + Discontinuity. If the accumulated group qualifies, use it. + If not, the top of the delete list is fragmented - give up entirely. + */ + if (run_count < min_run_records) + break; + hp_unlink_and_write_run(share, data_ptr, data_len, run_start, + run_count, &data_offset, + &first_run, &prev_run_start); + + pos= share->del_link; + total_records_needed-= run_count; + + /* This cannot be last run */ + DBUG_ASSERT(data_offset < data_len && pos); + DBUG_ASSERT(total_records_needed != 0); + + run_start= pos; + run_count= 1; + } + + /* Handle the last group after the loop ends */ + if (run_count >= min_run_records && data_offset < data_len) + hp_unlink_and_write_run(share, data_ptr, data_len, run_start, + run_count, &data_offset, + &first_run, &prev_run_start); + } + } + + /* + Step 2: Allocate remaining data from the block tail. + + Batch allocation: hp_alloc_from_tail() returns a contiguous + batch of records within a single leaf block in one call. + When we hit a block boundary, a new run starts. + */ + + is_only_run= (first_run == NULL && prev_run_start == NULL); + while (data_offset < data_len) + { + uchar *run_start; + uint run_rec_count; + uint32 remaining= data_len - data_offset; + + /* + Compute the number of records to request. + Case B (zero-copy) needs the most records per data byte, so + request that amount for is_only_run to give zero-copy the best + chance. hp_alloc_from_tail() caps at the remaining slots in + the current leaf block. + */ + if (is_only_run) + { + if (remaining <= visible) + run_rec_count= 1; + else + { + uint64 needed= ((uint64) remaining + recbuffer - 1) / recbuffer + 1; + run_rec_count= (uint) MY_MIN(needed, UINT_MAX16); + } + } + else + { + if (remaining <= first_payload) + run_rec_count= 1; + else + { + uint64 needed= 1 + ((uint64)(remaining - first_payload) + + recbuffer - 1) / recbuffer; + run_rec_count= (uint) MY_MIN(needed, UINT_MAX16); + } + } + + run_start= hp_alloc_from_tail(share, &run_rec_count); + if (!run_start) + break; + DBUG_ASSERT(run_rec_count >= 1); + + if (is_only_run && remaining <= visible) + { + /* Case A: single record, no header */ + hp_write_run_data(share, data_ptr, data_len, run_start, + (uint16) run_rec_count, HP_ROW_SINGLE_REC, + &data_offset); + } + else if (is_only_run && + (uint32)(run_rec_count - 1) * recbuffer >= remaining) + { + /* Case B: data in rec 1..N-1, contiguous for zero-copy reads */ + hp_write_run_data(share, data_ptr, data_len, run_start, + (uint16) run_rec_count, HP_ROW_CONT_ZEROCOPY, + &data_offset); + } + else + { + /* Case C: multi-run or partial run */ + hp_write_run_data(share, data_ptr, data_len, run_start, + (uint16) run_rec_count, HP_ROW_MULTIPLE_REC, + &data_offset); + } + + if (prev_run_start) + memcpy(prev_run_start, &run_start, sizeof(run_start)); + else + first_run= run_start; + prev_run_start= run_start; + is_only_run= 0; + } + + /* + Step 3: Free-list scavenge fallback. + + When the tail is full but there are deleted records on the free list, + walk the entire free list accepting any contiguous group (even a + single slot). This produces maximally fragmented chains (many short + runs, Case C), which are slower to read but correct. Failing with + table-full when free slots exist is worse than a fragmented chain. + */ + while (data_offset < data_len && share->del_link) + { + uchar *run_start= share->del_link; + uchar *pos= *((uchar**) run_start); + uint16 run_count= 1; + uint32 remaining= data_len - data_offset; + uint32 remaining_records= + (remaining <= first_payload ? 1 : + 1 + (remaining - first_payload + recbuffer - 1) / recbuffer); + uint32 max_run= MY_MIN(remaining_records, UINT_MAX16); + + /* Traverse the delete link list */ + for (; pos; pos= *((uchar**) pos)) + { + if (run_count >= max_run) + break; + if (pos == run_start - recbuffer) + { + /* Found block directly before current block. Combine them */ + run_start= pos; + run_count++; + continue; + } + break; + } + + hp_unlink_and_write_run(share, data_ptr, data_len, run_start, + run_count, &data_offset, + &first_run, &prev_run_start); + } + + if (data_offset < data_len) + { + /* + hp_alloc_from_tail() has set my_errno to HA_ERR_RECORD_FILE_FULL or + ENOMEM + */ + DBUG_ASSERT(my_errno); + goto err; + } + + *first_run_out= first_run; + return 0; + +err: + if (first_run) + { + hp_free_run_chain(share, first_run); + hp_shrink_tail(share); + } + *first_run_out= NULL; + return my_errno; +} + + +/* + Write blob data from the record buffer into continuation runs. + + For each blob column, reads the (length, pointer) descriptor from + the caller's record buffer, allocates variable-length continuation + runs, copies blob data into them, and overwrites the pointer in + the stored row (pos) to point to the first continuation run. + + @param info Table handle + @param record Source record buffer (caller's data) + @param pos Destination row in HP_BLOCK (already has memcpy'd record) + + @return 0 on success, my_errno on failure +*/ + +int hp_write_blobs(HP_INFO *info, const uchar *record, uchar *pos) +{ + HP_SHARE *share= info->s; + HP_BLOB_DESC *desc, *desc_end; + my_bool has_blob_data= FALSE; + DBUG_ENTER("hp_write_blobs"); + + for (desc= share->blob_descs, desc_end= desc + share->blob_count; + desc < desc_end; desc++) + { + uint32 data_len; + const uchar *data_ptr; + uchar *first_run; + + data_len= hp_blob_length(desc, record); + + if (data_len == 0) + { + bzero(pos + desc->offset + desc->packlength, sizeof(char*)); + continue; + } + + has_blob_data= TRUE; + memcpy(&data_ptr, record + desc->offset + desc->packlength, + sizeof(data_ptr)); + + if (hp_write_one_blob(share, data_ptr, data_len, &first_run)) + { + /* Rollback: free all previously completed blob columns */ + HP_BLOB_DESC *rd; + for (rd= share->blob_descs; rd < desc; rd++) + { + uchar *chain; + memcpy(&chain, pos + rd->offset + rd->packlength, sizeof(chain)); + if (chain) + hp_free_run_chain(share, chain); + bzero(pos + rd->offset + rd->packlength, sizeof(char*)); + } + hp_shrink_tail(share); + bzero(pos + desc->offset + desc->packlength, sizeof(char*)); + DBUG_RETURN(my_errno); + } + + memcpy(pos + desc->offset + desc->packlength, &first_run, + sizeof(first_run)); + } + + pos[share->visible]= has_blob_data ? + (HP_ROW_ACTIVE | HP_ROW_HAS_CONT) : HP_ROW_ACTIVE; + DBUG_RETURN(0); +} + + +/* + Reassemble blob data from a Case C multi-run continuation chain + into a contiguous output buffer. + + @param chain First run pointer + @param data_len Total blob data length + @param out Output buffer (must be >= data_len bytes) + @param visible share->visible + @param recbuffer share->block.recbuffer +*/ + +static void hp_reassemble_chain(const uchar *chain, uint32 data_len, + uchar *out, uint visible, uint recbuffer) +{ + uint32 remaining= data_len; + while (chain && remaining > 0) + { + uint16 rec; + uint16 run_rec_count; + uint32 chunk; + const uchar *next_cont; + + next_cont= hp_cont_next(chain); + run_rec_count= hp_cont_rec_count(chain); + + /* First record payload (after header) */ + chunk= visible - HP_CONT_HEADER_SIZE; + if (chunk > remaining) + chunk= remaining; + memcpy(out, chain + HP_CONT_HEADER_SIZE, chunk); + out+= chunk; + remaining-= chunk; + + /* Inner records: recbuffer stride, no flags byte */ + for (rec= 1; rec < run_rec_count; rec++) + { + const uchar *rec_ptr= chain + rec * recbuffer; + DBUG_ASSERT(remaining != 0); + chunk= recbuffer; + if (chunk > remaining) + chunk= remaining; + memcpy(out, rec_ptr, chunk); + out+= chunk; + remaining-= chunk; + } + + chain= next_cont; + } +} + + +/* + Read blob data from continuation runs into the reassembly buffer. + + After memcpy(record, pos, reclength), blob descriptor pointers in + record[] point into HP_BLOCK continuation run chains. This function + walks each chain, reassembles blob data into info->blob_buff, and + rewrites the pointers in record[] to point into blob_buff. + + @param info Table handle + @param record Record buffer (already has memcpy'd row data) + @param pos Row pointer in HP_BLOCK + + @return 0 on success, my_errno on failure +*/ + +int hp_read_blobs(HP_INFO *info, uchar *record, const uchar *pos) +{ + HP_SHARE *share= info->s; + HP_BLOB_DESC *desc, *desc_end; + uint visible= share->visible; + uint recbuffer= share->block.recbuffer; + ulonglong total_copy_size= 0; + my_bool force_copy= share->write_can_replace; + uchar *buff_ptr; + DBUG_ENTER("hp_read_blobs"); + + info->has_zerocopy_blobs= FALSE; + + if (!hp_has_cont(pos, share->visible)) + DBUG_RETURN(0); + + desc_end= share->blob_descs + share->blob_count; + + /* + Pass 1: sum data_len for blobs that need copying into blob_buff. + Cases A and B normally use zero-copy pointers into HP_BLOCK. + When write_can_replace is set (REPLACE statement), all blobs are + copied to avoid dangling pointers after the deferred chain free. + */ + for (desc= share->blob_descs; desc < desc_end; desc++) + { + uint32 data_len; + const uchar *chain; + + data_len= hp_blob_length(desc, record); + if (data_len == 0) + continue; + + memcpy(&chain, record + desc->offset + desc->packlength, sizeof(chain)); + + if (!force_copy && !hp_is_multi_run(chain, visible)) + { + info->has_zerocopy_blobs= TRUE; + continue; + } + total_copy_size+= data_len; + } + + /* Grow reassembly buffer */ + if (total_copy_size > 0) + { + if (total_copy_size > info->blob_buff_len) + { + uchar *new_buff= (uchar*) my_realloc(hp_key_memory_HP_BLOB, + info->blob_buff, + total_copy_size, + MYF(MY_ALLOW_ZERO_PTR)); + if (!new_buff) + DBUG_RETURN(my_errno= HA_ERR_OUT_OF_MEM); + info->blob_buff= new_buff; + info->blob_buff_len= total_copy_size; + } + } + + /* Pass 2: process each blob column */ + buff_ptr= info->blob_buff; + for (desc= share->blob_descs; desc < desc_end; desc++) + { + uint32 data_len; + const uchar *chain, *blob_data= buff_ptr; + + data_len= hp_blob_length(desc, record); + if (data_len == 0) + continue; + + memcpy(&chain, record + desc->offset + desc->packlength, sizeof(chain)); + + if (hp_is_single_rec(chain, visible)) + { + /* Case A: data at offset 0 -- record already has the right pointer */ + if (!force_copy) + continue; + memcpy(buff_ptr, chain, data_len); + buff_ptr+= data_len; + } + else if (hp_is_zerocopy(chain, visible)) + { + /* Case B: data in rec 1..N-1, contiguous -- adjust pointer past header */ + if (!force_copy) + blob_data= chain + recbuffer; /* Zero copy */ + else + { + memcpy(buff_ptr, chain + recbuffer, data_len); + buff_ptr+= data_len; + } + } + else + { + /* Case C: reassemble into blob_buff */ + hp_reassemble_chain(chain, data_len, buff_ptr, visible, recbuffer); + buff_ptr+= data_len; + } + /* Copy pointer to data to record */ + memcpy(record + desc->offset + desc->packlength, &blob_data, + sizeof(blob_data)); + } + + DBUG_RETURN(0); +} + + +/* + Materialize a single blob column's data from a continuation chain + into info->blob_buff. + + Used by hash comparison functions when comparing a stored record + (where the blob data pointer has been overwritten with a continuation + chain pointer) against an input record. + + @param info Table handle (provides blob_buff) + @param chain Pointer to first run of the continuation chain + @param data_len Total blob data length (from record's packlength bytes) + + @return Pointer into info->blob_buff with contiguous blob data, + or NULL on allocation failure. +*/ + +const uchar *hp_materialize_one_blob(HP_INFO *info, + const uchar *chain, + uint32 data_len) +{ + HP_SHARE *share= info->s; + uint visible= share->visible; + uint recbuffer= share->block.recbuffer; + + if (data_len == 0 || !chain) + return chain; + + if (hp_is_single_rec(chain, visible)) + return chain; /* Case A: no header, data at offset 0 */ + if (hp_is_zerocopy(chain, visible)) + return chain + recbuffer; /* Case B: data in rec 1..N-1 */ + + /* + Case C: multiple runs, reassemble into key_blob_buff. + Uses a separate buffer from blob_buff to avoid overwriting + blob data that hp_read_blobs() placed there and that record + pointers still reference. + */ + if (data_len > info->key_blob_buff_len) + { + uchar *new_buff= (uchar*) my_realloc(hp_key_memory_HP_BLOB, + info->key_blob_buff, + data_len, + MYF(MY_ALLOW_ZERO_PTR)); + if (!new_buff) + return NULL; + info->key_blob_buff= new_buff; + info->key_blob_buff_len= data_len; + } + + hp_reassemble_chain(chain, data_len, info->key_blob_buff, visible, recbuffer); + return info->key_blob_buff; +} + + +/* + Free continuation run chains for all blob columns of a row. + + Walks each blob column's run chain and adds all records back to the + delete list. + + @param share Table share + @param pos Primary record pointer in HP_BLOCK +*/ + +void hp_free_blobs(HP_SHARE *share, uchar *pos) +{ + HP_BLOB_DESC *desc, *desc_end; + DBUG_ENTER("hp_free_blobs"); + + if (!hp_has_cont(pos, share->visible)) + DBUG_VOID_RETURN; + + for (desc= share->blob_descs, desc_end= desc + share->blob_count; + desc < desc_end; desc++) + { + uchar *chain; + + if (hp_blob_length(desc, pos) == 0) + continue; + memcpy(&chain, pos + desc->offset + desc->packlength, sizeof(chain)); + hp_free_run_chain(share, chain); + } + + hp_shrink_tail(share); + DBUG_VOID_RETURN; +} diff --git a/storage/heap/hp_clear.c b/storage/heap/hp_clear.c index b0b263249a881..15a49b0e29a52 100644 --- a/storage/heap/hp_clear.c +++ b/storage/heap/hp_clear.c @@ -24,6 +24,7 @@ void heap_clear(HP_INFO *info) { + info->has_pending_blob_free= FALSE; hp_clear(info->s); } @@ -35,8 +36,10 @@ void hp_clear(HP_SHARE *info) (void) hp_free_level(&info->block,info->block.levels,info->block.root, (uchar*) 0); info->block.levels=0; + info->block.last_allocated=0; + info->block.high_water_allocated=0; hp_clear_keys(info); - info->records= info->deleted= 0; + info->records= info->deleted= info->total_records= 0; info->data_length= 0; info->blength=1; info->changed=0; @@ -100,6 +103,7 @@ void hp_clear_keys(HP_SHARE *info) (void) hp_free_level(block,block->levels,block->root,(uchar*) 0); block->levels=0; block->last_allocated=0; + block->high_water_allocated=0; keyinfo->hash_buckets= 0; } } diff --git a/storage/heap/hp_close.c b/storage/heap/hp_close.c index 82d6186340aa1..6c1f945540741 100644 --- a/storage/heap/hp_close.c +++ b/storage/heap/hp_close.c @@ -39,7 +39,13 @@ int hp_close(register HP_INFO *info) if (info->open_list.data) heap_open_list=list_delete(heap_open_list,&info->open_list); if (!--info->s->open_count && info->s->delete_on_close) + { + hp_flush_pending_blob_free(info); hp_free(info->s); /* Table was deleted */ + } + my_free(info->pending_blob_chains); + my_free(info->blob_buff); + my_free(info->key_blob_buff); my_free(info); DBUG_RETURN(error); } diff --git a/storage/heap/hp_create.c b/storage/heap/hp_create.c index 20a099d415951..73d0bccf988d0 100644 --- a/storage/heap/hp_create.c +++ b/storage/heap/hp_create.c @@ -37,7 +37,7 @@ static const ulong heap_min_allocation_block= 16384; int heap_create(const char *name, HP_CREATE_INFO *create_info, HP_SHARE **res, my_bool *created_new_share) { - uint i, j, key_segs, max_length, length; + uint i, key_segs, max_length, length; HP_SHARE *share= 0; HA_KEYSEG *keyseg; HP_KEYDEF *keydef= create_info->keydef; @@ -74,15 +74,29 @@ int heap_create(const char *name, HP_CREATE_INFO *create_info, so the visible_offset must be least at sizeof(uchar*) */ visible_offset= MY_MAX(reclength, sizeof (char*)); - + /* + Blob tables store continuation run headers (next_cont pointer + + run_slots count = HP_CONT_HEADER_SIZE bytes) in each run's first + slot. Ensure at least 1 byte of payload beyond the header, + otherwise hp_write_run_data() underflows computing + chunk = visible - HP_CONT_HEADER_SIZE. Only matters for + pathological single-TINYBLOB tables (reclength as low as 9). + */ + if (create_info->blob_count) + visible_offset= MY_MAX(visible_offset, HP_CONT_HEADER_SIZE + 1); + for (i= key_segs= max_length= 0, keyinfo= keydef; i < keys; i++, keyinfo++) { + HA_KEYSEG *keyseg, *keyseg_end; + bzero((char*) &keyinfo->block,sizeof(keyinfo->block)); bzero((char*) &keyinfo->rb_tree ,sizeof(keyinfo->rb_tree)); - for (j= length= 0; j < keyinfo->keysegs; j++) + for (keyseg= keyinfo->seg, keyseg_end= keyseg+ keyinfo->keysegs, length=0; + keyseg < keyseg_end ; + keyseg++) { - length+= keyinfo->seg[j].length; - if (keyinfo->seg[j].null_bit) + length+= keyseg->length; + if (keyseg->null_bit) { length++; if (!(keyinfo->flag & HA_NULL_ARE_EQUAL)) @@ -90,7 +104,7 @@ int heap_create(const char *name, HP_CREATE_INFO *create_info, if (keyinfo->algorithm == HA_KEY_ALG_BTREE) keyinfo->rb_tree.size_of_element++; } - switch (keyinfo->seg[j].type) { + switch (keyseg->type) { case HA_KEYTYPE_SHORT_INT: case HA_KEYTYPE_LONG_INT: case HA_KEYTYPE_FLOAT: @@ -102,45 +116,72 @@ int heap_create(const char *name, HP_CREATE_INFO *create_info, case HA_KEYTYPE_INT24: case HA_KEYTYPE_UINT24: case HA_KEYTYPE_INT8: - keyinfo->seg[j].flag|= HA_SWAP_KEY; + keyseg->flag|= HA_SWAP_KEY; break; case HA_KEYTYPE_VARBINARY1: /* Case-insensitiveness is handled in hash_sort */ - keyinfo->seg[j].type= HA_KEYTYPE_VARTEXT1; + keyseg->type= HA_KEYTYPE_VARTEXT1; /* fall through */ case HA_KEYTYPE_VARTEXT1: keyinfo->flag|= HA_VAR_LENGTH_KEY; + /* + Real blob fields always enter as VARTEXT4/VARBINARY4, never + as VARTEXT1/VARBINARY1. Strip any spurious HA_BLOB_PART + (e.g. from uninitialized key_part_flag in SJ weedout tables). + */ + DBUG_ASSERT(!(keyseg->flag & HA_BLOB_PART)); + keyseg->flag&= ~HA_BLOB_PART; /* For BTREE algorithm, key length, greater than or equal to 255, is packed on 3 bytes. */ if (keyinfo->algorithm == HA_KEY_ALG_BTREE) - length+= size_to_store_key_length(keyinfo->seg[j].length); + length+= size_to_store_key_length(keyseg->length); else length+= 2; - /* Save number of bytes used to store length */ - keyinfo->seg[j].bit_start= 1; + keyseg->bit_start= 1; /* Packlength for records */ + keyseg->bit_length= 2; /* Packlength for key */ break; + case HA_KEYTYPE_VARBINARY4: + /* fall through */ + case HA_KEYTYPE_VARTEXT4: + /* Key is stored as 4 byte length + pointer to data */ + DBUG_ASSERT(keyseg->flag & HA_BLOB_PART); + DBUG_ASSERT(keyinfo->algorithm != HA_KEY_ALG_BTREE); + DBUG_ASSERT(keyseg->length == 4+portable_sizeof_char_ptr); + DBUG_ASSERT(keyseg->bit_start >= 1 && keyseg->bit_start <= 4); + DBUG_ASSERT(keyseg->bit_length == 0); + + /* + bit_start holds the actual blob packlength (1-4), set by + heap_prepare_hp_create_info(). + */ + keyinfo->flag|= HA_VAR_LENGTH_KEY; + keyseg->type= HA_KEYTYPE_VARTEXT4; + break; + case HA_KEYTYPE_VARBINARY2: /* Case-insensitiveness is handled in hash_sort */ - /* fall_through */ + /* fall through */ case HA_KEYTYPE_VARTEXT2: keyinfo->flag|= HA_VAR_LENGTH_KEY; + /* key is stored as [length] + data */ + keyseg->bit_start= 2; + keyseg->bit_length= 2; + /* + Make future comparison simpler by only having to check for + one type + */ + keyseg->type= HA_KEYTYPE_VARTEXT1; + /* For BTREE algorithm, key length, greater than or equal to 255, is packed on 3 bytes. */ if (keyinfo->algorithm == HA_KEY_ALG_BTREE) - length+= size_to_store_key_length(keyinfo->seg[j].length); + length+= size_to_store_key_length(keyseg->length); else - length+= 2; - /* Save number of bytes used to store length */ - keyinfo->seg[j].bit_start= 2; - /* - Make future comparison simpler by only having to check for - one type - */ - keyinfo->seg[j].type= HA_KEYTYPE_VARTEXT1; + length+= keyseg->bit_start; break; case HA_KEYTYPE_BIT: /* @@ -174,7 +215,9 @@ int heap_create(const char *name, HP_CREATE_INFO *create_info, if (!(share= (HP_SHARE*) my_malloc(hp_key_memory_HP_SHARE, sizeof(HP_SHARE)+ keys*sizeof(HP_KEYDEF)+ - key_segs*sizeof(HA_KEYSEG), + key_segs*sizeof(HA_KEYSEG)+ + create_info->blob_count* + sizeof(HP_BLOB_DESC), MYF(MY_ZEROFILL | (create_info->internal_table ? MY_THREAD_SPECIFIC : 0))))) @@ -182,6 +225,13 @@ int heap_create(const char *name, HP_CREATE_INFO *create_info, share->keydef= (HP_KEYDEF*) (share + 1); share->key_stat_version= 1; keyseg= (HA_KEYSEG*) (share->keydef + keys); + if (create_info->blob_count) + { + share->blob_descs= (HP_BLOB_DESC*) (keyseg + key_segs); + memcpy(share->blob_descs, create_info->blob_descs, + create_info->blob_count * sizeof(HP_BLOB_DESC)); + share->blob_count= create_info->blob_count; + } init_block(&share->block, hp_memory_needed_per_row(reclength), min_records, max_records); /* Fix keys */ @@ -361,6 +411,7 @@ static void init_block(HP_BLOCK *block, size_t reclength, ulong min_records, block->records_in_block= records_in_block; block->recbuffer= recbuffer; block->last_allocated= 0L; + block->high_water_allocated= 0L; /* All allocations are done with this size, if possible */ block->alloc_size= alloc_size - MALLOC_OVERHEAD; diff --git a/storage/heap/hp_delete.c b/storage/heap/hp_delete.c index 9579fb51a7918..96cd5c7543440 100644 --- a/storage/heap/hp_delete.c +++ b/storage/heap/hp_delete.c @@ -26,6 +26,7 @@ int heap_delete(HP_INFO *info, const uchar *record) DBUG_PRINT("enter",("info: %p record: %p", info, record)); test_active(info); + hp_flush_pending_blob_free(info); if (info->opt_flag & READ_CHECK_USED && hp_rectest(info,record)) DBUG_RETURN(my_errno); /* Record changed */ @@ -42,11 +43,51 @@ int heap_delete(HP_INFO *info, const uchar *record) goto err; } + if (share->blob_count && hp_has_cont(pos, share->visible)) + { + if (share->internal) + { + /* + Internal temporary tables are never binlogged, so blob chains + can be freed immediately. + */ + hp_free_blobs(share, pos); + } + else + { + /* + Defer blob chain free: save chain pointers for later cleanup. + + The handler layer calls binlog_log_row() AFTER delete_row() + returns, reading blob data from the record buffer via zero-copy + pointers into HP_BLOCK chain records. Freeing chains here would + overwrite those records with del_link pointers, making the + zero-copy pointers dangle. + + Save the chain head pointers and free them on the next mutating + operation (write/update/delete) or on reset/close. + */ + HP_BLOB_DESC *desc; + uint i; + for (i= 0, desc= share->blob_descs; i < share->blob_count; i++, desc++) + { + if (hp_blob_length(desc, pos) == 0) + { + info->pending_blob_chains[i]= NULL; + continue; + } + memcpy(&info->pending_blob_chains[i], + pos + desc->offset + desc->packlength, sizeof(uchar*)); + } + info->has_pending_blob_free= TRUE; + } + } info->update=HA_STATE_DELETED; *((uchar**) pos)=share->del_link; share->del_link=pos; pos[share->visible]=0; /* Record deleted */ share->deleted++; + share->total_records--; share->key_version++; #if !defined(DBUG_OFF) && defined(EXTRA_HEAP_DEBUG) DBUG_EXECUTE("check_heap",heap_check_heap(info, 0);); @@ -104,7 +145,7 @@ int hp_rb_delete_key(HP_INFO *info, register HP_KEYDEF *keyinfo, int hp_delete_key(HP_INFO *info, register HP_KEYDEF *keyinfo, const uchar *record, uchar *recpos, int flag) { - ulong blength, pos2, pos_hashnr, lastpos_hashnr, key_pos; + ulong blength, pos2, pos_hashnr, lastpos_hashnr, key_pos, rec_hash; HASH_INFO *lastpos,*gpos,*pos,*pos3,*empty,*last_ptr; HP_SHARE *share=info->s; DBUG_ENTER("hp_delete_key"); @@ -116,14 +157,20 @@ int hp_delete_key(HP_INFO *info, register HP_KEYDEF *keyinfo, last_ptr=0; /* Search after record with key */ - key_pos= hp_mask(hp_rec_hashnr(keyinfo, record), blength, share->records + 1); + rec_hash= hp_rec_hashnr(0, keyinfo, record); + key_pos= hp_mask(rec_hash, blength, share->records + 1); pos= hp_find_hash(&keyinfo->block, key_pos); gpos = pos3 = 0; while (pos->ptr_to_rec != recpos) { - if (flag && !hp_rec_key_cmp(keyinfo, record, pos->ptr_to_rec)) + /* + Hash pre-check avoids expensive blob materialization + for non-matching entries. + */ + if (flag && pos->hash_of_key == rec_hash && + !hp_rec_key_cmp(keyinfo, record, pos->ptr_to_rec, info)) last_ptr=pos; /* Previous same key */ gpos=pos; if (!(pos=pos->next_key)) diff --git a/storage/heap/hp_extra.c b/storage/heap/hp_extra.c index 3c554fe98e780..901bace2fbef3 100644 --- a/storage/heap/hp_extra.c +++ b/storage/heap/hp_extra.c @@ -45,6 +45,12 @@ int heap_extra(register HP_INFO *info, enum ha_extra_function function) case HA_EXTRA_CHANGE_KEY_TO_DUP: heap_extra_keyflag(info, function); break; + case HA_EXTRA_WRITE_CAN_REPLACE: + info->s->write_can_replace= TRUE; + break; + case HA_EXTRA_WRITE_CANNOT_REPLACE: + info->s->write_can_replace= FALSE; + break; default: break; } @@ -54,11 +60,24 @@ int heap_extra(register HP_INFO *info, enum ha_extra_function function) int heap_reset(HP_INFO *info) { + hp_flush_pending_blob_free(info); info->lastinx= -1; info->current_record= (ulong) ~0L; info->current_hash_ptr=0; info->update=0; info->next_block=0; + if (info->blob_buff) + { + my_free(info->blob_buff); + info->blob_buff= NULL; + info->blob_buff_len= 0; + } + if (info->key_blob_buff) + { + my_free(info->key_blob_buff); + info->key_blob_buff= NULL; + info->key_blob_buff_len= 0; + } return 0; } diff --git a/storage/heap/hp_hash.c b/storage/heap/hp_hash.c index c83aaaedc2ebf..bcd0f1cd73506 100644 --- a/storage/heap/hp_hash.c +++ b/storage/heap/hp_hash.c @@ -20,6 +20,11 @@ #include "heapdef.h" #include +ulong hp_hashnr(HP_KEYDEF *keydef, const uchar *key); + +/* Size of a pointer, for use in memcpy to avoid -Wsizeof-pointer-memaccess */ +#define HP_PTR_SIZE sizeof(void*) + static inline size_t hp_charpos(CHARSET_INFO *cs, const uchar *b, const uchar *e, size_t num) @@ -28,7 +33,16 @@ hp_charpos(CHARSET_INFO *cs, const uchar *b, const uchar *e, size_t num) } -static ulong hp_hashnr(HP_KEYDEF *keydef, const uchar *key); +/* + Read blob data length using actual packlength stored in seg->bit_start. +*/ + +static inline uint32 hp_blob_key_length(uint packlength, const uchar *pos) +{ + return (uint32) read_lowendian(pos, packlength); +} + + /* Find out how many rows there is in the given range @@ -119,15 +133,23 @@ uchar *hp_search(HP_INFO *info, HP_KEYDEF *keyinfo, const uchar *key, if (share->records) { + ulong key_hash= hp_hashnr(keyinfo, key); ulong search_pos= - hp_mask(hp_hashnr(keyinfo, key), share->blength, share->records); + hp_mask(key_hash, share->blength, share->records); pos=hp_find_hash(&keyinfo->block, search_pos); if (search_pos != hp_mask(pos->hash_of_key, share->blength, share->records)) goto not_found; /* Wrong link */ + /* + Save hash for hp_search_next() to reuse without recomputing. + Pre-check hash_of_key before hp_key_cmp() to avoid expensive + blob materialization for non-matching entries. + */ + info->last_hash_of_key= key_hash; do { - if (!hp_key_cmp(keyinfo, pos->ptr_to_rec, key)) + if (pos->hash_of_key == key_hash && + !hp_key_cmp(keyinfo, pos->ptr_to_rec, key, info)) { switch (nextflag) { case 0: /* Search after key */ @@ -188,7 +210,8 @@ uchar *hp_search_next(HP_INFO *info, HP_KEYDEF *keyinfo, const uchar *key, while ((pos= pos->next_key)) { - if (! hp_key_cmp(keyinfo, pos->ptr_to_rec, key)) + if (pos->hash_of_key == info->last_hash_of_key && + ! hp_key_cmp(keyinfo, pos->ptr_to_rec, key, info)) { info->current_hash_ptr=pos; DBUG_RETURN (info->current_ptr= pos->ptr_to_rec); @@ -222,7 +245,7 @@ void hp_movelink(HASH_INFO *pos, HASH_INFO *next_link, HASH_INFO *newlink) /* Calc hashvalue for a key */ -static ulong hp_hashnr(HP_KEYDEF *keydef, const uchar *key) +ulong hp_hashnr(HP_KEYDEF *keydef, const uchar *key) { /*register*/ my_hasher_st hasher= my_hasher_mysql5x(); @@ -231,16 +254,15 @@ static ulong hp_hashnr(HP_KEYDEF *keydef, const uchar *key) for (seg=keydef->seg,endseg=seg+keydef->keysegs ; seg < endseg ; seg++) { uchar *pos=(uchar*) key; - key+=seg->length; + key+= seg->length; if (seg->null_bit) { key++; /* Skip null byte */ if (*pos) /* Found null */ { hasher.m_nr1^= (hasher.m_nr1 << 1) | 1; - /* Add key pack length (2) to key for VARCHAR segments */ - if (seg->type == HA_KEYTYPE_VARTEXT1) - key+= 2; + if (seg->type != HA_KEYTYPE_BIT) + key+= seg->bit_length; continue; } pos++; @@ -260,8 +282,9 @@ static ulong hp_hashnr(HP_KEYDEF *keydef, const uchar *key) else if (seg->type == HA_KEYTYPE_VARTEXT1) /* Any VARCHAR segments */ { CHARSET_INFO *cs= seg->charset; - size_t pack_length= 2; /* Key packing is constant */ + size_t pack_length= 2; /* Key packing is constant */ size_t length= uint2korr(pos); + if (cs->mbmaxlen > 1) { size_t char_length; @@ -271,7 +294,17 @@ static ulong hp_hashnr(HP_KEYDEF *keydef, const uchar *key) set_if_smaller(length, char_length); } my_ci_hash_sort(&hasher, cs, pos+pack_length, length); - key+= pack_length; + key+= seg->bit_length; + } + else if (seg->type == HA_KEYTYPE_VARTEXT4) /* All blob segments */ + { + /* Blob segment in pre-built key: 4-byte length + data pointer */ + CHARSET_INFO *cs= seg->charset; + uint32 blob_len= uint4korr(pos); + const uchar *blob_data; + memcpy(&blob_data, pos + 4, HP_PTR_SIZE); + if (blob_len > 0) + my_ci_hash_sort(&hasher, cs, blob_data, blob_len); } else { @@ -287,16 +320,33 @@ static ulong hp_hashnr(HP_KEYDEF *keydef, const uchar *key) return((ulong) hasher.m_nr1); } - /* Calc hashvalue for a key in a record */ +/* + Calc hashvalue for a key in a record -ulong hp_rec_hashnr(register HP_KEYDEF *keydef, register const uchar *rec) + This function should be kept in sync with ha_rec_hashnr_stored() + + @param info Table handler, in case of compute of hash for stored + record. 0 otherwise. + @param keydef Key definition + @param rec Stored record (ptr_to_rec from hash entry) + @return hash value, or 0 on OOM (forces hash mismatch) + + If info <> 0 we compute hash of a stored record, materializing blob + data from chains. The stored record's blob pointers point to chain + head records, not to raw data. + This avoids keeping multiple materialized blob pointers alive + simultaneously -- key_blob_buff holds only one blob at a time. +*/ + +ulong hp_rec_hashnr(HP_INFO *info, HP_KEYDEF *keydef, const uchar *rec) { my_hasher_st hasher= my_hasher_mysql5x(); HA_KEYSEG *seg,*endseg; for (seg=keydef->seg,endseg=seg+keydef->keysegs ; seg < endseg ; seg++) { - uchar *pos=(uchar*) rec+seg->start,*end=pos+seg->length; + const uchar *pos= rec+seg->start; + const uchar *end= pos+seg->length; if (seg->null_bit) { if (rec[seg->null_pos] & seg->null_bit) @@ -313,7 +363,7 @@ ulong hp_rec_hashnr(register HP_KEYDEF *keydef, register const uchar *rec) { char_length= hp_charpos(cs, pos, pos + char_length, char_length / cs->mbmaxlen); - set_if_smaller(char_length, seg->length); /* QQ: ok to remove? */ + set_if_smaller(char_length, seg->length); } my_ci_hash_sort(&hasher, cs, pos, char_length); } @@ -321,7 +371,11 @@ ulong hp_rec_hashnr(register HP_KEYDEF *keydef, register const uchar *rec) { CHARSET_INFO *cs= seg->charset; size_t pack_length= seg->bit_start; - size_t length= (pack_length == 1 ? (size_t) *(uchar*) pos : uint2korr(pos)); + size_t length= (pack_length == 1 ? + (size_t) *(uchar*) pos : + uint2korr(pos)); + DBUG_ASSERT(!(seg->flag & HA_BLOB_PART)); + if (cs->mbmaxlen > 1) { size_t char_length; @@ -334,11 +388,42 @@ ulong hp_rec_hashnr(register HP_KEYDEF *keydef, register const uchar *rec) set_if_smaller(length, seg->length); my_ci_hash_sort(&hasher, cs, pos+pack_length, length); } + else if (seg->type == HA_KEYTYPE_VARTEXT4 || + seg->type == HA_KEYTYPE_VARBINARY4) + { + /* Blob segment in input record: dereference data pointer */ + uint packlength= seg->bit_start; + uint32 blob_len= hp_blob_key_length(packlength, pos); + if (blob_len > 0) + { + const uchar *data; + if (info) + { + /* Compute blob for unpacked record */ + const uchar *chain; + memcpy(&chain, pos + packlength, sizeof(chain)); + data= hp_materialize_one_blob(info, chain, blob_len); + DBUG_ASSERT(data); + if (!data) + return 0; + my_ci_hash_sort(&hasher, seg->charset, data, blob_len); + } + else + { + /* Compute blob for record given by upper level */ + const uchar *data; + memcpy(&data, pos + packlength, HP_PTR_SIZE); + my_ci_hash_sort(&hasher, seg->charset, data, blob_len); + } + } + } else { if (seg->type == HA_KEYTYPE_BIT && seg->bit_length) { - MY_HASH_ADD_MARIADB(hasher.m_nr1, hasher.m_nr2, *pos); + uchar bits= get_rec_bits(rec + seg->bit_pos, + seg->bit_start, seg->bit_length); + MY_HASH_ADD_MARIADB(hasher.m_nr1, hasher.m_nr2, bits); end--; } @@ -356,24 +441,23 @@ ulong hp_rec_hashnr(register HP_KEYDEF *keydef, register const uchar *rec) /* - Compare keys for two records. Returns 0 if they are identical - - SYNOPSIS - hp_rec_key_cmp() - keydef Key definition - rec1 Record to compare - rec2 Other record to compare - - NOTES - diff_if_only_endspace_difference is used to allow us to insert - 'a' and 'a ' when there is an an unique key. - - RETURN - 0 Key is identical - <> 0 Key differs + Compare two records using key segments. + + @param keydef Key definition + @param rec1 First record (input) - blob fields contain direct data + pointers to caller-owned memory + @param rec2 Second record - when info is non-NULL, blob fields + contain continuation chain pointers (stored format) that + are materialized via hp_materialize_one_blob(). + When info is NULL, treated same as rec1. + @param info When non-NULL, enables stored-blob materialization for rec2. + Must be NULL when both records are input records. + + @return 0 if records are equal by all key segments, 1 otherwise */ -int hp_rec_key_cmp(HP_KEYDEF *keydef, const uchar *rec1, const uchar *rec2) +int hp_rec_key_cmp(HP_KEYDEF *keydef, const uchar *rec1, const uchar *rec2, + HP_INFO *info) { HA_KEYSEG *seg,*endseg; @@ -411,6 +495,51 @@ int hp_rec_key_cmp(HP_KEYDEF *keydef, const uchar *rec1, const uchar *rec2) pos2, char_length2)) return 1; } + else if (seg->type == HA_KEYTYPE_VARTEXT4) /* All blob segments */ + { + /* + Blob segment comparison. + rec1 always has valid blob pointers (input record). + rec2 may be stored (chain pointers) when info != NULL. + */ + uint packlength= seg->bit_start; + uchar *pos1= (uchar*) rec1 + seg->start; + uchar *pos2= (uchar*) rec2 + seg->start; + uint32 len1= hp_blob_key_length(packlength, pos1); + uint32 len2= hp_blob_key_length(packlength, pos2); + const uchar *data1; + const uchar *data2; + + if (len1 == 0 && len2 == 0) + continue; + /* + Only short-circuit on length mismatch for NO PAD collations. + PAD SPACE collations treat trailing spaces as insignificant, + so 'a' (len=1) and 'a ' (len=3) must compare equal. + */ + if ((seg->charset->state & MY_CS_NOPAD) && len1 != len2) + return 1; + + /* rec1: always input -- dereference pointer */ + memcpy(&data1, pos1 + packlength, HP_PTR_SIZE); + + /* rec2: if info != NULL, it's stored -- materialize from chain */ + if (info) + { + const uchar *chain2; + memcpy(&chain2, pos2 + packlength, HP_PTR_SIZE); + data2= hp_materialize_one_blob(info, chain2, (uint32) len2); + if (!data2) + return 1; + } + else + { + memcpy(&data2, pos2 + packlength, HP_PTR_SIZE); + } + + if (my_ci_strnncollsp(seg->charset, data1, len1, data2, len2)) + return 1; + } else if (seg->type == HA_KEYTYPE_VARTEXT1) /* Any VARCHAR segments */ { uchar *pos1= (uchar*) rec1 + seg->start; @@ -493,7 +622,8 @@ int hp_rec_key_cmp(HP_KEYDEF *keydef, const uchar *rec1, const uchar *rec2) /* Compare a key in a record to a whole key */ -int hp_key_cmp(HP_KEYDEF *keydef, const uchar *rec, const uchar *key) +int hp_key_cmp(HP_KEYDEF *keydef, const uchar *rec, const uchar *key, + HP_INFO *info) { HA_KEYSEG *seg,*endseg; @@ -508,9 +638,8 @@ int hp_key_cmp(HP_KEYDEF *keydef, const uchar *rec, const uchar *key) return 1; if (found_null) { - /* Add key pack length (2) to key for VARCHAR segments */ - if (seg->type == HA_KEYTYPE_VARTEXT1) - key+= 2; + if (seg->type != HA_KEYTYPE_BIT) + key+= seg->bit_length; continue; } } @@ -533,12 +662,47 @@ int hp_key_cmp(HP_KEYDEF *keydef, const uchar *rec, const uchar *key) char_length_key= seg->length; char_length_rec= seg->length; } - + if (my_ci_strnncollsp(seg->charset, pos, char_length_rec, key, char_length_key)) return 1; } + else if (seg->type == HA_KEYTYPE_VARTEXT4) + { + /* + Blob segment: rec side uses packlength-byte (1-4) length prefix + followed by a chain pointer; key side is the normalized format + from hp_make_key(): always 4-byte length + data pointer. + */ + uint packlength= seg->bit_start; + uchar *pos= (uchar*) rec + seg->start; + uint32 rec_blob_len= hp_blob_key_length(packlength, pos); + uint32 key_blob_len= uint4korr(key); + const uchar *key_data; + const uchar *rec_data; + + memcpy(&key_data, key + 4, HP_PTR_SIZE); + + if (rec_blob_len == 0 && key_blob_len == 0) + continue; + if ((seg->charset->state & MY_CS_NOPAD) && rec_blob_len != key_blob_len) + return 1; + + /* rec is stored -- materialize from chain */ + { + const uchar *chain; + memcpy(&chain, pos + packlength, HP_PTR_SIZE); + rec_data= hp_materialize_one_blob(info, chain, (uint32) rec_blob_len); + if (!rec_data) + return 1; + } + + if (my_ci_strnncollsp(seg->charset, + rec_data, rec_blob_len, + key_data, key_blob_len)) + return 1; + } else if (seg->type == HA_KEYTYPE_VARTEXT1) /* Any VARCHAR segments */ { uchar *pos= (uchar*) rec + seg->start; @@ -552,7 +716,7 @@ int hp_key_cmp(HP_KEYDEF *keydef, const uchar *rec, const uchar *key) key+= 2; /* skip key pack length */ if (cs->mbmaxlen > 1 && !(cs->state & MY_CS_NOPAD)) { - size_t nchars= seg->length / cs->mbmaxlen; + size_t nchars= seg->length / cs->mbmaxlen; if (my_ci_strnncollsp_nchars(cs, pos, char_length_rec, key, char_length_key, @@ -614,6 +778,22 @@ void hp_make_key(HP_KEYDEF *keydef, uchar *key, const uchar *rec) uchar *pos= (uchar*) rec + seg->start; if (seg->null_bit) *key++= MY_TEST(rec[seg->null_pos] & seg->null_bit); + if (seg->type == HA_KEYTYPE_VARTEXT4) + { + /* + Blob segment: read packlength-byte (1-4) length from input record, + normalize to 4-byte length + data pointer in key buffer for + hp_hashnr/hp_key_cmp. + */ + uint packlength= seg->bit_start; + uint32 blob_len= hp_blob_key_length(packlength, pos); + const uchar *blob_data; + memcpy(&blob_data, pos + packlength, HP_PTR_SIZE); + int4store(key, blob_len); + memcpy(key + 4, &blob_data, HP_PTR_SIZE); + key+= 4 + portable_sizeof_char_ptr; + continue; + } if (cs->mbmaxlen > 1) { char_length= hp_charpos(cs, pos, pos + seg->length, @@ -621,7 +801,24 @@ void hp_make_key(HP_KEYDEF *keydef, uchar *key, const uchar *rec) set_if_smaller(char_length, seg->length); /* QQ: ok to remove? */ } if (seg->type == HA_KEYTYPE_VARTEXT1) - char_length+= seg->bit_start; /* Copy also length */ + { + /* + Normalize VARCHAR to always use a 2-byte length prefix in the key + buffer, regardless of whether the record uses 1-byte or 2-byte + packing. This keeps the key format consistent with what + hp_hashnr() and hp_key_cmp() expect (they always read 2 bytes). + */ + uint native_pack= seg->bit_start; + size_t data_len= (native_pack == 1 ? (size_t) *(uchar*) pos + : uint2korr(pos)); + set_if_smaller(data_len, char_length); + int2store(key, (uint16) data_len); + memcpy(key + 2, pos + native_pack, data_len); + if (data_len < char_length) + bzero(key + 2 + data_len, char_length - data_len); + key+= 2 + char_length; + continue; + } else if (seg->type == HA_KEYTYPE_BIT && seg->bit_length) { *key++= get_rec_bits(rec + seg->bit_pos, @@ -767,6 +964,7 @@ uint hp_rb_pack_key(HP_KEYDEF *keydef, uchar *key, const uchar *old, } continue; } + DBUG_ASSERT(!(seg->flag & HA_BLOB_PART)); if (seg->flag & (HA_VAR_LENGTH_PART | HA_BLOB_PART)) { /* Length of key-part used with heap_rkey() always 2 */ diff --git a/storage/heap/hp_open.c b/storage/heap/hp_open.c index 272c4a3af230e..c512b88efd369 100644 --- a/storage/heap/hp_open.c +++ b/storage/heap/hp_open.c @@ -46,6 +46,19 @@ HP_INFO *heap_open_from_share(HP_SHARE *share, int mode) info->mode= mode; info->current_record= (ulong) ~0L; /* No current record */ info->lastinx= info->errkey= -1; + if (share->blob_count && !share->internal) + { + info->pending_blob_chains= + (uchar**) my_malloc(hp_key_memory_HP_BLOB, + share->blob_count * sizeof(uchar*), + MYF(MY_ZEROFILL)); + if (!info->pending_blob_chains) + { + share->open_count--; + my_free(info); + DBUG_RETURN(0); + } + } #ifndef DBUG_OFF info->opt_flag= READ_CHECK_USED; /* Check when changing */ #endif diff --git a/storage/heap/hp_rfirst.c b/storage/heap/hp_rfirst.c index 60596a2c650fd..903fd42a135ed 100644 --- a/storage/heap/hp_rfirst.c +++ b/storage/heap/hp_rfirst.c @@ -38,6 +38,8 @@ int heap_rfirst(HP_INFO *info, uchar *record, int inx) sizeof(uchar*)); info->current_ptr = pos; memcpy(record, pos, (size_t)share->reclength); + if (share->blob_count && hp_read_blobs(info, record, pos)) + DBUG_RETURN(my_errno); /* If we're performing index_first on a table that was taken from table cache, info->lastkey_len is initialized to previous query. diff --git a/storage/heap/hp_rkey.c b/storage/heap/hp_rkey.c index 2d9fae4c52097..8a9b515d8af04 100644 --- a/storage/heap/hp_rkey.c +++ b/storage/heap/hp_rkey.c @@ -29,6 +29,7 @@ int heap_rkey(HP_INFO *info, uchar *record, int inx, const uchar *key, { DBUG_RETURN(my_errno= HA_ERR_WRONG_INDEX); } + DBUG_ASSERT(!(keyinfo->flag & HA_NO_KEY_READ)); info->lastinx= inx; info->current_record= (ulong) ~0L; /* For heap_rrnd() */ info->key_version= info->s->key_version; @@ -69,6 +70,8 @@ int heap_rkey(HP_INFO *info, uchar *record, int inx, const uchar *key, memcpy(info->lastkey, key, (size_t) keyinfo->length); } memcpy(record, pos, (size_t) share->reclength); + if (share->blob_count && hp_read_blobs(info, record, pos)) + DBUG_RETURN(my_errno); info->update= HA_STATE_AKTIV; DBUG_RETURN(0); } diff --git a/storage/heap/hp_rlast.c b/storage/heap/hp_rlast.c index ed9c3499d5e84..5b31bfccf07c0 100644 --- a/storage/heap/hp_rlast.c +++ b/storage/heap/hp_rlast.c @@ -38,6 +38,8 @@ int heap_rlast(HP_INFO *info, uchar *record, int inx) sizeof(uchar*)); info->current_ptr = pos; memcpy(record, pos, (size_t)share->reclength); + if (share->blob_count && hp_read_blobs(info, record, pos)) + DBUG_RETURN(my_errno); info->update = HA_STATE_AKTIV; } else diff --git a/storage/heap/hp_rnext.c b/storage/heap/hp_rnext.c index ac21ed83da271..774731624fd96 100644 --- a/storage/heap/hp_rnext.c +++ b/storage/heap/hp_rnext.c @@ -127,6 +127,8 @@ int heap_rnext(HP_INFO *info, uchar *record) DBUG_RETURN(my_errno); } memcpy(record,pos,(size_t) share->reclength); + if (share->blob_count && hp_read_blobs(info, record, pos)) + DBUG_RETURN(my_errno); info->update=HA_STATE_AKTIV | HA_STATE_NEXT_FOUND; DBUG_RETURN(0); } diff --git a/storage/heap/hp_rprev.c b/storage/heap/hp_rprev.c index cc81d179570aa..948d1db15ec53 100644 --- a/storage/heap/hp_rprev.c +++ b/storage/heap/hp_rprev.c @@ -94,6 +94,8 @@ int heap_rprev(HP_INFO *info, uchar *record) DBUG_RETURN(my_errno); } memcpy(record,pos,(size_t) share->reclength); + if (share->blob_count && hp_read_blobs(info, record, pos)) + DBUG_RETURN(my_errno); info->update=HA_STATE_AKTIV | HA_STATE_PREV_FOUND; DBUG_RETURN(0); } diff --git a/storage/heap/hp_rrnd.c b/storage/heap/hp_rrnd.c index 3947946ce6706..045804a94afe7 100644 --- a/storage/heap/hp_rrnd.c +++ b/storage/heap/hp_rrnd.c @@ -44,6 +44,8 @@ int heap_rrnd(register HP_INFO *info, uchar *record, uchar *pos) } info->update=HA_STATE_PREV_FOUND | HA_STATE_NEXT_FOUND | HA_STATE_AKTIV; memcpy(record,info->current_ptr,(size_t) share->reclength); + if (share->blob_count && hp_read_blobs(info, record, info->current_ptr)) + DBUG_RETURN(my_errno); DBUG_PRINT("exit", ("found record at %p", info->current_ptr)); info->current_hash_ptr=0; /* Can't use rnext */ DBUG_RETURN(0); diff --git a/storage/heap/hp_rsame.c b/storage/heap/hp_rsame.c index 8bba4cd23a9c1..1ab2511d617ba 100644 --- a/storage/heap/hp_rsame.c +++ b/storage/heap/hp_rsame.c @@ -49,6 +49,8 @@ int heap_rsame(register HP_INFO *info, uchar *record, int inx) } } memcpy(record,info->current_ptr,(size_t) share->reclength); + if (share->blob_count && hp_read_blobs(info, record, info->current_ptr)) + DBUG_RETURN(my_errno); DBUG_RETURN(0); } info->update=0; diff --git a/storage/heap/hp_scan.c b/storage/heap/hp_scan.c index f07efe6cf671c..486a146eec9c8 100644 --- a/storage/heap/hp_scan.c +++ b/storage/heap/hp_scan.c @@ -43,6 +43,27 @@ int heap_scan(register HP_INFO *info, uchar *record) ulong pos; DBUG_ENTER("heap_scan"); + /* + Scan boundary: total_records + deleted == block.last_allocated. + + Every slot in the HP_BLOCK data area is either a live record (counted in + total_records) or a deleted/free slot (counted in deleted). This + includes blob continuation records allocated by hp_alloc_from_tail() + and freed by hp_free_run_chain(), both of which maintain the invariant + total_records + deleted == block.last_allocated. + + next_block is a cached upper bound for the current HP_BLOCK segment: + within one segment, current_ptr can be advanced by recbuffer without + calling hp_find_record(). It MUST satisfy + next_block <= total_records + deleted + at all times, otherwise the scan will walk past the last allocated + slot into unmapped memory. + + The else branch below recomputes next_block and caps it. Any code + that manipulates next_block externally (e.g. restart_rnd_next) must + also enforce this cap. + */ +retry: pos= ++info->current_record; if (pos < info->next_block) { @@ -50,12 +71,18 @@ int heap_scan(register HP_INFO *info, uchar *record) } else { - /* increase next_block to the next records_in_block boundary */ + /* Advance next_block to the next records_in_block boundary */ ulong rem= info->next_block % share->block.records_in_block; info->next_block+=share->block.records_in_block - rem; - if (info->next_block >= share->records+share->deleted) + /* + Cap next_block at the scan end (total_records + deleted). This is + essential: rows may have been deleted since next_block was last set + (e.g. remove_dup_with_compare deletes duplicates mid-scan), and + block boundaries can extend well past the last allocated slot. + */ + if (info->next_block >= share->total_records+share->deleted) { - info->next_block= share->records+share->deleted; + info->next_block= share->total_records+share->deleted; if (pos >= info->next_block) { info->update= 0; @@ -70,8 +97,32 @@ int heap_scan(register HP_INFO *info, uchar *record) info->update= HA_STATE_PREV_FOUND | HA_STATE_NEXT_FOUND; DBUG_RETURN(my_errno=HA_ERR_RECORD_DELETED); } + /* + Skip blob continuation runs internally -- advance past the entire run + and retry from the top rather than returning to the caller. + + Rec 0 of each run has the flags byte with HP_ROW_IS_CONT set; inner + records (rec 1..N-1) have no flags byte. Read run_rec_count from the + header and skip the entire run. + */ + if (hp_is_cont(info->current_ptr, share->visible)) + { + if (!hp_is_single_rec(info->current_ptr, share->visible)) + { + uint16 run_rec_count= hp_cont_rec_count(info->current_ptr); + if (run_rec_count > 1) + { + uint skip= run_rec_count - 1; + info->current_record+= skip; + info->current_ptr+= skip * share->block.recbuffer; + } + } + goto retry; + } info->update= HA_STATE_PREV_FOUND | HA_STATE_NEXT_FOUND | HA_STATE_AKTIV; memcpy(record,info->current_ptr,(size_t) share->reclength); + if (share->blob_count && hp_read_blobs(info, record, info->current_ptr)) + DBUG_RETURN(my_errno); info->current_hash_ptr=0; /* Can't use read_next */ DBUG_RETURN(0); } /* heap_scan */ diff --git a/storage/heap/hp_static.c b/storage/heap/hp_static.c index 9a4410eead9ea..07c9f25597122 100644 --- a/storage/heap/hp_static.c +++ b/storage/heap/hp_static.c @@ -28,6 +28,7 @@ PSI_memory_key hp_key_memory_HP_SHARE; PSI_memory_key hp_key_memory_HP_INFO; PSI_memory_key hp_key_memory_HP_PTRS; PSI_memory_key hp_key_memory_HP_KEYDEF; +PSI_memory_key hp_key_memory_HP_BLOB; #ifdef HAVE_PSI_INTERFACE @@ -36,7 +37,8 @@ static PSI_memory_info all_heap_memory[]= { & hp_key_memory_HP_SHARE, "HP_SHARE", 0}, { & hp_key_memory_HP_INFO, "HP_INFO", 0}, { & hp_key_memory_HP_PTRS, "HP_PTRS", 0}, - { & hp_key_memory_HP_KEYDEF, "HP_KEYDEF", 0} + { & hp_key_memory_HP_KEYDEF, "HP_KEYDEF", 0}, + { & hp_key_memory_HP_BLOB, "HP_BLOB", 0} }; void init_heap_psi_keys() diff --git a/storage/heap/hp_test_concurrent-t.c b/storage/heap/hp_test_concurrent-t.c new file mode 100644 index 0000000000000..c1d0fa8c5ac66 --- /dev/null +++ b/storage/heap/hp_test_concurrent-t.c @@ -0,0 +1,131 @@ +/* + Unit test: concurrent blob insert/delete on shared HEAP table. + + Two threads share one HP_SHARE via separate HP_INFO handles and + do insert/delete cycles with blobs. A pthread mutex serializes + all operations on the shared table, and hp_flush_pending_blob_free() + is called while the mutex is still held -- matching what + ha_heap::external_lock(F_UNLCK) does under thr_lock. + + Without the external_lock flush fix (i.e. if the mutex is released + before flushing), this test crashes with SIGSEGV from concurrent + del_link corruption. With the fix (flush under lock), it passes. +*/ + +#include "hp_test_helpers.h" + +#define ITERATIONS 5000 + +static pthread_mutex_t table_mutex; +static volatile int go; + +typedef struct +{ + HP_INFO *info; + int base_id; +} thread_arg_t; + + +static void spin_wait(void) +{ + while (!go) + ; +} + + +static void *thread_worker(void *arg) +{ + thread_arg_t *ta= (thread_arg_t *) arg; + HP_INFO *info= ta->info; + int base= ta->base_id; + uchar rec[REC_LENGTH]; + uchar blob_data[500]; + int i; + + spin_wait(); + + for (i= 0; i < ITERATIONS; i++) + { + uchar key[4]; + int32 id= base + (i % 500); + + memset(blob_data, 'A' + (i % 26), sizeof(blob_data)); + build_record(rec, id, blob_data, sizeof(blob_data)); + + pthread_mutex_lock(&table_mutex); + + if (heap_write(info, rec) != 0) + { + int4store(key, id); + if (heap_rkey(info, rec, 0, key, 4, HA_READ_KEY_EXACT) == 0) + { + heap_delete(info, rec); + hp_flush_pending_blob_free(info); + } + pthread_mutex_unlock(&table_mutex); + continue; + } + + int4store(key, id); + if (heap_rkey(info, rec, 0, key, 4, HA_READ_KEY_EXACT) == 0) + { + heap_delete(info, rec); + hp_flush_pending_blob_free(info); + } + + pthread_mutex_unlock(&table_mutex); + } + return NULL; +} + + +int main(int argc __attribute__((unused)), + char **argv __attribute__((unused))) +{ + HP_SHARE *share; + HP_INFO *info1, *info2; + pthread_t t1, t2; + thread_arg_t arg1, arg2; + + plan(1); + MY_INIT("hp_test_concurrent-t"); + pthread_mutex_init(&table_mutex, NULL); + + if (create_and_open("test_concurrent", &share, &info1)) + { + ok(0, "setup failed"); + return exit_status(); + } + + info2= heap_open("test_concurrent", 2); + if (!info2) + { + ok(0, "second open failed"); + heap_close(info1); + return exit_status(); + } + heap_extra(info2, HA_EXTRA_NO_READCHECK); + + go= 0; + + arg1.info= info1; + arg1.base_id= 10000; + arg2.info= info2; + arg2.base_id= 20000; + + pthread_create(&t1, NULL, thread_worker, &arg1); + pthread_create(&t2, NULL, thread_worker, &arg2); + + go= 1; + + pthread_join(t1, NULL); + pthread_join(t2, NULL); + + ok(1, "concurrent blob insert/delete completed without crash"); + + heap_close(info2); + heap_drop_table(info1); + pthread_mutex_destroy(&table_mutex); + my_end(0); + return exit_status(); +} diff --git a/storage/heap/hp_test_freelist-t.c b/storage/heap/hp_test_freelist-t.c new file mode 100644 index 0000000000000..d3eb16f9ca4e9 --- /dev/null +++ b/storage/heap/hp_test_freelist-t.c @@ -0,0 +1,835 @@ +/* + Unit tests for free-list contiguity detection in hp_write_one_blob(). + + Verifies that Step 1 (free-list peek) correctly identifies contiguous + groups of 3+ records, not just pairs. +*/ + +#include "hp_test_helpers.h" + + +/* + Test: free-list contiguity detection finds groups > 2 records. + + Scenario: + 1. Insert a row with a 100-byte blob (needs 8 continuation records + in Case B format: 1 header + 7 data records, recbuffer=16). + 2. Delete the row. The continuation chain's 8 records form a + contiguous group on the free list (pushed in ascending address + order by hp_free_run_chain, so LIFO yields descending addresses). + The primary record is pushed on top. + 3. Insert a new row with the same blob size. The primary record + reuses the old primary from the free list head. The blob + allocation (Step 1) should detect the remaining 8 contiguous + continuation records as a single group and reuse them. + 4. Assert that block.last_allocated did NOT grow: all records + came from the free list, nothing from the tail. + + With the prev_pos bug (prev_pos not updated in the contiguity loop), + Step 1 only detects 2-record groups. For a 100-byte blob: + total_records_needed = 7 + min_run_records = min(7, max(ceil(128/16), 2)) = 7 + A 2-record group < 7 causes Step 1 to give up, falling to tail + allocation, which grows last_allocated. +*/ + +static void test_freelist_contiguity_multirecord(void) +{ + HP_SHARE *share; + HP_INFO *info; + uchar rec[REC_LENGTH]; + ulong last_alloc_after_first_insert, last_alloc_after_delete; + + uchar blob_data[100]; + memset(blob_data, 'Q', sizeof(blob_data)); + blob_data[0]= '!'; + blob_data[99]= '?'; + + if (create_and_open("test_freelist_cont", &share, &info)) + { + ok(0, "setup failed: %d", my_errno); + skip(10, "setup failed"); + return; + } + + /* Insert row with 100-byte blob */ + build_record(rec, 1, blob_data, sizeof(blob_data)); + ok(heap_write(info, rec) == 0, "insert row 1 (100-byte blob)"); + + last_alloc_after_first_insert= (ulong) share->block.last_allocated; + ok(last_alloc_after_first_insert >= 9, + "allocated >= 9 records: 1 primary + 8 continuation (got %lu)", + last_alloc_after_first_insert); + + /* Delete row 1 */ + { + uchar key[4]; + int4store(key, 1); + ok(heap_rkey(info, rec, 0, key, 4, HA_READ_KEY_EXACT) == 0, + "found row 1 for deletion"); + ok(heap_delete(info, rec) == 0, "deleted row 1"); + } + + last_alloc_after_delete= (ulong) share->block.last_allocated; + ok(last_alloc_after_delete == last_alloc_after_first_insert, + "last_allocated unchanged after delete (%lu)", + last_alloc_after_delete); + + ok(share->deleted == 1, + "primary on free list, blob chains deferred (deleted=%lu)", + (ulong) share->deleted); + + /* Insert row 2 with same blob size - should fully reuse free list */ + build_record(rec, 2, blob_data, sizeof(blob_data)); + ok(heap_write(info, rec) == 0, "insert row 2 (100-byte blob, free-list reuse)"); + + /* + Key assertion: if contiguity detection works for groups > 2, + the entire continuation chain is recovered from the free list + without any tail allocation. + */ + ok(share->block.last_allocated == last_alloc_after_delete, + "last_allocated unchanged after reinsert: free list fully reused " + "(before=%lu, after=%lu)", + last_alloc_after_delete, (ulong) share->block.last_allocated); + + /* Verify data integrity of the reinserted row */ + { + uchar key[4]; + uchar read_buf[REC_LENGTH]; + uint32 read_len; + const uchar *read_ptr; + + int4store(key, 2); + ok(heap_rkey(info, read_buf, 0, key, 4, HA_READ_KEY_EXACT) == 0, + "found row 2 for verification"); + read_len= uint2korr(read_buf + BLOB_OFFSET); + memcpy(&read_ptr, read_buf + BLOB_OFFSET + BLOB_PACKLEN, sizeof(read_ptr)); + ok(read_len == 100, "blob length == 100 (got %u)", read_len); + ok(memcmp(read_ptr, blob_data, 100) == 0, "blob data matches after free-list reuse"); + } + + ok(heap_check_heap(info, 0) == 0, + "heap_check_heap validates after free-list reuse"); + + heap_drop_table(info); + heap_close(info); +} + + +/* + Test: free-list contiguity across multiple delete-reinsert cycles. + + Performs 3 rounds of insert-delete-reinsert with a 200-byte blob. + In each round, last_allocated must not grow, proving that contiguity + detection consistently reuses the freed continuation chain. +*/ + +static void test_freelist_contiguity_repeated_cycles(void) +{ + HP_SHARE *share; + HP_INFO *info; + uchar rec[REC_LENGTH]; + int round; + + uchar blob_data[200]; + memset(blob_data, 'R', sizeof(blob_data)); + + if (create_and_open("test_freelist_cycles", &share, &info)) + { + ok(0, "setup failed: %d", my_errno); + skip(12, "setup failed"); + return; + } + + /* Initial insert to establish baseline */ + build_record(rec, 1, blob_data, sizeof(blob_data)); + ok(heap_write(info, rec) == 0, "initial insert (200-byte blob)"); + + for (round= 0; round < 3; round++) + { + uchar key[4]; + ulong alloc_before; + + /* Delete current row */ + int4store(key, round + 1); + ok(heap_rkey(info, rec, 0, key, 4, HA_READ_KEY_EXACT) == 0, + "round %d: found row for deletion", round); + ok(heap_delete(info, rec) == 0, + "round %d: deleted row", round); + alloc_before= (ulong) share->block.last_allocated; + + /* Reinsert with new key */ + build_record(rec, round + 2, blob_data, sizeof(blob_data)); + ok(heap_write(info, rec) == 0, + "round %d: reinserted (free-list reuse)", round); + ok(share->block.last_allocated == alloc_before, + "round %d: last_allocated stable (before=%lu, after=%lu)", + round, alloc_before, (ulong) share->block.last_allocated); + } + + heap_drop_table(info); + heap_close(info); +} + + +/* + Test: Step 3 free-list scavenge fallback when tail is full. + + Fills the entire first HP_BLOCK leaf block with 0-byte-blob rows, + then deletes every other row to create a heavily fragmented free + list of non-contiguous individual slots. Locks out tail allocation + by setting max_table_size tight. + + Insert a row with a 50-byte blob. Step 1 gives up (1-slot groups + < min_run_records=4). Step 2 fails (tail at block boundary with + tight max_table_size). Step 3 scavenges individual free-list + slots, writing maximally fragmented Case C chains. +*/ + +static void test_freelist_scavenge_fallback(void) +{ + HP_SHARE *share; + HP_INFO *info; + uchar rec[REC_LENGTH]; + int32 i; + int32 inserted, deleted_count; + int32 records_in_block; + ulong last_alloc_after_fill; + + uchar blob_data[50]; + memset(blob_data, 'F', sizeof(blob_data)); + blob_data[0]= '<'; + blob_data[49]= '>'; + + if (create_and_open("test_scavenge", &share, &info)) + { + ok(0, "setup failed: %d", my_errno); + skip(8, "setup failed"); + return; + } + + records_in_block= (int32) share->block.records_in_block; + + /* Fill the first leaf block with 0-byte blob rows (1 record each) */ + for (i= 0, inserted= 0; i < records_in_block; i++) + { + build_record(rec, i, (const uchar*) "", 0); + if (heap_write(info, rec) != 0) + break; + inserted++; + } + + last_alloc_after_fill= (ulong) share->block.last_allocated; + ok(inserted == records_in_block, + "filled block: %d of %d rows inserted", + inserted, records_in_block); + + /* Delete every other row -- non-contiguous fragmentation */ + deleted_count= 0; + for (i= 0; i < inserted; i+= 2) + { + uchar key[4]; + int4store(key, i); + if (heap_rkey(info, rec, 0, key, 4, HA_READ_KEY_EXACT) == 0 && + heap_delete(info, rec) == 0) + deleted_count++; + } + ok(deleted_count == inserted / 2, + "deleted %d rows (every other)", deleted_count); + ok(share->deleted == (ulong) deleted_count, + "share->deleted == %d", deleted_count); + + /* Lock out tail allocation */ + share->max_table_size= share->data_length + share->index_length; + + ok(share->block.last_allocated == last_alloc_after_fill, + "last_allocated at block boundary (%lu)", last_alloc_after_fill); + + /* + Insert row with 50-byte blob. + total_records_needed = 1 + ceil((50 - 5) / 16) = 4 + min_run_records = min(4, max(ceil(128/16), 2)) = 4 + Step 1: free list has non-contiguous singles -> 1 < 4 -> gives up + Step 2: block boundary + tight max_table_size -> fails + Step 3: scavenges individual free-list slots -> succeeds + */ + build_record(rec, 99999, blob_data, sizeof(blob_data)); + ok(heap_write(info, rec) == 0, + "insert with 50-byte blob via Step 3 scavenge fallback"); + + ok(share->block.last_allocated == last_alloc_after_fill, + "last_allocated unchanged: all records from free list (%lu)", + (ulong) share->block.last_allocated); + + /* Verify data integrity */ + { + uchar key[4]; + uchar read_buf[REC_LENGTH]; + uint32 read_len; + const uchar *read_ptr; + + int4store(key, 99999); + ok(heap_rkey(info, read_buf, 0, key, 4, HA_READ_KEY_EXACT) == 0, + "found row via key lookup after scavenge insert"); + read_len= uint2korr(read_buf + BLOB_OFFSET); + memcpy(&read_ptr, read_buf + BLOB_OFFSET + BLOB_PACKLEN, sizeof(read_ptr)); + ok(read_len == 50, "blob length == 50 (got %u)", read_len); + ok(memcmp(read_ptr, blob_data, 50) == 0, "blob data matches"); + } + + ok(heap_check_heap(info, 0) == 0, + "heap_check_heap validates after Step 3 scavenge insert"); + + heap_drop_table(info); + heap_close(info); +} + + +/* + Test: true capacity exhaustion fails correctly. + + Same setup as test_freelist_scavenge_fallback, but insert a blob + large enough to exhaust BOTH the tail AND the free list. The + insert must fail with HA_ERR_RECORD_FILE_FULL. +*/ + +static void test_true_capacity_exhaustion(void) +{ + HP_SHARE *share; + HP_INFO *info; + uchar rec[REC_LENGTH]; + int32 i; + int32 inserted, deleted_count; + int32 records_in_block; + + /* + Blob large enough to need more records than available free slots. + With records_in_block=1024, deleting 512 rows gives 512 free slots. + A blob needing 600+ continuation records exceeds that. + 600 records x 16 bytes/rec ~ 9600 bytes, minus headers. + Use ~5000 bytes to be safe (needs ~315 records). + We only delete ~10 rows to create very few free slots. + */ + uchar blob_data[5000]; + memset(blob_data, 'X', sizeof(blob_data)); + + if (create_and_open("test_exhaust", &share, &info)) + { + ok(0, "setup failed: %d", my_errno); + skip(4, "setup failed"); + return; + } + + records_in_block= (int32) share->block.records_in_block; + + /* Fill the block */ + for (i= 0, inserted= 0; i < records_in_block; i++) + { + build_record(rec, i, (const uchar*) "", 0); + if (heap_write(info, rec) != 0) + break; + inserted++; + } + ok(inserted == records_in_block, "filled block: %d rows", inserted); + + /* Delete only 10 rows -- not enough free slots for the large blob */ + deleted_count= 0; + for (i= 0; i < 20 && i < inserted; i+= 2) + { + uchar key[4]; + int4store(key, i); + if (heap_rkey(info, rec, 0, key, 4, HA_READ_KEY_EXACT) == 0 && + heap_delete(info, rec) == 0) + deleted_count++; + } + ok(deleted_count == 10, "deleted 10 rows"); + + /* Lock out tail */ + share->max_table_size= share->data_length + share->index_length; + + /* Try to insert a 5000-byte blob -- should fail */ + build_record(rec, 99998, blob_data, sizeof(blob_data)); + ok(heap_write(info, rec) != 0, + "insert with 5000-byte blob correctly fails (not enough free slots)"); + ok(my_errno == HA_ERR_RECORD_FILE_FULL, + "error is HA_ERR_RECORD_FILE_FULL (got %d)", my_errno); + + /* Verify table is still consistent (no corruption from partial rollback) */ + ok(share->records == (ulong)(inserted - deleted_count), + "records count consistent after failed insert (%lu)", + (ulong) share->records); + + heap_drop_table(info); + heap_close(info); +} + + +/* + Test: tail reclaim on failed blob allocation (single block). + + Fills a block to within 5 slots of capacity, locks out new block + allocation, then inserts a blob needing more records than the + remaining tail. The blob partially allocates from the tail, then + fails. hp_shrink_tail() must reclaim the tail-allocated records. +*/ + +static void test_tail_reclaim_single_block(void) +{ + HP_SHARE *share; + HP_INFO *info; + uchar rec[REC_LENGTH]; + int32 i; + int32 rib; + ulong last_alloc_before; + + uchar blob_data[200]; + memset(blob_data, 'T', sizeof(blob_data)); + + if (create_and_open("test_tail_reclaim_sb", &share, &info)) + { + ok(0, "setup failed: %d", my_errno); + skip(8, "setup failed"); + return; + } + + rib= (int32) share->block.records_in_block; + + /* Fill block to rib - 5 */ + for (i= 0; i < rib - 5; i++) + { + build_record(rec, i, (const uchar*) "", 0); + if (heap_write(info, rec) != 0) + break; + } + ok(i == rib - 5, "filled block to rib-5 (%d rows)", i); + + last_alloc_before= (ulong) share->block.last_allocated; + + /* Lock out new block allocation */ + share->max_table_size= share->data_length + share->index_length; + + /* Insert blob needing ~14 records -- only 4 tail slots available for blob */ + build_record(rec, 99999, blob_data, sizeof(blob_data)); + ok(heap_write(info, rec) != 0, + "insert with 200-byte blob fails (not enough tail)"); + ok(my_errno == HA_ERR_RECORD_FILE_FULL, + "error is HA_ERR_RECORD_FILE_FULL (got %d)", my_errno); + + /* + After failed insert: 4 blob continuation records were tail-allocated + then reclaimed by hp_shrink_tail(). The primary record was allocated + (+1 to last_allocated) and freed to the delete list. + */ + ok(share->block.last_allocated == last_alloc_before + 1, + "last_allocated: 4 blob recs reclaimed, primary on free list " + "(expected %lu, got %lu)", + last_alloc_before + 1, (ulong) share->block.last_allocated); + ok(share->deleted == 1, + "deleted == 1 (primary on free list, got %lu)", + (ulong) share->deleted); + ok(share->total_records == (ulong)(rib - 5), + "total_records back to pre-insert (%lu)", + (ulong) share->total_records); + ok(share->total_records + share->deleted == share->block.last_allocated, + "invariant: total_records + deleted == last_allocated"); + + /* Verify existing data readable */ + { + uchar key[4]; + int4store(key, 0); + ok(heap_rkey(info, rec, 0, key, 4, HA_READ_KEY_EXACT) == 0, + "existing row 0 still readable"); + } + + /* Insert a small blob -- should succeed using reclaimed tail */ + { + uchar small_blob[30]; + memset(small_blob, 's', sizeof(small_blob)); + build_record(rec, 88888, small_blob, sizeof(small_blob)); + ok(heap_write(info, rec) == 0, + "small blob insert succeeds using reclaimed tail"); + } + + heap_drop_table(info); + heap_close(info); +} + + +/* + Test: tail reclaim across block boundaries. + + Fills a block to within 5 slots of capacity, allows exactly 2 blocks, + then inserts a blob large enough to span both blocks and require a third. + The allocation fails at the third block. hp_shrink_tail() must reclaim + all records across both blocks and update last_blocks to the first block. +*/ + +static void test_tail_reclaim_cross_block(void) +{ + HP_SHARE *share; + HP_INFO *info; + uchar rec[REC_LENGTH]; + int32 i; + int32 rib; + ulong last_alloc_before; + HP_PTRS *first_block; + uchar *blob_data; + uint32 blob_len; + + if (create_and_open("test_tail_reclaim_xb", &share, &info)) + { + ok(0, "setup failed: %d", my_errno); + skip(8, "setup failed"); + return; + } + + rib= (int32) share->block.records_in_block; + + /* Fill block to rib - 5 */ + for (i= 0; i < rib - 5; i++) + { + build_record(rec, i, (const uchar*) "", 0); + if (heap_write(info, rec) != 0) + break; + } + ok(i == rib - 5, "filled block to rib-5 (%d rows)", i); + + last_alloc_before= (ulong) share->block.last_allocated; + first_block= share->block.level_info[0].last_blocks; + + /* Allow exactly 2 blocks, fail at 3rd */ + share->max_records= (ulong)(2 * rib - 1); + + /* + Blob large enough to need more than 4 + rib continuation records. + Each Case C record holds 16 bytes of payload (inner records) plus + 5 bytes in the first record of each run. Use (rib + 20) * 16 bytes + to ensure it exceeds 2 blocks. + */ + blob_len= (uint32)((rib + 20) * 16); + blob_data= (uchar*) my_malloc(PSI_NOT_INSTRUMENTED, blob_len, MYF(0)); + memset(blob_data, 'X', blob_len); + + build_record(rec, 99999, blob_data, (uint16) MY_MIN(blob_len, 65535)); + ok(heap_write(info, rec) != 0, + "insert with %u-byte blob fails (spans 2 blocks, needs 3rd)", + blob_len); + ok(my_errno == HA_ERR_RECORD_FILE_FULL, + "error is HA_ERR_RECORD_FILE_FULL (got %d)", my_errno); + + /* + All blob continuation records (4 from first block + rib from second) + were reclaimed by hp_shrink_tail(). Only the primary record remains + on the free list. + */ + ok(share->block.last_allocated == last_alloc_before + 1, + "last_allocated: all blob recs reclaimed across blocks " + "(expected %lu, got %lu)", + last_alloc_before + 1, (ulong) share->block.last_allocated); + ok(share->deleted == 1, + "deleted == 1 (primary on free list, got %lu)", + (ulong) share->deleted); + ok(share->total_records + share->deleted == share->block.last_allocated, + "invariant: total_records + deleted == last_allocated"); + + /* last_blocks must have been restored to the first block */ + ok(share->block.level_info[0].last_blocks == first_block, + "last_blocks restored to first block after cross-block reclaim"); + + /* Verify existing data readable */ + { + uchar key[4]; + int4store(key, 0); + ok(heap_rkey(info, rec, 0, key, 4, HA_READ_KEY_EXACT) == 0, + "existing row 0 still readable"); + } + + /* Insert a small blob -- should succeed using reclaimed tail in first block */ + { + uchar small_blob[30]; + memset(small_blob, 's', sizeof(small_blob)); + build_record(rec, 88888, small_blob, sizeof(small_blob)); + ok(heap_write(info, rec) == 0, + "small blob insert succeeds using reclaimed tail in first block"); + } + + my_free(blob_data); + heap_drop_table(info); + heap_close(info); +} + + +/* + Test: tail reclaim across 3 block boundaries. + + Like test 6 but with a blob large enough to span 3 leaf blocks. + Verifies hp_find_block() correctly navigates the block tree when + hp_shrink_tail() crosses two block boundaries sequentially. +*/ + +static void test_tail_reclaim_three_blocks(void) +{ + HP_SHARE *share; + HP_INFO *info; + uchar rec[REC_LENGTH]; + int32 i; + int32 rib; + ulong last_alloc_before; + HP_PTRS *first_block; + uchar *blob_data; + uint32 blob_len; + + if (create_and_open("test_tail_reclaim_3b", &share, &info)) + { + ok(0, "setup failed: %d", my_errno); + skip(8, "setup failed"); + return; + } + + rib= (int32) share->block.records_in_block; + + /* Fill block to rib - 5 */ + for (i= 0; i < rib - 5; i++) + { + build_record(rec, i, (const uchar*) "", 0); + if (heap_write(info, rec) != 0) + break; + } + ok(i == rib - 5, "filled block to rib-5 (%d rows)", i); + + last_alloc_before= (ulong) share->block.last_allocated; + first_block= share->block.level_info[0].last_blocks; + + /* Allow exactly 3 blocks, fail at 4th */ + share->max_records= (ulong)(3 * rib - 1); + + /* + Blob needs more than 4 + 2*rib continuation records to span + 3 blocks and fail requesting the 4th. + */ + blob_len= (uint32)((2 * rib + 20) * 16); + blob_data= (uchar*) my_malloc(PSI_NOT_INSTRUMENTED, blob_len, MYF(0)); + memset(blob_data, 'Z', blob_len); + + build_record(rec, 99999, blob_data, (uint16) MY_MIN(blob_len, 65535)); + ok(heap_write(info, rec) != 0, + "insert with %u-byte blob fails (spans 3 blocks, needs 4th)", + blob_len); + ok(my_errno == HA_ERR_RECORD_FILE_FULL, + "error is HA_ERR_RECORD_FILE_FULL (got %d)", my_errno); + + /* + All blob records across 3 blocks reclaimed: 4 from block 1 + + rib from block 2 + rib from block 3. Two block boundary crossings. + */ + ok(share->block.last_allocated == last_alloc_before + 1, + "last_allocated: all recs reclaimed across 3 blocks " + "(expected %lu, got %lu)", + last_alloc_before + 1, (ulong) share->block.last_allocated); + ok(share->deleted == 1, + "deleted == 1 (got %lu)", (ulong) share->deleted); + ok(share->total_records + share->deleted == share->block.last_allocated, + "invariant: total_records + deleted == last_allocated"); + ok(share->block.level_info[0].last_blocks == first_block, + "last_blocks restored to first block after 2 boundary crossings"); + + /* Verify existing data and insert small blob */ + { + uchar key[4]; + int4store(key, 0); + ok(heap_rkey(info, rec, 0, key, 4, HA_READ_KEY_EXACT) == 0, + "existing row 0 still readable"); + } + { + uchar small_blob[30]; + memset(small_blob, 's', sizeof(small_blob)); + build_record(rec, 88888, small_blob, sizeof(small_blob)); + ok(heap_write(info, rec) == 0, + "small blob insert succeeds after 3-block reclaim"); + } + + my_free(blob_data); + heap_drop_table(info); + heap_close(info); +} + + +/* + Test: orphaned blocks are reused via high_water_allocated. + + After hp_shrink_tail() empties 2 leaf blocks, fills them back up + with non-blob rows. data_length must NOT grow (blocks are reused + via hp_find_block, not freshly allocated via hp_get_new_block). +*/ + +static void test_block_reuse_after_reclaim(void) +{ + HP_SHARE *share; + HP_INFO *info; + uchar rec[REC_LENGTH]; + int32 i; + int32 rib; + ulong last_alloc_before; + ulonglong data_len_after_shrink; + uchar *blob_data; + uint32 blob_len; + + if (create_and_open("test_block_reuse", &share, &info)) + { + ok(0, "setup failed: %d", my_errno); + skip(13, "setup failed"); + return; + } + + rib= (int32) share->block.records_in_block; + + /* Fill block to rib - 5 */ + for (i= 0; i < rib - 5; i++) + { + build_record(rec, i, (const uchar*) "", 0); + if (heap_write(info, rec) != 0) + break; + } + ok(i == rib - 5, "filled block to rib-5 (%d rows)", i); + + last_alloc_before= (ulong) share->block.last_allocated; + + /* Allow exactly 3 blocks, fail at 4th */ + share->max_records= (ulong)(3 * rib - 1); + + /* Blob that spans 3 blocks then fails */ + blob_len= (uint32)((2 * rib + 20) * 16); + blob_data= (uchar*) my_malloc(PSI_NOT_INSTRUMENTED, blob_len, MYF(0)); + memset(blob_data, 'R', blob_len); + + build_record(rec, 99999, blob_data, (uint16) MY_MIN(blob_len, 65535)); + ok(heap_write(info, rec) != 0, + "blob insert fails as expected"); + + ok(share->block.last_allocated == last_alloc_before + 1, + "tail reclaimed after failure"); + ok(share->block.high_water_allocated == (ulong)(3 * rib), + "high_water_allocated set to 3*rib (got %lu, expected %lu)", + (ulong) share->block.high_water_allocated, (ulong)(3 * rib)); + + data_len_after_shrink= share->data_length; + + /* Remove max_records limit so we can fill freely */ + share->max_records= 0; + + /* + Insert 2*rib non-blob rows. The first reuses the free-list slot + (primary from failed insert). The remaining 2*rib-1 extend the + tail, crossing block boundaries at rib and 2*rib. At each boundary, + hp_alloc_from_tail() must REUSE the orphaned block (not allocate new). + Key check: data_length must NOT grow. + */ + { + int32 inserted= 0; + int32 to_insert= 2 * rib; + for (i= 0; i < to_insert; i++) + { + build_record(rec, rib + i, (const uchar*) "", 0); + if (heap_write(info, rec) != 0) + break; + inserted++; + } + ok(inserted == to_insert, + "inserted %d more rows across 2 reused blocks", inserted); + } + + ok(share->data_length == data_len_after_shrink, + "data_length unchanged: blocks reused, not newly allocated " + "(before=%llu, after=%llu)", + data_len_after_shrink, share->data_length); + + /* + last_allocated = (rib - 4) + (2*rib - 1) = 3*rib - 5 + The -4 is the post-shrink starting point (rib-5 data + 1 primary + on free list); the -1 accounts for the first insert reusing the + free-list slot instead of extending the tail. + */ + ok(share->block.last_allocated == (ulong)(3 * rib - 5), + "last_allocated grew through reused blocks (got %lu, expected %lu)", + (ulong) share->block.last_allocated, (ulong)(3 * rib - 5)); + + ok(share->total_records + share->deleted == share->block.last_allocated, + "invariant: total_records + deleted == last_allocated"); + + /* Verify data integrity: read first and last inserted rows */ + { + uchar key[4]; + int4store(key, 0); + ok(heap_rkey(info, rec, 0, key, 4, HA_READ_KEY_EXACT) == 0, + "first row still readable"); + int4store(key, 3 * rib - 6); + ok(heap_rkey(info, rec, 0, key, 4, HA_READ_KEY_EXACT) == 0, + "last inserted row readable"); + } + + /* + Now delete a row and insert a blob that crosses a block boundary. + hp_write_one_blob() -> hp_alloc_from_tail() uses the same reuse path. + high_water_allocated is still 3*rib; current last_allocated is 3*rib-5. + A blob needing 6+ continuation records will cross into the tail of + block 3 (5 slots left). No new block should be allocated. + */ + { + uchar key[4]; + ulonglong data_len_before; + uchar reuse_blob[50]; + memset(reuse_blob, 'B', sizeof(reuse_blob)); + + int4store(key, rib); + ok(heap_rkey(info, rec, 0, key, 4, HA_READ_KEY_EXACT) == 0, + "found row for delete before blob reuse test"); + ok(heap_delete(info, rec) == 0, "deleted row"); + + data_len_before= share->data_length; + build_record(rec, 77777, reuse_blob, sizeof(reuse_blob)); + ok(heap_write(info, rec) == 0, + "blob insert succeeds using reused blocks"); + ok(share->data_length == data_len_before, + "data_length unchanged after blob insert (blocks reused)"); + } + + my_free(blob_data); + heap_drop_table(info); + heap_close(info); +} + + +int main(int argc __attribute__((unused)), + char **argv __attribute__((unused))) +{ + MY_INIT("hp_test_freelist"); + plan(81); + + diag("Test 1: free-list contiguity detects groups > 2 records"); + test_freelist_contiguity_multirecord(); + + diag("Test 2: contiguity across repeated delete-reinsert cycles"); + test_freelist_contiguity_repeated_cycles(); + + diag("Test 3: Step 3 free-list scavenge fallback"); + test_freelist_scavenge_fallback(); + + diag("Test 4: true capacity exhaustion fails correctly"); + test_true_capacity_exhaustion(); + + diag("Test 5: tail reclaim on failed blob (single block)"); + test_tail_reclaim_single_block(); + + diag("Test 6: tail reclaim across block boundaries"); + test_tail_reclaim_cross_block(); + + diag("Test 7: tail reclaim across 3 block boundaries"); + test_tail_reclaim_three_blocks(); + + diag("Test 8: orphaned blocks reused after reclaim"); + test_block_reuse_after_reclaim(); + + my_end(0); + return exit_status(); +} diff --git a/storage/heap/hp_test_hash-t.c b/storage/heap/hp_test_hash-t.c new file mode 100644 index 0000000000000..d640d9002000c --- /dev/null +++ b/storage/heap/hp_test_hash-t.c @@ -0,0 +1,1165 @@ +/* + Unit tests for HEAP hash functions with blob key segments. + + Validates that hp_rec_hashnr() (hashes from a record) and hp_hashnr() + (hashes from a pre-built key via hp_make_key()) produce identical + results for blob data. Also validates hp_rec_key_cmp() and hp_key_cmp() + for blob segments. + + The three blob storage cases (A, B, C) refer to how blobs are stored + in continuation chains, but for hashing purposes what matters is the + record format: packlength bytes of length + sizeof(ptr) bytes of + data pointer. The hash functions read blob data via pointer + dereference, so the tests verify that the pointer dereference and + length handling are correct for various configurations. +*/ + +#include +#include +#include +#include +#include "heap.h" +#include "heapdef.h" + +/* + Record layout for a table (int4, blob(N)): + byte 0: null bitmap (1 byte, bit 2 = blob null) + bytes 1-4: int4 field (4 bytes) + bytes 5-6: blob packlength=2 (length, little-endian) + bytes 7-14: blob data pointer (8 bytes on x86_64) + byte 15: flags byte (at offset = visible = 15) + Total: recbuffer = ALIGN(MAX(16, 8) + 1, 8) = 24 +*/ + +#define REC_NULL_OFFSET 0 +#define REC_INT_OFFSET 1 +#define REC_BLOB_OFFSET 5 +#define REC_BLOB_PACKLEN 2 +#define REC_LENGTH 16 /* reclength: through end of blob descriptor */ +#define REC_VISIBLE 15 /* flags byte offset */ +#define REC_BUFFER 24 /* aligned recbuffer */ + +/* Key buffer: null_byte + 4B_blob_len + 8B_blob_ptr = 13 bytes max */ +#define KEY_BUF_SIZE 64 + +/* + Use sizeof(void*) for memcpy of actual pointer values, matching + HP_PTR_SIZE in hp_hash.c. The record slot is portable_sizeof_char_ptr + bytes wide (always 8), but only sizeof(void*) bytes are meaningful. +*/ +#define PTR_SIZE sizeof(void*) + + +static void setup_blob_keyseg(HA_KEYSEG *seg, my_bool nullable) +{ + memset(seg, 0, sizeof(*seg)); + seg->type= HA_KEYTYPE_VARTEXT4; + seg->flag= HA_BLOB_PART | HA_VAR_LENGTH_PART; + seg->start= REC_BLOB_OFFSET; + seg->length= 4+portable_sizeof_char_ptr; /* Length of blob key */ + seg->bit_start= REC_BLOB_PACKLEN; /* actual packlength */ + seg->charset= &my_charset_latin1; + if (nullable) + { + seg->null_bit= 2; + seg->null_pos= REC_NULL_OFFSET; + } + else + { + seg->null_bit= 0; + } +} + + +static void setup_keydef(HP_KEYDEF *keydef, HA_KEYSEG *seg, uint keysegs) +{ + uint i; + memset(keydef, 0, sizeof(*keydef)); + keydef->keysegs= keysegs; + keydef->seg= seg; + keydef->algorithm= HA_KEY_ALG_HASH; + keydef->flag= HA_NOSAME; + keydef->length= 0; /* computed below */ + + /* Compute keydef->length: sum of key part sizes */ + for (i= 0; i < keysegs; i++) + { + if (seg[i].null_bit) + keydef->length++; + if (seg[i].flag & HA_BLOB_PART) + keydef->length+= 4 + PTR_SIZE; + else if (seg[i].flag & HA_VAR_LENGTH_PART) + keydef->length+= 2 + seg[i].length; + else + keydef->length+= seg[i].length; + } +} + + +/* + Build a record with blob data. + rec must be at least REC_LENGTH bytes. + Sets the blob field to point to blob_data with blob_len bytes. +*/ +static void build_record(uchar *rec, int32 int_val, + const uchar *blob_data, size_t blob_len, + my_bool blob_is_null) +{ + memset(rec, 0, REC_LENGTH); + + /* null bitmap */ + if (blob_is_null) + rec[REC_NULL_OFFSET]= 2; /* null_bit=2 for blob */ + else + rec[REC_NULL_OFFSET]= 0; + + /* int4 field */ + int4store(rec + REC_INT_OFFSET, int_val); + + /* blob field: packlength (2 bytes) + data pointer (8 bytes) */ + int2store(rec + REC_BLOB_OFFSET, blob_len); + memcpy(rec + REC_BLOB_OFFSET + REC_BLOB_PACKLEN, &blob_data, PTR_SIZE); +} + + +/* + Test 1: hp_rec_hashnr and hp_make_key + hp_hashnr produce same hash + for various blob data sizes. +*/ +static void test_hash_consistency(void) +{ + HA_KEYSEG seg; + HP_KEYDEF keydef; + uchar rec[REC_LENGTH]; + uchar key_buf[KEY_BUF_SIZE]; + ulong rec_hash_a, rec_hash_b, rec_hash_c; + + /* Case A: very small blob (fits in single record, <= visible - 10) */ + LEX_CUSTRING data_a= { USTRING_WITH_LEN("Hi") }; + + /* Case B: medium blob (fits in single run, zero-copy) */ + LEX_CUSTRING data_b= { USTRING_WITH_LEN("Hello World! This is a medium blob.") }; + + /* Case C: larger blob data (would need multiple runs in real storage) */ + uchar data_c[200]; + size_t len_c= sizeof(data_c); + memset(data_c, 'X', sizeof(data_c)); + /* Make it non-uniform so hash is more interesting */ + data_c[0]= 'A'; + data_c[50]= 'B'; + data_c[100]= 'C'; + data_c[199]= 'Z'; + + setup_blob_keyseg(&seg, FALSE); + setup_keydef(&keydef, &seg, 1); + + /* --- Case A: small blob --- */ + build_record(rec, 1, data_a.str, data_a.length, FALSE); + + rec_hash_a= hp_rec_hashnr(0, &keydef, rec); + hp_make_key(&keydef, key_buf, rec); + /* Now hash the pre-built key */ + { + /* Verify the key format produced by hp_make_key is correct. */ + uint32 key_blob_len= uint4korr(key_buf); + const uchar *key_blob_data; + memcpy(&key_blob_data, key_buf + 4, PTR_SIZE); + ok(key_blob_len == data_a.length, + "Case A: hp_make_key blob length = %u (expected %u)", + (uint) key_blob_len, (uint) data_a.length); + ok(key_blob_data == data_a.str, + "Case A: hp_make_key blob pointer matches source data"); + ok(memcmp(key_blob_data, data_a.str, data_a.length) == 0, + "Case A: hp_make_key blob data content matches"); + } + + /* --- Case B: medium blob --- */ + build_record(rec, 2, data_b.str, data_b.length, FALSE); + + rec_hash_b= hp_rec_hashnr(0, &keydef, rec); + hp_make_key(&keydef, key_buf, rec); + { + uint32 key_blob_len= uint4korr(key_buf); + const uchar *key_blob_data; + memcpy(&key_blob_data, key_buf + 4, PTR_SIZE); + ok(key_blob_len == data_b.length, + "Case B: hp_make_key blob length = %u (expected %u)", + (uint) key_blob_len, (uint) data_b.length); + ok(key_blob_data == data_b.str, + "Case B: hp_make_key blob pointer matches source data"); + ok(memcmp(key_blob_data, data_b.str, data_b.length) == 0, + "Case B: hp_make_key blob data content matches"); + } + + /* --- Case C: large blob --- */ + build_record(rec, 3, data_c, len_c, FALSE); + + rec_hash_c= hp_rec_hashnr(0, &keydef, rec); + hp_make_key(&keydef, key_buf, rec); + { + uint32 key_blob_len= uint4korr(key_buf); + const uchar *key_blob_data; + memcpy(&key_blob_data, key_buf + 4, PTR_SIZE); + ok(key_blob_len == len_c, + "Case C: hp_make_key blob length = %u (expected %u)", + (uint) key_blob_len, (uint) len_c); + ok(key_blob_data == data_c, + "Case C: hp_make_key blob pointer matches source data"); + ok(memcmp(key_blob_data, data_c, len_c) == 0, + "Case C: hp_make_key blob data content matches"); + } + + /* Different data must produce different hashes */ + ok(rec_hash_a != rec_hash_b, + "Hash A (%lu) != Hash B (%lu)", rec_hash_a, rec_hash_b); + ok(rec_hash_a != rec_hash_c, + "Hash A (%lu) != Hash C (%lu)", rec_hash_a, rec_hash_c); + ok(rec_hash_b != rec_hash_c, + "Hash B (%lu) != Hash C (%lu)", rec_hash_b, rec_hash_c); +} + + +/* + Test 2: hp_rec_key_cmp with blob segments. + Two records with same blob data must compare equal. + Two records with different blob data must compare unequal. +*/ +static void test_rec_key_cmp(void) +{ + HA_KEYSEG seg; + HP_KEYDEF keydef; + uchar rec1[REC_LENGTH], rec2[REC_LENGTH]; + + LEX_CUSTRING data1= { USTRING_WITH_LEN("same_data_value!") }; + LEX_CUSTRING data2= { USTRING_WITH_LEN("different_value!") }; + LEX_CUSTRING data3= { USTRING_WITH_LEN("short") }; + + setup_blob_keyseg(&seg, FALSE); + setup_keydef(&keydef, &seg, 1); + + /* Same data, same length */ + build_record(rec1, 1, data1.str, data1.length, FALSE); + build_record(rec2, 2, data1.str, data1.length, FALSE); + ok(hp_rec_key_cmp(&keydef, rec1, rec2, NULL) == 0, + "rec_key_cmp: same blob data compares equal"); + + /* Different data, same length */ + build_record(rec2, 2, data2.str, data2.length, FALSE); + ok(hp_rec_key_cmp(&keydef, rec1, rec2, NULL) != 0, + "rec_key_cmp: different blob data compares unequal"); + + /* Different length (PAD SPACE: "short" vs "short\0\0..." may differ) */ + build_record(rec2, 2, data3.str, data3.length, FALSE); + /* For binary charset, different lengths always means different */ + { + HA_KEYSEG seg_bin; + HP_KEYDEF keydef_bin; + setup_blob_keyseg(&seg_bin, FALSE); + seg_bin.charset= &my_charset_bin; + setup_keydef(&keydef_bin, &seg_bin, 1); + + build_record(rec1, 1, data1.str, data1.length, FALSE); + build_record(rec2, 2, data3.str, data3.length, FALSE); + ok(hp_rec_key_cmp(&keydef_bin, rec1, rec2, NULL) != 0, + "rec_key_cmp: different length blobs compare unequal (binary)"); + } +} + + +/* + Test 3: NULL blob handling. + Two NULL blobs must compare equal. + NULL vs non-NULL must compare unequal. + NULL blob must hash consistently. +*/ +static void test_null_blob(void) +{ + HA_KEYSEG seg; + HP_KEYDEF keydef; + uchar rec1[REC_LENGTH], rec2[REC_LENGTH]; + uchar key_buf[KEY_BUF_SIZE]; + ulong hash1, hash2; + + LEX_CUSTRING data1= { USTRING_WITH_LEN("not_null_data") }; + + setup_blob_keyseg(&seg, TRUE); /* nullable */ + setup_keydef(&keydef, &seg, 1); + + /* Both NULL */ + build_record(rec1, 1, NULL, 0, TRUE); + build_record(rec2, 2, NULL, 0, TRUE); + ok(hp_rec_key_cmp(&keydef, rec1, rec2, NULL) == 0, + "null_blob: two NULLs compare equal"); + + /* NULL vs non-NULL */ + build_record(rec2, 2, data1.str, data1.length, FALSE); + ok(hp_rec_key_cmp(&keydef, rec1, rec2, NULL) != 0, + "null_blob: NULL vs non-NULL compares unequal"); + + /* NULL hash consistency */ + hash1= hp_rec_hashnr(0, &keydef, rec1); + hash2= hp_rec_hashnr(0, &keydef, rec1); + ok(hash1 == hash2, + "null_blob: NULL blob hashes consistently (%lu == %lu)", hash1, hash2); + + /* NULL hash differs from empty non-NULL */ + { + LEX_CUSTRING empty= { USTRING_WITH_LEN("") }; + ulong hash_empty; + build_record(rec2, 2, empty.str, empty.length, FALSE); + hash_empty= hp_rec_hashnr(0, &keydef, rec2); + ok(hash1 != hash_empty, + "null_blob: NULL hash (%lu) != empty non-NULL hash (%lu)", + hash1, hash_empty); + } + + /* hp_make_key for NULL blob */ + build_record(rec1, 1, NULL, 0, TRUE); + hp_make_key(&keydef, key_buf, rec1); + ok(key_buf[0] == 1, + "null_blob: hp_make_key sets null flag byte to 1 for NULL"); +} + + +/* + Test 4: empty blob (non-NULL, length=0). +*/ +static void test_empty_blob(void) +{ + HA_KEYSEG seg; + HP_KEYDEF keydef; + uchar rec1[REC_LENGTH], rec2[REC_LENGTH]; + ulong h1, h2; + + LEX_CUSTRING empty= { USTRING_WITH_LEN("") }; + LEX_CUSTRING nonempty= { USTRING_WITH_LEN("x") }; + + setup_blob_keyseg(&seg, FALSE); + setup_keydef(&keydef, &seg, 1); + + /* Two empty blobs */ + build_record(rec1, 1, empty.str, empty.length, FALSE); + build_record(rec2, 2, empty.str, empty.length, FALSE); + ok(hp_rec_key_cmp(&keydef, rec1, rec2, NULL) == 0, + "empty_blob: two empty blobs compare equal"); + + /* Empty vs non-empty */ + build_record(rec2, 2, nonempty.str, nonempty.length, FALSE); + ok(hp_rec_key_cmp(&keydef, rec1, rec2, NULL) != 0, + "empty_blob: empty vs non-empty compares unequal"); + + /* Hash consistency for empty */ + h1= hp_rec_hashnr(0, &keydef, rec1); + h2= hp_rec_hashnr(0, &keydef, rec1); + ok(h1 == h2, "empty_blob: empty blob hashes consistently"); +} + + +/* + Test 5: Multi-segment key with int + blob. + Verifies that key advancement works correctly when blob segments + have seg->length=0. +*/ +static void test_multi_segment_key(void) +{ + HA_KEYSEG segs[2]; + HP_KEYDEF keydef; + uchar rec1[REC_LENGTH], rec2[REC_LENGTH]; + uchar key_buf[KEY_BUF_SIZE]; + LEX_CUSTRING blob_data= { USTRING_WITH_LEN("multi_seg_test_data") }; + LEX_CUSTRING blob_data2= { USTRING_WITH_LEN("different_blob_data") }; + + /* Segment 0: int4 at offset 1, length 4 */ + memset(&segs[0], 0, sizeof(segs[0])); + segs[0].type= HA_KEYTYPE_BINARY; + segs[0].start= REC_INT_OFFSET; + segs[0].length= 4; + segs[0].charset= &my_charset_bin; + segs[0].null_bit= 0; + + /* Segment 1: blob at offset 5, packlength 2 */ + setup_blob_keyseg(&segs[1], FALSE); + + setup_keydef(&keydef, segs, 2); + + /* Same int, same blob */ + build_record(rec1, 42, blob_data.str, blob_data.length, FALSE); + build_record(rec2, 42, blob_data.str, blob_data.length, FALSE); + ok(hp_rec_key_cmp(&keydef, rec1, rec2, NULL) == 0, + "multi_seg: same int + same blob compares equal"); + + /* Different int, same blob */ + build_record(rec2, 99, blob_data.str, blob_data.length, FALSE); + ok(hp_rec_key_cmp(&keydef, rec1, rec2, NULL) != 0, + "multi_seg: different int + same blob compares unequal"); + + /* Same int, different blob */ + build_record(rec2, 42, blob_data2.str, blob_data2.length, FALSE); + ok(hp_rec_key_cmp(&keydef, rec1, rec2, NULL) != 0, + "multi_seg: same int + different blob compares unequal"); + + /* Hash consistency: record hash matches after make_key round-trip */ + build_record(rec1, 42, blob_data.str, blob_data.length, FALSE); + (void) hp_rec_hashnr(0, &keydef, rec1); + + hp_make_key(&keydef, key_buf, rec1); + /* Verify the key contains int4 (4 bytes) + blob (4B len + 8B ptr) */ + { + int32 key_int= sint4korr(key_buf); + uint32 key_blob_len= uint4korr(key_buf + 4); + const uchar *key_blob_data; + memcpy(&key_blob_data, key_buf + 8, PTR_SIZE); + + ok(key_int == 42, + "multi_seg: hp_make_key int = %d (expected 42)", (int) key_int); + ok(key_blob_len == blob_data.length, + "multi_seg: hp_make_key blob length = %u (expected %u)", + (uint) key_blob_len, (uint) blob_data.length); + ok(key_blob_data == blob_data.str, + "multi_seg: hp_make_key blob pointer matches"); + } +} + + +/* + Test 6: PAD SPACE collation behavior. + With PAD SPACE (default for latin1), 'a' and 'a ' should + compare equal and produce the same hash. +*/ +static void test_pad_space(void) +{ + HA_KEYSEG seg; + HP_KEYDEF keydef; + uchar rec1[REC_LENGTH], rec2[REC_LENGTH]; + LEX_CUSTRING data_no_pad= { USTRING_WITH_LEN("abc") }; + LEX_CUSTRING data_padded= { USTRING_WITH_LEN("abc ") }; + ulong h1, h2; + + setup_blob_keyseg(&seg, FALSE); + seg.charset= &my_charset_latin1; /* PAD SPACE */ + setup_keydef(&keydef, &seg, 1); + + build_record(rec1, 1, data_no_pad.str, data_no_pad.length, FALSE); + build_record(rec2, 2, data_padded.str, data_padded.length, FALSE); + + ok(hp_rec_key_cmp(&keydef, rec1, rec2, NULL) == 0, + "pad_space: 'abc' == 'abc ' with PAD SPACE collation"); + + /* Hashes should also match for PAD SPACE */ + h1= hp_rec_hashnr(0, &keydef, rec1); + h2= hp_rec_hashnr(0, &keydef, rec2); + ok(h1 == h2, + "pad_space: hash('abc') == hash('abc ') (%lu == %lu)", h1, h2); + + /* With binary charset (NO PAD), they should differ */ + seg.charset= &my_charset_bin; + ok(hp_rec_key_cmp(&keydef, rec1, rec2, NULL) != 0, + "pad_space: 'abc' != 'abc ' with binary charset"); +} + + +/* + Test 7: DISTINCT key path -- varstring key format. + + The SQL layer builds lookup keys in varstring format (2B length prefix + + inline data) via Field_blob::new_key_field() -> Field_varstring. The HEAP + handler's rebuild_key_from_group_buff() converts this to + record[0]'s blob descriptor format, then hp_make_key() + builds the hash key. + + This test simulates the full round-trip: + 1. Build a record with blob data (as at INSERT time) + 2. Compute hp_rec_hashnr(0, ) (stored in HASH_INFO at write time) + 3. Build a varstring-format key (as the SQL layer would for lookup) + 4. Parse the varstring key into a record's blob field + (rebuild_key_from_group_buff) + 5. hp_make_key() from that record, then hp_rec_hashnr(0, ) on the record + 6. Verify the hashes match +*/ +static void test_distinct_key_format(void) +{ + HA_KEYSEG seg; + HP_KEYDEF keydef; + uchar rec_insert[REC_LENGTH]; /* record at INSERT time */ + uchar rec_lookup[REC_LENGTH]; /* record rebuilt from lookup key */ + ulong insert_hash, lookup_hash; + + LEX_CUSTRING blob_data= { USTRING_WITH_LEN("1 - 01xxxxxxxxxx") }; + + /* + Step 3: Build varstring-format key (what SQL layer produces). + Format: null_flag(1) + uint2_length(2) + inline_data(blob_len) + */ + uchar varstring_key[1 + 2 + 256]; + + setup_blob_keyseg(&seg, TRUE); /* nullable, like real DISTINCT keys */ + setup_keydef(&keydef, &seg, 1); + + /* Step 1-2: INSERT-time record and hash */ + build_record(rec_insert, 1, blob_data.str, blob_data.length, FALSE); + insert_hash= hp_rec_hashnr(0, &keydef, rec_insert); + + varstring_key[0]= 0; /* not null */ + int2store(varstring_key + 1, blob_data.length); + memcpy(varstring_key + 3, blob_data.str, blob_data.length); + + /* + Step 4: Parse varstring key into rec_lookup's blob field. + This is what rebuild_key_from_group_buff() does. + */ + memset(rec_lookup, 0, REC_LENGTH); + { + const uchar *key_pos= varstring_key; + uint16 varchar_len; + const uchar *varchar_data; + uint32 bl; + /* skip null byte */ + key_pos++; + /* read varstring: 2B length + data */ + varchar_len= uint2korr(key_pos); + varchar_data= key_pos + 2; + + /* Write into rec_lookup's blob field */ + bl= (uint32) varchar_len; + int2store(rec_lookup + REC_BLOB_OFFSET, bl); + memcpy(rec_lookup + REC_BLOB_OFFSET + REC_BLOB_PACKLEN, + &varchar_data, sizeof(varchar_data)); + } + + /* Step 5: hp_make_key from rec_lookup, then hash the record */ + lookup_hash= hp_rec_hashnr(0, &keydef, rec_lookup); + + /* Step 6: hashes must match */ + ok(insert_hash == lookup_hash, + "distinct_key: INSERT hash (%lu) == lookup hash (%lu)", + insert_hash, lookup_hash); + + /* Also verify comparison works */ + ok(hp_rec_key_cmp(&keydef, rec_insert, rec_lookup, NULL) == 0, + "distinct_key: INSERT record == lookup record via rec_key_cmp"); +} + + +/* + Test 8: DISTINCT key truncation bug. + + When the DISTINCT key path sets key_part.length = pack_length() = 10 + (blob descriptor size), and new_key_field() creates Field_varstring(10), + the outer value (e.g. 16 bytes) gets truncated to 10 bytes. The lookup + key then has only 10 bytes but the stored record was hashed with 16 bytes. + This must produce different hashes -- demonstrating the bug. +*/ +static void test_distinct_key_truncation(void) +{ + HA_KEYSEG seg; + HP_KEYDEF keydef; + uchar rec_full[REC_LENGTH]; + uchar rec_trunc[REC_LENGTH]; + ulong full_hash, trunc_hash; + + LEX_CUSTRING full_data= { USTRING_WITH_LEN("1 - 01xxxxxxxxxx") }; + uint16 trunc_len= 10; /* pack_length() = packlength(2) + sizeof(ptr)(8) */ + + setup_blob_keyseg(&seg, FALSE); + seg.charset= &my_charset_bin; /* binary: no PAD SPACE confusion */ + setup_keydef(&keydef, &seg, 1); + + /* Full record (as stored at INSERT time) */ + build_record(rec_full, 1, full_data.str, full_data.length, FALSE); + full_hash= hp_rec_hashnr(0, &keydef, rec_full); + + /* Truncated record (as rebuilt from truncated varstring key) */ + build_record(rec_trunc, 1, full_data.str, trunc_len, FALSE); + trunc_hash= hp_rec_hashnr(0, &keydef, rec_trunc); + + /* Hashes MUST differ -- this is the bug: truncation causes lookup miss */ + ok(full_hash != trunc_hash, + "distinct_trunc: full hash (%lu) != truncated hash (%lu) -- " + "truncation causes hash mismatch (the bug)", + full_hash, trunc_hash); + + /* Comparison must also differ */ + ok(hp_rec_key_cmp(&keydef, rec_full, rec_trunc, NULL) != 0, + "distinct_trunc: full vs truncated compares unequal"); +} + + +/* + Test 9: GROUP BY key format -- varstring with key_length override. + + The GROUP BY path overrides the key field length to max_length (not + key_length() which is 0 for blobs). This means the varstring key + holds the full data. Verify hash consistency. +*/ +static void test_group_by_key_format(void) +{ + HA_KEYSEG seg; + HP_KEYDEF keydef; + uchar rec_insert[REC_LENGTH]; + uchar rec_lookup[REC_LENGTH]; + ulong insert_hash, lookup_hash; + + /* GROUP BY on group_concat result: blob data */ + LEX_CUSTRING data= { USTRING_WITH_LEN("group_concat_result_data_here!!") }; + + uchar varstring_key[1 + 2 + 256]; + + setup_blob_keyseg(&seg, FALSE); + setup_keydef(&keydef, &seg, 1); + + /* INSERT-time hash */ + build_record(rec_insert, 1, data.str, data.length, FALSE); + insert_hash= hp_rec_hashnr(0, &keydef, rec_insert); + + /* + Simulate rebuild_key_from_group_buff: parse varstring + key, populate rec_lookup. + In GROUP BY, key_field_length = max_length (not 0, not pack_length). + */ + /* no null bit for this test */ + int2store(varstring_key, data.length); + memcpy(varstring_key + 2, data.str, data.length); + + memset(rec_lookup, 0, REC_LENGTH); + { + const uchar *key_pos= varstring_key; + uint16 varchar_len; + const uchar *varchar_data; + uint32 bl; + + varchar_len= uint2korr(key_pos); + varchar_data= key_pos + 2; + + bl= (uint32) varchar_len; + int2store(rec_lookup + REC_BLOB_OFFSET, bl); + memcpy(rec_lookup + REC_BLOB_OFFSET + REC_BLOB_PACKLEN, + &varchar_data, sizeof(varchar_data)); + } + + lookup_hash= hp_rec_hashnr(0, &keydef, rec_lookup); + + ok(insert_hash == lookup_hash, + "group_by_key: INSERT hash (%lu) == lookup hash (%lu)", + insert_hash, lookup_hash); + + ok(hp_rec_key_cmp(&keydef, rec_insert, rec_lookup, NULL) == 0, + "group_by_key: INSERT record == lookup record"); +} + + +/* + Test 10: Multi-segment DISTINCT key (varchar + blob). + + Tests the key advancement logic when a non-blob varchar segment + precedes a blob segment, both with seg->length handling. +*/ +static void test_multi_seg_distinct(void) +{ + HA_KEYSEG segs[2]; + HP_KEYDEF keydef; + uchar rec1[REC_LENGTH], rec2[REC_LENGTH]; + LEX_CUSTRING blob1= { USTRING_WITH_LEN("sj_materialize_value_1") }; + LEX_CUSTRING blob2= { USTRING_WITH_LEN("sj_materialize_value_2") }; + ulong h1, h2, h3; + + /* Segment 0: int4 at offset 1, length 4 */ + memset(&segs[0], 0, sizeof(segs[0])); + segs[0].type= HA_KEYTYPE_BINARY; + segs[0].start= REC_INT_OFFSET; + segs[0].length= 4; + segs[0].charset= &my_charset_bin; + segs[0].null_bit= 0; + + /* Segment 1: blob */ + setup_blob_keyseg(&segs[1], TRUE); /* nullable */ + setup_keydef(&keydef, segs, 2); + + /* Same int, same blob */ + build_record(rec1, 100, blob1.str, blob1.length, FALSE); + build_record(rec2, 100, blob1.str, blob1.length, FALSE); + + h1= hp_rec_hashnr(0, &keydef, rec1); + h2= hp_rec_hashnr(0, &keydef, rec2); + ok(h1 == h2, + "multi_distinct: same data hashes equal (%lu == %lu)", h1, h2); + ok(hp_rec_key_cmp(&keydef, rec1, rec2, NULL) == 0, + "multi_distinct: same data compares equal"); + + /* Same blob, different int */ + build_record(rec2, 200, blob1.str, blob1.length, FALSE); + h3= hp_rec_hashnr(0, &keydef, rec2); + ok(h1 != h3, + "multi_distinct: different int hashes differ (%lu != %lu)", h1, h3); + ok(hp_rec_key_cmp(&keydef, rec1, rec2, NULL) != 0, + "multi_distinct: same blob + different int compares unequal"); + + /* Same int, different blob */ + build_record(rec2, 100, blob2.str, blob2.length, FALSE); + h3= hp_rec_hashnr(0, &keydef, rec2); + ok(h1 != h3, + "multi_distinct: different blob hashes differ (%lu != %lu)", h1, h3); + ok(hp_rec_key_cmp(&keydef, rec1, rec2, NULL) != 0, + "multi_distinct: different blob compares unequal"); + + /* Same int, NULL blob vs non-NULL blob */ + build_record(rec2, 100, NULL, 0, TRUE); + ok(hp_rec_key_cmp(&keydef, rec1, rec2, NULL) != 0, + "multi_distinct: non-NULL vs NULL blob compares unequal"); +} + + +/* + Test 11: hp_hashnr (key-based) must equal hp_rec_hashnr (record-based). + + This directly tests that building a key via hp_make_key() and then + hashing it with hp_hashnr() produces the same hash as hp_rec_hashnr(0, ) + on the original record. This catches divergence bugs where the two + functions process segments differently (e.g. VARCHAR pack_length + hardcoded to 2 in hp_hashnr but read from seg->bit_start in + hp_rec_hashnr). +*/ + +extern ulong hp_hashnr(HP_KEYDEF *keydef, const uchar *key); + +/* + Record layout for mixed varchar + blob table: + + byte 0: null bitmap (null_bit 4 = city null, null_bit 8 = libname null) + bytes 1: varchar length_bytes=1 (libname: VARCHAR(22)) + bytes 2-23: varchar data (22 bytes) + bytes 24-25: blob packlength=2 (city: TEXT) + bytes 26-33: blob data pointer (8 bytes on x86_64) + byte 34: flags byte (visible offset) + Total reclength: 35, recbuffer: ALIGN(MAX(34,8)+1, 8) = 40 +*/ +#define MIX_NULL_OFFSET 0 +#define MIX_VARCHAR_OFFSET 1 +#define MIX_VARCHAR_LEN 22 +#define MIX_VARCHAR_LENBYTES 1 +#define MIX_BLOB_OFFSET 24 +#define MIX_BLOB_PACKLEN 2 +#define MIX_REC_LENGTH 35 +#define MIX_KEY_BUF_SIZE 256 + + +static void setup_mixed_keydef(HP_KEYDEF *keydef, HA_KEYSEG *segs) +{ + /* Segment 0: blob (city TEXT) at offset 24 */ + memset(&segs[0], 0, sizeof(segs[0])); + segs[0].type= HA_KEYTYPE_VARTEXT4; + segs[0].flag= HA_BLOB_PART | HA_VAR_LENGTH_PART; + segs[0].start= MIX_BLOB_OFFSET; + segs[0].length= 4+portable_sizeof_char_ptr; /* Length of blob key */ + segs[0].bit_start= MIX_BLOB_PACKLEN; + segs[0].charset= &my_charset_latin1; + segs[0].null_bit= 4; /* bit 2 in null bitmap */ + segs[0].null_pos= MIX_NULL_OFFSET; + + /* Segment 1: varchar (libname VARCHAR(21)) at offset 1 */ + memset(&segs[1], 0, sizeof(segs[1])); + segs[1].type= HA_KEYTYPE_VARTEXT1; + segs[1].flag= HA_VAR_LENGTH_PART; + segs[1].start= MIX_VARCHAR_OFFSET; + segs[1].length= MIX_VARCHAR_LEN; + segs[1].bit_start= MIX_VARCHAR_LENBYTES; /* 1-byte length prefix */ + segs[1].bit_length= 2; /* key uses 2-byte length prefix */ + segs[1].charset= &my_charset_latin1; + segs[1].null_bit= 8; /* bit 3 in null bitmap */ + segs[1].null_pos= MIX_NULL_OFFSET; + + setup_keydef(keydef, segs, 2); +} + + +static void build_mixed_record(uchar *rec, const uchar *blob_data, + size_t blob_len, const uchar *varchar_data, + size_t varchar_len, + my_bool blob_null, my_bool varchar_null) +{ + memset(rec, 0, MIX_REC_LENGTH); + + /* null bitmap */ + if (blob_null) + rec[MIX_NULL_OFFSET] |= 4; + if (varchar_null) + rec[MIX_NULL_OFFSET] |= 8; + + /* varchar: 1-byte length prefix + data */ + rec[MIX_VARCHAR_OFFSET]= (uchar) varchar_len; + if (varchar_data && varchar_len > 0) + memcpy(rec + MIX_VARCHAR_OFFSET + MIX_VARCHAR_LENBYTES, + varchar_data, varchar_len); + + /* blob: packlength(2) + data pointer */ + int2store(rec + MIX_BLOB_OFFSET, blob_len); + memcpy(rec + MIX_BLOB_OFFSET + MIX_BLOB_PACKLEN, &blob_data, PTR_SIZE); +} + + +static void test_key_vs_rec_hash_consistency(void) +{ + HA_KEYSEG segs[2]; + HP_KEYDEF keydef; + uchar rec[MIX_REC_LENGTH]; + uchar key_buf[MIX_KEY_BUF_SIZE]; + ulong rec_hash, key_hash; + + LEX_CUSTRING city= { USTRING_WITH_LEN("New York") }; + LEX_CUSTRING libname= { USTRING_WITH_LEN("New York Public Libra") }; + + setup_mixed_keydef(&keydef, segs); + + /* Build record and compute record-based hash (used at INSERT time) */ + build_mixed_record(rec, city.str, city.length, libname.str, libname.length, + FALSE, FALSE); + rec_hash= hp_rec_hashnr(0, &keydef, rec); + + /* Build key via hp_make_key and compute key-based hash (used at LOOKUP) */ + hp_make_key(&keydef, key_buf, rec); + key_hash= hp_hashnr(&keydef, key_buf); + + ok(rec_hash == key_hash, + "key_vs_rec_hash: rec_hash (%lu) == key_hash (%lu) " + "for mixed blob+varchar(1B) key", + rec_hash, key_hash); + + /* Second test: different data to ensure it's not a coincidence */ + { + LEX_CUSTRING city2= { USTRING_WITH_LEN("San Fran") }; + LEX_CUSTRING libname2= { USTRING_WITH_LEN("SF Public Library") }; + + build_mixed_record(rec, city2.str, city2.length, + libname2.str, libname2.length, FALSE, FALSE); + rec_hash= hp_rec_hashnr(0, &keydef, rec); + hp_make_key(&keydef, key_buf, rec); + key_hash= hp_hashnr(&keydef, key_buf); + + ok(rec_hash == key_hash, + "key_vs_rec_hash: rec_hash (%lu) == key_hash (%lu) " + "for second mixed blob+varchar(1B) data", + rec_hash, key_hash); + } + + /* Third test: varchar with 2-byte length prefix (field_length >= 256) */ + { + HA_KEYSEG segs2b[2]; + HP_KEYDEF keydef2b; + uchar rec2b[MIX_REC_LENGTH + 256]; + uchar key2b[MIX_KEY_BUF_SIZE + 256]; + + /* + Copy the setup but change varchar to 2-byte length prefix. + This should always work because hp_hashnr already hardcodes 2. + */ + memcpy(segs2b, segs, sizeof(segs)); + segs2b[1].bit_start= 2; /* 2-byte length prefix */ + segs2b[1].length--; /* Size for one less character */ + setup_keydef(&keydef2b, segs2b, 2); + + memset(rec2b, 0, sizeof(rec2b)); + /* blob */ + rec2b[MIX_NULL_OFFSET]= 0; + int2store(rec2b + MIX_BLOB_OFFSET, city.length); + memcpy(rec2b + MIX_BLOB_OFFSET + MIX_BLOB_PACKLEN, &city.str, PTR_SIZE); + /* varchar with 2B length prefix */ + int2store(rec2b + MIX_VARCHAR_OFFSET, libname.length); + memcpy(rec2b + MIX_VARCHAR_OFFSET + 2, libname.str, libname.length); + + rec_hash= hp_rec_hashnr(0, &keydef2b, rec2b); + hp_make_key(&keydef2b, key2b, rec2b); + key_hash= hp_hashnr(&keydef2b, key2b); + + ok(rec_hash == key_hash, + "key_vs_rec_hash: rec_hash (%lu) == key_hash (%lu) " + "for mixed blob+varchar(2B) key", + rec_hash, key_hash); + } + + /* Fourth test: blob-only key (no varchar) -- should always match */ + { + HA_KEYSEG seg_blob; + HP_KEYDEF kd_blob; + uchar rec_b[REC_LENGTH]; + uchar key_b[KEY_BUF_SIZE]; + + setup_blob_keyseg(&seg_blob, TRUE); + setup_keydef(&kd_blob, &seg_blob, 1); + + build_record(rec_b, 1, city.str, city.length, FALSE); + rec_hash= hp_rec_hashnr(0, &kd_blob, rec_b); + hp_make_key(&kd_blob, key_b, rec_b); + key_hash= hp_hashnr(&kd_blob, key_b); + + ok(rec_hash == key_hash, + "key_vs_rec_hash: rec_hash (%lu) == key_hash (%lu) " + "for blob-only key", + rec_hash, key_hash); + } +} + + +/* + Build a record with a blob field using an arbitrary packlength (1-4). + Record layout for packlength P: + byte 0: null bitmap + bytes 1-4: int4 field + bytes 5..(5+P-1): blob length (little-endian, P bytes) + bytes (5+P)..(5+P+PTR_SIZE-1): blob data pointer + Total reclength: 5 + P + PTR_SIZE +*/ +static void build_record_packN(uchar *rec, size_t rec_len, uint packlength, + int32 int_val, + const uchar *blob_data, size_t blob_len) +{ + memset(rec, 0, rec_len); + int4store(rec + REC_INT_OFFSET, int_val); + switch (packlength) + { + case 1: + rec[REC_BLOB_OFFSET]= (uchar) blob_len; + break; + case 2: + int2store(rec + REC_BLOB_OFFSET, blob_len); + break; + case 3: + int3store(rec + REC_BLOB_OFFSET, blob_len); + break; + case 4: + int4store(rec + REC_BLOB_OFFSET, blob_len); + break; + } + memcpy(rec + REC_BLOB_OFFSET + packlength, &blob_data, PTR_SIZE); +} + + +/* + Test 12: Packlength variants (1, 3, 4). + + The existing tests only exercise packlength=2. The HEAP engine supports + packlengths 1-4 via seg->bit_start. Verify that hp_rec_hashnr(0, ), + hp_rec_key_cmp(), and hp_make_key() work correctly for each. +*/ +static void test_packlength_variants(void) +{ + static const uint packlengths[]= {1, 3, 4}; + uint i; + + for (i= 0; i < array_elements(packlengths); i++) + { + uint pl= packlengths[i]; + /* + Record size: 5 (null + int4) + pl + PTR_SIZE, rounded up. + Use a generous buffer. + */ + uchar rec1[32], rec2[32]; + uchar key_buf[KEY_BUF_SIZE]; + HA_KEYSEG seg; + HP_KEYDEF keydef; + ulong hash1; + uint32 key_blob_len; + + uchar data_same[]= "packlength_test_data_ABCDEFGHIJ"; + size_t data_len= 30; /* well within all packlength maxima */ + uchar data_diff[]= "DIFFERENT_data_XYZXYZXYZXYZXYZ"; + + /* Set up keyseg for this packlength */ + memset(&seg, 0, sizeof(seg)); + seg.type= HA_KEYTYPE_VARTEXT4; + seg.flag= HA_BLOB_PART | HA_VAR_LENGTH_PART; + seg.start= REC_BLOB_OFFSET; + seg.length= 4 + portable_sizeof_char_ptr; + seg.bit_start= pl; + seg.charset= &my_charset_latin1; + seg.null_bit= 0; + setup_keydef(&keydef, &seg, 1); + + /* Build record and compute hash */ + build_record_packN(rec1, sizeof(rec1), pl, 1, data_same, data_len); + hash1= hp_rec_hashnr(0, &keydef, rec1); + + ok(hash1 != 0, + "packlen=%u: hp_rec_hashnr produces non-zero hash (%lu)", pl, hash1); + + /* Same data must compare equal */ + build_record_packN(rec2, sizeof(rec2), pl, 2, data_same, data_len); + ok(hp_rec_key_cmp(&keydef, rec1, rec2, NULL) == 0, + "packlen=%u: same blob data compares equal", pl); + + /* Different data must compare unequal */ + build_record_packN(rec2, sizeof(rec2), pl, 2, data_diff, data_len); + ok(hp_rec_key_cmp(&keydef, rec1, rec2, NULL) != 0, + "packlen=%u: different blob data compares unequal", pl); + + /* hp_make_key must produce correct blob length */ + hp_make_key(&keydef, key_buf, rec1); + key_blob_len= uint4korr(key_buf); + ok(key_blob_len == data_len, + "packlen=%u: hp_make_key blob length = %u (expected %u)", + pl, (uint) key_blob_len, (uint) data_len); + } +} + + +/* + Record layout for a two-blob table: + byte 0: null bitmap + bytes 1-4: int4 field (padding, not keyed) + bytes 5-6: blob1 packlength=2 + bytes 7-14: blob1 data pointer (8 bytes) + bytes 15-16: blob2 packlength=2 + bytes 17-24: blob2 data pointer (8 bytes) + byte 25: flags byte (visible offset) + Total reclength: 26, recbuffer: ALIGN(MAX(26,8)+1, 8) = 32 +*/ +#define BB_INT_OFFSET 1 +#define BB_BLOB1_OFFSET 5 +#define BB_BLOB2_OFFSET 15 +#define BB_BLOB_PACKLEN 2 +#define BB_REC_LENGTH 26 +#define BB_REC_BUFFER 32 + + +static void build_two_blob_record(uchar *rec, int32 int_val, + const uchar *blob1_data, size_t blob1_len, + const uchar *blob2_data, size_t blob2_len) +{ + memset(rec, 0, BB_REC_LENGTH); + int4store(rec + BB_INT_OFFSET, int_val); + int2store(rec + BB_BLOB1_OFFSET, blob1_len); + memcpy(rec + BB_BLOB1_OFFSET + BB_BLOB_PACKLEN, &blob1_data, PTR_SIZE); + int2store(rec + BB_BLOB2_OFFSET, blob2_len); + memcpy(rec + BB_BLOB2_OFFSET + BB_BLOB_PACKLEN, &blob2_data, PTR_SIZE); +} + + +/* + Test 13: Multi-segment key with two blob segments. + + Verifies hash, comparison, and hp_make_key() when a key has two + HA_KEYTYPE_VARTEXT4 (blob) segments. +*/ +static void test_blob_blob_multi_segment(void) +{ + HA_KEYSEG segs[2]; + HP_KEYDEF keydef; + uchar rec1[BB_REC_BUFFER], rec2[BB_REC_BUFFER]; + uchar key_buf[KEY_BUF_SIZE]; + ulong h1, h2; + + LEX_CUSTRING data_a1= { USTRING_WITH_LEN("first_blob_data_alpha") }; + LEX_CUSTRING data_a2= { USTRING_WITH_LEN("second_blob_data_beta") }; + LEX_CUSTRING data_b1= { USTRING_WITH_LEN("CHANGED_first_blob!!!") }; + LEX_CUSTRING data_b2= { USTRING_WITH_LEN("CHANGED_second_blob!!") }; + + /* Segment 0: blob1 at BB_BLOB1_OFFSET */ + memset(&segs[0], 0, sizeof(segs[0])); + segs[0].type= HA_KEYTYPE_VARTEXT4; + segs[0].flag= HA_BLOB_PART | HA_VAR_LENGTH_PART; + segs[0].start= BB_BLOB1_OFFSET; + segs[0].length= 4 + portable_sizeof_char_ptr; + segs[0].bit_start= BB_BLOB_PACKLEN; + segs[0].charset= &my_charset_latin1; + segs[0].null_bit= 0; + + /* Segment 1: blob2 at BB_BLOB2_OFFSET */ + memset(&segs[1], 0, sizeof(segs[1])); + segs[1].type= HA_KEYTYPE_VARTEXT4; + segs[1].flag= HA_BLOB_PART | HA_VAR_LENGTH_PART; + segs[1].start= BB_BLOB2_OFFSET; + segs[1].length= 4 + portable_sizeof_char_ptr; + segs[1].bit_start= BB_BLOB_PACKLEN; + segs[1].charset= &my_charset_latin1; + segs[1].null_bit= 0; + + setup_keydef(&keydef, segs, 2); + + /* Same data in both blobs: hash consistency */ + build_two_blob_record(rec1, 1, data_a1.str, data_a1.length, + data_a2.str, data_a2.length); + build_two_blob_record(rec2, 2, data_a1.str, data_a1.length, + data_a2.str, data_a2.length); + h1= hp_rec_hashnr(0, &keydef, rec1); + h2= hp_rec_hashnr(0, &keydef, rec2); + ok(h1 == h2, + "blob+blob: same data in both blobs hashes equal (%lu == %lu)", h1, h2); + + /* Same data: comparison must be equal */ + ok(hp_rec_key_cmp(&keydef, rec1, rec2, NULL) == 0, + "blob+blob: same data compares equal"); + + /* Change blob1 only: must compare unequal */ + build_two_blob_record(rec2, 2, data_b1.str, data_b1.length, + data_a2.str, data_a2.length); + ok(hp_rec_key_cmp(&keydef, rec1, rec2, NULL) != 0, + "blob+blob: different blob1 compares unequal"); + + /* Change blob2 only: must compare unequal */ + build_two_blob_record(rec2, 2, data_a1.str, data_a1.length, + data_b2.str, data_b2.length); + ok(hp_rec_key_cmp(&keydef, rec1, rec2, NULL) != 0, + "blob+blob: different blob2 compares unequal"); + + /* hp_make_key: verify both blob lengths in the key */ + hp_make_key(&keydef, key_buf, rec1); + { + uint32 key_blob1_len= uint4korr(key_buf); + uint32 key_blob2_len= uint4korr(key_buf + 4 + portable_sizeof_char_ptr); + + ok(key_blob1_len == data_a1.length, + "blob+blob: hp_make_key blob1 length = %u (expected %u)", + (uint) key_blob1_len, (uint) data_a1.length); + ok(key_blob2_len == data_a2.length, + "blob+blob: hp_make_key blob2 length = %u (expected %u)", + (uint) key_blob2_len, (uint) data_a2.length); + } +} + + +int main(int argc __attribute__((unused)), + char **argv __attribute__((unused))) +{ + MY_INIT("hp_test_hash"); + plan(67); + + diag("Test 1: Hash consistency between record and key formats"); + test_hash_consistency(); + + diag("Test 2: Record-to-record comparison with blobs"); + test_rec_key_cmp(); + + diag("Test 3: NULL blob handling"); + test_null_blob(); + + diag("Test 4: Empty blob handling"); + test_empty_blob(); + + diag("Test 5: Multi-segment key (int + blob)"); + test_multi_segment_key(); + + diag("Test 6: PAD SPACE collation"); + test_pad_space(); + + diag("Test 7: DISTINCT key format (varstring round-trip)"); + test_distinct_key_format(); + + diag("Test 8: DISTINCT key truncation bug"); + test_distinct_key_truncation(); + + diag("Test 9: GROUP BY key format"); + test_group_by_key_format(); + + diag("Test 10: Multi-segment DISTINCT key (sj-materialize)"); + test_multi_seg_distinct(); + + diag("Test 11: hp_hashnr vs hp_rec_hashnr consistency"); + test_key_vs_rec_hash_consistency(); + + diag("Test 12: Packlength variants (1, 3, 4)"); + test_packlength_variants(); + + diag("Test 13: Multi-segment key with two blob segments"); + test_blob_blob_multi_segment(); + + my_end(0); + return exit_status(); +} diff --git a/storage/heap/hp_test_helpers.h b/storage/heap/hp_test_helpers.h new file mode 100644 index 0000000000000..5e114df488c9b --- /dev/null +++ b/storage/heap/hp_test_helpers.h @@ -0,0 +1,87 @@ +/* + Shared helpers for HEAP blob unit tests. + + Record layout: (int4, blob(packlength=2)) + byte 0: null bitmap (1 byte) + bytes 1-4: int4 field (4 bytes) + bytes 5-6: blob packlength=2 (length, little-endian) + bytes 7-14: blob data pointer (portable_sizeof_char_ptr = 8 bytes always) + reclength = 15 + visible_offset = MAX(15, 8) = 15 + recbuffer = ALIGN(15 + 1, 8) = 16 + + The pointer slot is always portable_sizeof_char_ptr (8) bytes in the record + (same on 32-bit and 64-bit), but only sizeof(void*) bytes are meaningful. + The rest is zero-padded by memset. +*/ + +#ifndef HP_TEST_HELPERS_H +#define HP_TEST_HELPERS_H + +#include +#include +#include +#include +#include "heap.h" +#include "heapdef.h" + +#define REC_LENGTH 15 +#define INT_OFFSET 1 +#define BLOB_OFFSET 5 +#define BLOB_PACKLEN 2 + + +static void build_record(uchar *rec, int32 int_val, + const uchar *blob_data, uint16 blob_len) +{ + memset(rec, 0, REC_LENGTH); + int4store(rec + INT_OFFSET, int_val); + int2store(rec + BLOB_OFFSET, blob_len); + memcpy(rec + BLOB_OFFSET + BLOB_PACKLEN, &blob_data, sizeof(blob_data)); +} + + +static int create_and_open(const char *name, HP_SHARE **share, HP_INFO **info) +{ + HP_KEYDEF keydef; + HA_KEYSEG keyseg; + HP_CREATE_INFO ci; + HP_BLOB_DESC blob_desc; + my_bool unused; + + memset(&keyseg, 0, sizeof(keyseg)); + keyseg.type= HA_KEYTYPE_BINARY; + keyseg.start= INT_OFFSET; + keyseg.length= 4; + keyseg.charset= &my_charset_bin; + + memset(&keydef, 0, sizeof(keydef)); + keydef.keysegs= 1; + keydef.seg= &keyseg; + keydef.algorithm= HA_KEY_ALG_HASH; + keydef.flag= HA_NOSAME; + keydef.length= 4; + + blob_desc.offset= BLOB_OFFSET; + blob_desc.packlength= BLOB_PACKLEN; + + memset(&ci, 0, sizeof(ci)); + ci.keys= 1; + ci.keydef= &keydef; + ci.reclength= REC_LENGTH; + ci.max_records= 1000; + ci.min_records= 10; + ci.max_table_size= 1024 * 1024; + ci.blob_descs= &blob_desc; + ci.blob_count= 1; + + if (heap_create(name, &ci, share, &unused)) + return 1; + *info= heap_open(name, 2); + if (!*info) + return 1; + heap_extra(*info, HA_EXTRA_NO_READCHECK); + return 0; +} + +#endif /* HP_TEST_HELPERS_H */ diff --git a/storage/heap/hp_test_key_setup-t.cc b/storage/heap/hp_test_key_setup-t.cc new file mode 100644 index 0000000000000..733ed99ce0862 --- /dev/null +++ b/storage/heap/hp_test_key_setup-t.cc @@ -0,0 +1,675 @@ +/* + Unit tests for HEAP blob key handling in heap_prepare_hp_create_info(). + + Verifies blob segment normalization (seg->length, seg->bit_start, + seg->flag, seg->type) and correct key setup for non-blob types + (varchar, int, enum, mixed keys, geometry GROUP BY). +*/ + +#include +#include +#include +#include + +#include "sql_priv.h" +#include "sql_class.h" /* THD (full definition) */ +#include "ha_heap.h" +#include "heapdef.h" + +/* + RAII guard for the fake THD used by unit tests. + Allocates a zero-initialized THD (without calling the constructor), + sets max_heap_table_size, installs it as current_thd, and tears it + down on destruction. This is technically UB (no C++ construction), + but works because heap_prepare_hp_create_info only reads + thd->variables.max_heap_table_size from the zeroed memory. +*/ +class Fake_thd_guard +{ + char *m_buf; +public: + Fake_thd_guard(ulonglong max_heap_size= 1024*1024) + { + m_buf= (char*) calloc(1, sizeof(THD)); + THD *thd= (THD*) m_buf; + thd->variables.max_heap_table_size= max_heap_size; + set_current_thd(thd); + } + ~Fake_thd_guard() + { + set_current_thd(NULL); + free(m_buf); + } +}; + +static const LEX_CSTRING test_field_name= {STRING_WITH_LEN("")}; + +extern int heap_prepare_hp_create_info(TABLE *table_arg, + bool internal_table, + HP_CREATE_INFO *hp_create_info); + +/* + Record layout for test table (nullable tinyblob(16)): + byte 0: null bitmap (bit 2 = blob null) + bytes 1-2: blob packlength=2 (length, little-endian) + bytes 3-10: blob data pointer (8 bytes) + reclength = 11 +*/ +#define T_REC_NULL_OFFSET 0 +#define T_REC_BLOB_OFFSET 1 +#define T_REC_BLOB_PACKLEN 2 +#define T_REC_LENGTH 11 + + +/* + Helper: create a Field_blob using the full server constructor + (the same one make_table_field uses) via placement new. + Sets field_length = BLOB_PACK_LENGTH_TO_MAX_LENGH(packlength), + matching real server behavior. +*/ +static Field_blob * +make_test_field_blob(void *storage, uchar *ptr, uchar *null_ptr, + uchar null_bit, TABLE_SHARE *share, + uint packlength, CHARSET_INFO *cs) +{ + static const LEX_CSTRING fname= {STRING_WITH_LEN("")}; + return ::new (storage) Field_blob(ptr, null_ptr, null_bit, + Field::NONE, &fname, + share, packlength, + DTCollation(cs)); +} + + +/* + distinct_key_truncation: blob segment normalization. + + Verifies that heap_prepare_hp_create_info() correctly normalizes + blob key segments: seg->length = 4+ptr, seg->bit_start = packlength, + seg->flag has HA_BLOB_PART, seg->type is VARTEXT4/VARBINARY4. + + Uses the DISTINCT key path setup where key_part.length = pack_length() = 10. +*/ +static void test_distinct_key_truncation() +{ + uchar local_rec[T_REC_LENGTH]; + memset(local_rec, 0, sizeof(local_rec)); + + TABLE_SHARE share; + memset(static_cast(&share), 0, sizeof(share)); + share.fields= 1; + share.blob_fields= 0; /* Field_blob constructor increments this */ + share.keys= 1; + share.reclength= T_REC_LENGTH; + share.rec_buff_length= T_REC_LENGTH; + share.db_record_offset= 1; + + alignas(Field_blob) char bf_storage[sizeof(Field_blob)]; + Field_blob *bfp= make_test_field_blob(bf_storage, + local_rec + T_REC_BLOB_OFFSET, + local_rec + T_REC_NULL_OFFSET, + 2, &share, + T_REC_BLOB_PACKLEN, + &my_charset_bin); + Field_blob &bf= *bfp; + bf.field_index= 0; + + Field *field_array[2]= { &bf, NULL }; + + KEY_PART_INFO local_kpi; + memset(&local_kpi, 0, sizeof(local_kpi)); + local_kpi.field= &bf; + local_kpi.offset= T_REC_BLOB_OFFSET; + local_kpi.length= (uint16) bf.pack_length(); /* = 10 (the bug) */ + local_kpi.key_part_flag= bf.key_part_flag(); + local_kpi.type= bf.key_type(); + + KEY local_sql_key= {}; + local_sql_key.user_defined_key_parts= 1; + local_sql_key.usable_key_parts= 1; + local_sql_key.key_part= &local_kpi; + local_sql_key.algorithm= HA_KEY_ALG_HASH; + + TABLE test_table; + memset(static_cast(&test_table), 0, sizeof(test_table)); + test_table.record[0]= local_rec; + test_table.s= &share; + test_table.field= field_array; + test_table.key_info= &local_sql_key; + share.key_info= &local_sql_key; + + bf.table= &test_table; + + uint blob_offsets[1]= { 0 }; + share.blob_field= blob_offsets; + + /* + Simulate DISTINCT key path: set store_length and key_length + based on key_part.length = pack_length() = 10, same as finalize(). + */ + local_kpi.store_length= local_kpi.length; + if (bf.real_maybe_null()) + local_kpi.store_length+= HA_KEY_NULL_LENGTH; + local_kpi.store_length+= bf.key_part_length_bytes(); + local_sql_key.key_length= local_kpi.store_length; + + ok(local_kpi.length == bf.pack_length(), + "distinct_key_truncation setup: key_part.length = pack_length() = %u", + (uint) local_kpi.length); + + Fake_thd_guard thd_guard; + + HP_CREATE_INFO hp_ci; + memset(&hp_ci, 0, sizeof(hp_ci)); + hp_ci.max_table_size= 1024*1024; + hp_ci.keys= 1; + hp_ci.reclength= T_REC_LENGTH; + + int err= heap_prepare_hp_create_info(&test_table, TRUE, &hp_ci); + + ok(err == 0, + "distinct_key_truncation: heap_prepare succeeded (err=%d)", err); + + HP_KEYDEF *kd= &hp_ci.keydef[0]; + ok(kd->seg[0].length == 4 + portable_sizeof_char_ptr, + "distinct_key_truncation: seg->length = %u (expected %u = 4+ptr)", + (uint) kd->seg[0].length, (uint)(4 + portable_sizeof_char_ptr)); + ok(kd->seg[0].bit_start == bf.length_size(), + "distinct_key_truncation: seg->bit_start = %u (expected %u = packlength)", + (uint) kd->seg[0].bit_start, (uint) bf.length_size()); + ok(kd->seg[0].flag & HA_BLOB_PART, + "distinct_key_truncation: seg->flag (0x%x) has HA_BLOB_PART", + (uint) kd->seg[0].flag); + ok(kd->seg[0].type == HA_KEYTYPE_VARBINARY4 || + kd->seg[0].type == HA_KEYTYPE_VARTEXT4, + "distinct_key_truncation: seg->type = %u (expected VARTEXT4/VARBINARY4)", + (uint) kd->seg[0].type); + + my_free(hp_ci.keydef); + my_free(hp_ci.blob_descs); + bf.~Field_blob(); +} + + +/* + Test: heap_prepare_hp_create_info for various non-blob key types. + + Verifies that seg->flag does not contain HA_BLOB_PART for: + - VARCHAR-only keys (Field_varstring, length_bytes=1) + - Fixed-length keys (Field_long = INT) + - ENUM keys (Field_enum) + - Mixed VARCHAR + INT keys + + Also verifies seg->length, seg->type, seg->bit_start are correct. +*/ + +/* Helper: set up a single-field TABLE + KEY for heap_prepare testing */ +struct Hp_test_single_key +{ + TABLE_SHARE share; + TABLE test_table; + KEY_PART_INFO kpi; + KEY sql_key; + Field *field_array[2]; + uchar rec_buf[64]; + uint blob_offsets[1]; + + void init(Field *field, uint offset, uint rec_length) + { + memset(rec_buf, 0, sizeof(rec_buf)); + memset(static_cast(&share), 0, sizeof(share)); + share.fields= 1; + share.keys= 1; + share.reclength= rec_length; + share.rec_buff_length= rec_length; + share.db_record_offset= 1; + share.blob_fields= 0; + blob_offsets[0]= 0; + share.blob_field= blob_offsets; + + field_array[0]= field; + field_array[1]= NULL; + + memset(&kpi, 0, sizeof(kpi)); + kpi.field= field; + kpi.offset= offset; + kpi.length= (uint16) field->key_length(); + kpi.key_part_flag= field->key_part_flag(); + kpi.type= field->key_type(); + kpi.store_length= kpi.length; + if (field->real_maybe_null()) + kpi.store_length+= HA_KEY_NULL_LENGTH; + if (field->key_part_flag() & HA_VAR_LENGTH_PART) + kpi.store_length+= field->key_part_length_bytes(); + + sql_key= {}; + sql_key.user_defined_key_parts= 1; + sql_key.usable_key_parts= 1; + sql_key.key_part= &kpi; + sql_key.algorithm= HA_KEY_ALG_HASH; + sql_key.key_length= kpi.store_length; + + memset(static_cast(&test_table), 0, sizeof(test_table)); + test_table.record[0]= rec_buf; + test_table.s= &share; + test_table.field= field_array; + test_table.key_info= &sql_key; + share.key_info= &sql_key; + + field->table= &test_table; + } + + int run_hp_create(HP_CREATE_INFO *hp_ci) + { + Fake_thd_guard thd_guard; + + memset(hp_ci, 0, sizeof(*hp_ci)); + hp_ci->max_table_size= 1024*1024; + hp_ci->keys= 1; + hp_ci->reclength= share.reclength; + + return heap_prepare_hp_create_info(&test_table, TRUE, hp_ci); + } +}; + + +static void test_varchar_only_key() +{ + /* VARCHAR(28) NOT NULL, length_bytes=1 */ + static const LEX_CSTRING fname= {STRING_WITH_LEN("v1")}; + TABLE_SHARE dummy_share; + memset(static_cast(&dummy_share), 0, sizeof(dummy_share)); + alignas(Field_varstring) char vs_storage[sizeof(Field_varstring)]; + Field_varstring *vs= ::new (vs_storage) Field_varstring( + (uchar*) NULL + 1, 28, 1, (uchar*) 0, 0, + Field::NONE, &fname, &dummy_share, + DTCollation(&my_charset_latin1)); + vs->field_index= 0; + + Hp_test_single_key ctx; + ctx.init(vs, 1, 30); + + HP_CREATE_INFO hp_ci; + int err= ctx.run_hp_create(&hp_ci); + ok(err == 0, "varchar_only: heap_prepare succeeded (err=%d)", err); + + HA_KEYSEG *seg= hp_ci.keydef[0].seg; + ok(seg->length == 28, + "varchar_only: seg->length = %u (expected 28)", (uint) seg->length); + ok(seg->type == HA_KEYTYPE_VARTEXT1, + "varchar_only: seg->type = %d (expected VARTEXT1=%d)", + (int) seg->type, (int) HA_KEYTYPE_VARTEXT1); + /* + bit_start for varchar is set by hp_create(), not + heap_prepare_hp_create_info(). After prepare it's 0. + */ + ok(seg->bit_start == 0, + "varchar_only: seg->bit_start = %u (expected 0 -- set later by hp_create)", + (uint) seg->bit_start); + ok(!(seg->flag & HA_BLOB_PART), + "varchar_only: seg->flag (0x%x) has NO HA_BLOB_PART", + (uint) seg->flag); + ok((seg->flag & HA_VAR_LENGTH_PART), + "varchar_only: seg->flag (0x%x) has HA_VAR_LENGTH_PART", + (uint) seg->flag); + + my_free(hp_ci.keydef); + vs->~Field_varstring(); +} + + +static void test_int_only_key() +{ + /* INT NOT NULL */ + static const LEX_CSTRING fname= {STRING_WITH_LEN("i1")}; + TABLE_SHARE dummy_share; + memset(static_cast(&dummy_share), 0, sizeof(dummy_share)); + alignas(Field_long) char fl_storage[sizeof(Field_long)]; + Field_long *fl= ::new (fl_storage) Field_long( + (uchar*) NULL + 1, 11, (uchar*) 0, 0, + Field::NONE, &fname, false, false); + fl->field_index= 0; + + Hp_test_single_key ctx; + ctx.init(fl, 1, 5); + + HP_CREATE_INFO hp_ci; + int err= ctx.run_hp_create(&hp_ci); + ok(err == 0, "int_only: heap_prepare succeeded (err=%d)", err); + + HA_KEYSEG *seg= hp_ci.keydef[0].seg; + ok(seg->length == 4, + "int_only: seg->length = %u (expected 4)", (uint) seg->length); + ok(seg->type == HA_KEYTYPE_BINARY, + "int_only: seg->type = %d (expected BINARY=%d)", + (int) seg->type, (int) HA_KEYTYPE_BINARY); + ok(!(seg->flag & HA_BLOB_PART), + "int_only: seg->flag (0x%x) has NO HA_BLOB_PART", + (uint) seg->flag); + ok(!(seg->flag & HA_VAR_LENGTH_PART), + "int_only: seg->flag (0x%x) has NO HA_VAR_LENGTH_PART", + (uint) seg->flag); + + my_free(hp_ci.keydef); + fl->~Field_long(); +} + + +static void test_enum_key() +{ + /* ENUM('a','','b') NULLABLE */ + static const LEX_CSTRING fname= {STRING_WITH_LEN("e1")}; + static const char *enum_names[]= { "a", "", "b", NULL }; + static unsigned int enum_lengths[]= { 1, 0, 1 }; + TYPELIB enum_typelib= { 3, "", enum_names, enum_lengths, NULL }; + Type_typelib_attributes enum_typelib_attr(enum_typelib); + TABLE_SHARE dummy_share; + memset(static_cast(&dummy_share), 0, sizeof(dummy_share)); + alignas(Field_enum) char fe_storage[sizeof(Field_enum)]; + /* + Field_enum(ptr, len, null_ptr, null_bit, unireg, name, + packlength, typelib, collation) + */ + Field_enum *fe= ::new (fe_storage) Field_enum( + (uchar*) NULL + 1, 1, (uchar*) NULL, 2, + Field::NONE, &fname, 1, &enum_typelib_attr, + &my_charset_latin1); + fe->field_index= 0; + + Hp_test_single_key ctx; + ctx.init(fe, 1, 3); + + HP_CREATE_INFO hp_ci; + int err= ctx.run_hp_create(&hp_ci); + ok(err == 0, "enum: heap_prepare succeeded (err=%d)", err); + + HA_KEYSEG *seg= hp_ci.keydef[0].seg; + ok(seg->length == 1, + "enum: seg->length = %u (expected 1 = packlength)", (uint) seg->length); + ok(seg->type == HA_KEYTYPE_BINARY, + "enum: seg->type = %d (expected BINARY=%d)", + (int) seg->type, (int) HA_KEYTYPE_BINARY); + ok(!(seg->flag & HA_BLOB_PART), + "enum: seg->flag (0x%x) has NO HA_BLOB_PART", (uint) seg->flag); + + my_free(hp_ci.keydef); + fe->~Field_enum(); +} + + +static void test_mixed_int_varchar_key() +{ + /* + Two-part key: INT(4 bytes) + VARCHAR(20), simulating the + main.having GROUP BY (bigint, varchar(20)). + */ + static const LEX_CSTRING fname_i= {STRING_WITH_LEN("id")}; + static const LEX_CSTRING fname_v= {STRING_WITH_LEN("description")}; + TABLE_SHARE dummy_share; + memset(static_cast(&dummy_share), 0, sizeof(dummy_share)); + dummy_share.fields= 2; + dummy_share.keys= 1; + dummy_share.reclength= 26; /* 1 null + 4 int + 1 len + 20 varchar */ + dummy_share.rec_buff_length= 26; + dummy_share.db_record_offset= 1; + dummy_share.blob_fields= 0; + uint blob_offsets[1]= { 0 }; + dummy_share.blob_field= blob_offsets; + + alignas(Field_long) char fl_storage[sizeof(Field_long)]; + Field_long *fl= ::new (fl_storage) Field_long( + (uchar*) NULL + 1, 11, (uchar*) 0, 0, + Field::NONE, &fname_i, false, false); + fl->field_index= 0; + + alignas(Field_varstring) char vs_storage[sizeof(Field_varstring)]; + Field_varstring *vs= ::new (vs_storage) Field_varstring( + (uchar*) NULL + 5, 20, 1, (uchar*) 0, 0, + Field::NONE, &fname_v, &dummy_share, + DTCollation(&my_charset_latin1)); + vs->field_index= 1; + + Field *field_array[3]= { fl, vs, NULL }; + + KEY_PART_INFO kpis[2]; + memset(kpis, 0, sizeof(kpis)); + kpis[0].field= fl; + kpis[0].offset= 1; + kpis[0].length= 4; + kpis[0].key_part_flag= fl->key_part_flag(); + kpis[0].type= fl->key_type(); + kpis[0].store_length= 4; + + kpis[1].field= vs; + kpis[1].offset= 5; + kpis[1].length= 20; + kpis[1].key_part_flag= vs->key_part_flag(); + kpis[1].type= vs->key_type(); + kpis[1].store_length= 20 + 2; /* + key_part_length_bytes */ + + KEY sql_key= {}; + sql_key.user_defined_key_parts= 2; + sql_key.usable_key_parts= 2; + sql_key.key_part= kpis; + sql_key.algorithm= HA_KEY_ALG_HASH; + sql_key.key_length= 4 + 20 + 2; + + TABLE test_table; + uchar rec_buf[26]; + memset(rec_buf, 0, sizeof(rec_buf)); + memset(static_cast(&test_table), 0, sizeof(test_table)); + test_table.record[0]= rec_buf; + test_table.s= &dummy_share; + test_table.field= field_array; + test_table.key_info= &sql_key; + dummy_share.key_info= &sql_key; + + fl->table= &test_table; + vs->table= &test_table; + + Fake_thd_guard thd_guard; + + HP_CREATE_INFO hp_ci; + memset(&hp_ci, 0, sizeof(hp_ci)); + hp_ci.max_table_size= 1024*1024; + hp_ci.keys= 1; + hp_ci.reclength= 26; + + int err= heap_prepare_hp_create_info(&test_table, TRUE, &hp_ci); + + ok(err == 0, "int_varchar: heap_prepare succeeded (err=%d)", err); + + HP_KEYDEF *kd= &hp_ci.keydef[0]; + ok(kd->keysegs == 2, + "int_varchar: keysegs = %u (expected 2)", kd->keysegs); + { + my_bool any_blob= FALSE; + uint j; + for (j= 0; j < kd->keysegs; j++) + if (kd->seg[j].flag & HA_BLOB_PART) + any_blob= TRUE; + ok(!any_blob, + "int_varchar: no keydef seg has HA_BLOB_PART"); + } + + HA_KEYSEG *seg0= &kd->seg[0]; + ok(seg0->length == 4, + "int_varchar: seg[0].length = %u (expected 4)", (uint) seg0->length); + ok(!(seg0->flag & HA_BLOB_PART), + "int_varchar: seg[0] has NO HA_BLOB_PART"); + + HA_KEYSEG *seg1= &kd->seg[1]; + ok(seg1->length == 20, + "int_varchar: seg[1].length = %u (expected 20)", (uint) seg1->length); + ok(!(seg1->flag & HA_BLOB_PART), + "int_varchar: seg[1] has NO HA_BLOB_PART"); + ok((seg1->flag & HA_VAR_LENGTH_PART), + "int_varchar: seg[1] has HA_VAR_LENGTH_PART"); + + my_free(hp_ci.keydef); + vs->~Field_varstring(); + fl->~Field_long(); +} + + +/* + Test: geometry GROUP BY key must NOT trigger blob key widening. + + Field_geom::key_length() returns packlength (4), not 0 like Field_blob. + The widening condition in heap_prepare_hp_create_info must skip when + key_part->length <= pack_length_no_ptr(). Without this, len_delta + overflows (~4 billion), corrupting store_length and key_length, which + causes rebuild_key_from_group_buff() to read uninitialized memory. + + This test simulates a GROUP BY on a GEOMETRY(POINT) column: + - key_part->length = 4 (from Field_geom::key_length() = packlength) + - key_part->store_length = small (from GROUP BY buffer sizing) + After heap_prepare, key_part->length must still be 4 (not widened), + and store_length must not overflow. +*/ +static void test_geometry_group_by_no_widening() +{ + /* + Record layout: nullable geometry (POINT, packlength=4) + byte 0: null bitmap + bytes 1-4: blob packlength=4 + bytes 5-12: blob data pointer + reclength = 13 + */ + uchar rec[16]; + memset(rec, 0, sizeof(rec)); + + TABLE_SHARE share; + memset(static_cast(&share), 0, sizeof(share)); + share.fields= 1; + share.blob_fields= 0; + share.keys= 1; + share.reclength= 13; + share.rec_buff_length= 13; + share.db_record_offset= 1; + + /* GEOMETRY is a LONGBLOB (packlength=4) */ + alignas(Field_blob) char bf_storage[sizeof(Field_blob)]; + Field_blob *bfp= make_test_field_blob(bf_storage, + rec + 1, + rec + 0, + 2, &share, + 4 /* packlength for LONGBLOB */, + &my_charset_bin); + bfp->field_index= 0; + + Field *field_array[2]= { bfp, NULL }; + + KEY_PART_INFO kpi; + memset(&kpi, 0, sizeof(kpi)); + kpi.field= bfp; + kpi.offset= 1; + /* + GROUP BY path: Field_geom::key_length() returns packlength = 4. + finalize() sets m_key_part_info->length = field->key_length() = 4. + */ + kpi.length= 4; + kpi.key_part_flag= HA_BLOB_PART; + kpi.null_bit= 2; + kpi.null_offset= 0; + kpi.type= bfp->key_type(); + /* + GROUP BY store_length: set by finalize() from the group buffer + Field_varstring. Use a reasonable value (e.g. 100 + 2 + 1 = 103). + */ + kpi.store_length= 103; + + KEY sql_key= {}; + sql_key.user_defined_key_parts= 1; + sql_key.usable_key_parts= 1; + sql_key.key_part= &kpi; + sql_key.algorithm= HA_KEY_ALG_HASH; + sql_key.key_length= kpi.store_length; + + TABLE test_table; + memset(static_cast(&test_table), 0, sizeof(test_table)); + test_table.record[0]= rec; + test_table.s= &share; + test_table.field= field_array; + test_table.key_info= &sql_key; + share.key_info= &sql_key; + bfp->table= &test_table; + + uint blob_offsets[1]= { 0 }; + share.blob_field= blob_offsets; + + /* Set group to simulate GROUP BY path */ + ORDER group_item= {}; + test_table.group= &group_item; + + Fake_thd_guard thd_guard; + + HP_CREATE_INFO hp_ci; + memset(&hp_ci, 0, sizeof(hp_ci)); + hp_ci.max_table_size= 1024*1024; + hp_ci.keys= 1; + hp_ci.reclength= 13; + + uint16 orig_length= kpi.length; + uint orig_store_length= kpi.store_length; + uint orig_key_length= sql_key.key_length; + + int err= heap_prepare_hp_create_info(&test_table, TRUE, &hp_ci); + ok(err == 0, "geom_group_by: heap_prepare succeeded (err=%d)", err); + + /* key_part->length must NOT be widened -- must stay at packlength (4) */ + ok(kpi.length == orig_length, + "geom_group_by: key_part.length = %u (expected %u, NOT widened)", + (uint) kpi.length, (uint) orig_length); + + /* store_length must not overflow */ + ok(kpi.store_length == orig_store_length, + "geom_group_by: store_length = %u (expected %u, NOT overflowed)", + (uint) kpi.store_length, (uint) orig_store_length); + + /* key_length must not overflow */ + ok(sql_key.key_length == orig_key_length, + "geom_group_by: key_length = %u (expected %u, NOT overflowed)", + (uint) sql_key.key_length, (uint) orig_key_length); + + /* seg->length = 4+ptr (blob key format) */ + ok(hp_ci.keydef[0].seg[0].length == 4 + portable_sizeof_char_ptr, + "geom_group_by: seg->length = %u (expected %u = 4+ptr)", + (uint) hp_ci.keydef[0].seg[0].length, + (uint)(4 + portable_sizeof_char_ptr)); + + + my_free(hp_ci.keydef); + my_free(hp_ci.blob_descs); + bfp->~Field_blob(); +} + + +int main(int argc __attribute__((unused)), + char **argv __attribute__((unused))) +{ + MY_INIT("hp_test_key_setup"); + /* Field constructors reference system_charset_info via DTCollation */ + system_charset_info= &my_charset_latin1; + plan(34); + + diag("distinct_key_truncation: blob segment normalization"); + test_distinct_key_truncation(); + + diag("varchar_only: VARCHAR key has no blob flag"); + test_varchar_only_key(); + + diag("int_only: INT key has no blob flag"); + test_int_only_key(); + + diag("enum: ENUM key has no blob flag"); + test_enum_key(); + + diag("int_varchar: mixed INT+VARCHAR key has no blob flag"); + test_mixed_int_varchar_key(); + + diag("geom_group_by: geometry GROUP BY key must not trigger blob key widening"); + test_geometry_group_by_no_widening(); + + my_end(0); + return exit_status(); +} diff --git a/storage/heap/hp_test_scan-t.c b/storage/heap/hp_test_scan-t.c new file mode 100644 index 0000000000000..8b00992a4a38b --- /dev/null +++ b/storage/heap/hp_test_scan-t.c @@ -0,0 +1,545 @@ +/* + Unit tests for heap_scan() internal continuation record skipping. + + Verifies that heap_scan() skips blob continuation records internally + (via goto retry) rather than returning HA_ERR_RECORD_DELETED to the + caller. Uses real HEAP tables with blob columns. +*/ + +#include "hp_test_helpers.h" + + +/* + Test 1: scan with continuation records never returns HA_ERR_RECORD_DELETED. + + Inserts rows with blobs large enough to create continuation chains + (recbuffer=16, so >5 bytes needs continuations), then scans and + verifies that heap_scan returns only 0 or HA_ERR_END_OF_FILE. +*/ +static void test_scan_skips_continuations(void) +{ + HP_SHARE *share; + HP_INFO *info; + uchar rec[REC_LENGTH]; + uchar scan_buf[REC_LENGTH]; + int error; + uint row_count= 0; + my_bool got_record_deleted= FALSE; + + uchar blob1[50], blob2[80]; + memset(blob1, 'A', sizeof(blob1)); + memset(blob2, 'B', sizeof(blob2)); + blob1[0]= '1'; blob1[49]= 'Z'; + blob2[0]= '2'; blob2[79]= 'Z'; + + if (create_and_open("test_scan_cont", &share, &info)) + { + ok(0, "setup failed: %d", my_errno); + skip(5, "setup failed"); + return; + } + + build_record(rec, 1, blob1, sizeof(blob1)); + ok(heap_write(info, rec) == 0, "insert row 1 (50-byte blob)"); + + build_record(rec, 2, blob2, sizeof(blob2)); + ok(heap_write(info, rec) == 0, "insert row 2 (80-byte blob)"); + + ok(share->records == 2, + "records == 2 (got %lu)", (ulong) share->records); + ok(share->total_records > share->records, + "total_records (%lu) > records (%lu)", + (ulong) share->total_records, (ulong) share->records); + + ok(heap_check_heap(info, 0) == 0, + "heap_check_heap validates after inserts with continuations"); + + heap_scan_init(info); + while ((error= heap_scan(info, scan_buf)) != HA_ERR_END_OF_FILE) + { + if (error == HA_ERR_RECORD_DELETED) + got_record_deleted= TRUE; + else if (error == 0) + row_count++; + else + break; + } + + ok(!got_record_deleted, + "heap_scan never returned HA_ERR_RECORD_DELETED for continuations"); + ok(row_count == 2, + "scan returned 2 rows (got %u)", row_count); + + heap_drop_table(info); + heap_close(info); +} + + +/* + Test 2: scan with deleted rows AND continuation records. + + Inserts 3 rows with blobs, deletes the middle one, then scans. + Deleted rows should still return HA_ERR_RECORD_DELETED (existing + behavior), but continuation records must be skipped internally. +*/ +static void test_scan_deleted_plus_continuations(void) +{ + HP_SHARE *share; + HP_INFO *info; + uchar rec[REC_LENGTH]; + uchar scan_buf[REC_LENGTH]; + int error; + uint row_count= 0; + uint deleted_count= 0; + + uchar blob1[40], blob2[60], blob3[45]; + memset(blob1, 'X', sizeof(blob1)); + memset(blob2, 'Y', sizeof(blob2)); + memset(blob3, 'Z', sizeof(blob3)); + + if (create_and_open("test_scan_del", &share, &info)) + { + ok(0, "setup failed: %d", my_errno); + skip(5, "setup failed"); + return; + } + + build_record(rec, 10, blob1, sizeof(blob1)); + ok(heap_write(info, rec) == 0, "insert row 10"); + + build_record(rec, 20, blob2, sizeof(blob2)); + ok(heap_write(info, rec) == 0, "insert row 20"); + + build_record(rec, 30, blob3, sizeof(blob3)); + ok(heap_write(info, rec) == 0, "insert row 30"); + + /* Delete row 20 via key lookup */ + { + uchar key[4]; + int4store(key, 20); + ok(heap_rkey(info, rec, 0, key, 4, HA_READ_KEY_EXACT) == 0, + "found row 20 for deletion"); + ok(heap_delete(info, rec) == 0, "deleted row 20"); + } + + ok(share->records == 2, + "records == 2 after delete (got %lu)", (ulong) share->records); + + ok(heap_check_heap(info, 0) == 0, + "heap_check_heap validates after delete with continuations"); + + heap_scan_init(info); + while ((error= heap_scan(info, scan_buf)) != HA_ERR_END_OF_FILE) + { + if (error == HA_ERR_RECORD_DELETED) + deleted_count++; + else if (error == 0) + row_count++; + else + break; + } + + ok(row_count == 2, "scan returned 2 live rows (got %u)", row_count); + ok(deleted_count > 0, + "scan returned HA_ERR_RECORD_DELETED for deleted slots (%u times)", + deleted_count); + + heap_drop_table(info); + heap_close(info); +} + + +/* + Test 3: scan a non-blob table is unaffected. + + Inserts rows without blobs, scans, verifies existing behavior is + unchanged (deleted records still return HA_ERR_RECORD_DELETED). +*/ +static void test_scan_no_blobs(void) +{ + HP_KEYDEF keydef; + HA_KEYSEG keyseg; + HP_CREATE_INFO ci; + HP_SHARE *share; + HP_INFO *info; + my_bool unused; + uchar rec[REC_LENGTH]; + uchar scan_buf[REC_LENGTH]; + int error; + uint row_count= 0; + uint deleted_count= 0; + + memset(&keyseg, 0, sizeof(keyseg)); + keyseg.type= HA_KEYTYPE_BINARY; + keyseg.start= INT_OFFSET; + keyseg.length= 4; + keyseg.charset= &my_charset_bin; + + memset(&keydef, 0, sizeof(keydef)); + keydef.keysegs= 1; + keydef.seg= &keyseg; + keydef.algorithm= HA_KEY_ALG_HASH; + keydef.flag= HA_NOSAME; + keydef.length= 4; + + memset(&ci, 0, sizeof(ci)); + ci.keys= 1; + ci.keydef= &keydef; + ci.reclength= REC_LENGTH; + ci.max_records= 1000; + ci.min_records= 10; + ci.max_table_size= 1024 * 1024; + + if (heap_create("test_scan_noblob", &ci, &share, &unused)) + { + ok(0, "setup failed: %d", my_errno); + skip(4, "setup failed"); + return; + } + info= heap_open("test_scan_noblob", 2); + if (!info) + { + ok(0, "open failed: %d", my_errno); + skip(4, "open failed"); + return; + } + + /* Insert 3 rows (no blob data, just int) */ + memset(rec, 0, REC_LENGTH); + int4store(rec + INT_OFFSET, 100); + ok(heap_write(info, rec) == 0, "insert row 100"); + + int4store(rec + INT_OFFSET, 200); + ok(heap_write(info, rec) == 0, "insert row 200"); + + int4store(rec + INT_OFFSET, 300); + ok(heap_write(info, rec) == 0, "insert row 300"); + + /* Delete middle row */ + { + uchar key[4]; + int4store(key, 200); + heap_rkey(info, rec, 0, key, 4, HA_READ_KEY_EXACT); + heap_delete(info, rec); + } + + heap_scan_init(info); + while ((error= heap_scan(info, scan_buf)) != HA_ERR_END_OF_FILE) + { + if (error == HA_ERR_RECORD_DELETED) + deleted_count++; + else if (error == 0) + row_count++; + else + break; + } + + ok(row_count == 2, "no-blob scan returned 2 live rows (got %u)", row_count); + ok(deleted_count > 0, + "no-blob scan returned HA_ERR_RECORD_DELETED for deleted slots (%u)", + deleted_count); + + ok(heap_check_heap(info, 0) == 0, + "heap_check_heap validates non-blob table"); + + heap_drop_table(info); + heap_close(info); +} + + +/* + Test 4: heap_check_heap with Case A single-record continuations. + + Inserts rows with small blobs (<= 5 bytes for recbuffer=16) that use + Case A layout (HP_ROW_SINGLE_REC: no header, data at offset 0), and + verifies that heap_check_heap correctly counts them. Also inserts + a larger blob (Case B) to exercise both continuation paths. +*/ +static void test_check_heap_with_case_a(void) +{ + HP_SHARE *share; + HP_INFO *info; + uchar rec[REC_LENGTH]; + + uchar small_blob[3]= {'a', 'b', 'c'}; + uchar tiny_blob[5]= {'1', '2', '3', '4', '5'}; + uchar medium_blob[50]; + memset(medium_blob, 'M', sizeof(medium_blob)); + + if (create_and_open("test_check_case_a", &share, &info)) + { + ok(0, "setup failed: %d", my_errno); + skip(8, "setup failed"); + return; + } + + build_record(rec, 1, small_blob, sizeof(small_blob)); + ok(heap_write(info, rec) == 0, "insert row 1 (3-byte blob, Case A)"); + + build_record(rec, 2, tiny_blob, sizeof(tiny_blob)); + ok(heap_write(info, rec) == 0, "insert row 2 (5-byte blob, Case A)"); + + build_record(rec, 3, medium_blob, sizeof(medium_blob)); + ok(heap_write(info, rec) == 0, "insert row 3 (50-byte blob, Case B)"); + + ok(share->records == 3, "records == 3 (got %lu)", (ulong) share->records); + ok(share->total_records > share->records, + "total_records (%lu) > records (%lu) due to continuations", + (ulong) share->total_records, (ulong) share->records); + + ok(heap_check_heap(info, 0) == 0, + "heap_check_heap validates Case A + Case B continuations"); + + /* Delete a Case A row and verify check still passes */ + { + uchar key[4]; + int4store(key, 1); + ok(heap_rkey(info, rec, 0, key, 4, HA_READ_KEY_EXACT) == 0, + "found row 1 (Case A) for deletion"); + ok(heap_delete(info, rec) == 0, "deleted row 1"); + } + + ok(heap_check_heap(info, 0) == 0, + "heap_check_heap validates after deleting Case A row"); + + heap_drop_table(info); + heap_close(info); +} + + +/* + Create and open a blob-keyed table: hash key on the blob column. + + Record layout same as hp_test_helpers.h, but the hash key is on the + blob column instead of the int column. This exercises check_one_key() + blob hash materialization, because the stored record's blob pointers + point to continuation chain heads (not raw data). +*/ +static int create_and_open_blob_key(const char *name, + HP_SHARE **share, HP_INFO **info) +{ + HP_KEYDEF keydef; + HA_KEYSEG keyseg; + HP_CREATE_INFO ci; + HP_BLOB_DESC blob_desc; + my_bool unused; + + memset(&keyseg, 0, sizeof(keyseg)); + keyseg.type= HA_KEYTYPE_VARTEXT4; + keyseg.start= BLOB_OFFSET; + keyseg.length= 4 + portable_sizeof_char_ptr; + keyseg.bit_start= BLOB_PACKLEN; + keyseg.charset= &my_charset_bin; + keyseg.flag= HA_BLOB_PART; + + memset(&keydef, 0, sizeof(keydef)); + keydef.keysegs= 1; + keydef.seg= &keyseg; + keydef.algorithm= HA_KEY_ALG_HASH; + keydef.flag= 0; + keydef.length= keyseg.length; + + blob_desc.offset= BLOB_OFFSET; + blob_desc.packlength= BLOB_PACKLEN; + + memset(&ci, 0, sizeof(ci)); + ci.keys= 1; + ci.keydef= &keydef; + ci.reclength= REC_LENGTH; + ci.max_records= 1000; + ci.min_records= 10; + ci.max_table_size= 1024 * 1024; + ci.blob_descs= &blob_desc; + ci.blob_count= 1; + + if (heap_create(name, &ci, share, &unused)) + return 1; + *info= heap_open(name, 2); + if (!*info) + return 1; + heap_extra(*info, HA_EXTRA_NO_READCHECK); + return 0; +} + + +/* + Test 5: heap_check_heap with blob hash key. + + Creates a table with a hash key on the blob column, inserts rows + with Case A, B, and C blobs, and calls heap_check_heap(). This + exercises check_one_key() blob hash materialization: the stored + record's blob pointers point to chain heads (with headers for + Case B/C), not raw data, so the hash must be recomputed via + hp_materialize_one_blob(). +*/ +static void test_check_heap_blob_key(void) +{ + HP_SHARE *share; + HP_INFO *info; + uchar rec[REC_LENGTH]; + + uchar blob_a[3]= {'X', 'Y', 'Z'}; + uchar blob_b[60]; + uchar blob_c[200]; + memset(blob_b, 'B', sizeof(blob_b)); + memset(blob_c, 'C', sizeof(blob_c)); + blob_b[0]= '<'; blob_b[59]= '>'; + blob_c[0]= '{'; blob_c[199]= '}'; + + if (create_and_open_blob_key("test_blob_key", &share, &info)) + { + ok(0, "setup failed: %d", my_errno); + skip(9, "setup failed"); + return; + } + + /* Case A: blob <= 5 bytes */ + build_record(rec, 1, blob_a, sizeof(blob_a)); + ok(heap_write(info, rec) == 0, "insert row 1 (3-byte blob, Case A key)"); + + /* Case B: blob fits in single run, zero-copy layout */ + build_record(rec, 2, blob_b, sizeof(blob_b)); + ok(heap_write(info, rec) == 0, "insert row 2 (60-byte blob, Case B key)"); + + /* Case C: blob large enough for multi-run */ + build_record(rec, 3, blob_c, sizeof(blob_c)); + ok(heap_write(info, rec) == 0, "insert row 3 (200-byte blob, Case C key)"); + + ok(share->records == 3, "records == 3 (got %lu)", (ulong) share->records); + + ok(heap_check_heap(info, 0) == 0, + "heap_check_heap validates blob hash key (Case A + B + C)"); + + /* Delete Case B row and verify check still passes */ + { + uchar read_buf[REC_LENGTH]; + heap_scan_init(info); + while (heap_scan(info, read_buf) == 0) + { + uint16 len= uint2korr(read_buf + BLOB_OFFSET); + if (len == sizeof(blob_b)) + { + ok(heap_delete(info, read_buf) == 0, "deleted Case B row"); + break; + } + } + } + + ok(share->records == 2, "records == 2 after delete (got %lu)", + (ulong) share->records); + + ok(heap_check_heap(info, 0) == 0, + "heap_check_heap validates blob key after delete"); + + /* Insert new row to verify reuse + check */ + { + uchar blob_new[40]; + memset(blob_new, 'N', sizeof(blob_new)); + build_record(rec, 4, blob_new, sizeof(blob_new)); + ok(heap_write(info, rec) == 0, + "insert row 4 (40-byte blob, reuse after delete)"); + } + + ok(heap_check_heap(info, 0) == 0, + "heap_check_heap validates blob key after reinsert"); + + heap_drop_table(info); + heap_close(info); +} + + +/* + Test 6: hp_key_cmp() blob mismatch via hash collision. + + Two 2-byte blobs \x00\x69 and \x01\x07 produce identical + my_hash_sort_bin() output (hash 66889, initial state nr=1 nr2=4). + Inserting both into a blob hash key table puts them in the same + hash chain. Looking up the first-inserted value walks through the + second entry first (LIFO), calling hp_key_cmp() which returns 1 + (mismatch on the strnncollsp comparison), then finds the match. +*/ +static void test_key_cmp_blob_collision(void) +{ + HP_SHARE *share; + HP_INFO *info; + uchar rec[REC_LENGTH]; + uchar read_buf[REC_LENGTH]; + uchar key_buf[4 + sizeof(uchar*)]; + + /* + Pre-computed hash collision pair for my_hash_sort_bin() + with initial state nr=1, nr2=4. + */ + uchar blob_val1[2]= {0x00, 0x69}; + uchar blob_val2[2]= {0x01, 0x07}; + + if (create_and_open_blob_key("test_collision", &share, &info)) + { + ok(0, "setup failed: %d", my_errno); + skip(5, "setup failed"); + return; + } + + build_record(rec, 1, blob_val1, sizeof(blob_val1)); + ok(heap_write(info, rec) == 0, "insert blob \\x00\\x69"); + + build_record(rec, 2, blob_val2, sizeof(blob_val2)); + ok(heap_write(info, rec) == 0, "insert blob \\x01\\x07 (same hash)"); + + /* + Look up the first-inserted record. hp_search() walks the chain + starting from the most-recently-inserted entry (blob_val2). + hp_key_cmp() compares blob_val2 against the search key (blob_val1) + and returns 1 (mismatch), exercising hp_hash.c line 652. + */ + build_record(rec, 1, blob_val1, sizeof(blob_val1)); + hp_make_key(share->keydef, key_buf, rec); + ok(heap_rkey(info, read_buf, 0, key_buf, 1, HA_READ_KEY_EXACT) == 0, + "lookup blob \\x00\\x69 through collision chain"); + + /* Verify we got the right record with correct blob content */ + ok(sint4korr(read_buf + INT_OFFSET) == 1, + "found record has int_val=1 (got %d)", sint4korr(read_buf + INT_OFFSET)); + { + uint16 got_len= uint2korr(read_buf + BLOB_OFFSET); + const uchar *got_data; + memcpy(&got_data, read_buf + BLOB_OFFSET + BLOB_PACKLEN, sizeof(got_data)); + ok(got_len == 2 && got_data[0] == 0x00 && got_data[1] == 0x69, + "returned blob is \\x00\\x69 (got len=%u [%02x,%02x])", + got_len, got_data[0], got_data[1]); + } + + ok(heap_check_heap(info, 0) == 0, "heap_check_heap after collision test"); + + heap_drop_table(info); + heap_close(info); +} + + +int main(int argc __attribute__((unused)), + char **argv __attribute__((unused))) +{ + MY_INIT("hp_test_scan"); + plan(47); + + diag("Test 1: scan skips continuation records internally"); + test_scan_skips_continuations(); + + diag("Test 2: deleted rows + continuations"); + test_scan_deleted_plus_continuations(); + + diag("Test 3: non-blob table scan unchanged"); + test_scan_no_blobs(); + + diag("Test 4: heap_check_heap with Case A single-record continuations"); + test_check_heap_with_case_a(); + + diag("Test 5: heap_check_heap with blob hash key (materialization)"); + test_check_heap_blob_key(); + + diag("Test 6: hp_key_cmp blob mismatch via pre-computed hash collision"); + test_key_cmp_blob_collision(); + + my_end(0); + return exit_status(); +} diff --git a/storage/heap/hp_update.c b/storage/heap/hp_update.c index ad56ca979deb6..f822c8daaf803 100644 --- a/storage/heap/hp_update.c +++ b/storage/heap/hp_update.c @@ -28,6 +28,7 @@ int heap_update(HP_INFO *info, const uchar *old, const uchar *heap_new) DBUG_ENTER("heap_update"); test_active(info); + hp_flush_pending_blob_free(info); pos=info->current_ptr; if (info->opt_flag & READ_CHECK_USED && hp_rectest(info,old)) @@ -42,7 +43,7 @@ int heap_update(HP_INFO *info, const uchar *old, const uchar *heap_new) p_lastinx= share->keydef + info->lastinx; for (keydef= share->keydef, end= keydef + share->keys; keydef < end; keydef++) { - if (hp_rec_key_cmp(keydef, old, heap_new)) + if (hp_rec_key_cmp(keydef, heap_new, old, NULL)) { if ((*keydef->delete_key)(info, keydef, old, pos, keydef == p_lastinx) || (*keydef->write_key)(info, keydef, heap_new, pos)) @@ -52,7 +53,194 @@ int heap_update(HP_INFO *info, const uchar *old, const uchar *heap_new) } } - memcpy(pos,heap_new,(size_t) share->reclength); + /* + Blob update strategy: skip unchanged blobs, write-before-free for + changed ones. + + Compare each blob column (length, then pointer, then memcmp) to + detect changes. Unchanged blobs keep their existing chains. + Changed blobs get new chains written before old ones are freed. + + The bulk memcpy of heap_new into pos overwrites blob chain pointers + with SQL-layer data pointers, so we save old chain pointers first + and restore them for unchanged blobs afterward. + */ + if (share->blob_count) + { + my_bool had_cont= hp_has_cont(pos, share->visible); + uint alloc_size= share->blob_count * (sizeof(uchar*) + sizeof(my_bool)); + uchar **saved_chains= (uchar**) my_safe_alloca(alloc_size); + my_bool *blob_changed= (my_bool*)(saved_chains + share->blob_count); + my_bool any_changed= FALSE; + my_bool has_blob_data= FALSE; + HP_BLOB_DESC *desc; + uint32 new_len; + uint i; + + /* Save old chain pointers and detect which blobs changed */ + for (i= 0, desc= share->blob_descs; i < share->blob_count; i++, desc++) + { + uint32 old_len, cur_len; + + saved_chains[i]= NULL; + if (had_cont) + memcpy(&saved_chains[i], pos + desc->offset + desc->packlength, + sizeof(saved_chains[i])); + + old_len= hp_blob_length(desc, old); + cur_len= hp_blob_length(desc, heap_new); + + if (old_len != cur_len) + blob_changed[i]= TRUE; + else if (old_len == 0) + blob_changed[i]= FALSE; + else + { + const uchar *old_data, *new_data; + memcpy(&old_data, old + desc->offset + desc->packlength, + sizeof(old_data)); + memcpy(&new_data, heap_new + desc->offset + desc->packlength, + sizeof(new_data)); + blob_changed[i]= (old_data != new_data && + memcmp(old_data, new_data, old_len) != 0); + } + any_changed|= blob_changed[i]; + } + + memcpy(pos, heap_new, (size_t) share->reclength); + + /* Write new chains for changed blobs, restore old pointers for unchanged */ + for (i= 0, desc= share->blob_descs; i < share->blob_count; i++, desc++) + { + if (!blob_changed[i]) + { + /* + Restore old chain pointer, from the old current_ptr, where the blob + data is in heap memory. This is not the same as the pointer in 'old' + as this may have been allocated from a segmented blob. + + When there is no saved chain (zero-length blob with no continuation + data), NULL out the pointer that memcpy(pos, heap_new) left behind. + Without this, a stale SQL-layer pointer (e.g. from replication event + buffer) would be interpreted as a chain head by hp_free_blobs(). + */ + if (saved_chains[i]) + { + memcpy(pos + desc->offset + desc->packlength, + &saved_chains[i], sizeof(saved_chains[i])); + has_blob_data= TRUE; + } + else + bzero(pos + desc->offset + desc->packlength, sizeof(char*)); + continue; + } + + new_len= hp_blob_length(desc, heap_new); + if (new_len == 0) + bzero(pos + desc->offset + desc->packlength, sizeof(char*)); + else + { + const uchar *data_ptr; + uchar *first_run; + + has_blob_data= TRUE; + memcpy(&data_ptr, heap_new + desc->offset + desc->packlength, + sizeof(data_ptr)); + + if (hp_write_one_blob(share, data_ptr, new_len, &first_run)) + { + /* Rollback: free new chains already written, restore old record */ + uint j; + for (j= 0; j < i; j++) + { + if (blob_changed[j]) + { + uchar *chain; + memcpy(&chain, pos + share->blob_descs[j].offset + + share->blob_descs[j].packlength, sizeof(chain)); + if (chain) + hp_free_run_chain(share, chain); + } + } + hp_shrink_tail(share); + memcpy(pos, old, (size_t) share->reclength); + if (had_cont) + { + for (j= 0; j < share->blob_count; j++) + memcpy(pos + share->blob_descs[j].offset + + share->blob_descs[j].packlength, + &saved_chains[j], sizeof(saved_chains[j])); + pos[share->visible]|= HP_ROW_HAS_CONT; + } + my_safe_afree(saved_chains, alloc_size); + goto err; + } + memcpy(pos + desc->offset + desc->packlength, + &first_run, sizeof(first_run)); + } + } + + if (any_changed) + { + /* Set flags and free old chains for changed blobs */ + pos[share->visible]= has_blob_data ? + (HP_ROW_ACTIVE | HP_ROW_HAS_CONT) : HP_ROW_ACTIVE; + if (share->internal) + { + for (i= 0; i < share->blob_count; i++) + if (blob_changed[i] && saved_chains[i]) + hp_free_run_chain(share, saved_chains[i]); + hp_shrink_tail(share); + } + else + { + /* + Defer blob chain free for user-created tables. + + The handler layer calls binlog_log_row() AFTER update_row() + returns, reading old blob data from record[1] via zero-copy + pointers into HP_BLOCK chain records. Freeing chains here + would overwrite those records, making the pointers dangle. + */ + for (i= 0; i < share->blob_count; i++) + info->pending_blob_chains[i]= (blob_changed[i] ? + saved_chains[i] : NULL); + info->has_pending_blob_free= TRUE; + } + } + else if (had_cont) + pos[share->visible]|= HP_ROW_HAS_CONT; + + /* + Refresh blob pointers in the caller's record buffer. + + For changed blobs, pos has new chain pointers that heap_new + doesn't know about yet. Copy all chain pointers from pos into + heap_new and call hp_read_blobs() to re-materialize. + + Only do this for internal temporary tables. For user tables, + hp_read_blobs() may realloc blob_buff, invalidating Case C + pointers in old (record[1]) that binlog_log_row() still needs. + The SQL layer will re-materialize blobs on the next row fetch. + */ + if (share->internal && (any_changed || info->has_zerocopy_blobs)) + { + for (i= 0, desc= share->blob_descs; i < share->blob_count; i++, desc++) + { + uchar *chain; + memcpy(&chain, pos + desc->offset + desc->packlength, sizeof(chain)); + memcpy((uchar*) heap_new + desc->offset + desc->packlength, &chain, + sizeof(chain)); + } + hp_read_blobs(info, (uchar*) heap_new, pos); + } + + my_safe_afree(saved_chains, alloc_size); + } + else + { + memcpy(pos, heap_new, (size_t) share->reclength); + } if (++(share->records) == share->blength) share->blength+= share->blength; #if !defined(DBUG_OFF) && defined(EXTRA_HEAP_DEBUG) @@ -81,7 +269,7 @@ int heap_update(HP_INFO *info, const uchar *old, const uchar *heap_new) } while (keydef >= share->keydef) { - if (hp_rec_key_cmp(keydef, old, heap_new)) + if (hp_rec_key_cmp(keydef, heap_new, old, NULL)) { if ((*keydef->delete_key)(info, keydef, heap_new, pos, 0) || (*keydef->write_key)(info, keydef, old, pos)) diff --git a/storage/heap/hp_write.c b/storage/heap/hp_write.c index cb079eac75788..6229abc4ea89d 100644 --- a/storage/heap/hp_write.c +++ b/storage/heap/hp_write.c @@ -26,7 +26,6 @@ #define HIGHFIND 4 #define HIGHUSED 8 -static uchar *next_free_record_pos(HP_SHARE *info); static HASH_INFO *hp_find_free_hash(HP_SHARE *info, HP_BLOCK *block, ulong records); @@ -42,6 +41,7 @@ int heap_write(HP_INFO *info, const uchar *record) DBUG_RETURN(my_errno=EACCES); } #endif + hp_flush_pending_blob_free(info); if (!(pos=next_free_record_pos(share))) DBUG_RETURN(my_errno); share->changed=1; @@ -54,7 +54,26 @@ int heap_write(HP_INFO *info, const uchar *record) } memcpy(pos,record,(size_t) share->reclength); - pos[share->visible]= 1; /* Mark record as not deleted */ + if (share->blob_count) + { + if (hp_write_blobs(info, record, pos)) + { + /* + Blob write failed after all keys were written successfully. + Roll back all keys - unlike err: below, no key needs to be skipped. + + Do NOT call hp_free_blobs() here: hp_write_blobs() is self-cleaning + on failure - hp_write_one_blob() frees its own partial chain, and + hp_write_blobs() frees all previously completed columns (0..i-1) and + NULLs every chain pointer in pos. + */ + info->errkey= -1; + keydef= end - 1; + goto err_delete_written_keys; + } + } + else + pos[share->visible]= 1; /* Mark record as not deleted */ if (++share->records == share->blength) share->blength+= share->blength; info->s->key_version++; @@ -80,14 +99,23 @@ int heap_write(HP_INFO *info, const uchar *record) { keydef--; } + +err_delete_written_keys: while (keydef >= share->keydef) { if ((*keydef->delete_key)(info, keydef, record, pos, 0)) break; keydef--; - } + } + /* + Do NOT call hp_free_blobs here: the err: label is reached when a key + write fails (line 52), which is BEFORE memcpy(pos, record, reclength) + and hp_write_blobs(). The slot at pos still contains stale data from the + delete list, so hp_free_blobs would chase garbage chain pointers. + */ share->deleted++; + share->total_records--; *((uchar**) pos)=share->del_link; share->del_link=pos; pos[share->visible]= 0; /* Record deleted */ @@ -128,48 +156,111 @@ int hp_rb_write_key(HP_INFO *info, HP_KEYDEF *keyinfo, const uchar *record, return 0; } - /* Find where to place new record */ -static uchar *next_free_record_pos(HP_SHARE *info) +/* + Allocate a contiguous batch of records from the HP_BLOCK tail. + + *blocks is in/out: on entry, the maximum number of records to + allocate; on return, the actual count (capped at the remaining + slots in the current leaf block). All returned records are + contiguous in memory (run_start + i * recbuffer). + + Used by next_free_record_pos() (blocks=1) when no deleted records + are available, and by hp_write_one_blob() for blob continuation + chain allocation. + + Maintains the scan-boundary invariant: + total_records + deleted == block.last_allocated + by incrementing both last_allocated and total_records by the + allocated count. heap_scan() relies on this invariant. +*/ + +uchar *hp_alloc_from_tail(HP_SHARE *info, uint *blocks) { - int block_pos; - uchar *pos; + uint block_pos; + uint available, requested; size_t length; - DBUG_ENTER("next_free_record_pos"); + DBUG_ENTER("hp_alloc_from_tail"); - if (info->del_link) - { - pos=info->del_link; - info->del_link= *((uchar**) pos); - info->deleted--; - DBUG_PRINT("exit",("Used old position: %p", pos)); - DBUG_RETURN(pos); - } - if (!(block_pos=(info->records % info->block.records_in_block))) + DBUG_ASSERT(*blocks > 0); + if (!(block_pos= (uint)(info->block.last_allocated % + info->block.records_in_block))) { - if ((info->records > info->max_records && info->max_records) || + if ((info->block.last_allocated > info->max_records && + info->max_records) || (info->data_length + info->index_length >= info->max_table_size)) { DBUG_PRINT("error", - ("record file full. records: %lu max_records: %lu " + ("record file full. last_allocated: %lu max_records: %lu " "data_length: %llu index_length: %llu " "max_table_size: %llu", - info->records, info->max_records, + info->block.last_allocated, info->max_records, info->data_length, info->index_length, info->max_table_size)); - my_errno=HA_ERR_RECORD_FILE_FULL; + my_errno= HA_ERR_RECORD_FILE_FULL; DBUG_RETURN(NULL); } - if (hp_get_new_block(info, &info->block,&length)) - DBUG_RETURN(NULL); - info->data_length+=length; + if (info->block.last_allocated < info->block.high_water_allocated) + { + /* Block was freed by shrink_tail(). Reclaim block */ + info->block.level_info[0].last_blocks= + (HP_PTRS*) hp_find_block(&info->block, info->block.last_allocated); + } + else + { + /* No available blocks, allocate new ones */ + if (hp_get_new_block(info, &info->block, &length)) + DBUG_RETURN(NULL); + info->data_length+= length; + } } - DBUG_PRINT("exit",("Used new position: %p", + available= (uint)(info->block.records_in_block - block_pos); + requested= *blocks; + if (requested > available) + requested= available; + DBUG_ASSERT(block_pos + requested <= info->block.records_in_block); + info->block.last_allocated+= requested; + info->total_records+= requested; + *blocks= requested; + DBUG_PRINT("exit",("Used new position: %p blocks: %u", ((uchar*) info->block.level_info[0].last_blocks+ - block_pos * info->block.recbuffer))); + block_pos * info->block.recbuffer), + requested)); DBUG_RETURN((uchar*) info->block.level_info[0].last_blocks+ - block_pos*info->block.recbuffer); + block_pos * info->block.recbuffer); +} + + +/* + Find where to place a new record. + + Allocates from the free list (del_link) first; if empty, extends the + HP_BLOCK tail. Both paths maintain the scan-boundary invariant: + total_records + deleted == block.last_allocated + Free-list allocation does deleted-- + total_records++ (sum unchanged). + Tail allocation does last_allocated++ + total_records++ (sum grows by 1, + matching the new slot). heap_scan() relies on this sum to detect EOF. +*/ + +uchar *next_free_record_pos(HP_SHARE *info) +{ + uchar *pos; + DBUG_ENTER("next_free_record_pos"); + + if (info->del_link) + { + pos=info->del_link; + info->del_link= *((uchar**) pos); + info->deleted--; + info->total_records++; + DBUG_PRINT("exit",("Used old position: %p", pos)); + DBUG_RETURN(pos); + } + { + uint blocks= 1; + DBUG_RETURN(hp_alloc_from_tail(info, &blocks)); + } } @@ -347,7 +438,7 @@ int hp_write_key(HP_INFO *info, HP_KEYDEF *keyinfo, } /* Check if we are at the empty position */ - hash_of_key= hp_rec_hashnr(keyinfo, record); + hash_of_key= hp_rec_hashnr(0, keyinfo, record); pos=hp_find_hash(&keyinfo->block, hp_mask(hash_of_key, share->blength, share->records + 1)); if (pos == empty) @@ -385,7 +476,7 @@ int hp_write_key(HP_INFO *info, HP_KEYDEF *keyinfo, do { if (pos->hash_of_key == hash_of_key && - ! hp_rec_key_cmp(keyinfo, record, pos->ptr_to_rec)) + ! hp_rec_key_cmp(keyinfo, record, pos->ptr_to_rec, info)) { DBUG_RETURN(my_errno=HA_ERR_FOUND_DUPP_KEY); } diff --git a/storage/maria/ma_create.c b/storage/maria/ma_create.c index 8559655941f33..607a2a79afa62 100644 --- a/storage/maria/ma_create.c +++ b/storage/maria/ma_create.c @@ -589,6 +589,11 @@ int maria_create(const char *name, enum data_file_type datafile_type, keyseg->type == HA_KEYTYPE_VARBINARY1) ? 1 : 2); } + else + { + /* Ensure that that the blob has a fix max length */ + DBUG_ASSERT(keyseg->length > 0); + } break; default: break; diff --git a/storage/maria/ma_unique.c b/storage/maria/ma_unique.c index 548b8ac8904a8..bcb297bb9c747 100644 --- a/storage/maria/ma_unique.c +++ b/storage/maria/ma_unique.c @@ -146,7 +146,7 @@ ha_checksum _ma_unique_hash(MARIA_UNIQUEDEF *def, const uchar *record) } end= pos+length; if (type == HA_KEYTYPE_TEXT || type == HA_KEYTYPE_VARTEXT1 || - type == HA_KEYTYPE_VARTEXT2) + type == HA_KEYTYPE_VARTEXT2 || type == HA_KEYTYPE_VARTEXT4) { my_ci_hash_sort(&hasher, keyseg->charset, (const uchar*) pos, length); diff --git a/storage/mroonga/mysql-test/mroonga/storage/r/fulltext_order_boolean_mode_different_against.result b/storage/mroonga/mysql-test/mroonga/storage/r/fulltext_order_boolean_mode_different_against.result index d1f0d6bc0abb6..d41c80c4cd63f 100644 --- a/storage/mroonga/mysql-test/mroonga/storage/r/fulltext_order_boolean_mode_different_against.result +++ b/storage/mroonga/mysql-test/mroonga/storage/r/fulltext_order_boolean_mode_different_against.result @@ -15,7 +15,7 @@ Start groonga and Ruby SELECT *, MATCH(title) AGAINST("groonga" IN BOOLEAN MODE) AS score FROM diaries WHERE MATCH(title) AGAINST("groonga mroonga" IN BOOLEAN MODE) -ORDER BY MATCH(title) AGAINST("groonga" IN BOOLEAN MODE); +ORDER BY MATCH(title) AGAINST("groonga" IN BOOLEAN MODE), title; title score Start mroonga 0 Start groonga 1 diff --git a/storage/mroonga/mysql-test/mroonga/storage/r/fulltext_order_boolean_mode_no_where.result b/storage/mroonga/mysql-test/mroonga/storage/r/fulltext_order_boolean_mode_no_where.result index 125b35fb96e23..eb9cab648ba96 100644 --- a/storage/mroonga/mysql-test/mroonga/storage/r/fulltext_order_boolean_mode_no_where.result +++ b/storage/mroonga/mysql-test/mroonga/storage/r/fulltext_order_boolean_mode_no_where.result @@ -14,7 +14,7 @@ Start mroonga Start groonga and Ruby SELECT *, MATCH(title) AGAINST("groonga" IN BOOLEAN MODE) AS score FROM diaries -ORDER BY MATCH(title) AGAINST("groonga" IN BOOLEAN MODE); +ORDER BY MATCH(title) AGAINST("groonga" IN BOOLEAN MODE), title; title score Start mroonga 0 Start groonga 1 diff --git a/storage/mroonga/mysql-test/mroonga/storage/r/fulltext_order_boolean_mode_same_match_against.result b/storage/mroonga/mysql-test/mroonga/storage/r/fulltext_order_boolean_mode_same_match_against.result index a3a668c444509..8718c90f751ec 100644 --- a/storage/mroonga/mysql-test/mroonga/storage/r/fulltext_order_boolean_mode_same_match_against.result +++ b/storage/mroonga/mysql-test/mroonga/storage/r/fulltext_order_boolean_mode_same_match_against.result @@ -15,7 +15,7 @@ Start groonga and Ruby SELECT *, MATCH(title) AGAINST("groonga" IN BOOLEAN MODE) AS score FROM diaries WHERE MATCH(title) AGAINST("groonga" IN BOOLEAN MODE) -ORDER BY MATCH(title) AGAINST("groonga" IN BOOLEAN MODE); +ORDER BY MATCH(title) AGAINST("groonga" IN BOOLEAN MODE), title; title score Start groonga 1 Start groonga and Ruby 1 diff --git a/storage/mroonga/mysql-test/mroonga/storage/r/fulltext_order_natural_language_mode_different_match.result b/storage/mroonga/mysql-test/mroonga/storage/r/fulltext_order_natural_language_mode_different_match.result index 76abb660c19ef..3c3b3ca43b54f 100644 --- a/storage/mroonga/mysql-test/mroonga/storage/r/fulltext_order_natural_language_mode_different_match.result +++ b/storage/mroonga/mysql-test/mroonga/storage/r/fulltext_order_natural_language_mode_different_match.result @@ -24,13 +24,14 @@ Error host2 Warning host2 Error Error Error Error host2 Warning Warning Warning Warning host2 -SELECT *, MATCH(host) AGAINST("host2" IN NATURAL LANGUAGE MODE) AS score +SELECT *, MATCH(host) AGAINST("host2" IN NATURAL LANGUAGE MODE) AS score FROM logs WHERE MATCH(message) AGAINST("Error" IN NATURAL LANGUAGE MODE) -ORDER BY MATCH(host) AGAINST("host2" IN NATURAL LANGUAGE MODE); +ORDER BY MATCH(host) AGAINST("host2" IN NATURAL LANGUAGE MODE), +message; message host score Error Error Error host1 0 -Error Error host2 116509 Error host2 116509 +Error Error host2 116509 Error Error Error Error host2 116509 DROP TABLE logs; diff --git a/storage/mroonga/mysql-test/mroonga/storage/r/fulltext_order_natural_language_mode_no_where.result b/storage/mroonga/mysql-test/mroonga/storage/r/fulltext_order_natural_language_mode_no_where.result index 30130e6c85ba2..0d387954fc112 100644 --- a/storage/mroonga/mysql-test/mroonga/storage/r/fulltext_order_natural_language_mode_no_where.result +++ b/storage/mroonga/mysql-test/mroonga/storage/r/fulltext_order_natural_language_mode_no_where.result @@ -22,13 +22,13 @@ Error Warning Error Error Error Error Warning Warning Warning Warning -SELECT *, MATCH(message) AGAINST("Error" IN NATURAL LANGUAGE MODE) AS score +SELECT *, MATCH(message) AGAINST("Error" IN NATURAL LANGUAGE MODE) AS score FROM logs -ORDER BY MATCH(message) AGAINST("Error" IN NATURAL LANGUAGE MODE); +ORDER BY MATCH(message) AGAINST("Error" IN NATURAL LANGUAGE MODE), message; message score -Warning Warning Warning 0 -Warning Warning 0 Warning 0 +Warning Warning 0 +Warning Warning Warning 0 Warning Warning Warning Warning 0 Error 174763 Error Error 349526 diff --git a/storage/mroonga/mysql-test/mroonga/storage/t/fulltext_order_boolean_mode_different_against.test b/storage/mroonga/mysql-test/mroonga/storage/t/fulltext_order_boolean_mode_different_against.test index 90b8bc8fa106b..97a011fd0e7e3 100644 --- a/storage/mroonga/mysql-test/mroonga/storage/t/fulltext_order_boolean_mode_different_against.test +++ b/storage/mroonga/mysql-test/mroonga/storage/t/fulltext_order_boolean_mode_different_against.test @@ -35,7 +35,7 @@ SELECT * FROM diaries; SELECT *, MATCH(title) AGAINST("groonga" IN BOOLEAN MODE) AS score FROM diaries WHERE MATCH(title) AGAINST("groonga mroonga" IN BOOLEAN MODE) - ORDER BY MATCH(title) AGAINST("groonga" IN BOOLEAN MODE); + ORDER BY MATCH(title) AGAINST("groonga" IN BOOLEAN MODE), title; DROP TABLE diaries; diff --git a/storage/mroonga/mysql-test/mroonga/storage/t/fulltext_order_boolean_mode_no_where.test b/storage/mroonga/mysql-test/mroonga/storage/t/fulltext_order_boolean_mode_no_where.test index a421a31b160c7..e46d0f5bf7efc 100644 --- a/storage/mroonga/mysql-test/mroonga/storage/t/fulltext_order_boolean_mode_no_where.test +++ b/storage/mroonga/mysql-test/mroonga/storage/t/fulltext_order_boolean_mode_no_where.test @@ -34,7 +34,7 @@ SELECT * FROM diaries; SELECT *, MATCH(title) AGAINST("groonga" IN BOOLEAN MODE) AS score FROM diaries - ORDER BY MATCH(title) AGAINST("groonga" IN BOOLEAN MODE); + ORDER BY MATCH(title) AGAINST("groonga" IN BOOLEAN MODE), title; DROP TABLE diaries; diff --git a/storage/mroonga/mysql-test/mroonga/storage/t/fulltext_order_boolean_mode_same_match_against.test b/storage/mroonga/mysql-test/mroonga/storage/t/fulltext_order_boolean_mode_same_match_against.test index 3dbaa6bf15634..7ea9af71b7724 100644 --- a/storage/mroonga/mysql-test/mroonga/storage/t/fulltext_order_boolean_mode_same_match_against.test +++ b/storage/mroonga/mysql-test/mroonga/storage/t/fulltext_order_boolean_mode_same_match_against.test @@ -35,7 +35,7 @@ SELECT * FROM diaries; SELECT *, MATCH(title) AGAINST("groonga" IN BOOLEAN MODE) AS score FROM diaries WHERE MATCH(title) AGAINST("groonga" IN BOOLEAN MODE) - ORDER BY MATCH(title) AGAINST("groonga" IN BOOLEAN MODE); + ORDER BY MATCH(title) AGAINST("groonga" IN BOOLEAN MODE), title; DROP TABLE diaries; diff --git a/storage/mroonga/mysql-test/mroonga/storage/t/fulltext_order_natural_language_mode_different_match.test b/storage/mroonga/mysql-test/mroonga/storage/t/fulltext_order_natural_language_mode_different_match.test index 6c7eb0a64e3a0..50baf773d33bf 100644 --- a/storage/mroonga/mysql-test/mroonga/storage/t/fulltext_order_natural_language_mode_different_match.test +++ b/storage/mroonga/mysql-test/mroonga/storage/t/fulltext_order_natural_language_mode_different_match.test @@ -39,10 +39,11 @@ INSERT INTO logs VALUES("Warning Warning Warning Warning", "host2"); SELECT * FROM logs; -SELECT *, MATCH(host) AGAINST("host2" IN NATURAL LANGUAGE MODE) AS score +SELECT *, MATCH(host) AGAINST("host2" IN NATURAL LANGUAGE MODE) AS score FROM logs WHERE MATCH(message) AGAINST("Error" IN NATURAL LANGUAGE MODE) - ORDER BY MATCH(host) AGAINST("host2" IN NATURAL LANGUAGE MODE); + ORDER BY MATCH(host) AGAINST("host2" IN NATURAL LANGUAGE MODE), + message; DROP TABLE logs; diff --git a/storage/mroonga/mysql-test/mroonga/storage/t/fulltext_order_natural_language_mode_no_where.test b/storage/mroonga/mysql-test/mroonga/storage/t/fulltext_order_natural_language_mode_no_where.test index 917d437d0e9b9..492478b9326cf 100644 --- a/storage/mroonga/mysql-test/mroonga/storage/t/fulltext_order_natural_language_mode_no_where.test +++ b/storage/mroonga/mysql-test/mroonga/storage/t/fulltext_order_natural_language_mode_no_where.test @@ -37,10 +37,9 @@ INSERT INTO logs VALUES("Warning Warning Warning Warning"); SELECT * FROM logs; -SELECT *, MATCH(message) AGAINST("Error" IN NATURAL LANGUAGE MODE) AS score +SELECT *, MATCH(message) AGAINST("Error" IN NATURAL LANGUAGE MODE) AS score FROM logs - ORDER BY MATCH(message) AGAINST("Error" IN NATURAL LANGUAGE MODE); - + ORDER BY MATCH(message) AGAINST("Error" IN NATURAL LANGUAGE MODE), message; DROP TABLE logs; --source ../../include/mroonga/have_mroonga_deinit.inc diff --git a/storage/rocksdb/rdb_datadic.cc b/storage/rocksdb/rdb_datadic.cc index 0c81ff7997310..6659524ac0dd6 100644 --- a/storage/rocksdb/rdb_datadic.cc +++ b/storage/rocksdb/rdb_datadic.cc @@ -3301,7 +3301,7 @@ bool Rdb_field_packing::setup(const Rdb_key_def *const key_descr, m_max_image_len = key_length + (field->charset()->number == COLLATION_BINARY ? reinterpret_cast(field) - ->pack_length_no_ptr() + ->length_size() : 0); // Return false because indexes on text/blob will always require // a prefix. With a prefix, the optimizer will not be able to do an From eaffd7c5c16b255d872243729bb220bce5222311 Mon Sep 17 00:00:00 2001 From: Arcadiy Ivanov Date: Sat, 13 Jun 2026 13:12:29 -0400 Subject: [PATCH 2/4] Fix CI regressions from MDEV-38975 forward-port to main Seven code fixes, a new test, and test re-recordings for issues found by CI on PR #5222. **NULL dereference in `create_tmp_field()`**: `SYS_REFCURSOR` plugin returns NULL from `make_new_field()` (cursor values cannot be materialized). The feature added `result->flags |= FIELD_PART_OF_TMP_UNIQUE` without a NULL check. Added `if (result)` guard. **xmltype identity loss and recursive CTE reclength mismatch in `Item_type_holder::create_tmp_field_ex()`**: the blob_key dispatch now requires both: (1) `type_handler_for_tmp_table()` returns `blob_key_type_handler()`, AND (2) `dynamic_cast` confirms the original type is a native blob. Condition 1 excludes xmltype (its override returns itself). Condition 2 excludes VARCHAR types promoted via `varstring_type_handler()` -> `too_big_for_varchar()` -> `blob_type_handler()`. Without condition 2, wide VARCHAR in recursive CTEs (e.g. `cast('...' as varchar(1000))`) was promoted to `Field_blob_key` in the main UNION DISTINCT table (`part_of_unique_key=true`) but stayed as `Field_varchar` in the incremental table (`part_of_unique_key=false`), causing a `reclength` mismatch assertion in `select_union_recursive::send_data()` (`main.json_equals` crash). **Spurious `reclength > HA_MAX_REC_LENGTH` in `pick_engine()`**: the original `choose_engine()` (both 10.11 and upstream/main) never had a reclength check. MDEV-38975 introduced it when replacing the `blob_fields` condition. HEAP has no internal reclength limit -- `hp_create.c` stores `uint reclength` and allocates blocks of that size; `max_supported_record_length()` is only checked in `unireg.cc` during user-facing CREATE TABLE. I_S tables like SLAVE_STATUS routinely have reclength ~880KB (13 bare `Varchar()` columns). The check forced them to Aria where `fill_slave_status()` returned 0 rows. Removed the check and the unused `reclength` parameter from `pick_engine()`. **Multi-update `tmp_memory_table_size` override**: the 10.11 feature overrode `big_tables=FALSE` for multi-update dedup tables. The forward-port translated this as `tmp_memory_table_size=SIZE_T_MAX` when the variable was 0. But `big_tables=FALSE` was a soft "don't force disk" hint, while `tmp_memory_table_size=SIZE_T_MAX` overrides the user's explicit `tmp_memory_table_size=0` directive. Since main removed `big_tables` entirely (MDEV-19713), the override is not needed. Removed. **Zero-length key rejection in `check_tmp_key()`**: reject `key_len == 0` to prevent useless zero-length keys from being created by `add_tmp_key()`. Reachable when all key parts are CHAR(0) NOT NULL: `key_length()` returns 0, the field is not nullable (no HA_KEY_NULL_LENGTH) and not VARCHAR/BLOB/GEOMETRY (no HA_KEY_BLOB_LENGTH), so `fld_store_len` is 0 for every part. Without this guard, `check_tmp_key()` would accept the key (0 <= max_key_length), and the optimizer would create a ref key that cannot distinguish any rows. Added `heap.char0_key` test exercising this via a materialized derived table with CHAR(0) NOT NULL join columns. **Non-deterministic `column_compression` test**: HEAP blob support allows compressed VARCHAR/TEXT temp tables to stay in HEAP instead of falling to Aria, changing row iteration order. Added `--sorted_result` to the two MDEV-24726 subqueries that lack `ORDER BY`. Test changes: - `spatial_utility_function_collect`: added ORDER BY to window function that lacked it (results were engine-row-order-dependent) - `tmp_space_usage`: removed multi-update override; forced disk for MDEV-34016/34060 Aria-specific test sections (blob I_S tables now stay in MEMORY) - `blob_update_overflow`: replaced `SHOW STATUS LIKE 'Created_tmp_%'` with targeted I_S query (Created_tmp_files varies on sanitizer builds) - `column_compression`: added `--sorted_result` for MDEV-24726 queries - `char0_key` (new): CHAR(0) NOT NULL derived table ref key rejection - Re-recorded 8 tests for expected "temp table stays in MEMORY" changes --- mysql-test/main/column_compression.result | 4 +- mysql-test/main/column_compression.test | 2 + mysql-test/main/metadata.result | 2 +- mysql-test/main/opt_trace.result | 2 +- .../spatial_utility_function_collect.result | 4 +- .../spatial_utility_function_collect.test | 4 +- mysql-test/main/status.result | 2 +- mysql-test/main/tmp_space_usage.result | 4 +- mysql-test/main/tmp_space_usage.test | 4 +- mysql-test/suite/binlog/r/create_like.result | 4 +- .../funcs_1/r/processlist_priv_no_prot.result | 2 +- .../funcs_1/r/processlist_priv_ps.result | 2 +- .../suite/heap/blob_update_overflow.result | 44 ++++++------ .../suite/heap/blob_update_overflow.test | 24 +++++-- mysql-test/suite/heap/char0_key.result | 20 ++++++ mysql-test/suite/heap/char0_key.test | 25 +++++++ .../r/v_schema_object_overview.result | 6 +- sql/create_tmp_table.h | 2 +- sql/sql_select.cc | 71 +++++++++++-------- sql/sql_update.cc | 5 -- sql/table.cc | 6 ++ 21 files changed, 163 insertions(+), 76 deletions(-) create mode 100644 mysql-test/suite/heap/char0_key.result create mode 100644 mysql-test/suite/heap/char0_key.test diff --git a/mysql-test/main/column_compression.result b/mysql-test/main/column_compression.result index 708f5a709ef7e..53f23a68103d2 100644 --- a/mysql-test/main/column_compression.result +++ b/mysql-test/main/column_compression.result @@ -3035,9 +3035,9 @@ create algorithm=temptable view v1 as select * from t1; insert into t1 values ('foo'),('bar'),('foo'); select * from v1 where a in (select a from t1); a +bar foo foo -bar drop view v1; drop table t1; create table t1 (f1 varchar(8)) charset=eucjpms collate=eucjpms_nopad_bin; @@ -3054,9 +3054,9 @@ create algorithm=temptable view v1 as select * from t1; insert into t1 values ('foo'),('bar'),('foo'); select * from v1 where a in (select a from t1); a +bar foo foo -bar drop view v1; drop table t1; create table t1 (f1 text) charset=eucjpms collate=eucjpms_nopad_bin; diff --git a/mysql-test/main/column_compression.test b/mysql-test/main/column_compression.test index bba9f5d5ed1a1..5b3ee05405a66 100644 --- a/mysql-test/main/column_compression.test +++ b/mysql-test/main/column_compression.test @@ -533,6 +533,7 @@ DROP TABLE t1; create table t1 (a varchar(8) compressed) character set utf8mb4; create algorithm=temptable view v1 as select * from t1; insert into t1 values ('foo'),('bar'),('foo'); +--sorted_result select * from v1 where a in (select a from t1); # cleanup drop view v1; @@ -550,6 +551,7 @@ drop table t1, t2; create table t1 (a text compressed) character set utf8mb4; create algorithm=temptable view v1 as select * from t1; insert into t1 values ('foo'),('bar'),('foo'); +--sorted_result select * from v1 where a in (select a from t1); # cleanup drop view v1; diff --git a/mysql-test/main/metadata.result b/mysql-test/main/metadata.result index 5e41fadf05dab..81111b1146506 100644 --- a/mysql-test/main/metadata.result +++ b/mysql-test/main/metadata.result @@ -172,7 +172,7 @@ def 1 1 3 2 1 N 49157 0 63 1 select * from (select 1 union select 1) aaa; Catalog Database Table Table_alias Column Column_alias Type Length Max length Is_null Flags Decimals Charsetnr -def aaa 1 1 3 2 1 N 49153 0 63 +def aaa 1 1 3 2 1 N 49157 0 63 1 1 drop table t1; diff --git a/mysql-test/main/opt_trace.result b/mysql-test/main/opt_trace.result index 4f58dcfb3aebc..08b19865f93c8 100644 --- a/mysql-test/main/opt_trace.result +++ b/mysql-test/main/opt_trace.result @@ -13887,7 +13887,7 @@ OPTIMIZER_TRACE CREATE TEMPORARY TABLE `OPTIMIZER_TRACE` ( `TRACE` longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL, `MISSING_BYTES_BEYOND_MAX_MEM_SIZE` int(20) NOT NULL, `INSUFFICIENT_PRIVILEGES` tinyint(1) NOT NULL -) ENGINE=Aria DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci PAGE_CHECKSUM=0 +) ENGINE=MEMORY DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci set @@optimizer_switch= @save_optimizer_switch; set @@use_stat_tables= @save_use_stat_tables; set @@histogram_size= @save_histogram_size; diff --git a/mysql-test/main/spatial_utility_function_collect.result b/mysql-test/main/spatial_utility_function_collect.result index 07095862def21..1e19ad1984c75 100644 --- a/mysql-test/main/spatial_utility_function_collect.result +++ b/mysql-test/main/spatial_utility_function_collect.result @@ -62,8 +62,8 @@ INSERT INTO table_simple_aggregation (location) VALUES ( ST_GEOMFROMTEXT('POINT(0 -0)' ,4326)), ( NULL), ( NULL); -SELECT ST_ASTEXT(ST_COLLECT(location) OVER ( ROWS BETWEEN 1 PRECEDING AND -CURRENT ROW)) c FROM table_simple_aggregation; +SELECT ST_ASTEXT(ST_COLLECT(location) OVER ( ORDER BY running_number +ROWS BETWEEN 1 PRECEDING AND CURRENT ROW)) c FROM table_simple_aggregation; c MULTIPOINT(0 0) MULTIPOINT(0 0,0 0) diff --git a/mysql-test/main/spatial_utility_function_collect.test b/mysql-test/main/spatial_utility_function_collect.test index f9e6fbd141a70..91da2d529fcab 100644 --- a/mysql-test/main/spatial_utility_function_collect.test +++ b/mysql-test/main/spatial_utility_function_collect.test @@ -72,8 +72,8 @@ INSERT INTO table_simple_aggregation (location) VALUES ( ST_GEOMFROMTEXT('POINT(0 -0)' ,4326)), ( NULL), ( NULL); -SELECT ST_ASTEXT(ST_COLLECT(location) OVER ( ROWS BETWEEN 1 PRECEDING AND -CURRENT ROW)) c FROM table_simple_aggregation; +SELECT ST_ASTEXT(ST_COLLECT(location) OVER ( ORDER BY running_number +ROWS BETWEEN 1 PRECEDING AND CURRENT ROW)) c FROM table_simple_aggregation; --echo Excercising multiple code paths. diff --git a/mysql-test/main/status.result b/mysql-test/main/status.result index 94eca99904a54..64a7c164156f3 100644 --- a/mysql-test/main/status.result +++ b/mysql-test/main/status.result @@ -350,7 +350,7 @@ Created_tmp_tables 2 Handler_tmp_delete 0 Handler_tmp_update 2 Handler_tmp_write 6 -Max_tmp_space_used 32768 +Max_tmp_space_used 0 Rows_tmp_read 44 drop table t1; CREATE TABLE t1 (i int(11) DEFAULT NULL, KEY i (i) ) ENGINE=MyISAM; diff --git a/mysql-test/main/tmp_space_usage.result b/mysql-test/main/tmp_space_usage.result index fdaf5d7500332..f923c659209da 100644 --- a/mysql-test/main/tmp_space_usage.result +++ b/mysql-test/main/tmp_space_usage.result @@ -187,8 +187,9 @@ CREATE TABLE t1 (a varchar(1024)) engine=aria CHARSET=latin1; INSERT INTO t1 VALUES ('this'),('is'),('just'),('a'),('filling'),('for'),(REPEAT('a',500)); set @@global.max_tmp_total_space_usage=2*1024*1024; SET max_tmp_session_space_usage= 1024*1024, max_heap_table_size= 4*1024*1024; +SET tmp_memory_table_size= 0; SELECT DISTINCT a, seq FROM t1 JOIN seq_1_to_600; -ERROR HY000: Got error 201 "Local temporary space limit reached" when merging index +ERROR HY000: Local temporary space limit reached DROP TABLE t1; connection default; disconnect c1; @@ -210,6 +211,7 @@ CREATE OR REPLACE TABLE t1 (a DATETIME) ENGINE=InnoDB; BEGIN; INSERT INTO t1 SELECT NOW() FROM seq_1_to_6000; SET max_tmp_session_space_usage = 64*1024; +SET tmp_memory_table_size= 0; SELECT * FROM information_schema.ALL_PLUGINS LIMIT 2; ERROR HY000: Local temporary space limit reached ROLLBACK; diff --git a/mysql-test/main/tmp_space_usage.test b/mysql-test/main/tmp_space_usage.test index 85675b1450cdd..9480446acd509 100644 --- a/mysql-test/main/tmp_space_usage.test +++ b/mysql-test/main/tmp_space_usage.test @@ -247,7 +247,8 @@ CREATE TABLE t1 (a varchar(1024)) engine=aria CHARSET=latin1; INSERT INTO t1 VALUES ('this'),('is'),('just'),('a'),('filling'),('for'),(REPEAT('a',500)); set @@global.max_tmp_total_space_usage=2*1024*1024; SET max_tmp_session_space_usage= 1024*1024, max_heap_table_size= 4*1024*1024; ---error ER_NOT_KEYFILE +SET tmp_memory_table_size= 0; +--error HA_ERR_LOCAL_TMP_SPACE_FULL SELECT DISTINCT a, seq FROM t1 JOIN seq_1_to_600; DROP TABLE t1; connection default; @@ -277,6 +278,7 @@ BEGIN; INSERT INTO t1 SELECT NOW() FROM seq_1_to_6000; SET max_tmp_session_space_usage = 64*1024; +SET tmp_memory_table_size= 0; --error HA_ERR_LOCAL_TMP_SPACE_FULL SELECT * FROM information_schema.ALL_PLUGINS LIMIT 2; ROLLBACK; diff --git a/mysql-test/suite/binlog/r/create_like.result b/mysql-test/suite/binlog/r/create_like.result index 9614bea9c18e9..cd90c6068d417 100644 --- a/mysql-test/suite/binlog/r/create_like.result +++ b/mysql-test/suite/binlog/r/create_like.result @@ -60,7 +60,7 @@ master-bin.000001 # Query # # use `test`; CREATE TEMPORARY TABLE `t1` ( `INFO_BINARY` blob, `TID` bigint(10) NOT NULL, `TMP_SPACE_USED` bigint(10) NOT NULL -) ENGINE=Aria PAGE_CHECKSUM=0 +) ENGINE=MEMORY master-bin.000001 # Gtid # # GTID #-#-# master-bin.000001 # Query # # use `test`; CREATE TEMPORARY TABLE t2_tmp like t2 master-bin.000001 # Gtid # # GTID #-#-# @@ -91,7 +91,7 @@ master-bin.000001 # Query # # use `test`; CREATE TEMPORARY TABLE `t1` ( `INFO_BINARY` blob, `TID` bigint(10) NOT NULL, `TMP_SPACE_USED` bigint(10) NOT NULL -) ENGINE=Aria PAGE_CHECKSUM=0 +) ENGINE=MEMORY master-bin.000001 # Gtid # # GTID #-#-# master-bin.000001 # Query # # use `test`; CREATE TEMPORARY TABLE t2_tmp like t2 master-bin.000001 # Gtid # # GTID #-#-# diff --git a/mysql-test/suite/funcs_1/r/processlist_priv_no_prot.result b/mysql-test/suite/funcs_1/r/processlist_priv_no_prot.result index 4d07c7804a0a3..6d259359924f2 100644 --- a/mysql-test/suite/funcs_1/r/processlist_priv_no_prot.result +++ b/mysql-test/suite/funcs_1/r/processlist_priv_no_prot.result @@ -53,7 +53,7 @@ ID root HOST_NAME information_schema Query TIME starting SHOW processlist TIME_M ID ddicttestuser1 HOST_NAME information_schema Sleep TIME NULL TIME_MS SELECT * FROM processlist ORDER BY id; ID USER HOST DB COMMAND TIME STATE INFO TIME_MS STAGE MAX_STAGE PROGRESS MEMORY_USED MAX_MEMORY_USED EXAMINED_ROWS SENT_ROWS QUERY_ID INFO_BINARY TID TMP_SPACE_USED -ID root HOST_NAME information_schema Query TIME Filling schema table SELECT * FROM processlist ORDER BY id TIME_MS 0 0 0.000 MEMORY MAX_MEMORY E_ROWS S_ROWS QUERY_ID SELECT * FROM processlist ORDER BY id TID 16384 +ID root HOST_NAME information_schema Query TIME Filling schema table SELECT * FROM processlist ORDER BY id TIME_MS 0 0 0.000 MEMORY MAX_MEMORY E_ROWS S_ROWS QUERY_ID SELECT * FROM processlist ORDER BY id TID 0 ID ddicttestuser1 HOST_NAME information_schema Sleep TIME NULL TIME_MS 0 0 0.000 MEMORY MAX_MEMORY E_ROWS S_ROWS QUERY_ID NULL TID 0 SELECT ID, USER, HOST, DB, COMMAND, TIME, STATE, INFO, TIME_MS, STAGE, MAX_STAGE, PROGRESS, MEMORY_USED, EXAMINED_ROWS, QUERY_ID, INFO_BINARY FROM processlist ORDER BY id; ID USER HOST DB COMMAND TIME STATE INFO TIME_MS STAGE MAX_STAGE PROGRESS MEMORY_USED EXAMINED_ROWS QUERY_ID INFO_BINARY diff --git a/mysql-test/suite/funcs_1/r/processlist_priv_ps.result b/mysql-test/suite/funcs_1/r/processlist_priv_ps.result index 77e746fd59c91..5d3ff498073b2 100644 --- a/mysql-test/suite/funcs_1/r/processlist_priv_ps.result +++ b/mysql-test/suite/funcs_1/r/processlist_priv_ps.result @@ -53,7 +53,7 @@ ID root HOST_NAME information_schema Query TIME starting SHOW processlist TIME_M ID ddicttestuser1 HOST_NAME information_schema Sleep TIME NULL TIME_MS SELECT * FROM processlist ORDER BY id; ID USER HOST DB COMMAND TIME STATE INFO TIME_MS STAGE MAX_STAGE PROGRESS MEMORY_USED MAX_MEMORY_USED EXAMINED_ROWS SENT_ROWS QUERY_ID INFO_BINARY TID TMP_SPACE_USED -ID root HOST_NAME information_schema Execute TIME Filling schema table SELECT * FROM processlist ORDER BY id TIME_MS 0 0 0.000 MEMORY MAX_MEMORY E_ROWS S_ROWS QUERY_ID SELECT * FROM processlist ORDER BY id TID 16384 +ID root HOST_NAME information_schema Execute TIME Filling schema table SELECT * FROM processlist ORDER BY id TIME_MS 0 0 0.000 MEMORY MAX_MEMORY E_ROWS S_ROWS QUERY_ID SELECT * FROM processlist ORDER BY id TID 0 ID ddicttestuser1 HOST_NAME information_schema Sleep TIME NULL TIME_MS 0 0 0.000 MEMORY MAX_MEMORY E_ROWS S_ROWS QUERY_ID NULL TID 0 SELECT ID, USER, HOST, DB, COMMAND, TIME, STATE, INFO, TIME_MS, STAGE, MAX_STAGE, PROGRESS, MEMORY_USED, EXAMINED_ROWS, QUERY_ID, INFO_BINARY FROM processlist ORDER BY id; ID USER HOST DB COMMAND TIME STATE INFO TIME_MS STAGE MAX_STAGE PROGRESS MEMORY_USED EXAMINED_ROWS QUERY_ID INFO_BINARY diff --git a/mysql-test/suite/heap/blob_update_overflow.result b/mysql-test/suite/heap/blob_update_overflow.result index 4eb640d311173..dfaa2181d8a59 100644 --- a/mysql-test/suite/heap/blob_update_overflow.result +++ b/mysql-test/suite/heap/blob_update_overflow.result @@ -35,11 +35,12 @@ grp max_len 2 29600 3 29800 # Verify HEAP->Aria conversion happened -SHOW STATUS LIKE 'Created_tmp_%'; -Variable_name Value -Created_tmp_disk_tables 1 -Created_tmp_files 0 -Created_tmp_tables 2 +SELECT variable_name, variable_value FROM information_schema.session_status +WHERE variable_name IN ('Created_tmp_disk_tables', 'Created_tmp_tables') +ORDER BY variable_name; +variable_name variable_value +CREATED_TMP_DISK_TABLES 2 +CREATED_TMP_TABLES 4 DROP TABLE t1; # # Test 2: Verify result correctness after overflow @@ -62,11 +63,12 @@ grp max_len prefix suffix 1 30000 aaaaa aaaaa 2 29600 aaaaa aaaaa 3 29800 aaaaa aaaaa -SHOW STATUS LIKE 'Created_tmp_%'; -Variable_name Value -Created_tmp_disk_tables 1 -Created_tmp_files 0 -Created_tmp_tables 2 +SELECT variable_name, variable_value FROM information_schema.session_status +WHERE variable_name IN ('Created_tmp_disk_tables', 'Created_tmp_tables') +ORDER BY variable_name; +variable_name variable_value +CREATED_TMP_DISK_TABLES 2 +CREATED_TMP_TABLES 4 DROP TABLE t1; # # Test 3: Multiple blob aggregates (two MAX columns) @@ -85,11 +87,12 @@ grp max1_len max2_len 1 30000 22500 2 29600 22200 3 29800 22350 -SHOW STATUS LIKE 'Created_tmp_%'; -Variable_name Value -Created_tmp_disk_tables 1 -Created_tmp_files 0 -Created_tmp_tables 2 +SELECT variable_name, variable_value FROM information_schema.session_status +WHERE variable_name IN ('Created_tmp_disk_tables', 'Created_tmp_tables') +ORDER BY variable_name; +variable_name variable_value +CREATED_TMP_DISK_TABLES 2 +CREATED_TMP_TABLES 4 DROP TABLE t1; # # Test 4: MIN(TEXT) with monotonically shrinking minimum @@ -118,11 +121,12 @@ grp max_len min_len 1 30000 50 2 29600 50 3 29200 50 -SHOW STATUS LIKE 'Created_tmp_%'; -Variable_name Value -Created_tmp_disk_tables 1 -Created_tmp_files 0 -Created_tmp_tables 2 +SELECT variable_name, variable_value FROM information_schema.session_status +WHERE variable_name IN ('Created_tmp_disk_tables', 'Created_tmp_tables') +ORDER BY variable_name; +variable_name variable_value +CREATED_TMP_DISK_TABLES 2 +CREATED_TMP_TABLES 4 DROP TABLE t1; # # Cleanup diff --git a/mysql-test/suite/heap/blob_update_overflow.test b/mysql-test/suite/heap/blob_update_overflow.test index dd24e14690e47..4ba3a1473b2dc 100644 --- a/mysql-test/suite/heap/blob_update_overflow.test +++ b/mysql-test/suite/heap/blob_update_overflow.test @@ -44,7 +44,11 @@ SELECT grp, LENGTH(MAX(val)) AS max_len FROM t1 GROUP BY grp ORDER BY grp; --echo # Verify HEAP->Aria conversion happened -SHOW STATUS LIKE 'Created_tmp_%'; +--disable_warnings +SELECT variable_name, variable_value FROM information_schema.session_status + WHERE variable_name IN ('Created_tmp_disk_tables', 'Created_tmp_tables') + ORDER BY variable_name; +--enable_warnings --enable_cursor_protocol --enable_view_protocol --enable_ps2_protocol @@ -76,7 +80,11 @@ SELECT grp, RIGHT(MAX(val), 5) AS suffix FROM t1 GROUP BY grp ORDER BY grp; -SHOW STATUS LIKE 'Created_tmp_%'; +--disable_warnings +SELECT variable_name, variable_value FROM information_schema.session_status + WHERE variable_name IN ('Created_tmp_disk_tables', 'Created_tmp_tables') + ORDER BY variable_name; +--enable_warnings --enable_cursor_protocol --enable_view_protocol --enable_ps2_protocol @@ -103,7 +111,11 @@ SELECT grp, LENGTH(MAX(v2)) AS max2_len FROM t1 GROUP BY grp ORDER BY grp; -SHOW STATUS LIKE 'Created_tmp_%'; +--disable_warnings +SELECT variable_name, variable_value FROM information_schema.session_status + WHERE variable_name IN ('Created_tmp_disk_tables', 'Created_tmp_tables') + ORDER BY variable_name; +--enable_warnings --enable_cursor_protocol --enable_view_protocol --enable_ps2_protocol @@ -140,7 +152,11 @@ SELECT grp, LENGTH(MIN(val)) AS min_len FROM t1 GROUP BY grp ORDER BY grp; -SHOW STATUS LIKE 'Created_tmp_%'; +--disable_warnings +SELECT variable_name, variable_value FROM information_schema.session_status + WHERE variable_name IN ('Created_tmp_disk_tables', 'Created_tmp_tables') + ORDER BY variable_name; +--enable_warnings --enable_cursor_protocol --enable_view_protocol --enable_ps2_protocol diff --git a/mysql-test/suite/heap/char0_key.result b/mysql-test/suite/heap/char0_key.result new file mode 100644 index 0000000000000..eedc2cbd4ac39 --- /dev/null +++ b/mysql-test/suite/heap/char0_key.result @@ -0,0 +1,20 @@ +SET optimizer_switch='derived_merge=off'; +CREATE TABLE t1 (a CHAR(0) NOT NULL, b INT) ENGINE=MEMORY; +INSERT INTO t1 VALUES ('', 1), ('', 2), ('', 3); +CREATE TABLE t2 (c CHAR(0) NOT NULL, d INT) ENGINE=MEMORY; +INSERT INTO t2 VALUES ('', 10), ('', 20); +EXPLAIN SELECT * FROM t2, (SELECT * FROM t1) dt WHERE dt.a = t2.c; +id select_type table type possible_keys key key_len ref rows Extra +1 PRIMARY t2 ALL NULL NULL NULL NULL 2 +1 PRIMARY ALL NULL NULL NULL NULL 3 Using where; Using join buffer (flat, BNL join) +2 DERIVED t1 ALL NULL NULL NULL NULL 3 +SELECT * FROM t2, (SELECT * FROM t1) dt WHERE dt.a = t2.c; +c d a b + 10 1 + 10 2 + 10 3 + 20 1 + 20 2 + 20 3 +DROP TABLE t1, t2; +SET optimizer_switch=DEFAULT; diff --git a/mysql-test/suite/heap/char0_key.test b/mysql-test/suite/heap/char0_key.test new file mode 100644 index 0000000000000..a5b2a9c3762e6 --- /dev/null +++ b/mysql-test/suite/heap/char0_key.test @@ -0,0 +1,25 @@ +# +# Test that check_tmp_key rejects zero-length keys from CHAR(0) NOT NULL. +# The optimizer tries to add a ref key on a materialized derived table, +# but CHAR(0) NOT NULL has key_length()=0, producing a useless zero-length +# key that must be rejected. +# + +SET optimizer_switch='derived_merge=off'; + +CREATE TABLE t1 (a CHAR(0) NOT NULL, b INT) ENGINE=MEMORY; +INSERT INTO t1 VALUES ('', 1), ('', 2), ('', 3); + +CREATE TABLE t2 (c CHAR(0) NOT NULL, d INT) ENGINE=MEMORY; +INSERT INTO t2 VALUES ('', 10), ('', 20); + +# Materialized derived table: the optimizer sees the equi-join dt.a = t2.c +# and tries to add a key on column a. check_tmp_key() must reject the +# zero-length key. Without the guard the server would create a key with +# key_length=0 which is invalid. +EXPLAIN SELECT * FROM t2, (SELECT * FROM t1) dt WHERE dt.a = t2.c; +--sorted_result +SELECT * FROM t2, (SELECT * FROM t1) dt WHERE dt.a = t2.c; + +DROP TABLE t1, t2; +SET optimizer_switch=DEFAULT; diff --git a/mysql-test/suite/sysschema/r/v_schema_object_overview.result b/mysql-test/suite/sysschema/r/v_schema_object_overview.result index 1c6144e956f2e..1b6132a67b768 100644 --- a/mysql-test/suite/sysschema/r/v_schema_object_overview.result +++ b/mysql-test/suite/sysschema/r/v_schema_object_overview.result @@ -1,6 +1,6 @@ DESC sys.schema_object_overview; Field Type Null Key Default Extra -db varchar(64) NO -object_type varchar(64) YES NULL -count bigint(21) NO 0 +db varchar(64) NO UNI +object_type varchar(64) YES UNI NULL +count bigint(21) NO UNI 0 SELECT * FROM sys.schema_object_overview; diff --git a/sql/create_tmp_table.h b/sql/create_tmp_table.h index 6a9f7b07d7e72..8af7f70890476 100644 --- a/sql/create_tmp_table.h +++ b/sql/create_tmp_table.h @@ -60,7 +60,7 @@ class Create_tmp_table: public Data_type_statistics Create_tmp_table(ORDER *group, bool distinct, bool save_sum_fields, ulonglong select_options, ha_rows rows_limit); virtual ~Create_tmp_table() {} - handlerton *pick_engine(THD *thd, uint reclength); + handlerton *pick_engine(THD *thd); virtual bool choose_engine(THD *thd, TABLE *table, TMP_TABLE_PARAM *param); void add_field(TABLE *table, Field *field, uint fieldnr, bool force_not_null_cols); diff --git a/sql/sql_select.cc b/sql/sql_select.cc index f079f8994d576..da0f6c6e82536 100644 --- a/sql/sql_select.cc +++ b/sql/sql_select.cc @@ -21796,21 +21796,37 @@ Field *Item_type_holder::create_tmp_field_ex(MEM_ROOT *root, TABLE *table, Tmp_field_src *src, const Tmp_field_param *param) { - Type_handler const *type_handler= Item_type_holder::real_type_handler(); - Type_handler_blob_common const *blob_handler; - if (param->part_of_unique_key() && - (blob_handler= - dynamic_cast(type_handler))) + const Type_handler *type_handler= Item_type_holder::real_type_handler(); + const Type_handler *tmp_handler= + type_handler->type_handler_for_tmp_table(this, param); + /* + Only enter the blob_key path when BOTH conditions hold: + 1. type_handler_for_tmp_table() returned blob_key_type_handler() + (implies part_of_unique_key is true) + 2. The original type IS a blob (dynamic_cast succeeds) + + Condition 1 excludes xmltype (its override returns itself, never + blob_key). Condition 2 excludes VARCHAR types that were promoted + to blob_key via varstring_type_handler() -> too_big_for_varchar() + -> blob_type_handler(). Without condition 2, wide VARCHAR columns + would get promoted to blob in the main table (which has + part_of_unique_key=true for UNION DISTINCT) but stay as VARCHAR in + the incremental table of a recursive CTE (which has + part_of_unique_key=false), causing a reclength mismatch assertion. + */ + if (tmp_handler == Type_handler::blob_key_type_handler() && + dynamic_cast(type_handler)) { - Field_blob *blob_field= (Field_blob*) type_handler_blob_key. + Field *field= tmp_handler-> make_and_init_table_field(root, &name, Record_addr(maybe_null()), *this, table); - if (blob_field) + if (field) { - /* Fix length of blob to be able to return the original blob type */ - blob_field->set_pack_length(blob_handler->length_bytes()); + ((Field_blob*) field)->set_pack_length( + static_cast( + Type_handler::blob_type_handler(max_length, NULL))->length_bytes()); } - return blob_field; + return field; } return type_handler-> make_and_init_table_field(root, &name, Record_addr(maybe_null()), @@ -21953,10 +21969,13 @@ Field *create_tmp_field(TABLE *table, Item *item, *default_field= src.default_field(); if (src.item_result_field()) *((*copy_func)++)= src.item_result_field(); - if (part_of_unique_key) - result->flags|= FIELD_PART_OF_TMP_UNIQUE | UNIQUE_KEY_FLAG; - if (group) - result->flags|= UNIQUE_KEY_FLAG; + if (result) + { + if (part_of_unique_key) + result->flags|= FIELD_PART_OF_TMP_UNIQUE | UNIQUE_KEY_FLAG; + if (group) + result->flags|= UNIQUE_KEY_FLAG; + } return result; } @@ -22130,20 +22149,17 @@ TABLE *Create_tmp_table::start(THD *thd, fn_format(path, path, mysql_tmpdir, "", MY_REPLACE_EXT|MY_UNPACK_FILENAME); /* - Early engine prediction: reclength is not known yet (fields haven't been - added), so pass 0 -- this is safe because pick_engine()'s only reclength - check is "> HA_MAX_REC_LENGTH", which 0 never triggers. Returns - heap_hton unless session-level overrides (tmp_memory_table_size=0, - etc.) force a disk-based engine. We use this - to avoid the too_big_for_varchar() / group_length >= MAX_BLOB_WIDTH - bail-outs that would force m_using_unique_constraint for HEAP tables - that natively support blob keys. + Early engine prediction. Returns heap_hton unless session-level + overrides (tmp_memory_table_size=0, etc.) force a disk-based engine. + We use this to avoid the too_big_for_varchar() / group_length >= + MAX_BLOB_WIDTH bail-outs that would force m_using_unique_constraint + for HEAP tables that natively support blob keys. Note: pick_engine() also reads m_using_unique_constraint, which is false at this point. The guards below that set it to true are gated by !m_heap_expected, so there is no circular dependency in practice. */ - m_heap_expected= (pick_engine(thd, 0) == heap_hton); + m_heap_expected= (pick_engine(thd) == heap_hton); if (m_group) { @@ -22492,14 +22508,13 @@ bool Create_tmp_table::add_fields(THD *thd, /* Predict which engine a temporary table will use, based on session variables and the current m_using_unique_constraint / m_select_options - state. Called early (before fields are added) with reclength=0 to set - m_heap_expected, and again from choose_engine() with the real reclength. + state. Called early (before fields are added) to set m_heap_expected, + and again from choose_engine(). */ -handlerton *Create_tmp_table::pick_engine(THD *thd, uint reclength) +handlerton *Create_tmp_table::pick_engine(THD *thd) { if (m_using_unique_constraint || - reclength > HA_MAX_REC_LENGTH || (m_select_options & TMP_TABLE_FORCE_MYISAM) || thd->variables.tmp_memory_table_size == 0) return TMP_ENGINE_HTON; @@ -22513,7 +22528,7 @@ bool Create_tmp_table::choose_engine(THD *thd, TABLE *table, TABLE_SHARE *share= table->s; DBUG_ENTER("Create_tmp_table::choose_engine"); - handlerton *engine= pick_engine(thd, share->reclength); + handlerton *engine= pick_engine(thd); share->db_plugin= ha_lock_engine(0, engine); table->file= get_new_handler(share, &table->mem_root, share->db_type()); diff --git a/sql/sql_update.cc b/sql/sql_update.cc index 0aeda5cc50043..9293d5f6102f1 100644 --- a/sql/sql_update.cc +++ b/sql/sql_update.cc @@ -2308,15 +2308,10 @@ multi_update::initialize_tables(JOIN *join) tmp_param->field_count= temp_fields.elements; tmp_param->func_count= temp_fields.elements - 1; calc_group_buffer(tmp_param, group); - /* small table, ignore @@tmp_memory_table_size=0 */ - ulonglong save_tmp_memory_table_size= thd->variables.tmp_memory_table_size; - if (!save_tmp_memory_table_size) - thd->variables.tmp_memory_table_size= SIZE_T_MAX; tmp_tables[index]=create_tmp_table(thd, tmp_param, temp_fields, (ORDER*) group, 0, 0, TMP_TABLE_ALL_COLUMNS, HA_POS_ERROR, &empty_clex_str); - thd->variables.tmp_memory_table_size= save_tmp_memory_table_size; if (!tmp_tables[index]) DBUG_RETURN(1); tmp_tables[index]->file->extra(HA_EXTRA_WRITE_CACHE); diff --git a/sql/table.cc b/sql/table.cc index 3676393244380..3f929b3918ef3 100644 --- a/sql/table.cc +++ b/sql/table.cc @@ -8715,6 +8715,12 @@ bool TABLE::check_tmp_key(uint key, uint key_parts, key_len+= fld_store_len; } + /* + Reject zero-length keys (e.g. all parts are CHAR(0) NOT NULL): + such a key cannot distinguish rows and would be useless for lookup. + */ + if (!key_len) + return FALSE; // We use the on-disk storage engine's limit return key_len <= tmp_table_max_key_length() && key_parts <= tmp_table_max_key_parts(); From 147a2f9256ff9f46b4113c36567a7d16d1111169 Mon Sep 17 00:00:00 2001 From: Monty Date: Sat, 13 Jun 2026 16:38:29 +0300 Subject: [PATCH 3/4] MDEV-40030 Add support for CHECK TABLE for memory tables CHECK TABLE is now supported, but will only return ok or fail. Still good enough for testing heap table consistenty. --- .../main/alter_table_combinations,heap.rdiff | 24 +++---------------- mysql-test/main/ctype_utf8mb4_heap.result | 16 ++++++------- mysql-test/suite/heap/blob.result | 2 +- storage/heap/ha_heap.cc | 10 ++++++++ 4 files changed, 22 insertions(+), 30 deletions(-) diff --git a/mysql-test/main/alter_table_combinations,heap.rdiff b/mysql-test/main/alter_table_combinations,heap.rdiff index c9c7907f7d6d7..f211a0119a6ef 100644 --- a/mysql-test/main/alter_table_combinations,heap.rdiff +++ b/mysql-test/main/alter_table_combinations,heap.rdiff @@ -1,14 +1,5 @@ ---- main/alter_table_combinations.result -+++ main/alter_table_combinations.reject -@@ -11,7 +11,7 @@ - alter table t1 change x xx int, algorithm=inplace; - check table t1; - Table Op Msg_type Msg_text --test.t1 check status OK -+test.t1 check note The storage engine for the table doesn't support check - drop table t1; - # - # End of 10.3 tests +--- /home/my/maria-blob/mysql-test/main/alter_table_combinations.result 2026-06-12 16:33:21.229261067 +0300 ++++ /home/my/maria-blob/mysql-test/main/alter_table_combinations,heap.reject 2026-06-13 18:15:57.691863880 +0300 @@ -173,8 +173,7 @@ t3 CREATE TABLE `t3` ( `a` int(11) DEFAULT NULL, @@ -30,7 +21,7 @@ CREATE TABLE t4(a int); ALTER TABLE t4 RENAME COLUMN a TO aa, ALGORITHM = INPLACE; @@ -231,36 +229,6 @@ - ERROR 42S22: Unknown column 'd' in 'field list' + ERROR 42S22: Unknown column 'd' in 'INSERT INTO' DROP TRIGGER trg1; DROP PROCEDURE sp1; -CREATE TABLE t_gen(a INT, b DOUBLE GENERATED ALWAYS AS (SQRT(a))); @@ -66,12 +57,3 @@ SHOW CREATE TABLE t1; Table Create Table t1 CREATE TABLE `t1` ( -@@ -316,7 +284,7 @@ - alter table t1 change x xx int, algorithm=inplace; - check table t1; - Table Op Msg_type Msg_text --test.t1 check status OK -+test.t1 check note The storage engine for the table doesn't support check - drop table t1; - # - # End of 10.5 tests diff --git a/mysql-test/main/ctype_utf8mb4_heap.result b/mysql-test/main/ctype_utf8mb4_heap.result index 2455f9ae99719..059d9f6cd7dfb 100644 --- a/mysql-test/main/ctype_utf8mb4_heap.result +++ b/mysql-test/main/ctype_utf8mb4_heap.result @@ -1426,27 +1426,27 @@ INSERT INTO t1 VALUES('uuABCDEFGHIGKLMNOPRSTUVWXYZ̈bbbbbbbbbbbbbbbbbbbbbbbbbbbb INSERT INTO t1 VALUES('uu'); check table t1; Table Op Msg_type Msg_text -test.t1 check note The storage engine for the table doesn't support check +test.t1 check status OK INSERT INTO t1 VALUES('uU'); check table t1; Table Op Msg_type Msg_text -test.t1 check note The storage engine for the table doesn't support check +test.t1 check status OK INSERT INTO t1 VALUES('uu'); check table t1; Table Op Msg_type Msg_text -test.t1 check note The storage engine for the table doesn't support check +test.t1 check status OK INSERT INTO t1 VALUES('uuABC'); check table t1; Table Op Msg_type Msg_text -test.t1 check note The storage engine for the table doesn't support check +test.t1 check status OK INSERT INTO t1 VALUES('UuABC'); check table t1; Table Op Msg_type Msg_text -test.t1 check note The storage engine for the table doesn't support check +test.t1 check status OK INSERT INTO t1 VALUES('uuABC'); check table t1; Table Op Msg_type Msg_text -test.t1 check note The storage engine for the table doesn't support check +test.t1 check status OK alter table t1 add b int; INSERT INTO t1 VALUES('uuABCDEFGHIGKLMNOPRSTUVWXYZ̈bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb',1); INSERT INTO t1 VALUES('uuABCDEFGHIGKLMNOPRSTUVWXYZ̈bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb',2); @@ -1454,14 +1454,14 @@ delete from t1 where b=1; INSERT INTO t1 VALUES('UUABCDEFGHIGKLMNOPRSTUVWXYZ̈bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb',1); check table t1; Table Op Msg_type Msg_text -test.t1 check note The storage engine for the table doesn't support check +test.t1 check status OK INSERT INTO t1 VALUES('uuABCDEFGHIGKLMNOPRSTUVWXYZ̈bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb',3); INSERT INTO t1 VALUES('uuABCDEFGHIGKLMNOPRSTUVWXYZ̈bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb',4); delete from t1 where b=3; INSERT INTO t1 VALUES('uUABCDEFGHIGKLMNOPRSTUVWXYZ̈bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb',3); check table t1; Table Op Msg_type Msg_text -test.t1 check note The storage engine for the table doesn't support check +test.t1 check status OK drop table t1; set names utf8mb4; create table t1 (s1 char(5) character set utf8mb4) engine heap; diff --git a/mysql-test/suite/heap/blob.result b/mysql-test/suite/heap/blob.result index b1c04665d8639..7faf23da83fd0 100644 --- a/mysql-test/suite/heap/blob.result +++ b/mysql-test/suite/heap/blob.result @@ -527,7 +527,7 @@ a 2 check table t1; Table Op Msg_type Msg_text -test.t1 check note The storage engine for the table doesn't support check +test.t1 check status OK drop table t1; # # Large blob exceeding uint16 run_rec_count cap (65535 records) diff --git a/storage/heap/ha_heap.cc b/storage/heap/ha_heap.cc index 065b7e6b97bbb..5a3fb72d56b2b 100644 --- a/storage/heap/ha_heap.cc +++ b/storage/heap/ha_heap.cc @@ -1044,6 +1044,16 @@ int ha_heap::find_unique_row(uchar *record, uint unique_idx) DBUG_RETURN(result); } +int ha_heap::check(THD* thd, HA_CHECK_OPT* check_opt) +{ + if (!file) + return HA_ADMIN_INTERNAL_ERROR; + if (heap_check_heap(file, 0)) + return HA_ADMIN_CORRUPT; + return HA_ADMIN_OK; +}; + + struct st_mysql_storage_engine heap_storage_engine= { MYSQL_HANDLERTON_INTERFACE_VERSION }; From fc6212aba0979e79d671dd27e6b88d943a429d56 Mon Sep 17 00:00:00 2001 From: Monty Date: Sat, 13 Jun 2026 20:51:37 +0300 Subject: [PATCH 4/4] MDEV-40029 Add support for bit fields to HEAP This adds support for BIT_FIELD in record and keys for HEAP tables. The HEAP engine now has HA_CAN_BIT_FIELD set in table_flags() Multiple bugs in BIT field handling was found fixed. Some in HEAP table code, other bugs was affecting usage of BIT fields as keys. --- mysql-test/main/type_bit.result | 125 +++++++++++---------- mysql-test/main/type_bit.test | 10 +- mysql-test/suite/heap/heap_bit_hash.result | 86 ++++++++++++++ mysql-test/suite/heap/heap_bit_hash.test | 67 +++++++++++ sql/field.cc | 12 +- storage/heap/ha_heap.h | 3 +- storage/heap/hp_hash.c | 20 ++-- 7 files changed, 248 insertions(+), 75 deletions(-) create mode 100644 mysql-test/suite/heap/heap_bit_hash.result create mode 100644 mysql-test/suite/heap/heap_bit_hash.test diff --git a/mysql-test/main/type_bit.result b/mysql-test/main/type_bit.result index 8cfcc4eef5c54..6fcd1e5c9baa3 100644 --- a/mysql-test/main/type_bit.result +++ b/mysql-test/main/type_bit.result @@ -108,9 +108,16 @@ id select_type table type possible_keys key key_len ref rows Extra select a+0 from t1; a+0 0 -4 -5 -9 +104 +106 +108 +111 +116 +118 +119 +122 +123 +127 23 24 28 @@ -118,8 +125,10 @@ a+0 30 31 34 +4 44 49 +5 56 57 59 @@ -133,70 +142,68 @@ a+0 79 87 88 +9 94 94 -104 -106 -108 -111 -116 -118 -119 -122 -123 -127 explain select b+0 from t1; id select_type table type possible_keys key key_len ref rows Extra 1 SIMPLE t1 index NULL a 5 NULL 38 Using index select b+0 from t1; b+0 +118 +123 +133 +135 +152 177 -245 178 -363 -36 -398 -499 -399 -83 -438 +188 202 +206 +245 +280 307 -345 -379 -135 -188 343 -152 -206 -454 -42 -133 -123 +345 349 351 +36 +363 +368 +368 +379 +380 +390 +398 +399 +403 411 +411 +42 +438 +446 +454 46 468 -280 -446 +499 67 -368 -390 -380 -368 -118 -411 -403 +83 explain select a+0, b+0 from t1; id select_type table type possible_keys key key_len ref rows Extra 1 SIMPLE t1 index NULL a 5 NULL 38 Using index select a+0, b+0 from t1; a+0 b+0 0 177 -4 245 -5 178 -9 363 +104 280 +106 446 +108 67 +111 368 +116 390 +118 380 +119 368 +122 118 +123 411 +127 403 23 36 24 398 28 499 @@ -204,8 +211,10 @@ a+0 b+0 30 83 31 438 34 202 +4 245 44 307 49 345 +5 178 56 379 57 135 59 188 @@ -219,18 +228,9 @@ a+0 b+0 79 349 87 351 88 411 +9 363 94 46 94 468 -104 280 -106 446 -108 67 -111 368 -116 390 -118 380 -119 368 -122 118 -123 411 -127 403 explain select a+0, b+0 from t1 where a > 40 and b > 200 order by 1; id select_type table type possible_keys key key_len ref rows Extra 1 SIMPLE t1 range a a 2 NULL 27 Using where; Using index; Using filesort @@ -296,17 +296,20 @@ create table t1 (a int not null, b bit, c bit(9), key(a, b, c)); insert into t1 values (4, NULL, 1), (4, 0, 3), (2, 1, 4), (1, 1, 100), (4, 0, 23), (4, 0, 54), (56, 0, 22), (4, 1, 100), (23, 0, 1), (4, 0, 34); +explain select a+0, b+0, c+0 from t1; +id select_type table type possible_keys key key_len ref rows Extra +1 SIMPLE t1 index NULL a 9 NULL 10 Using index select a+0, b+0, c+0 from t1; a+0 b+0 c+0 1 1 100 2 1 4 -4 NULL 1 -4 0 3 +23 0 1 4 0 23 +4 0 3 4 0 34 4 0 54 4 1 100 -23 0 1 +4 NULL 1 56 0 22 select hex(min(b)) from t1 where a = 4; hex(min(b)) @@ -327,19 +330,19 @@ a+0 b+0 c+0 select a+0, b+0, c+0 from t1 where a = 4 and b = 1 and c=100; a+0 b+0 c+0 4 1 100 -select a+0, b+0, c+0 from t1 order by b desc; +select a+0, b+0, c+0 from t1 order by b desc, a, c; a+0 b+0 c+0 -2 1 4 1 1 100 +2 1 4 4 1 100 4 0 3 4 0 23 +4 0 34 4 0 54 -56 0 22 23 0 1 -4 0 34 +56 0 22 4 NULL 1 -select a+0, b+0, c+0 from t1 order by c; +select a+0, b+0, c+0 from t1 order by c,a,b; a+0 b+0 c+0 4 NULL 1 23 0 1 diff --git a/mysql-test/main/type_bit.test b/mysql-test/main/type_bit.test index 14e6fee300526..438b71f5cc8b2 100644 --- a/mysql-test/main/type_bit.test +++ b/mysql-test/main/type_bit.test @@ -48,6 +48,7 @@ insert into t1 values (b'00'), (b'01'), (b'10'), (b'100'); select a+0 from t1; alter table t1 add key (a); explain select a+0 from t1; +--sorted_result select a+0 from t1; drop table t1; @@ -60,10 +61,13 @@ insert into t1 values (30, 83), (5, 178), (60, 343), (4, 245), (104, 280), (106, 446), (127, 403), (44, 307), (68, 454), (57, 135); explain select a+0 from t1; +--sorted_result select a+0 from t1; explain select b+0 from t1; +--sorted_result select b+0 from t1; explain select a+0, b+0 from t1; +--sorted_result select a+0, b+0 from t1; explain select a+0, b+0 from t1 where a > 40 and b > 200 order by 1; select a+0, b+0 from t1 where a > 40 and b > 200 order by 1; @@ -80,6 +84,8 @@ create table t1 (a int not null, b bit, c bit(9), key(a, b, c)); insert into t1 values (4, NULL, 1), (4, 0, 3), (2, 1, 4), (1, 1, 100), (4, 0, 23), (4, 0, 54), (56, 0, 22), (4, 1, 100), (23, 0, 1), (4, 0, 34); +explain select a+0, b+0, c+0 from t1; +--sorted_result select a+0, b+0, c+0 from t1; select hex(min(b)) from t1 where a = 4; select hex(min(c)) from t1 where a = 4 and b = 0; @@ -87,8 +93,8 @@ select hex(max(b)) from t1; select a+0, b+0, c+0 from t1 where a = 4 and b = 0 limit 2; select a+0, b+0, c+0 from t1 where a = 4 and b = 1; select a+0, b+0, c+0 from t1 where a = 4 and b = 1 and c=100; -select a+0, b+0, c+0 from t1 order by b desc; -select a+0, b+0, c+0 from t1 order by c; +select a+0, b+0, c+0 from t1 order by b desc, a, c; +select a+0, b+0, c+0 from t1 order by c,a,b; drop table t1; create table t1(a bit(2), b bit(2)); diff --git a/mysql-test/suite/heap/heap_bit_hash.result b/mysql-test/suite/heap/heap_bit_hash.result new file mode 100644 index 0000000000000..6fe906a3aa0c6 --- /dev/null +++ b/mysql-test/suite/heap/heap_bit_hash.result @@ -0,0 +1,86 @@ +# +# HEAP hash index on a BIT column whose uneven (X & 7) +# high bits are stored in the record null-map header. +# +# hp_rec_hashnr() (used to place a row in a hash bucket on INSERT) +# must hash the same bits that hp_make_key()/hp_hashnr() use when +# searching. The bug hashed a record-body byte instead of the +# null-map bits, so placement and lookup disagreed and rows became +# unreachable through the index. +# +# Needs ENGINE=MEMORY advertising HA_CAN_BIT_FIELD so the uneven +# bits really live in the null map (seg->bit_length > 0), a key on +# the BIT column, and enough rows to force more than one hash bucket. +# +# BIT(10): 2 high bits in the null map, 1 byte in the record body. +# Column a is NULLable and stores NULLs. +CREATE TABLE t1 (a INT, b BIT(10), KEY(b)) ENGINE=MEMORY; +INSERT INTO t1 SELECT seq, seq FROM seq_1_to_400; +INSERT INTO t1 VALUES (NULL, b'1100000000'), (NULL, NULL); +# Every row must be reachable through the hash index. Under the bug +# only a handful are found (placement vs. lookup hash disagree). +SELECT COUNT(*) AS total FROM t1; +total +402 +SELECT COUNT(*) AS found_via_index FROM t1 t +WHERE EXISTS (SELECT 1 FROM t1 f FORCE INDEX(b) WHERE f.b <=> t.b); +found_via_index +402 +# Point lookups via the index must agree with a full scan. +SELECT COUNT(*) AS via_index FROM t1 FORCE INDEX(b) +WHERE b = CAST(123 AS UNSIGNED); +via_index +1 +SELECT COUNT(*) AS via_scan FROM t1 IGNORE INDEX(b) +WHERE b = CAST(123 AS UNSIGNED); +via_scan +1 +SELECT COUNT(*) AS dup_value_via_index FROM t1 FORCE INDEX(b) +WHERE b = b'1100000000'; +dup_value_via_index +1 +CHECK TABLE t1; +Table Op Msg_type Msg_text +test.t1 check status OK +DROP TABLE t1; +# BIT(3): all 3 bits live in the null map (0 body bytes), with a +# NULLable column that also stores NULLs. +CREATE TABLE t2 (a INT, b BIT(3), KEY(b)) ENGINE=MEMORY; +INSERT INTO t2 VALUES (1,b'000'),(2,b'001'),(3,b'010'),(4,b'011'), +(5,b'100'),(6,b'101'),(7,b'110'),(8,b'111'), +(NULL,b'101'),(NULL,NULL),(9,NULL); +SELECT COUNT(*) AS total FROM t2; +total +11 +SELECT COUNT(*) AS found_via_index FROM t2 t +WHERE EXISTS (SELECT 1 FROM t2 f FORCE INDEX(b) WHERE f.b <=> t.b); +found_via_index +11 +SELECT a FROM t2 FORCE INDEX(b) WHERE b=b'101' ORDER BY a; +a +NULL +6 +CHECK TABLE t2; +Table Op Msg_type Msg_text +test.t2 check status OK +DROP TABLE t2; +# Multi-part key with two BIT segments (b BIT(1), c BIT(9)) and a +# NULLable BIT column that stores NULLs. A stray key-pointer bump +# in hp_key_cmp() used to shift the second BIT segment's offset, so +# the lookup on the full key found nothing. +CREATE TABLE t3 (a INT NOT NULL, b BIT, c BIT(9), KEY(a,b,c)) ENGINE=MEMORY; +INSERT INTO t3 VALUES +(4, NULL, 1), (4, 0, 3), (2, 1, 4), (1, 1, 100), (4, 0, 23), (4, 0, 54), +(56, 0, 22), (4, 1, 100), (23, 0, 1), (4, 0, 34); +SELECT a+0, b+0, c+0 FROM t3 WHERE a=4 AND b=1 AND c=100; +a+0 b+0 c+0 +4 1 100 +SELECT COUNT(*) AS found_via_index FROM t3 t WHERE EXISTS +(SELECT 1 FROM t3 f FORCE INDEX(a) +WHERE f.a<=>t.a AND f.b<=>t.b AND f.c<=>t.c); +found_via_index +10 +CHECK TABLE t3; +Table Op Msg_type Msg_text +test.t3 check status OK +DROP TABLE t3; diff --git a/mysql-test/suite/heap/heap_bit_hash.test b/mysql-test/suite/heap/heap_bit_hash.test new file mode 100644 index 0000000000000..2a4f4925d29c5 --- /dev/null +++ b/mysql-test/suite/heap/heap_bit_hash.test @@ -0,0 +1,67 @@ +--source include/have_sequence.inc + +--echo # +--echo # HEAP hash index on a BIT column whose uneven (X & 7) +--echo # high bits are stored in the record null-map header. +--echo # +--echo # hp_rec_hashnr() (used to place a row in a hash bucket on INSERT) +--echo # must hash the same bits that hp_make_key()/hp_hashnr() use when +--echo # searching. The bug hashed a record-body byte instead of the +--echo # null-map bits, so placement and lookup disagreed and rows became +--echo # unreachable through the index. +--echo # +--echo # Needs ENGINE=MEMORY advertising HA_CAN_BIT_FIELD so the uneven +--echo # bits really live in the null map (seg->bit_length > 0), a key on +--echo # the BIT column, and enough rows to force more than one hash bucket. +--echo # + +--echo # BIT(10): 2 high bits in the null map, 1 byte in the record body. +--echo # Column a is NULLable and stores NULLs. +CREATE TABLE t1 (a INT, b BIT(10), KEY(b)) ENGINE=MEMORY; +INSERT INTO t1 SELECT seq, seq FROM seq_1_to_400; +INSERT INTO t1 VALUES (NULL, b'1100000000'), (NULL, NULL); + +--echo # Every row must be reachable through the hash index. Under the bug +--echo # only a handful are found (placement vs. lookup hash disagree). +SELECT COUNT(*) AS total FROM t1; +SELECT COUNT(*) AS found_via_index FROM t1 t + WHERE EXISTS (SELECT 1 FROM t1 f FORCE INDEX(b) WHERE f.b <=> t.b); + +--echo # Point lookups via the index must agree with a full scan. +SELECT COUNT(*) AS via_index FROM t1 FORCE INDEX(b) + WHERE b = CAST(123 AS UNSIGNED); +SELECT COUNT(*) AS via_scan FROM t1 IGNORE INDEX(b) + WHERE b = CAST(123 AS UNSIGNED); +SELECT COUNT(*) AS dup_value_via_index FROM t1 FORCE INDEX(b) + WHERE b = b'1100000000'; + +CHECK TABLE t1; +DROP TABLE t1; + +--echo # BIT(3): all 3 bits live in the null map (0 body bytes), with a +--echo # NULLable column that also stores NULLs. +CREATE TABLE t2 (a INT, b BIT(3), KEY(b)) ENGINE=MEMORY; +INSERT INTO t2 VALUES (1,b'000'),(2,b'001'),(3,b'010'),(4,b'011'), + (5,b'100'),(6,b'101'),(7,b'110'),(8,b'111'), + (NULL,b'101'),(NULL,NULL),(9,NULL); +SELECT COUNT(*) AS total FROM t2; +SELECT COUNT(*) AS found_via_index FROM t2 t + WHERE EXISTS (SELECT 1 FROM t2 f FORCE INDEX(b) WHERE f.b <=> t.b); +SELECT a FROM t2 FORCE INDEX(b) WHERE b=b'101' ORDER BY a; +CHECK TABLE t2; +DROP TABLE t2; + +--echo # Multi-part key with two BIT segments (b BIT(1), c BIT(9)) and a +--echo # NULLable BIT column that stores NULLs. A stray key-pointer bump +--echo # in hp_key_cmp() used to shift the second BIT segment's offset, so +--echo # the lookup on the full key found nothing. +CREATE TABLE t3 (a INT NOT NULL, b BIT, c BIT(9), KEY(a,b,c)) ENGINE=MEMORY; +INSERT INTO t3 VALUES + (4, NULL, 1), (4, 0, 3), (2, 1, 4), (1, 1, 100), (4, 0, 23), (4, 0, 54), + (56, 0, 22), (4, 1, 100), (23, 0, 1), (4, 0, 34); +SELECT a+0, b+0, c+0 FROM t3 WHERE a=4 AND b=1 AND c=100; +SELECT COUNT(*) AS found_via_index FROM t3 t WHERE EXISTS + (SELECT 1 FROM t3 f FORCE INDEX(a) + WHERE f.a<=>t.a AND f.b<=>t.b AND f.c<=>t.c); +CHECK TABLE t3; +DROP TABLE t3; diff --git a/sql/field.cc b/sql/field.cc index ecabbb899ead1..6c796cbf36c97 100644 --- a/sql/field.cc +++ b/sql/field.cc @@ -10533,18 +10533,22 @@ int Field_bit::cmp_offset(my_ptrdiff_t row_offset) } -uint Field_bit::get_key_image(uchar *buff, uint length, const uchar *ptr_arg, imagetype type_arg) const +uint Field_bit::get_key_image(uchar *buff, uint length, + const uchar *ptr_arg, imagetype type_arg) const { + uint bit_byte= 0; if (bit_len) { const uchar *bit_ptr_for_arg= ptr_arg + (bit_ptr - ptr); uchar bits= get_rec_bits(bit_ptr_for_arg, bit_ofs, bit_len); *buff++= bits; length--; + bit_byte= 1; } - uint tmp_data_length = MY_MIN(length, bytes_in_rec); - memcpy(buff, ptr, tmp_data_length); - return tmp_data_length + 1; + /* We don't support prefix read on bit fields */ + DBUG_ASSERT(length == bytes_in_rec); + memcpy(buff, ptr_arg, bytes_in_rec); + return bytes_in_rec + bit_byte; } diff --git a/storage/heap/ha_heap.h b/storage/heap/ha_heap.h index c1cf5b7e35440..22488c0fa57d1 100644 --- a/storage/heap/ha_heap.h +++ b/storage/heap/ha_heap.h @@ -43,7 +43,7 @@ class ha_heap final : public handler HA_CAN_SQL_HANDLER | HA_CAN_ONLINE_BACKUPS | HA_REC_NOT_IN_SEQ | HA_CAN_INSERT_DELAYED | HA_NO_TRANSACTIONS | HA_HAS_RECORDS | HA_STATS_RECORDS_IS_EXACT | HA_CAN_HASH_KEYS | - HA_CAN_GEOMETRY); + HA_CAN_GEOMETRY | HA_CAN_BIT_FIELD); } ulong index_flags(uint inx, uint part, bool all_parts) const override { @@ -101,6 +101,7 @@ class ha_heap final : public handler int delete_table(const char *from) override; void drop_table(const char *name) override; int rename_table(const char * from, const char * to) override; + int check(THD* thd, HA_CHECK_OPT* check_opt) override; int create(const char *name, TABLE *form, HA_CREATE_INFO *create_info) override; void update_create_info(HA_CREATE_INFO *create_info) override; diff --git a/storage/heap/hp_hash.c b/storage/heap/hp_hash.c index bcd0f1cd73506..83eff09b79702 100644 --- a/storage/heap/hp_hash.c +++ b/storage/heap/hp_hash.c @@ -315,7 +315,7 @@ ulong hp_hashnr(HP_KEYDEF *keydef, const uchar *key) } } #ifdef ONLY_FOR_HASH_DEBUGGING - DBUG_PRINT("exit", ("hash: 0x%lx", nr)); + DBUG_PRINT("exit", ("hash: 0x%lx", hasher.m_nr1)); #endif return((ulong) hasher.m_nr1); } @@ -754,10 +754,9 @@ int hp_key_cmp(HP_KEYDEF *keydef, const uchar *rec, const uchar *key, if (bits != (*key)) return 1; dec= 1; - key++; } - if (bcmp(rec + seg->start, key, seg->length - dec)) + if (bcmp(rec + seg->start, key + dec, seg->length - dec)) return 1; } } @@ -847,6 +846,7 @@ uint hp_rb_make_key(HP_KEYDEF *keydef, uchar *key, for (seg= keydef->seg, endseg= seg + keydef->keysegs; seg < endseg; seg++) { size_t char_length; + size_t offset; if (seg->null_bit) { if (!(*key++= 1 - MY_TEST(rec[seg->null_pos] & seg->null_bit))) @@ -906,10 +906,12 @@ uint hp_rb_make_key(HP_KEYDEF *keydef, uchar *key, continue; } + offset= 0; char_length= seg->length; + if (seg->charset->mbmaxlen > 1) { - char_length= hp_charpos(seg->charset, + char_length= hp_charpos(seg->charset, rec + seg->start, rec + seg->start + char_length, char_length / seg->charset->mbmaxlen); set_if_smaller(char_length, seg->length); /* QQ: ok to remove? */ @@ -919,11 +921,15 @@ uint hp_rb_make_key(HP_KEYDEF *keydef, uchar *key, } if (seg->type == HA_KEYTYPE_BIT && seg->bit_length) { - *key++= get_rec_bits(rec + seg->bit_pos, + /* + The uneven high bits are stored in the record null-map header. + Put them in the first key byte. + */ + *key= get_rec_bits(rec + seg->bit_pos, seg->bit_start, seg->bit_length); - char_length--; + offset= 1; } - memcpy(key, rec + seg->start, (size_t) char_length); + memcpy(key + offset, rec + seg->start, char_length - offset); key+= seg->length; } memcpy(key, &recpos, sizeof(uchar*));