Commit graph

79445 commits

Author SHA1 Message Date
Sergei Golubchik
8f49fb8cc3 MDEV-35191 Assertion failure in Create_tmp_table::finalize upon DISTINCT with vector type
test only
2024-11-05 14:00:51 -08:00
Sergei Golubchik
cfbf065893 MDEV-35176 ASAN errors in Field_vector::store with optimizer_trace enabled 2024-11-05 14:00:51 -08:00
Sergei Golubchik
425aa95655 MDEV-35178 Assertion failure in Field_vector::store upon INSERT IGNORE with a wrong data 2024-11-05 14:00:51 -08:00
Sergei Golubchik
88adcbf35a MDEV-35182 crash in online_alter_end_trans with XA over vector indexes
ONLINE ALTER didn't expect XA PREPARE to fail.
Mark rollback on failed prepare with the XA_ROLLBACK_ONLY state,
detect that in ONLINE ALTER
2024-11-05 14:00:51 -08:00
Sergei Golubchik
5bde23990b MDEV-35159 Assertion `tab->join->select_limit < (~ (ha_rows) 0)' fails upon forcing vector key
init_from_binary_frm_image() wrongly assumed that
* if a table has primary key
* and it has the HA_PRIMARY_KEY_IN_READ_INDEX flag
* than ORDER BY any index automatically implies ORDER BY pk at the end,
   that is for an index (a,b,c) ORDER BY a,b,c means ORDER BY a,b,c,pk

which is wrong, it holds not for _any index_ but only for indexes
that can be used for ORDER BY.

So, don't do `field->part_of_sortkey= share->keys_in_use`
but introduce `sort_keys_in_use` and use that.
2024-11-05 14:00:51 -08:00
Sergei Golubchik
88119addff Vec_ToText was underestimating max_length of the result
switch to a more predictable, shorter, and more correct output
that is, print as many significant digits as necessary.
but not more (they'd be just zeros) and not less (it'd lose precision)
2024-11-05 14:00:51 -08:00
Sergei Golubchik
91720da9be MDEV-35158 Assertion `res->length() > 0 && res->length() % 4 == 0' fails upon increasing length of vector column 2024-11-05 14:00:51 -08:00
Sergei Golubchik
6634c88480 MDEV-35150 Column containing non-vector tables can be modified to VECTOR type without warnings 2024-11-05 14:00:51 -08:00
Sergei Golubchik
ca28761066 MDEV-35147 Inconsistent NULL handling in vector type 2024-11-05 14:00:51 -08:00
Sergei Golubchik
f274cf1c25 MDEV-35141 Server crashes in Field_vector::report_wrong_value upon statistic collection 2024-11-05 14:00:51 -08:00
Sergei Golubchik
78119d1ae5 MDEV-33410 VECTOR data type 2024-11-05 14:00:51 -08:00
Sergei Golubchik
b56ca29f89 MDEV-35105 Assertion `tab->join->order' fails upon vector search with DISTINCT
don't apply distinct optimization to order by a vector index
2024-11-05 14:00:51 -08:00
Sergei Golubchik
9ddb94f60e MDEV-35104 Invalid (old?) table or database name upon DDL on table with vector key and unique key
InnoDB rename needs the same workaround for hlindexes
as it has for partitions
2024-11-05 14:00:51 -08:00
Sergei Golubchik
7d9c0e4f62 MDEV-35092 Server crash, hang or ASAN errors in mysql_create_frm_image upon using non-default table options and system variables
extend the option_list expicitly on CREATE/ALTER, not implicitly
on parsing.
2024-11-05 14:00:51 -08:00
Sergei Golubchik
cdc7253787 make MyISAM and Aria report correct reflength to the server
MyISAM and Aria used to lie to the server about the reflength value.
One value was used internally, it was stored on disk, e.g. in indexes,
and couldn't be changed without full table rebuild. A differently
calculated value was reported to the server - that value was sometimes
larger than the true reflength.

That caused the server to allocate more memory per position than
necessary - affecting filesort, join buffer usage, optimizer cost
calculations, and may be more.
2024-11-05 14:00:51 -08:00
Sergei Golubchik
ea1e720391 MDEV-35078 Server crash or ASAN errors in mhnsw_insert
when adding a column or index that uses plugin-defined
sysvar-based options with ALTER TABLE ... ADD, the server
was using the default value of the sysvar, not the current one.

CREATE TABLE was correctly using the current sysvar value.

Fix it so that new columns/indexes added in ALTER TABLE ... ADD
would use a current sysvar value. Existing columns/indexes
in ALTER TABLE would keep using the default sysvar value
(unless they had an explicit value in frm).
2024-11-05 14:00:51 -08:00
Sergei Golubchik
855aefb7b5 mysqldump and mariadb-backup tests of vector indexes 2024-11-05 14:00:51 -08:00
Sergei Golubchik
eb4ab2ce8f MDEV-35061 XA PREPARE "not supported by the engine" from storage engine mhnsw, memory leak
disallow explicit XA PREPARE over mhnsw indexes
2024-11-05 14:00:51 -08:00
Sergei Golubchik
09cd817f5d MDEV-35060 Assertion failure upon DML on table with vector under lock 2024-11-05 14:00:51 -08:00
Sergei Golubchik
09889d417b MDEV-35055 ASAN errors in TABLE_SHARE::lock_share upon committing transaction after FLUSH on table with vector key
MHNSW_Trx cannot store a pointer to the TABLE_SHARE for the duration
of a transaction, because the share can be evicted from the cache any
time.

Use a pointer to the MDL_ticket instead, it is stored in the THD
and has a duration of MDL_TRANSACTION, so won't go away.

When we need a TABLE_SHARE - on commit - get a new one from tdc.
Normally, it'll be already in the cache, so it'd be fast.
We don't optimize for the edge case when TABLE_SHARE was evicted.
2024-11-05 14:00:51 -08:00
Sergei Golubchik
d396fb9226 MDEV-35021 Behavior for RTREE indexes changed, assertion fails
disallow USING RTREE for not SPATIAL index
2024-11-05 14:00:51 -08:00
Sergei Golubchik
b3afd9f640 MDEV-35042 Vector indexes are allowed for MERGE tables, but do not
disallow hlindexes in MERGE - because we cannot create the secondary
table *in the same engine*
2024-11-05 14:00:51 -08:00
Sergei Golubchik
0932c3a27e MDEV-35044 ALTER on a table with vector index attempts to bypass unsupported locking limitation, server crashes in THD::free_tmp_table_share
open secondary tables early enough for the cleanup on error to see
them and remove their underlying files
2024-11-05 14:00:51 -08:00
Sergei Golubchik
824a63852b MDEV-35043 Unsuitable error upon an attempt to create MEMORY table with vector key
MEMORY engine doesn't support blobs
2024-11-05 14:00:51 -08:00
Sergei Golubchik
9f80e3fbb7 MDEV-35032 streaming mode for mhnsw search
support SQL semantics for SELECT ... WHERE ... ORDER BY ... LIMIT

* switch from returning k nearest neighbors to returning
  as many as needed, in k-neighbor chunks, with increasing distance
* make search_layer() skips nodes that are closer than a threshold
* read_next keeps a search context - list of k found nodes,
  threshold, ctx, etc.
* when the list of found nodes is exhausted, it repeats the search
  starting from last found nodes and a threshold
* search context kepts ctx->refcount incremented, so ctx won't go away
* but commit_lock is unlocked between calls, so InnoDB can modify the table
* use ctx version to detect that, switch to MHNSW_Trx when it happens

bugfix:
* use the correct lock in ha_external_lock() for the graph table
* InnoDB didn't reset locks on ha_external_lock(F_UNLCK) and previous
  LOCK_X leaked into the next statement
2024-11-05 14:00:51 -08:00
Sergei Golubchik
be69716287 MDEV-35029 ASAN errors in Lex_ident<Compare_ident_ci>::is_valid_ident upon DDL on table with vector index
in ALTER TABLE or CREATE TABLE LIKE, a create a copy of key->option_list,
because it can be extended later on the thd->mem_root, so it has
to be a copy
2024-11-05 14:00:51 -08:00
Sergei Golubchik
a6499049af MDEV-35033 LeakSanitizer errors in my_malloc / safe_mutex_lazy_init_deadlock_detection / MHNSW_Context::alloc_node and alike
ctx wasn't released on errors
2024-11-05 14:00:51 -08:00
Sergei Golubchik
6837bb54f4 MDEV-35020 After a failed attempt to create vector index temporary file remains and prevents further operation 2024-11-05 14:00:50 -08:00
Sergei Golubchik
ec2ff9f2a0 MDEV-35035 Assertion failure in ha_blackhole::position upon INSERT into blackhole table with vector index
let's allow ::position() and ::rnd_pos() in blackhole.
::position() can be called directly after insert, it doesn't need
a search to happen, so it's possible.

::rnd_pos() can be called with a value that ::position() produced,
so, possible too.
2024-11-05 14:00:50 -08:00
Sergei Golubchik
8ac3f0b1d4 MDEV-35038 Server crash in Index_statistics::get_avg_frequency upon EITS collection for vector index
don't collect eits for vector indexes
2024-11-05 14:00:50 -08:00
Sergei Golubchik
0bd01f4a95 MDEV-35039 Number of indexes inside InnoDB differs from that defined in MariaDB after altering table with vector key
don't show table->s->total_keys to engine in inplace alter
2024-11-05 14:00:50 -08:00
Sergei Golubchik
8253650aaa MDEV-35006 Using varbinary as vector-storing column results in assertion failures
* hlindexes cannot be extended with pk
* hlindexes cannot be covering
2024-11-05 14:00:50 -08:00
Sergei Golubchik
fb04cad37e trying to stabilize floating-point tests 2024-11-05 14:00:50 -08:00
Sergey Vojtovich
f867c2a21e Disabled high-level indexes with Aria
... until a few bugs that cause server crash are fixed.
2024-11-05 14:00:50 -08:00
Sergey Vojtovich
ca17b68bb6 ALTER TABLE fixes for high-level indexes (iii)
quick_rm_table() expects .frm to exist when it removes high-level indexes.
For cases like ALTER TABLE t1 RENAME TO t2, ENGINE=other_engine .frm was
removed earlier.

Another option would be removing high-level indexes explicitly before the
first quick_rm_table() and skipping high-level indexes for subsequent
quick_rm_table(NO_FRM_RENAME).

But this suggested order may also help with ddl log recovery. That is
if we crash before high-level indexes are removed, .frm is going to
exist.
2024-11-05 14:00:50 -08:00
Sergey Vojtovich
7aa6bb3aa3 ALTER TABLE fixes for high-level indexes (ii)
Disable non-copy ALTER algorithms when VECTOR index is affected. Engines
are not supposed to handle high-level indexes anyway.

Also fixed misbehaving IF [NOT] EXISTS variants.
2024-11-05 14:00:50 -08:00
Sergey Vojtovich
a90fa3f397 ALTER TABLE fixes for high-level indexes (i)
Fixes for ALTER TABLE ... ADD/DROP COLUMN, ALGORITHM=COPY.

Let quick_rm_table() remove high-level indexes along with original table.

Avoid locking uninitialized LOCK_share for INTERNAL_TMP_TABLEs.

Don't enable bulk insert when altering a table containing vector index.
InnoDB can't handle situation when bulk insert is enabled for one table
but disabled for another. We can't do bulk insert on vector index as it
does table updates currently.
2024-11-05 14:00:50 -08:00
Sergei Golubchik
2ad9df8c9b VEC_Distance_Cosine() 2024-11-05 14:00:50 -08:00
Sergei Golubchik
2e1fcc6a80 rename VEC_Distance to VEC_Distance_Euclidean
and create a parent Item_func_vec_distance_common class
2024-11-05 14:00:50 -08:00
Sergei Golubchik
0da820cb12 mhnsw: use plugin index options and transaction_participant API 2024-11-05 14:00:50 -08:00
Sergei Golubchik
126d6d787c cleanup: handlerton
remove unused methods, reorder methods, add comments
2024-11-05 14:00:50 -08:00
Sergei Golubchik
8087cefc07 make rename test to work with InnoDB too 2024-11-05 14:00:50 -08:00
Sergei Golubchik
445198c10e pos-fixes for rename 2024-11-05 14:00:50 -08:00
Sergey Vojtovich
97e112fb82 VECTOR indexes support for RENAME TABLE
Rename high-level indexes along with a table.
2024-11-05 14:00:49 -08:00
Sergei Golubchik
ebcbed6d74 post-fixes for TRUNCATE
* fix the truncate-by-handler variant, used by InnoDB
* test that insert works after truncate, meaning graph table was emptied
* test that the vector index size is zero after truncate in MyISAM
2024-11-05 14:00:49 -08:00
Sergey Vojtovich
70575defb7 Fixed TRUNCATE TABLE against VECTOR indexes
This patch fixes only TRUNCATE by recreate variant, there seem to be no
reasonable engine that uses TRUNCATE by handler method for testing.

Reset index_cinfo so that mi_create is not confused by garbage passed via
index_file_name and sets MY_DELETE_OLD flag.

Review question: can we add a test case to make sure VECTOR index is empty
indeed?
2024-11-05 14:00:49 -08:00
Sergey Vojtovich
91a24ddc5d Test insert ... select with vector index 2024-11-05 14:00:49 -08:00
Sergey Vojtovich
4aa1968b89 Disable VECTOR indexes with partitioned tables 2024-11-05 14:00:49 -08:00
Sergey Vojtovich
7c16bba71d CREATE TABLE ... LIKE loses VECTOR index 2024-11-05 14:00:49 -08:00
Vicențiu Ciorbaru
eec1339f5d MDEV-32886 Vec_FromText and Vec_ToText
This commit introduces two utility functions meant to make working with
vectors simpler.

Vec_ToText converts a binary vector into a json array of numbers
(floats).
Vec_FromText takes in a json array of numbers and converts it into a
little-endian IEEE float sequence of bytes (4 bytes per float).
2024-11-05 14:00:49 -08:00