MyISAM and Aria used to lie to the server about the reflength value.
One value was used internally, it was stored on disk, e.g. in indexes,
and couldn't be changed without full table rebuild. A differently
calculated value was reported to the server - that value was sometimes
larger than the true reflength.
That caused the server to allocate more memory per position than
necessary - affecting filesort, join buffer usage, optimizer cost
calculations, and may be more.
when adding a column or index that uses plugin-defined
sysvar-based options with ALTER TABLE ... ADD, the server
was using the default value of the sysvar, not the current one.
CREATE TABLE was correctly using the current sysvar value.
Fix it so that new columns/indexes added in ALTER TABLE ... ADD
would use a current sysvar value. Existing columns/indexes
in ALTER TABLE would keep using the default sysvar value
(unless they had an explicit value in frm).
MHNSW_Trx cannot store a pointer to the TABLE_SHARE for the duration
of a transaction, because the share can be evicted from the cache any
time.
Use a pointer to the MDL_ticket instead, it is stored in the THD
and has a duration of MDL_TRANSACTION, so won't go away.
When we need a TABLE_SHARE - on commit - get a new one from tdc.
Normally, it'll be already in the cache, so it'd be fast.
We don't optimize for the edge case when TABLE_SHARE was evicted.
support SQL semantics for SELECT ... WHERE ... ORDER BY ... LIMIT
* switch from returning k nearest neighbors to returning
as many as needed, in k-neighbor chunks, with increasing distance
* make search_layer() skips nodes that are closer than a threshold
* read_next keeps a search context - list of k found nodes,
threshold, ctx, etc.
* when the list of found nodes is exhausted, it repeats the search
starting from last found nodes and a threshold
* search context kepts ctx->refcount incremented, so ctx won't go away
* but commit_lock is unlocked between calls, so InnoDB can modify the table
* use ctx version to detect that, switch to MHNSW_Trx when it happens
bugfix:
* use the correct lock in ha_external_lock() for the graph table
* InnoDB didn't reset locks on ha_external_lock(F_UNLCK) and previous
LOCK_X leaked into the next statement
in ALTER TABLE or CREATE TABLE LIKE, a create a copy of key->option_list,
because it can be extended later on the thd->mem_root, so it has
to be a copy
let's allow ::position() and ::rnd_pos() in blackhole.
::position() can be called directly after insert, it doesn't need
a search to happen, so it's possible.
::rnd_pos() can be called with a value that ::position() produced,
so, possible too.
make generosity depend on
1. M. Keep small M's fast, increase generosity for larger M's to get
better recall.
2. distance. Keep generosity small when vectors are far from the
target, increase generosity when the search gets closer. This
allows to examine more relevant vectors but doesn't waste time
examining irrelevant vectors. Particularly important with cosine
metric when the distance is bounded
quick_rm_table() expects .frm to exist when it removes high-level indexes.
For cases like ALTER TABLE t1 RENAME TO t2, ENGINE=other_engine .frm was
removed earlier.
Another option would be removing high-level indexes explicitly before the
first quick_rm_table() and skipping high-level indexes for subsequent
quick_rm_table(NO_FRM_RENAME).
But this suggested order may also help with ddl log recovery. That is
if we crash before high-level indexes are removed, .frm is going to
exist.
Disable non-copy ALTER algorithms when VECTOR index is affected. Engines
are not supposed to handle high-level indexes anyway.
Also fixed misbehaving IF [NOT] EXISTS variants.
Fixes for ALTER TABLE ... ADD/DROP COLUMN, ALGORITHM=COPY.
Let quick_rm_table() remove high-level indexes along with original table.
Avoid locking uninitialized LOCK_share for INTERNAL_TMP_TABLEs.
Don't enable bulk insert when altering a table containing vector index.
InnoDB can't handle situation when bulk insert is enabled for one table
but disabled for another. We can't do bulk insert on vector index as it
does table updates currently.
into a separate transaction_participant structure
handlerton inherits it, so handlerton itself doesn't change.
but entities that only need to participate in a transaction,
like binlog or online alter log, use a transaction_participant
and no longer need to pretend to be a full-blown but invisible
storage engine which doesn't support create table.
* fix the truncate-by-handler variant, used by InnoDB
* test that insert works after truncate, meaning graph table was emptied
* test that the vector index size is zero after truncate in MyISAM
This patch fixes only TRUNCATE by recreate variant, there seem to be no
reasonable engine that uses TRUNCATE by handler method for testing.
Reset index_cinfo so that mi_create is not confused by garbage passed via
index_file_name and sets MY_DELETE_OLD flag.
Review question: can we add a test case to make sure VECTOR index is empty
indeed?
This commit introduces two utility functions meant to make working with
vectors simpler.
Vec_ToText converts a binary vector into a json array of numbers
(floats).
Vec_FromText takes in a json array of numbers and converts it into a
little-endian IEEE float sequence of bytes (4 bytes per float).
This method will write out a float to a String object, keeping the
charset of the original string.
Also have Float::to_string make use of String::append_float
introduced a generosity factor that makes the search less greedy.
it dramatically improves the recall by making the search a bit slower
(for the same recall one can use half the M and smaller ef).
had to add Queue::safe_push() method that removes one of the
furthest elements (not necessarily the furthest) in the queue
to keep it from overflowing.
use int16_t instead of floats, they're faster and smaller.
but perform intermediate SIMD calculations with floats to avoid overflows.
recall drop with such scheme is below 0.002, often none.
int8_t would've been better but the precision loss is too big
and recall degrades too much.
