MDEV-14021 Failing assertion: table->get_ref_count() == 0 on ALTER TABLE with implicit ALGORITHM=INPLACE
Post-merge fix: Remove the now-useless use of table reference count.
The dict_operation_lock S-latch whose use was introduced in the 10.0
fix will prevent the table from disappearing due to concurrent
DDL operations (DROP, table-rebuilding ALTER, TRUNCATE).
InnoDB was writing unnecessary information to the
update undo log records. Most notably, if an indexed column is updated,
the old value of the column would be logged twice: first as part of
the update vector, and then another time because it is an indexed column.
Because the InnoDB undo log record must fit in a single page,
this would cause unnecessary failure of certain updates.
Even after this fix, InnoDB still seems to be unnecessarily logging
indexed column values for non-updated columns. It seems that non-updated
secondary index columns only need to be logged when a PRIMARY KEY
column is updated. To reduce risk, we are not fixing this remaining flaw
in GA versions.
trx_undo_page_report_modify(): Log updated indexed columns only once.
remove remnants of 10.0 bugfix, incorrectly merged into 10.2
Using col_names[i] was obviously, wrong, must've been col_names[ifield->col_no].
incorrect column name resulted in innodb having index unique_id2(id1),
while the server thought it's unique_id2(id4).
But col_names[ifield->col_no] is wrong too, because `table` has non-renamed
columns, so the correct column name is always dict_table_get_col_name(table, ifield->col_no)
i_s_sys_tables_fill_table_stats(): Acquire dict_operation_lock
S-latch before acquiring dict_sys->mutex, to prevent the table
from being removed from the data dictionary cache and from
being freed while i_s_dict_fill_sys_tablestats() is accessing
the table handle.
Issue
=====
The original issue was that the size of a fil_per_table tablespace was calculated
incorrectly during truncate in the presence of an fts index. This incorrect calculation
was fixed as part of BUG#25053705 along with a testcase to reproduce the bug. The
assert that was added as part of it to reproduce the bug was wrong and resulted in
this bug.
Fix
===
Although the assert was removed earlier in a seperate commit as it was blocking the
ntest, this patch replaces the other parts of the code that were added to reproduce
the bug and replaces it with code that tries to reproduce the bug in a different way.
The new code basically tries to tweak conditions so as to simulate the random read
where a page that doesn't exist is tried to be read.
RB: 15890
Reviewed-by: Jimmy Yang <Jimmy.Yang@oracle.com>
Reviewed-by: Satya Bodapati <satya.bodapati@oracle.com>
The ownership of the field query->intersection usually transfers
to query->doc_ids. In some error scenario, it could be possible
that fts_query_free() would be invoked with query->intersection!=NULL.
Let us handle that case, instead of intentionally crashing the server.
MySQL 5.7 added code to push down the LIMIT to fulltext search
in InnoDB:
commit 2cd0ebf97e1b265e2282d7ceb5d8dfb663ffc48f
Author: Thirunarayanan Balathandayuthapani <thirunarayanan.balathandayuth@oracle.com>
Date: Fri May 27 13:49:28 2016 +0530
Bug #22709692 FTS QUERY EXCEEDS RESULT CACHE LIMIT
The code was disabled when MySQL 5.7.9 was merged to MariaDB 10.2.2.
We shall remove the disabled code and unnecessary variables.
When MySQL 5.6.10 introduced innodb_read_only mode, it skipped the
creation of the InnoDB buffer pool dump/restore subsystem in that mode.
Attempts to set the variable innodb_buf_pool_dump_now would have
no effect in innodb_read_only mode, but the corresponding condition
was forgotten in from the other two update functions.
MySQL 5.7.20 would fix the innodb_buffer_pool_load_now,
but not innodb_buffer_pool_load_abort. Let us fix both in MariaDB.
Port the previous patch:
- Implement MariaDB's Group Commit API. This is a first
attempt which lacks the expected performance.
To newer MariaDB (which includes newer MyRocks)
fts_get_next_doc_id(): Assign the first and subsequent FTS_DOC_ID
in the same way: by post-incrementing the cached value.
If there is a user-specified FTS_DOC_ID, do not touch the internal
sequence.
There are two bugs related to failed ADD INDEX and
the InnoDB table cache eviction.
dict_table_close(): Try dropping failed ADD INDEX when releasing
the last table handle, not when releasing the last-but-one.
dict_table_remove_from_cache_low(): Do not invoke
row_merge_drop_indexes() after freeing all index metadata.
Instead, directly invoke row_merge_drop_indexes_dict() to
remove the metadata from the persistent data dictionary
and to free the index pages.
This was the cause of the bug in CalculateArray
modified: storage/connect/jsonudf.cpp
modified: storage/connect/tabjson.cpp
modified: storage/connect/value.cpp
- Avoid some compiler warnings
modified: storage/connect/mongo.cpp
modified: storage/connect/tabjmg.cpp
modified: storage/connect/tabtbl.cpp
modified: storage/connect/tabutil.cpp
modified: storage/connect/ha_connect.cc
- Include MONGO in all Java enabled distributions
Mongo will be enabled only for 10.2 and 10.3
modified: storage/connect/CMakeLists.txt
- Change JDBC_SUPPORT to JAVA_SUPPORT which also replaces MONGO_SUPPORT
MONGO_SUPPORT is now just used to enable the MONGO table type
modified: storage/connect/filter.cpp
modified: storage/connect/ha_connect.cc
modified: storage/connect/ha_connect.h
modified: storage/connect/mongo.cpp
modified: storage/connect/mycat.cc
modified: storage/connect/plgdbutl.cpp
modified: storage/connect/tabjson.cpp
modified: storage/connect/tabjson.h
- Move MakeSelector function from FILTER to mongo.cpp
modified: storage/connect/filter.cpp
modified: storage/connect/filter.h
modified: storage/connect/cmgoconn.cpp
modified: storage/connect/jmgoconn.cpp
modified: storage/connect/mongo.cpp
- Do mongo_init only on first use of the MongoDB C Driver
This will permit to delay load the mongo lib on Windows
modified: storage/connect/cmgoconn.cpp
modified: storage/connect/cmgoconn.h
modified: storage/connect/ha_connect.cc
- Replace NEW_VAR by a test on MYSQL_VERSION_ID
modified: storage/connect/ha_connect.cc
- Suppress enable_mongo session variable
modified: storage/connect/ha_connect.cc
modified: storage/connect/mycat.cc
- Make some function headers identical in .h and .cc file
(replacing const char* by PCSZ)
modified: storage/connect/ha_connect.cc
modified: storage/connect/ha_connect.h
- Change a parameter type from uchar* to const uchar*
(for ScanRecord and CheckRecord)
modified: storage/connect/ha_connect.cc
modified: storage/connect/ha_connect.h
- Changes on LIKE and NOT LIKE does not fix a bug yet
modified: storage/connect/ha_connect.cc
- Suppress PIVOT_SUPPORT (PIVOT type is unconditionnal)
modified: storage/connect/ha_connect.cc
modified: storage/connect/mycat.cc
- Change the strz function from inline to static
modified: storage/connect/ha_connect.cc
modified: storage/connect/ha_connect.h
- export the JavaConn class and the MgoColumns and IsNum functions
modified: storage/connect/javaconn.h
modified: storage/connect/json.h
modified: storage/connect/mongo.h
- Fix MDEV-13924
modified: storage/connect/jdbconn.cpp
- Make a temporary fix for the compiler bug in CalculateArray
modified: storage/connect/jsonudf.cpp
modified: storage/connect/tabjson.cpp
- Typo
modified: storage/connect/jdbccat.h
modified: storage/connect/reldef.h
modified: storage/connect/tabext.h
modified: storage/connect/tabjmg.cpp
modified: storage/connect/tabxml.h
modified: storage/connect/valblk.h
modified: storage/connect/value.h
modified: storage/connect/xtable.h
- Fix a bug in MONGO tests by changing 'MONGO' to $TYPE
modified: storage/connect/mysql-test/connect/t/mongo_test.inc
- Record test results to reflect all changes
modified: storage/connect/mysql-test/connect/r/json_java_2.result
modified: storage/connect/mysql-test/connect/r/json_java_3.result
modified: storage/connect/mysql-test/connect/r/json_mongo_c.result
modified: storage/connect/mysql-test/connect/r/mongo_c.result
modified: storage/connect/mysql-test/connect/r/mongo_java_2.result
modified: storage/connect/mysql-test/connect/r/mongo_java_3.result
This was a false alarm in a debug check that was introduced in
commit 48192f963a which was a
10.2 code refactoring in preparation for
MDEV-11369 (instant ADD COLUMN) in 10.3.2. The code refactoring
only affected debug builds.
InnoDB B-tree record locks are only supposed to exist on leaf page
records. An assertion failed, because the debug function lock_validate()
was invoking lock_rec_block_validate() on a page for which there were
no locks set in the record lock bitmap. This could happen on a page split.
Especially when the index size grows from a single page to multiple pages,
the root page would transform from a leaf node into an internal node,
and its record lock bitmap would be emptied.
lock_validate(): Skip empty lock bitmaps.
Problem was that we could take page latches on different
order than wat is entitled with SX-lock. To follow the
latching order defined in WL#6326, acquire index->lock X-latch.
This entitles us to acquire page latches in any order for the index.
btr0btr.cc
Document latch rules before and after MariaDB 10.2.2
sync0rw.cc
Document latch compatibility rules better.
btr_defragment_merge_pages
Fix parameter value.
btr_defragment_thread
Acquire X-lock to dict_index_t::lock before restoring
cursor position and continuing defragmentation.
ha_innobase::optimize
Restore defragment feature.
Testing
Add GIS-index and FT-index to table being defragmented.
Defragmentation is not done to GIS-indexes and FT auxiliary
tables.
in 10.1 innodb was basically ignoring virtual columns. In particular,
information about them was not stored in system tables. To make 10.1
table usable in 10.2 it needs to be rebuilt to have virtual colunm
metadata properly recreated.
See also a followup:
MDEV-14046 Allow ALGORITHM=INPLACE for 10.1 tables that contain virtual columns
The fix 716f97e271
was inadvertently reverted in
commit 2e814d4702
"Merge InnoDB 5.7 from mysql-5.7.9".
Reapply the fix, because the test of the bug would fail
after merging MDEV-13838, which replaced an earlier incorrect
bug fix with a correct one.
Reverted incorrect changes done on MDEV-7367 and MDEV-9469. Fixes properly
also related bugs:
MDEV-13668: InnoDB unnecessarily rebuilds table when renaming a column and adding index
MDEV-9469: 'Incorrect key file' on ALTER TABLE
MDEV-9548: Alter table (renaming and adding index) fails with "Incorrect key file for table"
MDEV-10535: ALTER TABLE causes standalone/wsrep cluster crash
MDEV-13640: ALTER TABLE CHANGE and ADD INDEX on auto_increment column fails with "Incorrect key file for table..."
Root cause for all these bugs is the fact that MariaDB .frm file
can contain virtual columns but InnoDB dictionary does not and
previous fixes were incorrect or unnecessarily forced table
rebuilt. In index creation key_part->fieldnr can be bigger than
number of columns in InnoDB data dictionary. We need to skip not
stored fields when calculating correct column number for InnoDB
data dictionary.
dict_table_get_col_name_for_mysql
Remove
innobase_match_index_columns
Revert incorrect change done on MDEV-7367
innobase_need_rebuild
Remove unnecessary rebuild force when column is renamed.
innobase_create_index_field_def
Calculate InnoDB column number correctly and remove
unnecessary column name set.
innobase_create_index_def, innobase_create_key_defs
Remove unneeded fields parameter. Revert unneeded memset.
prepare_inplace_alter_table_dict
Remove unneeded col_names parameter
index_field_t
Remove unneeded col_name member.
row_merge_create_index
Remove unneeded col_names parameter and resolution.
Effected tests:
innodb-alter-table : Add test case for MDEV-13668
innodb-alter : Remove MDEV-13668, MDEV-9469 FIXMEs
and restore original tests
innodb-wl5980-alter : Remove MDEV-13668, MDEV-9469 FIXMEs
and restore original tests
