When MariaDB 10.1.0 introduced table options for encryption and
compression, it unnecessarily changed
ha_innobase::check_if_supported_inplace_alter() so that ALGORITHM=COPY
is forced when these parameters differ.
A better solution is to move the check to innobase_need_rebuild().
In that way, the ALGORITHM=INPLACE interface (yes, the syntax is
very misleading) can be used for rebuilding the table much more
efficiently, with merge sort, with no undo logging, and allowing
concurrent DML operations.
Some test are still disabled as they provide different results,
fail on different error or crash.
Fixes also MDEV-14060 InnoDB: Foreign key constraint to
geometry type does not work where test intentionally tried
incorrect cases.
InnoDB was writing unnecessary information to the
update undo log records. Most notably, if an indexed column is updated,
the old value of the column would be logged twice: first as part of
the update vector, and then another time because it is an indexed column.
Because the InnoDB undo log record must fit in a single page,
this would cause unnecessary failure of certain updates.
Even after this fix, InnoDB still seems to be unnecessarily logging
indexed column values for non-updated columns. It seems that non-updated
secondary index columns only need to be logged when a PRIMARY KEY
column is updated. To reduce risk, we are not fixing this remaining flaw
in GA versions.
trx_undo_page_report_modify(): Log updated indexed columns only once.
remove remnants of 10.0 bugfix, incorrectly merged into 10.2
Using col_names[i] was obviously, wrong, must've been col_names[ifield->col_no].
incorrect column name resulted in innodb having index unique_id2(id1),
while the server thought it's unique_id2(id4).
But col_names[ifield->col_no] is wrong too, because `table` has non-renamed
columns, so the correct column name is always dict_table_get_col_name(table, ifield->col_no)
Issue
=====
The original issue was that the size of a fil_per_table tablespace was calculated
incorrectly during truncate in the presence of an fts index. This incorrect calculation
was fixed as part of BUG#25053705 along with a testcase to reproduce the bug. The
assert that was added as part of it to reproduce the bug was wrong and resulted in
this bug.
Fix
===
Although the assert was removed earlier in a seperate commit as it was blocking the
ntest, this patch replaces the other parts of the code that were added to reproduce
the bug and replaces it with code that tries to reproduce the bug in a different way.
The new code basically tries to tweak conditions so as to simulate the random read
where a page that doesn't exist is tried to be read.
RB: 15890
Reviewed-by: Jimmy Yang <Jimmy.Yang@oracle.com>
Reviewed-by: Satya Bodapati <satya.bodapati@oracle.com>
When MySQL 5.6.10 introduced innodb_read_only mode, it skipped the
creation of the InnoDB buffer pool dump/restore subsystem in that mode.
Attempts to set the variable innodb_buf_pool_dump_now would have
no effect in innodb_read_only mode, but the corresponding condition
was forgotten in from the other two update functions.
MySQL 5.7.20 would fix the innodb_buffer_pool_load_now,
but not innodb_buffer_pool_load_abort. Let us fix both in MariaDB.
Partition wasn't setting HA_OPTION_PACK_RECORD on ALTER TABLE
if the row format was PAGE.
(so one bit in the null bitmap was reserved for a deleted bit -
see make_empty_rec - and all actual null bits were one off)
fts_get_next_doc_id(): Assign the first and subsequent FTS_DOC_ID
in the same way: by post-incrementing the cached value.
If there is a user-specified FTS_DOC_ID, do not touch the internal
sequence.
There are two bugs related to failed ADD INDEX and
the InnoDB table cache eviction.
dict_table_close(): Try dropping failed ADD INDEX when releasing
the last table handle, not when releasing the last-but-one.
dict_table_remove_from_cache_low(): Do not invoke
row_merge_drop_indexes() after freeing all index metadata.
Instead, directly invoke row_merge_drop_indexes_dict() to
remove the metadata from the persistent data dictionary
and to free the index pages.
Problem was that we could take page latches on different
order than wat is entitled with SX-lock. To follow the
latching order defined in WL#6326, acquire index->lock X-latch.
This entitles us to acquire page latches in any order for the index.
btr0btr.cc
Document latch rules before and after MariaDB 10.2.2
sync0rw.cc
Document latch compatibility rules better.
btr_defragment_merge_pages
Fix parameter value.
btr_defragment_thread
Acquire X-lock to dict_index_t::lock before restoring
cursor position and continuing defragmentation.
ha_innobase::optimize
Restore defragment feature.
Testing
Add GIS-index and FT-index to table being defragmented.
Defragmentation is not done to GIS-indexes and FT auxiliary
tables.
During show create procedure we ommited to check the current role, if it
is the actual definer of the procedure. In addition, we should support
indirectly granted roles to the current role. Implemented a recursive
lookup to search the tree of grants if the rolename is present.
SQL Standard 2016, Part 5 Section 53 View I_S.ROUTINES selects
ROUTINE_BODY and its WHERE clause says that the GRANTEE must be
either PUBLIC, or CURRENT_USER or in the ENABLED_ROLES.
Reverted incorrect changes done on MDEV-7367 and MDEV-9469. Fixes properly
also related bugs:
MDEV-13668: InnoDB unnecessarily rebuilds table when renaming a column and adding index
MDEV-9469: 'Incorrect key file' on ALTER TABLE
MDEV-9548: Alter table (renaming and adding index) fails with "Incorrect key file for table"
MDEV-10535: ALTER TABLE causes standalone/wsrep cluster crash
MDEV-13640: ALTER TABLE CHANGE and ADD INDEX on auto_increment column fails with "Incorrect key file for table..."
Root cause for all these bugs is the fact that MariaDB .frm file
can contain virtual columns but InnoDB dictionary does not and
previous fixes were incorrect or unnecessarily forced table
rebuilt. In index creation key_part->fieldnr can be bigger than
number of columns in InnoDB data dictionary. We need to skip not
stored fields when calculating correct column number for InnoDB
data dictionary.
dict_table_get_col_name_for_mysql
Remove
innobase_match_index_columns
Revert incorrect change done on MDEV-7367
innobase_need_rebuild
Remove unnecessary rebuild force when column is renamed.
innobase_create_index_field_def
Calculate InnoDB column number correctly and remove
unnecessary column name set.
innobase_create_index_def, innobase_create_key_defs
Remove unneeded fields parameter. Revert unneeded memset.
prepare_inplace_alter_table_dict
Remove unneeded col_names parameter
index_field_t
Remove unneeded col_name member.
row_merge_create_index
Remove unneeded col_names parameter and resolution.
Effected tests:
innodb-alter-table : Add test case for MDEV-13668
innodb-alter : Remove MDEV-13668, MDEV-9469 FIXMEs
and restore original tests
innodb-wl5980-alter : Remove MDEV-13668, MDEV-9469 FIXMEs
and restore original tests
Mariabackup 10.2.7 would delete the redo log files after a successful
--prepare operation. If the user is manually copying the prepared files
instead of using the --copy-back option, it could happen that some old
redo log file would be preserved in the restored location. These old
redo log files could cause corruption of the restored data files when
the server is started up.
We prevent this scenario by creating a "poisoned" redo log file
ib_logfile0 at the end of the --prepare step. The poisoning consists
of simply truncating the file to an empty file. InnoDB will refuse
to start up on an empty redo log file.
copy_back(): Delete all redo log files in the target if the source
file ib_logfile0 is empty. (Previously we did this if the source
file is missing.)
SRV_OPERATION_RESTORE_EXPORT: A new variant of SRV_OPERATION_RESTORE
when the --export option is specified. In this mode, we will keep
deleting all redo log files, instead of truncating the first one.
delete_log_files(): Add a parameter for the first file to delete,
to be passed as 0 or 1.
innobase_start_or_create_for_mysql(): In mariabackup --prepare,
tolerate an empty ib_logfile0 file. Otherwise, require the first
redo log file to be longer than 4 blocks (2048 bytes). Unless
--export was specified, truncate the first log file at the
end of --prepare.
fil_space_extend_must_retry(): If the table is being truncated,
do not call fil_flush_low(). The operation is covered by the
truncate log. File extension during TRUNCATE only occurs
if there are many indexes on the table. With smaller innodb_page_size,
the file extension occurs already with fewer indexes on the table.
Problem:- This crash happens because of thd = NULL , and while checking
for wsrep_on , we no longer check for thd != NULL (MDEV-7955). So this
problem is regression of MDEV-7955. However this patch not only solves
this regression , It solves all regression caused by MDEV-7955 patch.
To get all possible cases when thd can be null , assert(thd)/
assert(trx->mysql_thd) is place just before all wsrep_on and innodb test
suite is run. And the assert which caused failure are removed with a physical
check for thd != NULL. Rest assert are removed. Hopefully this method will
remove all current/potential regression of MDEV-7955.
Ensure that no adaptive hash index exists for any system tables,
so that the blocked TRUNCATE TABLE t1 will not block the concurrent
TRUNCATE TABLE t2.
This bug is a regression caused by the code refactoring in
commit f5a833c3e0. It was not present
in any release of the MariaDB server. The bug affects table-rebuilding
ALTER TABLE when the source table is in ROW_FORMAT=REDUNDANT and
contains no virtual columns.
row_log_table_low_redundant(): Log virtual column data only if
virtual columns are present.
Implement a special Copy_field method for timestamps, that copies
timestamps without converting them to MYSQL_TIME (the conversion
is lossy around DST change dates).
MDEV-13851 Always check table options in ALTER TABLE…ALGORITHM=INPLACE
In the merge of MySQL 5.7.9 to MariaDB 10.2.2, some code was included
that prevents ADD SPATIAL INDEX from being executed with ALGORITHM=INPLACE.
Also, the constant ADD_SPATIAL_INDEX was introduced as an alias
to ADD_INDEX. We will remove that alias now, and properly implement
the same ADD SPATIAL INDEX restrictions as MySQL 5.7 does:
1. table-rebuilding operations are not allowed if SPATIAL INDEX survive it
2. ALTER TABLE…ADD SPATIAL INDEX…LOCK=NONE is not allowed
ha_innobase::prepare_inplace_alter_table(): If the ALTER TABLE
requires actions within InnoDB, enforce the table options (MDEV-13851).
In this way, we will keep denying ADD SPATIAL INDEX for tables
that use encryption (MDEV-11974), even if ALGORITHM=INPLACE is used.
Another fix (work around MDEV-12699): Ensure that the 1234-byte
truncated page is all zero, so that after data file extension
pads the page with zeroes to full page size, the page will read
as a valid one (consisting of zero bytes only).
InnoDB was too eager to forget the open table (m_mysql_table=NULL)
and that caused it to try to open a table which was opened by the user
not FK-prelocked. The server didn't expect that.
After fixing this, it crashed in gcol.innodb_virtual_fk test, trying to
compute virtual columns for a table that didn't have them. Because
row_upd_store_row() was deleting a row from node->table, while computing
virtual columns in thr->prebuilt->m_mysql_table. Which wasn't necessarily
the same table, and might've not even had virtual columns, even if
node->table did.
multi-update first runs a select to find affected rows, then performs
a separate update step. On the second step WITH CHECK OPTION rows
are read with rnd_read, but the first step might've been done with
keyread.
keyread over indexed virtual columns only reads the column's value, not
dependent base columns. This is reflected in the read_set too. But on
the rnd_read step base columns must be read - thus we need to update the
read_set before doing updates.
* created tests focusing in multi-master conflicts during cascading foreign key
processing
* in row0upd.cc, calling wsrep_row_ups_check_foreign_constraints only when
running in cluster
* in row0ins.cc fixed regression from MW-369, which caused crash with MW-402.test
The purpose of the test is to ensure that redo log apply will
extend data files before applying page-level redo log records.
The test intermittently failed, because the doublewrite buffer
would sometimes contain data for the pages that the test
truncated. When the test truncates data files, it must also remove
the doublewrite buffer entries, because under normal operation, the
doublewrite buffer would only be written to if the data file already
has been extended.
MySQL 5.7 introduced some optimizations to avoid file I/O during
ALGORITHM=INPLACE operations. While both innodb-index-online and
innodb-table-online will exercise both the merge sort files and
the online log files in 10.1, in 10.2 they would only exercise the
online log files.
Modify one test case in innodb.innodb-table-online so that
skip_pk_sort will not hold. In this way, this test case will
write and read the merge sort files. The other instrumented tests
in innodb-index-online and innodb-table-online will only write
and read online_log files.
This should also fix the MariaDB 10.2.2 bug
MDEV-13826 CREATE FULLTEXT INDEX on encrypted table fails.
MDEV-12634 FIXME: Modify innodb-index-online, innodb-table-online
so that they will write and read merge sort files. InnoDB 5.7
introduced some optimizations to avoid using the files for small tables.
Many collation test results have been adjusted for MDEV-10191.
Introduce innodb_encrypt_log.combinations and prove that
the encryption and decryption take place during both
online ADD INDEX (WL#5266) and online table-rebuilding ALTER (WL#6625).
Skip rollback and other redo-log-generating tasks if
srv_apply_log_only is set.
Instead of assigning the debug variable recv_no_log_write = FALSE,
assign it to srv_apply_log_only, so that any unwanted writes are caught.
Import the changes to innodb.innodb-index innodb.innodb-index-debug
Note: As noted in MDEV-13613, due to the behaviour change in MDEV-11114,
DROP COLUMN will not imply DROP/ADD PRIMARY/UNIQUE KEY, like it does
in MySQL. The tests have been adjusted accordingly.
…porary file
Fixed by removing writing key version to start of every block that
was encrypted. Instead we will use single key version from log_sys
crypt info.
After this MDEV also blocks writen to row log are encrypted and blocks
read from row log aren decrypted if encryption is configured for the
table.
innodb_status_variables[], struct srv_stats_t
Added status variables for merge block and row log block
encryption and decryption amounts.
Removed ROW_MERGE_RESERVE_SIZE define.
row_merge_fts_doc_tokenize
Remove ROW_MERGE_RESERVE_SIZE
row_log_t
Add index, crypt_tail, crypt_head to be used in case of
encryption.
row_log_online_op, row_log_table_close_func
Before writing a block encrypt it if encryption is enabled
row_log_table_apply_ops, row_log_apply_ops
After reading a block decrypt it if encryption is enabled
row_log_allocate
Allocate temporary buffers crypt_head and crypt_tail
if needed.
row_log_free
Free temporary buffers crypt_head and crypt_tail if they
exist.
row_merge_encrypt_buf, row_merge_decrypt_buf
Removed.
row_merge_buf_create, row_merge_buf_write
Remove ROW_MERGE_RESERVE_SIZE
row_merge_build_indexes
Allocate temporary buffer used in decryption and encryption
if needed.
log_tmp_blocks_crypt, log_tmp_block_encrypt, log_temp_block_decrypt
New functions used in block encryption and decryption
log_tmp_is_encrypted
New function to check is encryption enabled.
Added test case innodb-rowlog to force creating a row log and
verify that operations are done using introduced status
variables.
The InnoDB purge subsystem can be best stopped by opening a read view,
for example by START TRANSACTION WITH CONSISTENT SNAPSHOT.
To ensure that everything is purged, use wait_all_purged.inc,
which waits for the History list length in SHOW ENGINE INNODB STATUS
to reach 0. Setting innodb_purge_run_now never guaranteed this.
Implement lock-ddl-per-table option that locks tables before it
is copied to backup, and helds the lock until backup finished
The "DDL-lock" itself is implemented as "SELECT * from <table> LIMIT 0",
inside a transaction, and "COMMIT" of this transaction is the DDL-unlock.
The redo log is divided into 512-byte blocks. Sometimes, some of
the strings end up at a block boundary and will not be found.
Look for only one match.
full server recovery is performed .
We start "mysqld" with --bootstrap
and pass bootstrap script consisting of several FLUSH TABLES FOR export/
UNLOCK TABLES
