SECONDARY INDEX IN INNODB
The patches for Bug#11751388 and Bug#11784056 enabled concurrent
reads while creating secondary indexes in InnoDB. However, they
introduced a regression. This regression occured if ALTER TABLE
failed after the index had been added, for example during the
lock upgrade needed to update .FRM. If this happened, InnoDB
and the server got out of sync with regards to which indexes
actually existed. Therefore the patch for Bug#11815600 again
disabled concurrent reads.
This patch re-enables concurrent reads. The original regression
is fixed by splitting the ADD INDEX operation into two parts.
First the new index is created but not made active. This is
done while concurrent reads are allowed. The second part of
the operation makes the index active (or reverts the change).
This is done after lock upgrade, which prevents the original
regression.
In order to implement this change, the patch changes the storage
API for in-place index creation. handler::add_index() is split
into two functions, handler_add_index() and
handler::final_add_index(). The former for creating indexes without
making them visible and the latter for commiting (i.e. making
visible) new indexes or reverting the changes.
Large parts of this patch were written by Marko Mäkelä.
Test case added to innodb_mysql_lock.test.
Manual merge from mysql-5.1 into mysql-5.5.
Conflicts
=========
Text conflict in mysql-test/suite/rpl/t/rpl_row_until.test
Text conflict in sql/handler.h
Text conflict in storage/archive/ha_archive.cc
Manual merge from mysql-5.1 into mysql-5.5.
Conflicts
=========
Text conflict in mysql-test/suite/rpl/t/rpl_row_until.test
Text conflict in sql/handler.h
Text conflict in storage/archive/ha_archive.cc
This makes it possible to do safe multi volume snapshots as long as one snapshots the volume with the transaction logs last.
include/mysql_com.h:
Added REFRESH_CHECKPOINT
mysql-test/r/flush.result:
Added test of new FLUSH TABLES syntax + calls to checkpoint_status handler calls
mysql-test/t/flush.test:
Added test of new FLUSH TABLES syntax + calls to checkpoint_status handler calls
sql/handler.cc:
Added code to call checkpoint_state for all handlertons that supports it
sql/handler.h:
Added new checkpoint_state() handlerton call to temporarly disable checkpoints.
sql/lex.h:
Added CHECKPOINT keyword
sql/sql_yacc.yy:
Added support for FLUSH TABLES WITH READ LOCK AND DISABLE CHECKPOINT
storage/maria/ha_maria.cc:
Added handlerton call to disable checkpoints.
storage/maria/ma_checkpoint.c:
Don't do checkpoint if checkpoints are disabled.
storage/maria/ma_static.c:
Added maria_checkpoint_disabled
storage/maria/maria_def.h:
Added maria_checkpoint_disabled
storage/xtradb/handler/ha_innodb.cc:
Added handlerton call to disable checkpoints.
storage/xtradb/include/log0log.h:
Added option to log_checkpoint() to allow one to ignore not critical checkpoints during the time checkpoints are disabled.
storage/xtradb/log/log0log.c:
Added code to allow one to disable checkpoints during a FLUSH TABLES ... DISABLE CHECKPOINT
This was done by adding a new argument to log_checkpoint() which tells us when the checkpoint is called by srv_master_thread (which are safe to ignore)
storage/xtradb/srv/srv0srv.c:
Tell log_checkpoint() that checkpoints from srv_master_thread() are safe to ignore (will just delay recovery time a bit).
- "Using MRR" is no longer shown with range access.
- Instead, both range and BKA accesses will show one of the following:
= "Rowid-ordered scan"
= "Key-ordered scan"
= "Key-ordered Rowid-ordered scan"
depending on whether DS-MRR implementation will do scan keys in order, rowids in order,
or both.
- The patch also introduces a way for other storage engines/MRR implementations to
pass information to EXPLAIN output about the properties of employed MRR scans.
The LGPL license is used in some legacy code, and to
adhere to current licensing polity, we remove those
files that are no longer used, and reorganize the
remaining LGPL code so it will be GPL licensed from
now on.
Note: This patch only removed LGPL licensed files
in MySQL 5.5 and later, and is the third of a
set of patches to remove LGPL from all trees.
(See Bug# 11840513 for details)
The LGPL license is used in some legacy code, and to
adhere to current licensing polity, we remove those
files that are no longer used, and reorganize the
remaining LGPL code so it will be GPL licensed from
now on.
Note: This patch only removed LGPL licensed files
in MySQL 5.5 and later, and is the third of a
set of patches to remove LGPL from all trees.
(See Bug# 11840513 for details)
The LGPL license is used in some legacy code, and to
adhere to current licensing polity, we remove those
files that are no longer used, and reorganize the
remaining LGPL code so it will be GPL licensed from
now on.
Note: This patch only removed LGPL licensed files
in MySQL 5.1, and is the second of a set of
patches to remove LGPL from all trees.
(See Bug# 11840513 for details)
The LGPL license is used in some legacy code, and to
adhere to current licensing polity, we remove those
files that are no longer used, and reorganize the
remaining LGPL code so it will be GPL licensed from
now on.
Note: This patch only removed LGPL licensed files
in MySQL 5.1, and is the second of a set of
patches to remove LGPL from all trees.
(See Bug# 11840513 for details)
When executing row-ordered-retrieval index merge,
the handler was cloned, but it used the wrong
memory root, so instead of allocating memory
on the thread/query's mem_root, it used the table's
mem_root, resulting in non released memory in the
table object, and was not freed until the table was
closed.
Solution was to ensure that memory used during cloning
of a handler was allocated from the correct memory root.
This was implemented by fixing handler::clone() to also
take a name argument, so it can be used with partitioning.
And in ha_partition only allocate the ha_partition's ref, and
call the original ha_partition partitions clone() and set at cloned
partitions.
Fix of .bzrignore on Windows with VS 2010
When executing row-ordered-retrieval index merge,
the handler was cloned, but it used the wrong
memory root, so instead of allocating memory
on the thread/query's mem_root, it used the table's
mem_root, resulting in non released memory in the
table object, and was not freed until the table was
closed.
Solution was to ensure that memory used during cloning
of a handler was allocated from the correct memory root.
This was implemented by fixing handler::clone() to also
take a name argument, so it can be used with partitioning.
And in ha_partition only allocate the ha_partition's ref, and
call the original ha_partition partitions clone() and set at cloned
partitions.
Fix of .bzrignore on Windows with VS 2010
MAP 'REPAIR TABLE' TO RECREATE +ANALYZE FOR ENGINES NOT
SUPPORTING NATIVE REPAIR
Executing 'mysqlcheck --check-upgrade --auto-repair ...' will first issue
'CHECK TABLE FOR UPGRADE' for all tables in the database in order to check if the
tables are compatible with the current version of MySQL. Any tables that are
found incompatible are then upgraded using 'REPAIR TABLE'.
The problem was that some engines (e.g. InnoDB) do not support 'REPAIR TABLE'.
This caused any such tables to be left incompatible. As a result such tables were
not properly fixed by the mysql_upgrade tool.
This patch fixes the problem by first changing 'CHECK TABLE FOR UPGRADE' to return
a different error message if the engine does not support REPAIR. Instead of
"Table upgrade required. Please do "REPAIR TABLE ..." it will report
"Table rebuild required. Please do "ALTER TABLE ... FORCE ..."
Second, the patch changes mysqlcheck to do 'ALTER TABLE ... FORCE' instead of
'REPAIR TABLE' in these cases.
This patch also fixes 'ALTER TABLE ... FORCE' to actually rebuild the table.
This change should be reflected in the documentation. Before this patch,
'ALTER TABLE ... FORCE' was unused (See Bug#11746162)
Test case added to mysqlcheck.test
client/mysqlcheck.c:
Changed mysqlcheck to do 'ALTER TABLE ... FORCE' if
'CHECK TABLE FOR UPGRADE' reports ER_TABLE_NEEDS_REBUILD
and not ER_TABLE_NEEDS_UPGRADE.
mysql-test/r/mysqlcheck.result:
Added regression test.
mysql-test/std_data/bug47205.frm:
InnoDB 5.0 FRM which contains a varchar primary key using
utf8_general_ci. This is an incompatible FRM for 5.5.
mysql-test/t/mysqlcheck.test:
Added regression test.
sql/handler.h:
Added new HA_CAN_REPAIR flag.
sql/share/errmsg-utf8.txt:
Added new error message ER_TABLE_NEEDS_REBUILD
sql/sql_admin.cc:
Changed 'CHECK TABLE FOR UPDATE' to give ER_TABLE_NEEDS_REBUILD
instead of ER_TABLE_NEEDS_UPGRADE if the engine does not support
REPAIR (as indicated by the new HA_CAN_REPAIR flag).
sql/sql_lex.h:
Remove unused ALTER_FORCE flag.
sql/sql_yacc.yy:
Make sure ALTER TABLE ... FORCE recreates the table
by setting the ALTER_RECREATE flag as the ALTER_FORCE
flag was unused.
storage/archive/ha_archive.h:
Added new HA_CAN_REPAIR flag to Archive
storage/csv/ha_tina.h:
Added new HA_CAN_REPAIR flag to CSV
storage/federated/ha_federated.h:
Added new HA_CAN_REPAIR flag to Federated
storage/myisam/ha_myisam.cc:
Added new HA_CAN_REPAIR flag to MyISAM
MAP 'REPAIR TABLE' TO RECREATE +ANALYZE FOR ENGINES NOT
SUPPORTING NATIVE REPAIR
Executing 'mysqlcheck --check-upgrade --auto-repair ...' will first issue
'CHECK TABLE FOR UPGRADE' for all tables in the database in order to check if the
tables are compatible with the current version of MySQL. Any tables that are
found incompatible are then upgraded using 'REPAIR TABLE'.
The problem was that some engines (e.g. InnoDB) do not support 'REPAIR TABLE'.
This caused any such tables to be left incompatible. As a result such tables were
not properly fixed by the mysql_upgrade tool.
This patch fixes the problem by first changing 'CHECK TABLE FOR UPGRADE' to return
a different error message if the engine does not support REPAIR. Instead of
"Table upgrade required. Please do "REPAIR TABLE ..." it will report
"Table rebuild required. Please do "ALTER TABLE ... FORCE ..."
Second, the patch changes mysqlcheck to do 'ALTER TABLE ... FORCE' instead of
'REPAIR TABLE' in these cases.
This patch also fixes 'ALTER TABLE ... FORCE' to actually rebuild the table.
This change should be reflected in the documentation. Before this patch,
'ALTER TABLE ... FORCE' was unused (See Bug#11746162)
Test case added to mysqlcheck.test
- MWL#47, allowing to annotate row-based binlog events with the SQL test of
the originating query (eg. in mysqlbinlog output).
- row_based_replication_without_primary_key.patch, providing more intelligent
selection of index to use on slave when applying row-based binlog events
for tables with no primary key.
- Make mysqlbinlog omit redundant `use` around BEGIN/SAVEPOINT/COMMIT/
ROLLBACK in 5.0 binlogs.
This will also enable us in the future to collect statistics for
writes to internal tmp tables.
sql/handler.h:
Added ha_write_tmp_row()
sql/opt_subselect.cc:
ha_write_row -> ha_write_tmp_row
sql/sql_class.h:
Added ha_write_tmp_row()
sql/sql_select.cc:
ha_write_row -> ha_write_tmp_row
mysql-test/suite/handler/innodb.result:
Added test case
mysql-test/suite/handler/innodb.test:
Added test case
sql/handler.h:
Move setting/resetting of active_index to ha_index_init()/ha_index_end() to simplify handler functions index_init()/index_end()
Fixed that get_index() returns MAX_KEY if index is not inited (this fixed LP#697610)
storage/federated/ha_federated.cc:
Settting of active_index is not needed anymore
storage/maria/ma_pagecache.c:
Added error message if we have too little memory for Maria page cache
that implement add_index
The problem was that ALTER TABLE blocked reads on an InnoDB table
while adding a secondary index, even if this was not needed. It is
only needed for the final step where the .frm file is updated.
The reason queries were blocked, was that ALTER TABLE upgraded the
metadata lock from MDL_SHARED_NO_WRITE (which blocks writes) to
MDL_EXCLUSIVE (which blocks all accesses) before index creation.
The way the server handles index creation, is that storage engines
publish their capabilities to the server and the server determines
which of the following three ways this can be handled: 1) build a
new version of the table; 2) change the existing table but with
exclusive metadata lock; 3) change the existing table but without
metadata lock upgrade.
For InnoDB and secondary index creation, option 3) should have been
selected. However this failed for two reasons. First, InnoDB did
not publish this capability properly.
Second, the ALTER TABLE code failed to made proper use of the
information supplied by the storage engine. A variable
need_lock_for_indexes was set accordingly, but was not later used.
This patch fixes this problem by only doing metadata lock upgrade
before index creation/deletion if this variable has been set.
This patch also changes some of the related terminology used
in the code. Specifically the use of "fast" and "online" with
respect to ALTER TABLE. "Fast" was used to indicate that an
ALTER TABLE operation could be done without involving a
temporary table. "Fast" has been renamed "in-place" to more
accurately describe the behavior.
"Online" meant that the operation could be done without taking
a table lock. However, in the current implementation writes
are always prohibited during ALTER TABLE and an exclusive
metadata lock is held while updating the .frm, so ALTER TABLE
is not completely online. This patch replaces "online" with
"in-place", with additional comments indicating if concurrent
reads are allowed during index creation/deletion or not.
An important part of this update of terminology is renaming
of the handler flags used by handlers to indicate if index
creation/deletion can be done in-place and if concurrent reads
are allowed. For example, the HA_ONLINE_ADD_INDEX_NO_WRITES
flag has been renamed to HA_INPLACE_ADD_INDEX_NO_READ_WRITE,
while HA_ONLINE_ADD_INDEX is now HA_INPLACE_ADD_INDEX_NO_WRITE.
Note that this is a rename to clarify current behavior, the
flag values have not changed and no flags have been removed or
added.
Test case added to innodb_mysql_sync.test.
that implement add_index
The problem was that ALTER TABLE blocked reads on an InnoDB table
while adding a secondary index, even if this was not needed. It is
only needed for the final step where the .frm file is updated.
The reason queries were blocked, was that ALTER TABLE upgraded the
metadata lock from MDL_SHARED_NO_WRITE (which blocks writes) to
MDL_EXCLUSIVE (which blocks all accesses) before index creation.
The way the server handles index creation, is that storage engines
publish their capabilities to the server and the server determines
which of the following three ways this can be handled: 1) build a
new version of the table; 2) change the existing table but with
exclusive metadata lock; 3) change the existing table but without
metadata lock upgrade.
For InnoDB and secondary index creation, option 3) should have been
selected. However this failed for two reasons. First, InnoDB did
not publish this capability properly.
Second, the ALTER TABLE code failed to made proper use of the
information supplied by the storage engine. A variable
need_lock_for_indexes was set accordingly, but was not later used.
This patch fixes this problem by only doing metadata lock upgrade
before index creation/deletion if this variable has been set.
This patch also changes some of the related terminology used
in the code. Specifically the use of "fast" and "online" with
respect to ALTER TABLE. "Fast" was used to indicate that an
ALTER TABLE operation could be done without involving a
temporary table. "Fast" has been renamed "in-place" to more
accurately describe the behavior.
"Online" meant that the operation could be done without taking
a table lock. However, in the current implementation writes
are always prohibited during ALTER TABLE and an exclusive
metadata lock is held while updating the .frm, so ALTER TABLE
is not completely online. This patch replaces "online" with
"in-place", with additional comments indicating if concurrent
reads are allowed during index creation/deletion or not.
An important part of this update of terminology is renaming
of the handler flags used by handlers to indicate if index
creation/deletion can be done in-place and if concurrent reads
are allowed. For example, the HA_ONLINE_ADD_INDEX_NO_WRITES
flag has been renamed to HA_INPLACE_ADD_INDEX_NO_READ_WRITE,
while HA_ONLINE_ADD_INDEX is now HA_INPLACE_ADD_INDEX_NO_WRITE.
Note that this is a rename to clarify current behavior, the
flag values have not changed and no flags have been removed or
added.
Test case added to innodb_mysql_sync.test.
- Added test case for Aria
- Tested HANDLER with HEAP (changes to HEAP code will be pushed in 5.3)
- Moved all HANDLER test to suite/handler.
mysql-test/Makefile.am:
Added suite/handler
mysql-test/mysql-test-run.pl:
Added suite/handler
mysql-test/r/lock_multi.result:
Remove test that is already in handler test suite
mysql-test/suite/handler/aria.result:
Test for HANDLER with Aria storage engine
mysql-test/suite/handler/aria.test:
Test for HANDLER with Aria storage engine
mysql-test/suite/handler/handler.inc:
Extended the general handler test
Moved interface testing to 'interface.test'
mysql-test/suite/handler/init.inc:
Common init for handler tests.
mysql-test/suite/handler/innodb.result:
New results
mysql-test/suite/handler/innodb.test:
Update to use new include files
mysql-test/suite/handler/interface.result:
Test of HANDLER interface (not storage engine dependent parts)
mysql-test/suite/handler/interface.test:
Test of HANDLER interface (not storage engine dependent parts)
mysql-test/suite/handler/myisam.result:
New results
mysql-test/suite/handler/myisam.test:
Update to use new include files
mysql-test/t/lock_multi.test:
Remove test that is already in handler test suite
mysys/tree.c:
Added missing handling of read previous (showed up in HEAP testing)
sql/handler.cc:
Don't marka 'HA_ERR_RECORD_CHANGED' as fatal (can be used with HANDLER READ, especially with MEMORY ENGINE)
sql/handler.h:
Added prototype for can_continue_handler_scan()
sql/sql_handler.cc:
Re-initialize search if we switch from key to table search.
Check if handler can continue searching between calls (via can_continue_handler_scan())
Don't write common not fatal errors to log
storage/maria/ma_extra.c:
Don't set index 0 as default. This forces call to ma_check_index() to set up index variables.
storage/maria/ma_ft_boolean_search.c:
Ensure that info->last_key.keyinfo is set
storage/maria/ma_open.c:
Don't set index 0 as default. This forces call to ma_check_index() to set up index variables.
storage/maria/ma_rkey.c:
Trivial optimization
storage/maria/ma_rnext.c:
Added missing code from mi_rnext.c to ensure that handler next/prev works.
storage/maria/ma_rsame.c:
Simple optimizations
storage/maria/ma_search.c:
Initialize info->last_key once and for all when we change keys.
storage/maria/ma_unique.c:
Ensure that info->last_key.keyinfo is up to date.
* move a capability from a virtual handler method to table_flags()
* rephrase error messages to avoid hard-coded English parts
* admit in test cases that they need xtradb, not innodb
mysql-test/suite/vcol/t/rpl_vcol.test:
this test needs xtradb, it will fail with innodb
mysql-test/suite/vcol/t/vcol_blocked_sql_funcs_innodb.test:
this test needs xtradb, it will fail with innodb
mysql-test/suite/vcol/t/vcol_column_def_options_innodb.test:
this test needs xtradb, it will fail with innodb
mysql-test/suite/vcol/t/vcol_handler_innodb.test:
this test needs xtradb, it will fail with innodb
mysql-test/suite/vcol/t/vcol_ins_upd_innodb.test:
this test needs xtradb, it will fail with innodb
mysql-test/suite/vcol/t/vcol_keys_innodb.test:
this test needs xtradb, it will fail with innodb
mysql-test/suite/vcol/t/vcol_non_stored_columns_innodb.test:
this test needs xtradb, it will fail with innodb
mysql-test/suite/vcol/t/vcol_partition_innodb.test:
this test needs xtradb, it will fail with innodb
mysql-test/suite/vcol/t/vcol_select_innodb.test:
this test needs xtradb, it will fail with innodb
mysql-test/suite/vcol/t/vcol_supported_sql_funcs_innodb.test:
this test needs xtradb, it will fail with innodb
mysql-test/suite/vcol/t/vcol_trigger_sp_innodb.test:
this test needs xtradb, it will fail with innodb
mysql-test/suite/vcol/t/vcol_view_innodb.test:
this test needs xtradb, it will fail with innodb
sql/ha_partition.h:
check_if_supported_virtual_columns() -> HA_CAN_VIRTUAL_COLUMNS
sql/handler.h:
check_if_supported_virtual_columns() -> HA_CAN_VIRTUAL_COLUMNS
sql/share/errmsg.txt:
no hard-coded english parts in the error messages (ER_UNSUPPORTED_ACTION_ON_VIRTUAL_COLUMN)
sql/sql_table.cc:
no hard-coded english parts in the error messages
sql/table.cc:
* check_if_supported_virtual_columns() -> HA_CAN_VIRTUAL_COLUMNS
* no "csv workaround" is needed
* no hard-coded english parts in the error messages
storage/maria/ha_maria.cc:
check_if_supported_virtual_columns() -> HA_CAN_VIRTUAL_COLUMNS
storage/maria/ha_maria.h:
check_if_supported_virtual_columns() -> HA_CAN_VIRTUAL_COLUMNS
storage/myisam/ha_myisam.cc:
check_if_supported_virtual_columns() -> HA_CAN_VIRTUAL_COLUMNS
storage/myisam/ha_myisam.h:
check_if_supported_virtual_columns() -> HA_CAN_VIRTUAL_COLUMNS
storage/xtradb/handler/ha_innodb.cc:
check_if_supported_virtual_columns() -> HA_CAN_VIRTUAL_COLUMNS
storage/xtradb/handler/ha_innodb.h:
check_if_supported_virtual_columns() -> HA_CAN_VIRTUAL_COLUMNS
- Fixed problem with oqgraph and 'make dist'
Note that after this merge we have a problem show in join_outer where we examine too many rows in one specific case (related to BUG#57024).
This will be fixed when mwl#128 is merged into 5.3.
