MySQL Bug#71095: Wrong results with PARTITION BY LIST COLUMNS()
MySQL Bug#72803: Wrong "Impossible where" with LIST partitioning
MDEV-6240: Wrong "Impossible where" with LIST partitioning
- Backprot the fix from MySQL Bug#71095.
(This is attempt at fix#2) (re-commit with fixed typo)
- Moved the testcase from partition_test to partition_innodb.test where it can really work.
- Made ordered index scans over ha_partition tables to satisfy ROR property for
the case where underlying table uses extended keys.
- Backport MySQL's fix: do set ha_partition::m_pkey_is_clustered for ha_partition
objects created with handler->clone() call.
- Also, include a testcase.
includes:
* remove some remnants of "Bug#14521864: MYSQL 5.1 TO 5.5 BUGS PARTITIONING"
* introduce LOCK_share, now LOCK_ha_data is strictly for engines
* rea_create_table() always creates .par file (even in "frm-only" mode)
* fix a 5.6 bug, temp file leak on dummy ALTER TABLE
from MariaDB 10.0.
The bug in mdev-3948 was an instance of the problem fixed by Sergey's patch
in 10.0 - namely that the range optimizer could change table->[read | write]_set,
and not restore it.
revno: 3471
committer: Sergey Petrunya <psergey@askmonty.org>
branch nick: 10.0-serg-fix-imerge
timestamp: Sat 2012-11-03 12:24:36 +0400
message:
# MDEV-3817: Wrong result with index_merge+index_merge_intersection, InnoDB table, join, AND and OR conditions
Reconcile the fixes from:
#
# guilhem.bichot@oracle.com-20110805143029-ywrzuz15uzgontr0
# Fix for BUG#12698916 - "JOIN QUERY GIVES WRONG RESULT AT 2ND EXEC. OR
# AFTER FLUSH TABLES [-INT VS NULL]"
#
# guilhem.bichot@oracle.com-20111209150650-tzx3ldzxe1yfwji6
# Fix for BUG#12912171 - ASSERTION FAILED: QUICK->HEAD->READ_SET == SAVE_READ_SET
# and
#
and related fixes from: BUG#1006164, MDEV-376:
Now, ROR-merged QUICK_RANGE_SELECT objects make no assumptions about the values
of table->read_set and table->write_set.
Each QUICK_ROR_SELECT has (and had before) its own column bitmap, but now, all
QUICK_ROR_SELECT's functions that care: reset(), init_ror_merged_scan(), and
get_next() will set table->read_set when invoked and restore it back to what
it was before the call before they return.
This allows to avoid the mess when somebody else modifies table->read_set for
some reason.
Reconcile the fixes from:
#
# guilhem.bichot@oracle.com-20110805143029-ywrzuz15uzgontr0
# Fix for BUG#12698916 - "JOIN QUERY GIVES WRONG RESULT AT 2ND EXEC. OR
# AFTER FLUSH TABLES [-INT VS NULL]"
#
# guilhem.bichot@oracle.com-20111209150650-tzx3ldzxe1yfwji6
# Fix for BUG#12912171 - ASSERTION FAILED: QUICK->HEAD->READ_SET == SAVE_READ_SET
# and
#
and related fixes from: BUG#1006164, MDEV-376:
Now, ROR-merged QUICK_RANGE_SELECT objects make no assumptions about the values
of table->read_set and table->write_set.
Each QUICK_ROR_SELECT has (and had before) its own column bitmap, but now, all
QUICK_ROR_SELECT's functions that care: reset(), init_ror_merged_scan(), and
get_next() will set table->read_set when invoked and restore it back to what
it was before the call before they return.
This allows to avoid the mess when somebody else modifies table->read_set for
some reason.
The main problem was a bug in CSV where it provided wrong statistics (it claimed the table was empty when it wasn't)
I also fixed wrong freeing of blob's in the CSV handler. (Any call to handler::read_first_row() on a CSV table with blobs would fail)
mysql-test/r/csv.result:
Added new test case
mysql-test/r/partition_innodb.result:
Updated test results after fixing bug with impossible partitions and const tables
mysql-test/t/csv.test:
Added new test case
sql/sql_select.cc:
Cleaned up code for handling of partitions.
Fixed also a bug where we didn't threat a table with impossible partitions as a const table.
storage/csv/ha_tina.cc:
Allocate blobroot onces.
PARTITION STATISTICS
Problem was the fix for bug#11756867; It always used the first
partitions, and stopped after it checked 10 [sub]partitions.
(or until it found a partition which would contain a match).
This results in bad statistics for tables where the first 10 partitions
don't represent the majority of the data (like when the first 10
partitions only contained a few rows in total).
The solution was to take statisics from the partitions containing
the most rows instead:
Added an array of partition ids which is sorted by number of records
in descending order.
this array is used in records_in_range to cover as many records as
possible in as few calls as possible.
Also changed the limit of how many partitions to use for the statistics
from a static max of 10 partitions, into a dynamic model:
Maximum number of partitions is now log2(total number of partitions)
taken from the ordered array.
It will continue calling partitions records_in_range until it has
checked:
(total rows in matching partitions) * (maximum number of partitions)
/ (number of used partitions)
Also reverted the changes for ha_partition::scan_time() and
ha_partition::estimate_rows_upper_bound() to before
the fix of bug#11756867. Since they are not as slow as
records_in_range.
leave the table unusable".
Failing ALTER statement on partitioned table could have left
this table in an unusable state. This has happened in cases
when ALTER was executed using "fast" algorithm, which doesn't
involve copying of data between old and new versions of table,
and the resulting new table was incompatible with partitioning
function in some way.
The problem stems from the fact that discrepancies between new
table definition and partitioning function are discovered only
when the table is opened. In case of "fast" algorithm this has
happened too late during ALTER's execution, at the moment when
all changes were already done and couldn't have been reverted.
In the cases when "slow" algorithm, which copies data, is used
such discrepancies are detected at the moment new table
definition is opened implicitly when new version of table is
created in storage engine. As result ALTER is aborted before
any changes to table were done.
This fix tries to address this issue by ensuring that "fast"
algorithm behaves similarly to "slow" algorithm and checks
compatibility between new definition and partitioning function
by trying to open new definition after .FRM file for it has
been created.
Long term we probably should implement some way to check
compatibility between partitioning function and new table
definition which won't involve opening it, as this should
allow much cleaner fix for this problem.
mysql-test/r/partition_innodb.result:
Added test for bug #57985 "ONLINE/FAST ALTER PARTITION can
fail and leave the table unusable".
mysql-test/t/partition_innodb.test:
Added test for bug #57985 "ONLINE/FAST ALTER PARTITION can
fail and leave the table unusable".
sql/sql_table.cc:
Ensure that in cases when .FRM for partitioned table is
created without creating table in storage engine (e.g.
during "fast" ALTER TABLE) we still open table definition.
This allows to check that definition of created table/.FRM
is compatible with its partitioning function.
When having a sub query in partitioned innodb one could
make the partitioning engine to search for a 'index_next_same'
on a partition that had not been initialized.
Problem was that the subselect function looks at table->status
which was not set in the partitioning handler when it skipped
scanning due to no matching partitions found.
Fixed by setting table->status = STATUS_NOT_FOUND when
there was no partitions to scan. (If there are partitions to
scan, it will be set in the partitions handler.)
mysql-test/r/partition_innodb.result:
added result
mysql-test/t/partition_innodb.test:
added test
sql/ha_partition.cc:
set table status to not found, if there ar no partitions to scan.
Bug#46754: 'rows' field doesn't reflect partition pruning
Update of test results after fixing the above bugs.
(fix in separate commit).
mysql-test/r/partition.result:
Updated test result after fixing bugs 46754 and 53806
mysql-test/r/partition_hash.result:
Updated test result after fixing bugs 46754 and 53806
mysql-test/r/partition_innodb.result:
Updated test result after fixing bugs 46754 and 53806
mysql-test/r/partition_range.result:
Updated test result after fixing bugs 46754 and 53806
mysql-test/suite/parts/r/partition_alter3_innodb.result:
Updated test result after fixing bugs 46754 and 53806
mysql-test/suite/parts/r/partition_alter3_myisam.result:
Updated test result after fixing bugs 46754 and 53806
The ALTER PARTITION and SELECT seemed to be deadlocked
when having innodb_thread_concurrency = 1.
Problem was that there was unreleased latches
in the ALTER PARTITION thread which was needed
by the SELECT thread to be able to continue.
Solution was to release the latches by commit
before requesting upgrade to exclusive MDL lock.
Updated according to reviewers comments (3).
mysql-test/r/partition_innodb.result:
updated test result
mysql-test/t/partition_innodb.test:
added test
sql/sql_partition.cc:
Moved implicit commit into mysql_change_partition
so that if latches are taken, they are always released
before waiting on exclusive lock.
sql/sql_table.cc:
refactored the code to prepare and commit
around copy_data_between_tables, to be able
to reuse it in mysql_change_partitions
sql/sql_table.h:
exporting mysql_trans_prepare/commit_alter_copy_data
corruption on ADD PARTITION and LOCK TABLE
Bug#53770: Server crash at handler.cc:2076 on
LOAD DATA after timed out COALESCE PARTITION
5.5 fix for:
Bug#51042: REORGANIZE PARTITION can leave table in an
inconsistent state in case of crash
Needs to be back-ported to 5.1
5.5 fix for:
Bug#50418: DROP PARTITION does not interact with
transactions
Main problem was non-persistent operations done
before meta-data lock was taken (53770+53676).
And 53676 needed to keep the table/partitions opened and locked
while copying the data to the new partitions.
Also added thorough tests to spot some additional bugs
in the ddl_log code, which could result in bad state
between the .frm and partitions.
Collapsed patch, includes all fixes required from the reviewers.
mysql-test/r/partition_innodb.result:
updated result with new test
mysql-test/suite/parts/inc/partition_crash.inc:
crash test include file
mysql-test/suite/parts/inc/partition_crash_add.inc:
test all states in fast_alter_partition_table
ADD PARTITION branch
mysql-test/suite/parts/inc/partition_crash_change.inc:
test all states in fast_alter_partition_table
CHANGE PARTITION branch
mysql-test/suite/parts/inc/partition_crash_drop.inc:
test all states in fast_alter_partition_table
DROP PARTITION branch
mysql-test/suite/parts/inc/partition_fail.inc:
recovery test including injecting errors
mysql-test/suite/parts/inc/partition_fail_add.inc:
test all states in fast_alter_partition_table
ADD PARTITION branch
mysql-test/suite/parts/inc/partition_fail_change.inc:
test all states in fast_alter_partition_table
CHANGE PARTITION branch
mysql-test/suite/parts/inc/partition_fail_drop.inc:
test all states in fast_alter_partition_table
DROP PARTITION branch
mysql-test/suite/parts/inc/partition_mgm_crash.inc:
include file that runs all crash and failure injection tests.
mysql-test/suite/parts/r/partition_debug_innodb.result:
new test result file
mysql-test/suite/parts/r/partition_debug_myisam.result:
new test result file
mysql-test/suite/parts/r/partition_special_innodb.result:
updated result
mysql-test/suite/parts/r/partition_special_myisam.result:
updated result
mysql-test/suite/parts/t/partition_debug_innodb-master.opt:
opt file for using with crashing tests of partitioned innodb
mysql-test/suite/parts/t/partition_debug_innodb.test:
partitioned innodb test that require debug builds
mysql-test/suite/parts/t/partition_debug_myisam-master.opt:
opt file for using with crashing tests of partitioned myisam
mysql-test/suite/parts/t/partition_debug_myisam.test:
partitioned myisam test that require debug builds
mysql-test/suite/parts/t/partition_special_innodb-master.opt:
added innodb-file-per-table to easier verify partition status on disk
mysql-test/suite/parts/t/partition_special_innodb.test:
added test case
mysql-test/suite/parts/t/partition_special_myisam.test:
added test case
mysql-test/t/partition_innodb.test:
added test case
sql/sql_base.cc:
Moved alter_close_tables to sql_partition.cc
sql/sql_base.h:
removed some non existing and duplicated functions.
sql/sql_partition.cc:
fast_alter_partition_table:
Spletted abort_and_upgrad_lock_and_close_table
to its parts (wait_while_table_is_used and
alter_close_tables) and always have
wait_while_table_is_used before any persistent
operations (including logs, which will be executed
on failure) and alter_close_tables after
create/read/write operations and before
drop operations.
moved alter_close_tables here from sql_base.cc
Added error injections for better test coverage.
write_log_final_change_partition:
fixed a log_entry linking bug (delete_frm was not
linked to change/drop partition)
and drop partition must be executed before
change partition (change partition can rename a
partition to an old name, like REORG p1 INTO (p1,p2).
write_log_add_change_partition:
need to use drop_frm first, and relinking that entry
and reusing its execute entry.
sql/sql_table.cc:
added initialization of next_active_log_entry.
sql/table.h:
removed a duplicate declaration.
As we check for the impossible partitions earlier, it's possible that we don't find any
suitable partitions at all. So this assertion just has to be corrected for this case.
per-file comments:
mysql-test/r/partition_innodb.result
Bug#55146 Assertion `m_part_spec.start_part == m_part_spec.end_part' in index_read_idx_map
test result updated.
mysql-test/t/partition_innodb.test
Bug#55146 Assertion `m_part_spec.start_part == m_part_spec.end_part' in index_read_idx_map
test case added.
sql/ha_partition.cc
Bug#55146 Assertion `m_part_spec.start_part == m_part_spec.end_part' in index_read_idx_map
Assertion changed to '>=' as the prune_partition_set() in the get_partition_set() can
do start_part= end_part+1 if no possible partitions were found.
value and NO_ZERO_DATE
The problem was that a older version of the error path for a
failed admin statement relied upon a few error conditions being
met in order to access a table handler, the first one being that
the table object pointer was not NULL. Probably due to chance,
in all cases a table object was closed but the reference wasn't
reset, the other conditions didn't evaluate to true. With the
addition of a new check on the error path, the handler started
being dereferenced whenever it was not reset to NULL, causing
problems for code paths which closed the table but didn't reset
the reference.
The solution is to reset the reference whenever a admin statement
fails and the tables are closed.
mysql-test/r/partition_innodb.result:
Add test case result for Bug#54783
mysql-test/t/partition_innodb.test:
Add test case for Bug#54783
sql/sql_table.cc:
In case table recreate failed, set a appropriate result code.
Reset reference to a closed table object, otherwise the error
path might attempt to access it.
The default storage engine is changed from MyISAM to
InnoDB, in all builds except for the embedded server.
In addition, the following system variables are
changed:
* innodb_file_per_table is enabled
* innodb_strict_mode is enabled
* innodb_file_format_name_update is changed
to 'Barracuda'
The test suite is changed so that tests that do not
explicitly include the have_innodb.inc are run with
--default-storage-engine=MyISAM. This is to ease the
transition, so that most regression tests are run
with the same engine as before.
Some tests are disabled for the embedded server
regression test, as the output of certain statements
will be different that for the regular server
(i.e SELECT @@default_storage_engine). This is to
ease transition.
mysql-test/mysql-test-run.pl:
The regression test suite now adds a
--default-storage-engine=MyISAM for all non-innodb
tests. This behaviour can be controlled by the
default-myisam switch in mysql-test-run
mysql-test/t/bootstrap-master.opt:
The bootstrap test can only be run without InnoDB
as it starts several mysqld instances on the same
datadir. This is possible with MyISAM, but not
with InnoDB.
storage/innobase/CMakeLists.txt:
Build InnoDB per default
storage/innobase/handler/ha_innodb.cc:
Change default values for system variables
Enable file_per_table
Enable strict_mode
Upgrade default file format to Barracuda
Conflicts:
Text conflict in client/mysqlbinlog.cc
Text conflict in mysql-test/Makefile.am
Text conflict in mysql-test/collections/default.daily
Text conflict in mysql-test/r/mysqlbinlog_row_innodb.result
Text conflict in mysql-test/suite/rpl/r/rpl_typeconv_innodb.result
Text conflict in mysql-test/suite/rpl/t/rpl_get_master_version_and_clock.test
Text conflict in mysql-test/suite/rpl/t/rpl_row_create_table.test
Text conflict in mysql-test/suite/rpl/t/rpl_slave_skip.test
Text conflict in mysql-test/suite/rpl/t/rpl_typeconv_innodb.test
Text conflict in mysys/charset.c
Text conflict in sql/field.cc
Text conflict in sql/field.h
Text conflict in sql/item.h
Text conflict in sql/item_func.cc
Text conflict in sql/log.cc
Text conflict in sql/log_event.cc
Text conflict in sql/log_event_old.cc
Text conflict in sql/mysqld.cc
Text conflict in sql/rpl_utility.cc
Text conflict in sql/rpl_utility.h
Text conflict in sql/set_var.cc
Text conflict in sql/share/Makefile.am
Text conflict in sql/sql_delete.cc
Text conflict in sql/sql_plugin.cc
Text conflict in sql/sql_select.cc
Text conflict in sql/sql_table.cc
Text conflict in storage/example/ha_example.h
Text conflict in storage/federated/ha_federated.cc
Text conflict in storage/myisammrg/ha_myisammrg.cc
Text conflict in storage/myisammrg/myrg_open.c
Conflicts:
Text conflict in mysql-test/r/partition_innodb.result
Text conflict in sql/field.h
Text conflict in sql/item.h
Text conflict in sql/item_cmpfunc.h
Text conflict in sql/item_sum.h
Text conflict in sql/log_event_old.cc
Text conflict in sql/protocol.cc
Text conflict in sql/sql_select.cc
Text conflict in sql/sql_yacc.yy
(regression)
Problem was that partition pruning did not exclude the
last partition if the range was beyond it
(i.e. not using MAXVALUE)
Fix was to not include the last partition if the
partitioning function value was not within the partition
range.
mysql-test/r/partition_innodb.result:
Bug#51830: Incorrect partition pruning on range partition
(regression)
Updated result
mysql-test/r/partition_pruning.result:
Bug#51830: Incorrect partition pruning on range partition
(regression)
Updated result
mysql-test/t/partition_innodb.test:
Bug#51830: Incorrect partition pruning on range partition
(regression)
Added test for pruning in InnoDB, since it does not show
for MyISAM due to 'Impossible WHERE noticed after reading
const tables'.
mysql-test/t/partition_pruning.test:
Bug#51830: Incorrect partition pruning on range partition
(regression)
Added test
sql/sql_partition.cc:
Bug#51830: Incorrect partition pruning on range partition
(regression)
Also increase the partition id if not inside the last partition
(and no MAXVALUE is defined).
Added comments and DBUG_ASSERT.
There was no check for foreign keys when altering partitioned
tables.
Added check for FK when altering partitioned tables.
mysql-test/r/partition_innodb.result:
Bug#50104: Partitioned table with just 1 partion works with fk
Updated test result
mysql-test/t/partition_innodb.test:
Bug#50104: Partitioned table with just 1 partion works with fk
Added test for adding FK on partitioned tables (both 1 and 2
partitions)
sql/sql_partition.cc:
Bug#50104: Partitioned table with just 1 partion works with fk
Disabled adding foreign key when altering a partitioned table.
REORGANIZE PARTITION
There were several problems which lead to this this,
all related to bad error handling.
1) There was several bugs preventing the ddl-log to be used for
cleaning up created files on error.
2) The error handling after the copy partition rows did not close
and unlock the tables, resulting in deletion of partitions
which were in use, which lead InnoDB to put the partition to
drop in a background queue.
sql/ha_partition.cc:
Bug#47343: InnoDB fails to clean-up after lock wait timeout on
REORGANIZE PARTITION
Better error handling, if partition has been created/opened/locked
then make sure it is unlocked and closed before returning error.
The delete of the newly created partition is handled by the ddl-log.
sql/sql_parse.cc:
Bug#47343: InnoDB fails to clean-up after lock wait timeout on
REORGANIZE PARTITION
Fix a bug found when experimenting, thd could really be NULL here,
as mentioned in the function header.
sql/sql_partition.cc:
Bug#47343: InnoDB fails to clean-up after lock wait timeout on
REORGANIZE PARTITION
Used the correct .frm shadow name to put into the ddl-log.
Really use the ddl-log to handle errors.
sql/sql_table.cc:
Bug#47343: InnoDB fails to clean-up after lock wait timeout on
REORGANIZE PARTITION
Fixes of the ddl-log when used as error recovery (no crash).
When executing an entry from memory (not read from disk)
the name_len was not set correctly.
mysql-test/r/log_tables.result:
Bug #35765 ALTER TABLE produces wrong error when non-existent storage engine used
Updated result
mysql-test/r/partition.result:
Bug #35765 ALTER TABLE produces wrong error when non-existent storage engine used
Updated result
mysql-test/r/partition_innodb.result:
Bug #35765 ALTER TABLE produces wrong error when non-existent storage engine used
Updated result
mysql-test/t/log_tables.test:
Bug #35765 ALTER TABLE produces wrong error when non-existent storage engine used
Updated test
mysql-test/t/partition.test:
Bug #35765 ALTER TABLE produces wrong error when non-existent storage engine used
Added test case
mysql-test/t/partition_innodb.test:
Bug #35765 ALTER TABLE produces wrong error when non-existent storage engine used
Updated test
sql/protocol.cc:
Bug #35765 ALTER TABLE produces wrong error when non-existent storage engine used
(fix of bug#48939 to avoid test failures on my test build).
sql/sql_yacc.yy:
Bug #35765 ALTER TABLE produces wrong error when non-existent storage engine used
if no existing engine was given, don't set HA_CREATE_USED_ENGINE
