Fix Bug#50555 "handler commands crash server in my_hash_first()"
as a post-merge fix (the new handler tests are not passing
otherwise).
- in hash.c, don't call calc_hash if ! my_hash_inited().
- add tests and results for the test case for Bug#50555
caused by patch which implemented new type-of-operation-aware
metadata locks and added a wait-for graph based deadlock
detector to the MDL subsystem (this patch fixed bug #46272
"MySQL 5.4.4, new MDL: unnecessary deadlock" and bug #37346
"innodb does not detect deadlock between update and alter
table").
Crashes were caused by a race in MDL_context::try_acquire_lock().
This method added MDL_ticket to the list of granted tickets and
released lock protecting list before setting MDL_ticket::m_lock.
Thus some other thread was able to see ticket without properly
set m_lock member for some short period of time. If this thread
called method involving this member during this period crash
happened.
This fix ensures that MDL_ticket::m_lock is set in all cases
when ticket is added to granted/pending lists in MDL_lock.
Add a wait-for graph based deadlock detector to the
MDL subsystem.
Fixes bug #46272 "MySQL 5.4.4, new MDL: unnecessary deadlock" and
bug #37346 "innodb does not detect deadlock between update and
alter table".
The first bug manifested itself as an unwarranted abort of a
transaction with ER_LOCK_DEADLOCK error by a concurrent ALTER
statement, when this transaction tried to repeat use of a
table, which it has already used in a similar fashion before
ALTER started.
The second bug showed up as a deadlock between table-level
locks and InnoDB row locks, which was "detected" only after
innodb_lock_wait_timeout timeout.
A transaction would start using the table and modify a few
rows.
Then ALTER TABLE would come in, and start copying rows
into a temporary table. Eventually it would stumble on
the modified records and get blocked on a row lock.
The first transaction would try to do more updates, and get
blocked on thr_lock.c lock.
This situation of circular wait would only get resolved
by a timeout.
Both these bugs stemmed from inadequate solutions to the
problem of deadlocks occurring between different
locking subsystems.
In the first case we tried to avoid deadlocks between metadata
locking and table-level locking subsystems, when upgrading shared
metadata lock to exclusive one.
Transactions holding the shared lock on the table and waiting for
some table-level lock used to be aborted too aggressively.
We also allowed ALTER TABLE to start in presence of transactions
that modify the subject table. ALTER TABLE acquires
TL_WRITE_ALLOW_READ lock at start, and that block all writes
against the table (naturally, we don't want any writes to be lost
when switching the old and the new table). TL_WRITE_ALLOW_READ
lock, in turn, would block the started transaction on thr_lock.c
lock, should they do more updates. This, again, lead to the need
to abort such transactions.
The second bug occurred simply because we didn't have any
mechanism to detect deadlocks between the table-level locks
in thr_lock.c and row-level locks in InnoDB, other than
innodb_lock_wait_timeout.
This patch solves both these problems by moving lock conflicts
which are causing these deadlocks into the metadata locking
subsystem, thus making it possible to avoid or detect such
deadlocks inside MDL.
To do this we introduce new type-of-operation-aware metadata
locks, which allow MDL subsystem to know not only the fact that
transaction has used or is going to use some object but also what
kind of operation it has carried out or going to carry out on the
object.
This, along with the addition of a special kind of upgradable
metadata lock, allows ALTER TABLE to wait until all
transactions which has updated the table to go away.
This solves the second issue.
Another special type of upgradable metadata lock is acquired
by LOCK TABLE WRITE. This second lock type allows to solve the
first issue, since abortion of table-level locks in event of
DDL under LOCK TABLES becomes also unnecessary.
Below follows the list of incompatible changes introduced by
this patch:
- From now on, ALTER TABLE and CREATE/DROP TRIGGER SQL (i.e. those
statements that acquire TL_WRITE_ALLOW_READ lock)
wait for all transactions which has *updated* the table to
complete.
- From now on, LOCK TABLES ... WRITE, REPAIR/OPTIMIZE TABLE
(i.e. all statements which acquire TL_WRITE table-level lock) wait
for all transaction which *updated or read* from the table
to complete.
As a consequence, innodb_table_locks=0 option no longer applies
to LOCK TABLES ... WRITE.
- DROP DATABASE, DROP TABLE, RENAME TABLE no longer abort
statements or transactions which use tables being dropped or
renamed, and instead wait for these transactions to complete.
- Since LOCK TABLES WRITE now takes a special metadata lock,
not compatible with with reads or writes against the subject table
and transaction-wide, thr_lock.c deadlock avoidance algorithm
that used to ensure absence of deadlocks between LOCK TABLES
WRITE and other statements is no longer sufficient, even for
MyISAM. The wait-for graph based deadlock detector of MDL
subsystem may sometimes be necessary and is involved. This may
lead to ER_LOCK_DEADLOCK error produced for multi-statement
transactions even if these only use MyISAM:
session 1: session 2:
begin;
update t1 ... lock table t2 write, t1 write;
-- gets a lock on t2, blocks on t1
update t2 ...
(ER_LOCK_DEADLOCK)
- Finally, support of LOW_PRIORITY option for LOCK TABLES ... WRITE
was abandoned.
LOCK TABLE ... LOW_PRIORITY WRITE from now on has the same
priority as the usual LOCK TABLE ... WRITE.
SELECT HIGH PRIORITY no longer trumps LOCK TABLE ... WRITE in
the wait queue.
- We do not take upgradable metadata locks on implicitly
locked tables. So if one has, say, a view v1 that uses
table t1, and issues:
LOCK TABLE v1 WRITE;
FLUSH TABLE t1; -- (or just 'FLUSH TABLES'),
an error is produced.
In order to be able to perform DDL on a table under LOCK TABLES,
the table must be locked explicitly in the LOCK TABLES list.
condition variable per context instead of one mutex and one conditional
variable for the whole subsystem.
This should increase concurrency in this subsystem.
It also opens the way for further changes which are necessary to solve
such bugs as bug #46272 "MySQL 5.4.4, new MDL: unnecessary deadlock"
and bug #37346 "innodb does not detect deadlock between update and alter
table".
Two other notable changes done by this patch:
- MDL subsystem no longer implicitly acquires global intention exclusive
metadata lock when per-object metadata lock is acquired. Now this has
to be done by explicit calls outside of MDL subsystem.
- Instead of using separate MDL_context for opening system tables/tables
for purposes of I_S we now create MDL savepoint in the main context
before opening tables and rollback to this savepoint after closing
them. This means that it is now possible to get ER_LOCK_DEADLOCK error
even not inside a transaction. This might happen in unlikely case when
one runs DDL on one of system tables while also running DDL on some
other tables. Cases when this ER_LOCK_DEADLOCK error is not justified
will be addressed by advanced deadlock detector for MDL subsystem which
we plan to implement.
Diagnostics_area::set_ok_status at PREPARE
The problem occured during processing of stored routines.
Routines are loaded from mysql.proc, parsed and put into the sp cache by
sp_cache_routine(). The assert occured because the return value from
sp_cache_routine() was not checked for top level CALLs. This meant that any
errors during sp_cache_routine() went unoticed and triggered the assert when
my_ok() was later called.
This is a regression introduced by the patch for Bug#30977, only visible in
source trees with MDL and using debug builds of the server.
This patch fixes the problem by checking the return value from sp_cache_routine()
for top level CALLs and propagating any errors similar to what is done for other
calls to sp_cache_routine().
No test case added.
This was a deadlock between LOCK TABLES/CREATE DATABASE in one connection
and DROP DATABASE in another. It only happened if the table locked by
LOCK TABLES was in the database to be dropped. The deadlock is similar
to the one in Bug#48940, but with LOCK TABLES instead of an active
transaction.
The order of events needed to trigger the deadlock was:
1) Connection 1 locks table db1.t1 using LOCK TABLES. It will now
have a metadata lock on the table name.
2) Connection 2 issues DROP DATABASE db1. This will wait inside
the MDL subsystem for the lock on db1.t1 to go away. While waiting, it
will hold the LOCK_mysql_create_db mutex.
3) Connection 1 issues CREATE DATABASE (database name irrelevant).
This will hang trying to lock the same mutex. Since this is the connection
holding the metadata lock blocking Connection 2, we have a deadlock.
This deadlock would also happen for earlier trees without MDL, but
there DROP DATABASE would wait for a table to be removed from the
table definition cache.
This patch fixes the problem by prohibiting CREATE DATABASE in LOCK TABLES
mode. In the example above, this prevents Connection 1 from hanging trying
to get the LOCK_mysql_create_db mutex. Note that other commands that use
LOCK_mysql_create_db (ALTER/DROP DATABASE) are already prohibited in
LOCK TABLES mode.
Incompatible change: CREATE DATABASE is now disallowed in LOCK TABLES mode.
Test case added to schema.test.
Bug#45523 "Objects of class base_ilist should not be copyable".
Suppress the compiler-generated public copy constructor
and assignment operator of class base_ilist; instead, implement
move_elements_to() function which transfers ownership of elements
from one list to another.
This change is supposed to reduce number of ER_LOCK_DEADLOCK
errors which occur when multi-statement transaction encounters
conflicting metadata lock in cases when waiting is possible.
The idea is not to fail ER_LOCK_DEADLOCK error immediately when
we encounter conflicting metadata lock. Instead we release all
metadata locks acquired by current statement and start to wait
until conflicting lock go away. To avoid deadlocks we use simple
empiric which aborts waiting with ER_LOCK_DEADLOCK error if it
turns out that somebody is waiting for metadata locks owned by
this transaction.
This patch also fixes bug #46273 "MySQL 5.4.4 new MDL: Bug#989
is not fully fixed in case of ALTER".
The bug was that concurrent execution of UPDATE or MULTI-UPDATE
statement as a part of multi-statement transaction that already
has used table being updated and ALTER TABLE statement might have
resulted of loss of isolation between this transaction and ALTER
TABLE statement, which manifested itself as changes performed by
ALTER TABLE becoming visible in transaction and wrong binary log
order as a consequence.
This problem occurred when UPDATE or MULTI-UPDATE's wait in
mysql_lock_tables() call was aborted due to metadata lock
upgrade performed by concurrent ALTER TABLE. After such abort all
metadata locks held by transaction were released but transaction
silently continued to be executed as if nothing has happened.
We solve this problem by changing our code not to release all
locks in such case. Instead we release only locks which were
acquired by current statement and then try to reacquire them
by restarting open/lock tables process. We piggyback on simple
deadlock detector implementation since this change has to be
done anyway for it.
3655 Jon Olav Hauglid 2009-10-19
Bug #30977 Concurrent statement using stored function and DROP FUNCTION
breaks SBR
Bug #48246 assert in close_thread_table
Implement a fix for:
Bug #41804 purge stored procedure cache causes mysterious hang for many
minutes
Bug #49972 Crash in prepared statements
The problem was that concurrent execution of DML statements that
use stored functions and DDL statements that drop/modify the same
function might result in incorrect binary log in statement (and
mixed) mode and therefore break replication.
This patch fixes the problem by introducing metadata locking for
stored procedures and functions. This is similar to what is done
in Bug#25144 for views. Procedures and functions now are
locked using metadata locks until the transaction is either
committed or rolled back. This prevents other statements from
modifying the procedure/function while it is being executed. This
provides commit ordering - guaranteeing serializability across
multiple transactions and thus fixes the reported binlog problem.
Note that we do not take locks for top-level CALLs. This means
that procedures called directly are not protected from changes by
simultaneous DDL operations so they are executed at the state they
had at the time of the CALL. By not taking locks for top-level
CALLs, we still allow transactions to be started inside
procedures.
This patch also changes stored procedure cache invalidation.
Upon a change of cache version, we no longer invalidate the entire
cache, but only those routines which we use, only when a statement
is executed that uses them.
This patch also changes the logic of prepared statement validation.
A stored procedure used by a prepared statement is now validated
only once a metadata lock has been acquired. A version mismatch
causes a flush of the obsolete routine from the cache and
statement reprepare.
Incompatible changes:
1) ER_LOCK_DEADLOCK is reported for a transaction trying to access
a procedure/function that is locked by a DDL operation in
another connection.
2) Procedure/function DDL operations are now prohibited in LOCK
TABLES mode as exclusive locks must be taken all at once and
LOCK TABLES provides no way to specifiy procedures/functions to
be locked.
Test cases have been added to sp-lock.test and rpl_sp.test.
Work on this bug has very much been a team effort and this patch
includes and is based on contributions from Davi Arnaut, Dmitry
Lenev, Magne Mæhre and Konstantin Osipov.
Unlike other platforms --mfpmath=sse is the default
gcc mode on Mac OS X Intel. So it is unnecessary
to switch FPU to double precision mode (in fact,
it even breaks some math library functions).
to string conversions and vice versa"
Initial import of the dtoa.c code and custom wrappers around it
to allow its usage from the server code.
Conversion of FLOAT/DOUBLE values to DECIMAL ones or strings
and vice versa has been significantly reworked. As the new
algoritms are more precise than the older ones, results of such
conversions may not always match those obtained from older
server versions. This in turn may break compatibility for some
applications.
This patch also fixes the following bugs:
- bug #12860 "Difference in zero padding of exponent between
Unix and Windows"
- bug #21497 "DOUBLE truncated to unusable value"
- bug #26788 "mysqld (debug) aborts when inserting specific
numbers into char fields"
- bug #24541 "Data truncated..." on decimal type columns
without any good reason"
"HANDLER statements within a transaction might lead to deadlocks".
Introduce a notion of a sentinel to MDL_context. A sentinel
is a ticket that separates all tickets in the context into two
groups: before and after it. Currently we can have (and need) only
one designated sentinel -- it separates all locks taken by LOCK
TABLE or HANDLER statement, which must survive COMMIT and ROLLBACK
and all other locks, which must be released at COMMIT or ROLLBACK.
The tricky part is maintaining the sentinel up to date when
someone release its corresponding ticket. This can happen, e.g.
if someone issues DROP TABLE under LOCK TABLES (generally,
see all calls to release_all_locks_for_name()).
MDL_context::release_ticket() is modified to take care of it.
******
A fix and a test case for Bug#46224 "HANDLER statements within a
transaction might lead to deadlocks".
An attempt to mix HANDLER SQL statements, which are transaction-
agnostic, an open multi-statement transaction,
and DDL against the involved tables (in a concurrent connection)
could lead to a deadlock. The deadlock would occur when
HANDLER OPEN or HANDLER READ would have to wait on a conflicting
metadata lock. If the connection that issued HANDLER statement
also had other metadata locks (say, acquired in scope of a
transaction), a classical deadlock situation of mutual wait
could occur.
Incompatible change: entering LOCK TABLES mode automatically
closes all open HANDLERs in the current connection.
Incompatible change: previously an attempt to wait on a lock
in a connection that has an open HANDLER statement could wait
indefinitely/deadlock. After this patch, an error ER_LOCK_DEADLOCK
is produced.
The idea of the fix is to merge thd->handler_mdl_context
with the main mdl_context of the connection, used for transactional
locks. This makes deadlock detection possible, since all waits
with locks are "visible" and available to analysis in a single
MDL context of the connection.
Since HANDLER locks and transactional locks have a different life
cycle -- HANDLERs are explicitly open and closed, and so
are HANDLER locks, explicitly acquired and released, whereas
transactional locks "accumulate" till the end of a transaction
and are released only with COMMIT, ROLLBACK and ROLLBACK TO SAVEPOINT,
a concept of "sentinel" was introduced to MDL_context.
All locks, HANDLER and others, reside in the same linked list.
However, a selected element of the list separates locks with
different life cycle. HANDLER locks always reside at the
end of the list, after the sentinel. Transactional locks are
prepended to the beginning of the list, before the sentinel.
Thus, ROLLBACK, COMMIT or ROLLBACK TO SAVEPOINT, only
release those locks that reside before the sentinel. HANDLER locks
must be released explicitly as part of HANDLER CLOSE statement,
or an implicit close.
The same approach with sentinel
is also employed for LOCK TABLES locks. Since HANDLER and LOCK TABLES
statement has never worked together, the implementation is
made simple and only maintains one sentinel, which is used either
for HANDLER locks, or for LOCK TABLES locks.
If the handler (or delayed insert) thread failed to lock a table due
to being killed, the "dead" flag was used to notify the connection thread
of this failure. However, with the changes introduced by Bug#45949,
the handler thread will no longer try to lock the table if it was killed.
This meant that the "dead" flag would not be set, and the connection
thread would not notice that the handler thread had failed.
This could happen with concurrent INSERT DELAYED and FLUSH TABLES.
FLUSH TABLES would kill any active INSERT DELAYED that had opened any
table(s) to be flushed. This could cause the INSERT DELAYED connection
thread to be stuck waiting for the handler thread to lock its table,
while the handler thread would be looping, trying to get the connection
thread to notice the error.
The root of the problem was that the handler thread had both the "dead"
flag and "thd->killed" to indicate that it had been killed. Most places
both were set, but some only set "thd->killed". And
Delayed_insert::get_local_table() only checked "dead" while waiting for
the table to be locked.
This patch removes the "dead" variable and replaces its usage with
"thd->killed", thereby resolving the issue.
When compressed myisam files are opened, they are always memory mapped
sometimes causing memory swapping problems.
When we mmap the myisam compressed tables of size greater than the memory
available, the kswapd0 process utilization is very high consuming 30-40% of
the cpu. This happens only with linux kernels older than 2.6.9
With newer linux kernels, we don't have this problem of high cpu consumption
and this option may not be required.
The option 'myisam_mmap_size' is added to limit the amount of memory used for
memory mapping of myisam files. This option is not dynamic.
The default value on 32 bit system is 4294967295 bytes and on 64 bit system it
is 18446744073709547520 bytes.
Note: Testcase only tests the option variable. The actual bug has be to
tested manually.
returns incorrect results with where
An outer join of a const table (outer) and a normal table
(inner) with GROUP BY on a field from the outer table would
optimize away GROUP BY, and thus trigger the optimization to
do away with a temporary table if grouping was performed on
columns from the const table, hence executing the query with
filesort without temporary table. But this should not be
done if there is a non-indexed access to the inner table,
since filesort does not handle joins. It expects either ref
access, range ditto or table scan. The join condition will
thus not be applied.
Fixed by always forcing execution with temporary table in
the case of ROLLUP with a query involving an outer join. This
is a slightly broader class of queries than need fixing, but
it is hard to ascertain the position of a ROLLUP field wrt
outer join with current query representation.
Problem: inserting a record we don't set unused null bits in the
record buffer if no default field values used.
That may lead to wrong live checksum calculation.
Fix: set unused null bits in the record buffer in such cases.
The reason for the deadlock was an improper exit from
MDL_context::wait_for_locks() which caused mysys_var->current_mutex to remain
LOCK_mdl even though LOCK_mdl was no longer held by that connection.
This could for example lead to a deadlock in the following way:
1) INSERT DELAYED tries to open a table but fails, and trying to recover it
calls wait_for_locks().
2) Due to a pending exclusive request, wait_for_locks() fails and exits without
resetting mysys_var->current_mutex for the delayed insert handler thread. So it
continues to point to LOCK_mdl.
3) The handler thread manages to open a table.
4) A different connection takes LOCK_open and tries to take LOCK_mdl.
5) FLUSH TABLES from a third connection notices that the handler thread has a
table open, and tries to kill it. This involves locking mysys_var->current_mutex
while having LOCK_open locked. Since current_mutex mistakenly points to LOCK_mdl,
we have a deadlock.
This patch makes sure MDL_EXIT_COND() is called before exiting wait_for_locks().
This clears mysys->current_mutex which resolves the issue.
An assert is added to recover_from_failed_open_table_attempt() after
wait_for_locks() is called, to check that current_mutex is indeed reset.
With this assert in place, existing tests in (e.g.) mdl_sync.test will fail
without this patch.
