bug #57006 "Deadlock between HANDLER and FLUSH TABLES WITH READ
LOCK" and bug #54673 "It takes too long to get readlock for
'FLUSH TABLES WITH READ LOCK'".
The first bug manifested itself as a deadlock which occurred
when a connection, which had some table open through HANDLER
statement, tried to update some data through DML statement
while another connection tried to execute FLUSH TABLES WITH
READ LOCK concurrently.
What happened was that FTWRL in the second connection managed
to perform first step of GRL acquisition and thus blocked all
upcoming DML. After that it started to wait for table open
through HANDLER statement to be flushed. When the first connection
tried to execute DML it has started to wait for GRL/the second
connection creating deadlock.
The second bug manifested itself as starvation of FLUSH TABLES
WITH READ LOCK statements in cases when there was a constant
stream of concurrent DML statements (in two or more
connections).
This has happened because requests for protection against GRL
which were acquired by DML statements were ignoring presence of
pending GRL and thus the latter was starved.
This patch solves both these problems by re-implementing GRL
using metadata locks.
Similar to the old implementation acquisition of GRL in new
implementation is two-step. During the first step we block
all concurrent DML and DDL statements by acquiring global S
metadata lock (each DML and DDL statement acquires global IX
lock for its duration). During the second step we block commits
by acquiring global S lock in COMMIT namespace (commit code
acquires global IX lock in this namespace).
Note that unlike in old implementation acquisition of
protection against GRL in DML and DDL is semi-automatic.
We assume that any statement which should be blocked by GRL
will either open and acquires write-lock on tables or acquires
metadata locks on objects it is going to modify. For any such
statement global IX metadata lock is automatically acquired
for its duration.
The first problem is solved because waits for GRL become
visible to deadlock detector in metadata locking subsystem
and thus deadlocks like one in the first bug become impossible.
The second problem is solved because global S locks which
are used for GRL implementation are given preference over
IX locks which are acquired by concurrent DML (and we can
switch to fair scheduling in future if needed).
Important change:
FTWRL/GRL no longer blocks DML and DDL on temporary tables.
Before this patch behavior was not consistent in this respect:
in some cases DML/DDL statements on temporary tables were
blocked while in others they were not. Since the main use cases
for FTWRL are various forms of backups and temporary tables are
not preserved during backups we have opted for consistently
allowing DML/DDL on temporary tables during FTWRL/GRL.
Important change:
This patch changes thread state names which are used when
DML/DDL of FTWRL is waiting for global read lock. It is now
either "Waiting for global read lock" or "Waiting for commit
lock" depending on the stage on which FTWRL is.
Incompatible change:
To solve deadlock in events code which was exposed by this
patch we have to replace LOCK_event_metadata mutex with
metadata locks on events. As result we have to prohibit
DDL on events under LOCK TABLES.
This patch also adds extensive test coverage for interaction
of DML/DDL and FTWRL.
Performance of new and old global read lock implementations
in sysbench tests were compared. There were no significant
difference between new and old implementations.
This patch:
- Moves all definitions from the mysql_priv.h file into
header files for the component where the variable is
defined
- Creates header files if the component lacks one
- Eliminates all include directives from mysql_priv.h
- Eliminates all circular include cycles
- Rename time.cc to sql_time.cc
- Rename mysql_priv.h to sql_priv.h
The event scheduler was not designed to work in embedded mode. This
patch disables and excludes the event scheduler when the server is
compiled for embedded build.
- BUG#11986: Stored routines and triggers can fail if the code
has a non-ascii symbol
- BUG#16291: mysqldump corrupts string-constants with non-ascii-chars
- BUG#19443: INFORMATION_SCHEMA does not support charsets properly
- BUG#21249: Character set of SP-var can be ignored
- BUG#25212: Character set of string constant is ignored (stored routines)
- BUG#25221: Character set of string constant is ignored (triggers)
There were a few general problems that caused these bugs:
1. Character set information of the original (definition) query for views,
triggers, stored routines and events was lost.
2. mysqldump output query in client character set, which can be
inappropriate to encode definition-query.
3. INFORMATION_SCHEMA used strings with mixed encodings to display object
definition;
1. No query-definition-character set.
In order to compile query into execution code, some extra data (such as
environment variables or the database character set) is used. The problem
here was that this context was not preserved. So, on the next load it can
differ from the original one, thus the result will be different.
The context contains the following data:
- client character set;
- connection collation (character set and collation);
- collation of the owner database;
The fix is to store this context and use it each time we parse (compile)
and execute the object (stored routine, trigger, ...).
2. Wrong mysqldump-output.
The original query can contain several encodings (by means of character set
introducers). The problem here was that we tried to convert original query
to the mysqldump-client character set.
Moreover, we stored queries in different character sets for different
objects (views, for one, used UTF8, triggers used original character set).
The solution is
- to store definition queries in the original character set;
- to change SHOW CREATE statement to output definition query in the
binary character set (i.e. without any conversion);
- introduce SHOW CREATE TRIGGER statement;
- to dump special statements to switch the context to the original one
before dumping and restore it afterwards.
Note, in order to preserve the database collation at the creation time,
additional ALTER DATABASE might be used (to temporary switch the database
collation back to the original value). In this case, ALTER DATABASE
privilege will be required. This is a backward-incompatible change.
3. INFORMATION_SCHEMA showed non-UTF8 strings
The fix is to generate UTF8-query during the parsing, store it in the object
and show it in the INFORMATION_SCHEMA.
Basically, the idea is to create a copy of the original query convert it to
UTF8. Character set introducers are removed and all text literals are
converted to UTF8.
This UTF8 query is intended to provide user-readable output. It must not be
used to recreate the object. Specialized SHOW CREATE statements should be
used for this.
The reason for this limitation is the following: the original query can
contain symbols from several character sets (by means of character set
introducers).
Example:
- original query:
CREATE VIEW v1 AS SELECT _cp1251 'Hello' AS c1;
- UTF8 query (for INFORMATION_SCHEMA):
CREATE VIEW v1 AS SELECT 'Hello' AS c1;
streamline the event worker thread work flow and try to eliminate
possibilities for memory corruptions that might have been
lurking in previous (complicated) code.
This patch:
* removes Event_job_data::compile that was never used
* cleans up Event_job_data::execute to minimize juggling with
thread context and eliminate unneded code paths
* Implements Security_context::change/restore_security_context
to be able to re-use these methods in all stored programs
This is to maybe fix Bug#27733 "Valgrind failures in
remove_table_from_cache".
Review comments applied.
when there are no up-to-date system tables to support it:
- initialize the scheduler before reporting "Ready for connections".
This ensures that warnings, if any, are printed before "Ready for
connections", and this message is not mangled.
- do not abort the scheduler if there are no system tables
- check the tables once at start up, remember the status and disable
the scheduler if the tables are not up to date.
If one attempts to use the scheduler with bad tables,
issue an error message.
- clean up the behaviour of the module under LOCK TABLES and pre-locking
mode
- make sure implicit commit of Events DDL works as expected.
- add more tests
Collateral clean ups in the events code.
This patch fixes Bug#23631 Events: SHOW VARIABLES doesn't work
when mysql.event is damaged
the lexer API which internally uses unsigned char variables to
address its state map. The implementation of the lexer should be
internal to the lexer, and not influence the rest of the code.
BUG#26429: SHOW CREATE EVENT is incorrect for an event that
STARTS NOW()
BUG#26431: Impossible to re-create an event from backup if its
STARTS clause is in the past
WL#3698: Events: execution in local time zone
The problem was that local times specified by the user in AT, STARTS
and ENDS of CREATE EVENT/ALTER EVENT statement were converted to UTC,
and the original time zone was forgotten. This way, event scheduler
couldn't honor Daylight Saving Time shifts, and times shown to the
user were also in UTC. Additionally, CREATE EVENT didn't allow times
in the past, thus preventing straightforward event restoration from
old backups.
This patch reworks event scheduler time computations, performing them
in the time zone associated with the event. Also it allows times to
be in the past.
The patch adds time_zone column to mysql.event table.
NOTE: The patch is almost final, but the bug#9953 should be pushed
first.
This changeset adds replication of events and user-defined functions.
There are several bug reports involved in this change:
BUG#16421, BUG#17857, BUG#20384:
This patch modifies the mysql.events table to permit the addition of
another enum value for the status column. The column now has values
of ('DISABLED','SLAVESIDE_DISABLED','ENABLED'). A status of
SLAVESIDE_DISABLED is set on the slave during replication of events.
This enables users to determine which events werereplicated from the
master and to later enable them if they promote the slave to a master.
The CREATE, ALTER, and DROP statements are binlogged.
A new test was added for replication of events (rpl_events).
BUG#17671:
This patch modifies the code to permit logging of user-defined functions.
Note: this is the CREATE FUNCTION ... SONAME variety. A more friendly error
message to be displayed should a replicated user-defined function not be
found in the loadable library or if the library is missing from the
slave.The CREATE andDROP statements are binlogged. A new test was added
for replication of user-defined functions (rpl_udf).
The patch also adds a new column to the mysql.event table named
'originator' that is used to store the server_id of the server that
the event originated on. This enables users to promote a slave to a
master and later return the promoted slave to a slave and disable the
replicated events.
This patch implements the idea of the bug report by making Event_queue
unaware of Event_db_repository by making a higher level class - Events,
which is aware of most of all classes, responsible for passing all data
needed for adding/updating/deleting an event to/from the queue.
Introduces few new classes :
- Event_worker_thread
- Event_queue_element_for_exec
Added missing DBUG_RETURN statements (in mysqldump.c)
Added missing enums
Fixed a lot of wrong DBUG_PRINT() statements, some of which could cause crashes
Removed usage of %lld and %p in printf strings as these are not portable or produces different results on different systems.
More small fixes to the API : use LEX_STRING instead of LEX_STRING* and if error
then return bool(true) instead of error code.
Merged functions. Reduced usage of sp_name.
Fixed a lot of function documentation errors.
Added function documentation wherever needed.
Removed some unused defines and error codes.
Next to come is batch rename of Event_scheduler_ng to Event_scheduler.
This patch introduces specialized Event data objects
Event_basic as parent.
Event_queue_element used for queue storage
Event_timed used for SHOW EVENTS/ I_S.EVENTS / SHOW CREATE EVENT
Event_job_data using during execution.
Methods were moved out of Event_timed to other classes.
This patch also introduces Events::LOCK_event_metadata.
This patch gives new implementation of Events::dump_internal_status().
Now both the Event_scheduler and Event_queue return information during
their ::dump_internal_status().
Shortened a bit the runtime for executing events test cases.
Cleaned up the code a bit. Fixed few leaks.
This code still does not load events on server startup
from disk. The problem is that there is a need for a THD instance, which
does not exist during server boot. This will be solved soon.
Still Event_timed is used both for the memory queue and for exectution.
This will be changed according to WL#3337 probably in the next commit.
Cut Nr. 8.
All tests pass.
Separated Event_scheduler into Event_queue and Event_scheduler.
Added new Event_scheduler_ng which is the new scheduler and is used
system-wide. Will be moved to the event_scheduler.cc in the future.
Using Event_timed in Event_queue as well as cloned during execution.
Next step is to have Event_worker_data which will be used during execution
and will take ::compile()/::execute() out of Event_timed.
This cut No 7 should finish the part of fixing the parsing of the events :
- Event_timed is no more used during parsing. Less problems because it has
a mutex. Event_parse_data class is used during parsing. It is suited only
for this purpose. It's pretty lightweight
- Late checking of data from parsing is being performed. This should solve
the problems of nested events in SP or other events (for the situation
of no nested bodies). Before if an ALTER EVENT was in a SP, then when the
SP was compiled, and not executed, the actual init_xxx methods of Event_timed
were called, which is wrong.
- It could be a side effect of using a specialized class, but test events_stress is
now 25% quicker.
Cut No8 will start splitting Event_scheduler into 2 parts, the QUEUE will be moved
to Event_queue.
5th cut, moved DB related code to Event_db_repository and
updated accordingly the remanining code.
Moved change/restore_security_context() to class THD
Removed events_priv.h
Next step is to reorganize create/update_event() and parsing for them.
But probably some other refactoring could be done in the meanwhile.
The changes so far pass the test suite.
