Analysis:
---------
When the server is out of memory, an error is raised
to indicate the same. Handling the error requires
more memory to be allocated which fails, hence the
error handling loops in a recursion and causes the
server to crash.
Fix:
---
a) Prevents pushing the 'out of memory' error condition
to the diagnostic area as it requires memory allocation.
GET DIAGNOSTICS, SHOW WARNINGS and SHOW ERRORS statements
will not show information about this error. However the
'out of memory' error is returned to the client.
b) It sets the ME_FATALERROR flag when 'out of memory' errors
are reported (for places where the flag is not already set).
This flag prevents activation of SP error handlers which also
require memory allocation and therefore are likely to fail.
sql/sql_insert.cc:
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
******
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
sql/sql_table.cc:
small cleanup
******
small cleanup
In sql_class.cc, 'row_count', of type 'ha_rows', was used as last argument for
ER_TRUNCATED_WRONG_VALUE_FOR_FIELD which is
"Incorrect %-.32s value: '%-.128s' for column '%.192s' at row %ld".
So 'ha_rows' was used as 'long'.
On SPARC32 Solaris builds, 'long' is 4 bytes and 'ha_rows' is 'longlong' i.e. 8 bytes.
So the printf-like code was reading only the first 4 bytes.
Because the CPU is big-endian, 1LL is 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x01
so the first four bytes yield 0. So the warning message had "row 0" instead of
"row 1" in test outfile_loaddata.test:
-Warning 1366 Incorrect string value: '\xE1\xE2\xF7' for column 'b' at row 1
+Warning 1366 Incorrect string value: '\xE1\xE2\xF7' for column 'b' at row 0
All error-messaging functions which internally invoke some printf-life function
are potential candidate for such mistakes.
One apparently easy way to catch such mistakes is to use
ATTRIBUTE_FORMAT (from my_attribute.h).
But this works only when call site has both:
a) the format as a string literal
b) the types of arguments.
So:
func(ER(ER_BLAH), 10);
will silently not be checked, because ER(ER_BLAH) is not known at
compile time (it is known at run-time, and depends on the chosen
language).
And
func("%s", a va_list argument);
has the same problem, as the *real* type of arguments is not
known at this site at compile time (it's known in some caller).
Moreover,
func(ER(ER_BLAH));
though possibly correct (if ER(ER_BLAH) has no '%' markers), will not
compile (gcc says "error: format not a string literal and no format
arguments").
Consequences:
1) ATTRIBUTE_FORMAT is here added only to functions which in practice
take "string literal" formats: "my_error_reporter" and "print_admin_msg".
2) it cannot be added to the other functions: my_error(),
push_warning_printf(), Table_check_intact::report_error(),
general_log_print().
To do a one-time check of functions listed in (2), the following
"static code analysis" has been done:
1) replace
my_error(ER_xxx, arguments for substitution in format)
with the equivalent
my_printf_error(ER_xxx,ER(ER_xxx), arguments for substitution in
format),
so that we have ER(ER_xxx) and the arguments *in the same call site*
2) add ATTRIBUTE_FORMAT to push_warning_printf(),
Table_check_intact::report_error(), general_log_print()
3) replace ER(xxx) with the hard-coded English text found in
errmsg.txt (like: ER(ER_UNKNOWN_ERROR) is replaced with
"Unknown error"), so that a call site has the format as string literal
4) this way, ATTRIBUTE_FORMAT can effectively do its job
5) compile, fix errors detected by ATTRIBUTE_FORMAT
6) revert steps 1-2-3.
The present patch has no compiler error when submitted again to the
static code analysis above.
It cannot catch all problems though: see Field::set_warning(), in
which a call to push_warning_printf() has a variable error
(thus, not replacable by a string literal); I checked set_warning() calls
by hand though.
See also WL 5883 for one proposal to avoid such bugs from appearing
again in the future.
The issues fixed in the patch are:
a) mismatch in types (like 'int' passed to '%ld')
b) more arguments passed than specified in the format.
This patch resolves mismatches by changing the type/number of arguments,
not by changing error messages of sql/share/errmsg.txt. The latter would be wrong,
per the following old rule: errmsg.txt must be as stable as possible; no insertions
or deletions of messages, no changes of type or number of printf-like format specifiers,
are allowed, as long as the change impacts a message already released in a GA version.
If this rule is not followed:
- Connectors, which use error message numbers, will be confused (by insertions/deletions
of messages)
- using errmsg.sys of MySQL 5.1.n with mysqld of MySQL 5.1.(n+1)
could produce wrong messages or crash; such usage can easily happen if
installing 5.1.(n+1) while /etc/my.cnf still has --language=/path/to/5.1.n/xxx;
or if copying mysqld from 5.1.(n+1) into a 5.1.n installation.
When fixing b), I have verified that the superfluous arguments were not used in the format
in the first 5.1 GA (5.1.30 'bteam@astra04-20081114162938-z8mctjp6st27uobm').
Had they been used, then passing them today, even if the message doesn't use them
anymore, would have been necessary, as explained above.
include/my_getopt.h:
this function pointer is used only with "string literal" formats, so we can add
ATTRIBUTE_FORMAT.
mysql-test/collections/default.experimental:
test should pass now
sql/derror.cc:
by having a format as string literal, ATTRIBUTE_FORMAT check becomes effective.
sql/events.cc:
Change justified by the following excerpt from sql/share/errmsg.txt:
ER_EVENT_SAME_NAME
eng "Same old and new event name"
ER_EVENT_SET_VAR_ERROR
eng "Error during starting/stopping of the scheduler. Error code %u"
sql/field.cc:
ER_TOO_BIG_SCALE 42000 S1009
eng "Too big scale %d specified for column '%-.192s'. Maximum is %lu."
ER_TOO_BIG_PRECISION 42000 S1009
eng "Too big precision %d specified for column '%-.192s'. Maximum is %lu."
ER_TOO_BIG_DISPLAYWIDTH 42000 S1009
eng "Display width out of range for column '%-.192s' (max = %lu)"
sql/ha_ndbcluster.cc:
ER_OUTOFMEMORY HY001 S1001
eng "Out of memory; restart server and try again (needed %d bytes)"
(sizeof() returns size_t)
sql/ha_ndbcluster_binlog.cc:
Too many arguments for:
ER_GET_ERRMSG
eng "Got error %d '%-.100s' from %s"
Patch by Jonas Oreland.
sql/ha_partition.cc:
print_admin_msg() is used only with a literal as format, so ATTRIBUTE_FORMAT
works.
sql/handler.cc:
ER_OUTOFMEMORY HY001 S1001
eng "Out of memory; restart server and try again (needed %d bytes)"
(sizeof() returns size_t)
sql/item_create.cc:
ER_TOO_BIG_SCALE 42000 S1009
eng "Too big scale %d specified for column '%-.192s'. Maximum is %lu."
ER_TOO_BIG_PRECISION 42000 S1009
eng "Too big precision %d specified for column '%-.192s'. Maximum is %lu."
'c_len' and 'c_dec' are char*, passed as %d !! We don't know their value
(as strtoul() failed), but they are likely big, so we use INT_MAX.
'len' is ulong.
sql/item_func.cc:
ER_WARN_DATA_OUT_OF_RANGE 22003
eng "Out of range value for column '%s' at row %ld"
ER_CANT_FIND_UDF
eng "Can't load function '%-.192s'"
sql/item_strfunc.cc:
ER_TOO_BIG_FOR_UNCOMPRESS
eng "Uncompressed data size too large; the maximum size is %d (probably, length of uncompressed data was corrupted)"
max_allowed_packet is ulong.
sql/mysql_priv.h:
sql_print_message_func is a function _pointer_.
sql/sp_head.cc:
ER_SP_RECURSION_LIMIT
eng "Recursive limit %d (as set by the max_sp_recursion_depth variable) was exceeded for routine %.192s"
max_sp_recursion_depth is ulong
sql/sql_acl.cc:
ER_PASSWORD_NO_MATCH 42000
eng "Can't find any matching row in the user table"
ER_CANT_CREATE_USER_WITH_GRANT 42000
eng "You are not allowed to create a user with GRANT"
sql/sql_base.cc:
ER_NOT_KEYFILE
eng "Incorrect key file for table '%-.200s'; try to repair it"
ER_TOO_MANY_TABLES
eng "Too many tables; MySQL can only use %d tables in a join"
MAX_TABLES is size_t.
sql/sql_binlog.cc:
ER_UNKNOWN_ERROR
eng "Unknown error"
sql/sql_class.cc:
ER_TRUNCATED_WRONG_VALUE_FOR_FIELD
eng "Incorrect %-.32s value: '%-.128s' for column '%.192s' at row %ld"
WARN_DATA_TRUNCATED 01000
eng "Data truncated for column '%s' at row %ld"
sql/sql_connect.cc:
ER_HANDSHAKE_ERROR 08S01
eng "Bad handshake"
ER_BAD_HOST_ERROR 08S01
eng "Can't get hostname for your address"
sql/sql_insert.cc:
ER_WRONG_VALUE_COUNT_ON_ROW 21S01
eng "Column count doesn't match value count at row %ld"
sql/sql_parse.cc:
ER_WARN_HOSTNAME_WONT_WORK
eng "MySQL is started in --skip-name-resolve mode; you must restart it without this switch for this grant to work"
ER_TOO_HIGH_LEVEL_OF_NESTING_FOR_SELECT
eng "Too high level of nesting for select"
ER_UNKNOWN_ERROR
eng "Unknown error"
sql/sql_partition.cc:
ER_OUTOFMEMORY HY001 S1001
eng "Out of memory; restart server and try again (needed %d bytes)"
sql/sql_plugin.cc:
ER_OUTOFMEMORY HY001 S1001
eng "Out of memory; restart server and try again (needed %d bytes)"
sql/sql_prepare.cc:
ER_OUTOFMEMORY HY001 S1001
eng "Out of memory; restart server and try again (needed %d bytes)"
ER_UNKNOWN_STMT_HANDLER
eng "Unknown prepared statement handler (%.*s) given to %s"
length value (for '%.*s') must be 'int', per the doc of printf()
and the code of my_vsnprintf().
sql/sql_show.cc:
ER_OUTOFMEMORY HY001 S1001
eng "Out of memory; restart server and try again (needed %d bytes)"
sql/sql_table.cc:
ER_TOO_BIG_FIELDLENGTH 42000 S1009
eng "Column length too big for column '%-.192s' (max = %lu); use BLOB or TEXT instead"
sql/table.cc:
ER_NOT_FORM_FILE
eng "Incorrect information in file: '%-.200s'"
ER_COL_COUNT_DOESNT_MATCH_PLEASE_UPDATE
eng "Column count of mysql.%s is wrong. Expected %d, found %d. Created with MySQL %d, now running %d. Please use mysql_upgrade to fix this error."
table->s->mysql_version is ulong.
sql/unireg.cc:
ER_TOO_LONG_TABLE_COMMENT
eng "Comment for table '%-.64s' is too long (max = %lu)"
ER_TOO_LONG_FIELD_COMMENT
eng "Comment for field '%-.64s' is too long (max = %lu)"
ER_TOO_BIG_ROWSIZE 42000
eng "Row size too large. The maximum row size for the used table type, not counting BLOBs, is %ld. You have to change some columns to TEXT or BLOBs"
- Fixed some issues with partitions and connection_string, which also fixed lp:716890 "Pre- and post-recovery crash in Aria"
- Fixed wrong assert in Aria
Now need to merge with latest xtradb before pushing
sql/ha_partition.cc:
Ensure that m_ordered_rec_buffer is not freed before close.
sql/mysqld.cc:
Changed to use opt_stack_trace instead of opt_pstack.
Removed references to pstack
sql/partition_element.h:
Ensure that connect_string is initialized
storage/maria/ma_key_recover.c:
Fixed wrong assert
This patch adds options to annotate the binlog (and the mysqlbinlog
output) with the original SQL query for queries that are logged
using row-based replication.
When using BINLOG statement to execute rows log events, session variables
foreign_key_checks and unique_checks are changed temporarily. As each rows
log event has their own special session environment and its own
foreign_key_checks and unique_checks can be different from current session
which executing the BINLOG statement. But these variables are not restored
correctly after BINLOG statement. This problem will cause that the following
statements fail or generate unexpected data.
In this patch, code is added to backup and restore these two variables.
So BINLOG statement will not affect current session's variables again.
mysql-test/extra/binlog_tests/binlog.test:
Add test to verify this patch.
mysql-test/suite/binlog/r/binlog_row_binlog.result:
Add test to verify this patch.
mysql-test/suite/binlog/r/binlog_stm_binlog.result:
Add test to verify this patch.
sql/sql_binlog.cc:
Add code to backup and restore thd->options.
Essentially, the problem is that safemalloc is excruciatingly
slow as it checks all allocated blocks for overrun at each
memory management primitive, yielding a almost exponential
slowdown for the memory management functions (malloc, realloc,
free). The overrun check basically consists of verifying some
bytes of a block for certain magic keys, which catches some
simple forms of overrun. Another minor problem is violation
of aliasing rules and that its own internal list of blocks
is prone to corruption.
Another issue with safemalloc is rather the maintenance cost
as the tool has a significant impact on the server code.
Given the magnitude of memory debuggers available nowadays,
especially those that are provided with the platform malloc
implementation, maintenance of a in-house and largely obsolete
memory debugger becomes a burden that is not worth the effort
due to its slowness and lack of support for detecting more
common forms of heap corruption.
Since there are third-party tools that can provide the same
functionality at a lower or comparable performance cost, the
solution is to simply remove safemalloc. Third-party tools
can provide the same functionality at a lower or comparable
performance cost.
The removal of safemalloc also allows a simplification of the
malloc wrappers, removing quite a bit of kludge: redefinition
of my_malloc, my_free and the removal of the unused second
argument of my_free. Since free() always check whether the
supplied pointer is null, redudant checks are also removed.
Also, this patch adds unit testing for my_malloc and moves
my_realloc implementation into the same file as the other
memory allocation primitives.
client/mysqldump.c:
Pass my_free directly as its signature is compatible with the
callback type -- which wasn't the case for free_table_ent.
mysql_client_binlog_statement
Problem: server may read from unassigned memory performing
"wrong" BINLOG queries.
Fix: never read from unassigned memory.
mysql-test/suite/binlog/r/binlog_base64_flag.result:
Fix for bug #54393: crash and/or valgrind errors in
mysql_client_binlog_statement
- test result.
mysql-test/suite/binlog/t/binlog_base64_flag.test:
Fix for bug #54393: crash and/or valgrind errors in
mysql_client_binlog_statement
- test case.
sql/sql_binlog.cc:
Fix for bug #54393: crash and/or valgrind errors in
mysql_client_binlog_statement
- coded_len should not count trailing '/0';
- never read from unassigned memory.
This patch:
- Moves all definitions from the mysql_priv.h file into
header files for the component where the variable is
defined
- Creates header files if the component lacks one
- Eliminates all include directives from mysql_priv.h
- Eliminates all circular include cycles
- Rename time.cc to sql_time.cc
- Rename mysql_priv.h to sql_priv.h
Backport of:
------------------------------------------------------------
revno: 2630.4.1
committer: Dmitry Lenev <dlenev@mysql.com>
branch nick: mysql-6.0-3726-w
timestamp: Fri 2008-05-23 17:54:03 +0400
message:
WL#3726 "DDL locking for all metadata objects".
After review fixes in progress.
------------------------------------------------------------
This is the first patch in series. It transforms the metadata
locking subsystem to use a dedicated module (mdl.h,cc). No
significant changes in the locking protocol.
The import passes the test suite with the exception of
deprecated/removed 6.0 features, and MERGE tables. The latter
are subject to a fix by WL#4144.
Unfortunately, the original changeset comments got lost in a merge,
thus this import has its own (largely insufficient) comments.
This patch fixes Bug#25144 "replication / binlog with view breaks".
Warning: this patch introduces an incompatible change:
Under LOCK TABLES, it's no longer possible to FLUSH a table that
was not locked for WRITE.
Under LOCK TABLES, it's no longer possible to DROP a table or
VIEW that was not locked for WRITE.
******
Backport of:
------------------------------------------------------------
revno: 2630.4.2
committer: Dmitry Lenev <dlenev@mysql.com>
branch nick: mysql-6.0-3726-w
timestamp: Sat 2008-05-24 14:03:45 +0400
message:
WL#3726 "DDL locking for all metadata objects".
After review fixes in progress.
******
Backport of:
------------------------------------------------------------
revno: 2630.4.3
committer: Dmitry Lenev <dlenev@mysql.com>
branch nick: mysql-6.0-3726-w
timestamp: Sat 2008-05-24 14:08:51 +0400
message:
WL#3726 "DDL locking for all metadata objects"
Fixed failing Windows builds by adding mdl.cc to the lists
of files needed to build server/libmysqld on Windows.
******
Backport of:
------------------------------------------------------------
revno: 2630.4.4
committer: Dmitry Lenev <dlenev@mysql.com>
branch nick: mysql-6.0-3726-w
timestamp: Sat 2008-05-24 21:57:58 +0400
message:
WL#3726 "DDL locking for all metadata objects".
Fix for assert failures in kill.test which occured when one
tried to kill ALTER TABLE statement on merge table while it
was waiting in wait_while_table_is_used() for other connections
to close this table.
These assert failures stemmed from the fact that cleanup code
in this case assumed that temporary table representing new
version of table was open with adding to THD::temporary_tables
list while code which were opening this temporary table wasn't
always fulfilling this.
This patch changes code that opens new version of table to
always do this linking in. It also streamlines cleanup process
for cases when error occurs while we have new version of table
open.
******
WL#3726 "DDL locking for all metadata objects"
Add libmysqld/mdl.cc to .bzrignore.
******
Backport of:
------------------------------------------------------------
revno: 2630.4.6
committer: Dmitry Lenev <dlenev@mysql.com>
branch nick: mysql-6.0-3726-w
timestamp: Sun 2008-05-25 00:33:22 +0400
message:
WL#3726 "DDL locking for all metadata objects".
Addition to the fix of assert failures in kill.test caused by
changes for this worklog.
Make sure we close the new table only once.
.bzrignore:
Add libmysqld/mdl.cc
libmysqld/CMakeLists.txt:
Added mdl.cc to the list of files needed for building of libmysqld.
libmysqld/Makefile.am:
Added files implementing new meta-data locking subsystem to the server.
mysql-test/include/handler.inc:
Use separate connection for waiting while threads performing DDL
operations conflicting with open HANDLER tables reach blocked
state. This is required because now we check and close tables open
by HANDLER statements in this connection conflicting with DDL in
another each time open_tables() is called and thus select from I_S
which is used for waiting will unblock DDL operations if issued
from connection with open HANDLERs.
mysql-test/r/create.result:
Adjusted test case after change in implementation of CREATE TABLE
... SELECT. We no longer have special check in open_table() which
catches the case when we select from the table created. Instead we
rely on unique_table() call which happens after opening and
locking all tables.
mysql-test/r/flush.result:
FLUSH TABLES WITH READ LOCK can no longer happen under LOCK
TABLES. Updated test accordingly.
mysql-test/r/flush_table.result:
Under LOCK TABLES we no longer allow to do FLUSH TABLES for tables
locked for read. Updated test accordingly.
mysql-test/r/handler_innodb.result:
Use separate connection for waiting while threads performing DDL
operations conflicting with open HANDLER tables reach blocked
state. This is required because now we check and close tables open
by HANDLER statements in this connection conflicting with DDL in
another each time open_tables() is called and thus select from I_S
which is used for waiting will unblock DDL operations if issued
from connection with open HANDLERs.
mysql-test/r/handler_myisam.result:
Use separate connection for waiting while threads performing DDL
operations conflicting with open HANDLER tables reach blocked
state. This is required because now we check and close tables open
by HANDLER statements in this connection conflicting with DDL in
another each time open_tables() is called and thus select from I_S
which is used for waiting will unblock DDL operations if issued
from connection with open HANDLERs.
mysql-test/r/information_schema.result:
Additional test for WL#3726 "DDL locking for all metadata
objects". Check that we use high-priority metadata lock requests
when filling I_S tables.
Rearrange tests to match 6.0 better (fewer merge conflicts).
mysql-test/r/kill.result:
Added tests checking that DDL and DML statements waiting for
metadata locks can be interrupted by KILL command.
mysql-test/r/lock.result:
One no longer is allowed to do DROP VIEW under LOCK TABLES even if
this view is locked by LOCK TABLES. The problem is that in such
situation write locks on view are not mutually exclusive so
upgrading metadata lock which is required for dropping of view
will lead to deadlock.
mysql-test/r/partition_column_prune.result:
Update results (same results in 6.0), WL#3726
mysql-test/r/partition_pruning.result:
Update results (same results in 6.0), WL#3726
mysql-test/r/ps_ddl.result:
We no longer invalidate prepared CREATE TABLE ... SELECT statement
if target table changes. This is OK since it is not strictly
necessary.
The first change is wrong, is caused by FLUSH TABLE
now flushing all unused tables. This is a regression that
Dmitri fixed in 6.0 in a follow up patch.
mysql-test/r/sp.result:
Under LOCK TABLES we no longer allow accessing views which were
not explicitly locked. To access view we need to obtain metadata
lock on it and doing this under LOCK TABLES may lead to deadlocks.
mysql-test/r/view.result:
One no longer is allowed to do DROP VIEW under LOCK TABLES even if
this view is locked by LOCK TABLES. The problem is that in such
situation even "write locks" on view are not mutually exclusive so
upgrading metadata lock which is required for dropping of view
will lead to deadlock
mysql-test/r/view_grant.result:
ALTER VIEW implementation was changed to open a view only after
checking that user which does alter has appropriate privileges on
it. This means that in case when user's privileges are
insufficient for this we won't check that new view definer is the
same as original one or user performing alter has SUPER privilege.
Adjusted test case accordingly.
mysql-test/r/view_multi.result:
Added test case for bug#25144 "replication / binlog with view
breaks".
mysql-test/suite/rpl/t/disabled.def:
Disable test for deprecated features (they don't work with new MDL).
mysql-test/t/create.test:
Adjusted test case after change in implementation of CREATE TABLE
... SELECT. We no longer have special check in open_table() which
catches the case when we select from the table created. Instead we
rely on unique_table() call which happens after opening and
locking all tables.
mysql-test/t/disabled.def:
Disable merge.test, subject of WL#4144
mysql-test/t/flush.test:
FLUSH TABLES WITH READ LOCK can no longer happen under LOCK
TABLES. Updated test accordingly.
mysql-test/t/flush_table.test:
Under LOCK TABLES we no longer allow to do FLUSH TABLES for tables
locked for read. Updated test accordingly.
mysql-test/t/information_schema.test:
Additional test for WL#3726 "DDL locking for all metadata
objects". Check that we use high-priority metadata lock requests
when filling I_S tables.
Rearrange the results for easier merges with 6.0.
mysql-test/t/kill.test:
Added tests checking that DDL and DML statements waiting for
metadata locks can be interrupted by KILL command.
mysql-test/t/lock.test:
One no longer is allowed to do DROP VIEW under LOCK TABLES even if
this view is locked by LOCK TABLES. The problem is that in such
situation write locks on view are not mutually exclusive so
upgrading metadata lock which is required for dropping of view
will lead to deadlock.
mysql-test/t/lock_multi.test:
Adjusted test case to the changes of status in various places
caused by change in implementation FLUSH TABLES WITH READ LOCK,
which is now takes global metadata lock before flushing tables and
therefore waits on at these places.
mysql-test/t/ps_ddl.test:
We no longer invalidate prepared CREATE TABLE ... SELECT statement
if target table changes. This is OK since it is not strictly
necessary.
The first change is wrong, is caused by FLUSH TABLE
now flushing all unused tables. This is a regression that
Dmitri fixed in 6.0 in a follow up patch.
mysql-test/t/sp.test:
Under LOCK TABLES we no longer allow accessing views which were
not explicitly locked. To access view we need to obtain metadata
lock on it and doing this under LOCK TABLES may lead to deadlocks.
mysql-test/t/trigger_notembedded.test:
Adjusted test case to the changes of status in various places
caused by change in implementation FLUSH TABLES WITH READ LOCK,
which is now takes global metadata lock before flushing tables and
therefore waits on at these places.
mysql-test/t/view.test:
One no longer is allowed to do DROP VIEW under LOCK TABLES even if
this view is locked by LOCK TABLES. The problem is that in such
situation even "write locks" on view are not mutually exclusive so
upgrading metadata lock which is required for dropping of view
will lead to deadlock.
mysql-test/t/view_grant.test:
ALTER VIEW implementation was changed to open a view only after
checking that user which does alter has appropriate privileges on
it. This means that in case when user's privileges are
insufficient for this we won't check that new view definer is the
same as original one or user performing alter has SUPER privilege.
Adjusted test case accordingly.
mysql-test/t/view_multi.test:
Added test case for bug#25144 "replication / binlog with view
breaks".
sql/CMakeLists.txt:
Added mdl.cc to the list of files needed for building of server.
sql/Makefile.am:
Added files implementing new meta-data locking subsystem to the
server.
sql/event_db_repository.cc:
Allocate metadata lock requests objects (MDL_LOCK) on execution
memory root in cases when TABLE_LIST objects is also allocated
there or on stack.
sql/ha_ndbcluster.cc:
Adjusted code to work nicely with new metadata locking subsystem.
close_cached_tables() no longer has wait_for_placeholder argument.
Instead of relying on this parameter and related behavior FLUSH
TABLES WITH READ LOCK now takes global shared metadata lock.
sql/ha_ndbcluster_binlog.cc:
Adjusted code to work with new metadata locking subsystem.
close_cached_tables() no longer has wait_for_placeholder argument.
Instead of relying on this parameter and related behavior FLUSH
TABLES WITH READ LOCK now takes global shared metadata lock.
sql/handler.cc:
update_frm_version():
Directly update TABLE_SHARE::mysql_version member instead of
going through all TABLE instances for this table (old code was a
legacy from pre-table-definition-cache days).
sql/lock.cc:
Use new metadata locking subsystem. Threw away most of functions
related to name locking as now one is supposed to use metadata
locking API instead. In lock_global_read_lock() and
unlock_global_read_lock() in order to avoid problems with global
read lock sneaking in at the moment when we perform FLUSH TABLES
or ALTER TABLE under LOCK TABLES and when tables being reopened
are protected only by metadata locks we also have to take global
shared meta data lock.
sql/log_event.cc:
Adjusted code to work with new metadata locking subsystem. For
tables open by slave thread for applying RBR events allocate
memory for lock request object in the same chunk of memory as
TABLE_LIST objects for them. In order to ensure that we keep these
objects around until tables are open always close tables before
calling Relay_log_info::clear_tables_to_lock(). Use new auxiliary
Relay_log_info::slave_close_thread_tables() method to enforce
this.
sql/log_event_old.cc:
Adjusted code to work with new metadata locking subsystem. Since
for tables open by slave thread for applying RBR events memory for
lock request object is allocated in the same chunk of memory as
TABLE_LIST objects for them we have to ensure that we keep these
objects around until tables are open. To ensure this we always
close tables before calling
Relay_log_info::clear_tables_to_lock(). To enfore this we use
new auxiliary Relay_log_info::slave_close_thread_tables()
method.
sql/mdl.cc:
Implemented new metadata locking subsystem and API described in
WL3726 "DDL locking for all metadata objects".
sql/mdl.h:
Implemented new metadata locking subsystem and API described in
WL3726 "DDL locking for all metadata objects".
sql/mysql_priv.h:
- close_thread_tables()/close_tables_for_reopen() now has one more
argument which indicates that metadata locks should be released
but not removed from the context in order to be used later in
mdl_wait_for_locks() and tdc_wait_for_old_version().
- close_cached_table() routine is no longer public.
- Thread waiting in wait_while_table_is_used() can be now killed
so this function returns boolean to make caller aware of such
situation.
- We no longer have table cache as separate entity instead used
and unused TABLE instances are linked to TABLE_SHARE objects in
table definition cache.
- Now third argument of open_table() is also used for requesting
table repair or auto-discovery of table's new definition. So its
type was changed from bool to enum.
- Added tdc_open_view() function for opening view by getting its
definition from disk (and table cache in future).
- reopen_name_locked_table() no longer needs "link_in" argument as
now we have exclusive metadata locks instead of dummy TABLE
instances when this function is called.
- find_locked_table() now takes head of list of TABLE instances
instead of always scanning through THD::open_tables list. Also
added find_write_locked_table() auxiliary.
- reopen_tables(), close_cached_tables() no longer have
mark_share_as_old and wait_for_placeholder arguments. Instead of
relying on this parameters and related behavior FLUSH TABLES
WITH READ LOCK now takes global shared metadata lock.
- We no longer need drop_locked_tables() and
abort_locked_tables().
- mysql_ha_rm_tables() now always assume that LOCK_open is not
acquired by caller.
- Added notify_thread_having_shared_lock() callback invoked by
metadata locking subsystem when acquiring an exclusive lock, for
each thread that has a conflicting shared metadata lock.
- Introduced expel_table_from_cache() as replacement for
remove_table_from_cache() (the main difference is that this new
function assumes that caller follows metadata locking protocol
and never waits).
- Threw away most of functions related to name locking. One should
use new metadata locking subsystem and API instead.
sql/mysqld.cc:
Got rid of call initializing/deinitializing table cache since now
it is embedded into table definition cache. Added calls for
initializing/ deinitializing metadata locking subsystem.
sql/rpl_rli.cc:
Introduced auxiliary Relay_log_info::slave_close_thread_tables()
method which is used for enforcing that we always close tables
open for RBR before deallocating TABLE_LIST elements and MDL_LOCK
objects for them.
sql/rpl_rli.h:
Introduced auxiliary Relay_log_info::slave_close_thread_tables()
method which is used for enforcing that we always close tables
open for RBR before deallocating TABLE_LIST elements and MDL_LOCK
objects for them.
sql/set_var.cc:
close_cached_tables() no longer has wait_for_placeholder argument.
Instead of relying on this parameter and related behavior FLUSH
TABLES WITH READ LOCK now takes global shared metadata lock.
sql/sp_head.cc:
For tables added to the statement's table list by prelocking
algorithm we allocate these objects either on the same memory as
corresponding table list elements or on THD::locked_tables_root
(if we are building table list for LOCK TABLES).
sql/sql_acl.cc:
Allocate metadata lock requests objects (MDL_LOCK) on execution
memory root in cases when we use stack TABLE_LIST objects to open
tables. Got rid of redundant code by using unlock_locked_tables()
function.
sql/sql_base.cc:
Changed code to use new MDL subsystem. Got rid of separate table
cache. Now used and unused TABLE instances are linked to the
TABLE_SHAREs in table definition cache.
check_unused():
Adjusted code to the fact that we no longer have separate table
cache. Removed dead code.
table_def_free():
Free TABLE instances referenced from TABLE_SHARE objects before
destroying table definition cache.
get_table_share():
Added assert which ensures that noone will be able to access
table (and its share) without acquiring some kind of metadata
lock first.
close_handle_and_leave_table_as_lock():
Adjusted code to the fact that TABLE instances now are linked to
list in TABLE_SHARE.
list_open_tables():
Changed this function to use table definition cache instead of
table cache.
free_cache_entry():
Unlink freed TABLE elements from the list of all TABLE instances
for the table in TABLE_SHARE.
kill_delayed_thread_for_table():
Added auxiliary for killing delayed insert threads for
particular table.
close_cached_tables():
Got rid of wait_for_refresh argument as we now rely on global
shared metadata lock to prevent FLUSH WITH READ LOCK sneaking in
when we are reopening tables. Heavily reworked this function to
use new MDL code and not to rely on separate table cache entity.
close_open_tables():
We no longer have separate table cache.
close_thread_tables():
Release metadata locks after closing all tables. Added skip_mdl
argument which allows us not to remove metadata lock requests
from the context in case when we are going to use this requests
later in mdl_wait_for_locks() and tdc_wait_for_old_versions().
close_thread_table()/close_table_for_reopen():
Since we no longer have separate table cache and all TABLE
instances are linked to TABLE_SHARE objects in table definition
cache we have to link/unlink TABLE object to/from appropriate
lists in the share.
name_lock_locked_table():
Moved redundant code to find_write_locked_table() function and
adjusted code to the fact that wait_while_table_is_used() can
now return with an error if our thread is killed.
reopen_table_entry():
We no longer need "link_in" argument as with MDL we no longer
call this function with dummy TABLE object pre-allocated and
added to the THD::open_tables. Also now we add newly-open TABLE
instance to the list of share's used TABLE instances.
table_cache_insert_placeholder():
Got rid of name-locking legacy.
lock_table_name_if_not_cached():
Moved to sql_table.cc the only place where it is used. It was
also reimplemented using new MDL API.
open_table():
- Reworked this function to use new MDL subsystem.
- Changed code to deal with table definition cache directly
instead of going through separate table cache.
- Now third argument is also used for requesting table repair
or auto-discovery of table's new definition. So its type was
changed from bool to enum.
find_locked_table()/find_write_locked_table():
Accept head of list of TABLE objects as first argument and use
this list instead of always searching in THD::open_tables list.
Also added auxiliary for finding write-locked locked tables.
reopen_table():
Adjusted function to work with new MDL subsystem and to properly
manuipulate with lists of used/unused TABLE instaces in
TABLE_SHARE.
reopen_tables():
Removed mark_share_as_old parameter. Instead of relying on it
and related behavior FLUSH TABLES WITH READ LOCK now takes
global shared metadata lock. Changed code after removing
separate table cache.
drop_locked_tables()/abort_locked_tables():
Got rid of functions which are no longer needed.
unlock_locked_tables():
Moved this function from sql_parse.cc and changed it to release
memory which was used for allocating metadata lock requests for
tables open and locked by LOCK TABLES.
tdc_open_view():
Intoduced function for opening a view by getting its definition
from disk (and table cache in future).
reopen_table_entry():
Introduced function for opening table definitions while holding
exclusive metatadata lock on it.
open_unireg_entry():
Got rid of this function. Most of its functionality is relocated
to open_table() and open_table_fini() functions, and some of it
to reopen_table_entry() and tdc_open_view(). Also code
resposible for auto-repair and auto-discovery of tables was
moved to separate function.
open_table_entry_fini():
Introduced function which contains common actions which finalize
process of TABLE object creation.
auto_repair_table():
Moved code responsible for auto-repair of table being opened
here.
handle_failed_open_table_attempt()
Moved code responsible for handling failing attempt to open
table to one place (retry due to lock conflict/old version,
auto-discovery and repair).
open_tables():
- Flush open HANDLER tables if they have old version of if there
is conflicting metadata lock against them (before this moment
we had this code in open_table()).
- When we open view which should be processed via derived table
on the second execution of prepared statement or stored
routine we still should call open_table() for it in order to
obtain metadata lock on it and prepare its security context.
- In cases when we discover that some special handling of
failure to open table is needed call
handle_failed_open_table_attempt() which handles all such
scenarios.
open_ltable():
Handling of various special scenarios of failure to open a table
was moved to separate handle_failed_open_table_attempt()
function.
remove_db_from_cache():
Removed this function as it is no longer used.
notify_thread_having_shared_lock():
Added callback which is invoked by MDL subsystem when acquiring
an exclusive lock, for each thread that has a conflicting shared
metadata lock.
expel_table_from_cache():
Introduced function for removing unused TABLE instances. Unlike
remove_table_from_cache() it relies on caller following MDL
protocol and having appropriate locks when calling it and thus
does not do any waiting if table is still in use.
tdc_wait_for_old_version():
Added function which allows open_tables() to wait in cases when
we discover that we should back-off due to presence of old
version of table.
abort_and_upgrade_lock():
Use new MDL calls.
mysql_wait_completed_table():
Got rid of unused function.
open_system_tables_for_read/for_update()/performance_schema_table():
Allocate MDL_LOCK objects on execution memory root in cases when
TABLE_LIST objects for corresponding tables is allocated on
stack.
close_performance_schema_table():
Release metadata locks after closing tables.
******
Use I_P_List for free/used tables list in the table share.
sql/sql_binlog.cc:
Use Relay_log_info::slave_close_thread_tables() method to enforce
that we always close tables open for RBR before deallocating
TABLE_LIST elements and MDL_LOCK objects for them.
sql/sql_class.cc:
Added meta-data locking contexts as part of Open_tables_state
context. Also introduced THD::locked_tables_root memory root
which is to be used for allocating MDL_LOCK objects for tables in
LOCK TABLES statement (end of lifetime for such objects is UNLOCK
TABLES so we can't use statement or execution root for them).
sql/sql_class.h:
Added meta-data locking contexts as part of Open_tables_state
context. Also introduced THD::locked_tables_root memory root
which is to be used for allocating MDL_LOCK objects for tables in
LOCK TABLES statement (end of lifetime for such objects is UNLOCK
TABLES so we can't use statement or execution root for them).
Note: handler_mdl_context and locked_tables_root and
mdl_el_root will be removed by subsequent patches.
sql/sql_db.cc:
mysql_rm_db() does not really need to call remove_db_from_cache()
as it drops each table in the database using
mysql_rm_table_part2(), which performs all necessary operations on
table (definition) cache.
sql/sql_delete.cc:
Use the new metadata locking API for TRUNCATE.
sql/sql_handler.cc:
Changed HANDLER implementation to use new metadata locking
subsystem. Note that MDL_LOCK objects for HANDLER tables are
allocated in the same chunk of heap memory as TABLE_LIST object
for those tables.
sql/sql_insert.cc:
mysql_insert():
find_locked_table() now takes head of list of TABLE object as
its argument instead of always scanning through THD::open_tables
list.
handle_delayed_insert():
Allocate metadata lock request object for table open by delayed
insert thread on execution memroot. create_table_from_items():
We no longer allocate dummy TABLE objects for tables being
created if they don't exist. As consequence
reopen_name_locked_table() no longer has link_in argument.
open_table() now has one more argument which is not relevant for
temporary tables.
sql/sql_parse.cc:
- Moved unlock_locked_tables() routine to sql_base.cc and made
available it in other files. Got rid of some redundant code by
using this function.
- Replaced boolean TABLE_LIST::create member with enum
open_table_type member.
- Use special memory root for allocating MDL_LOCK objects for
tables open and locked by LOCK TABLES (these object should live
till UNLOCK TABLES so we can't allocate them on statement nor
execution memory root). Also properly set metadata lock
upgradability attribure for those tables.
- Under LOCK TABLES it is no longer allowed to flush tables which
are not write-locked as this breaks metadata locking protocol
and thus potentially might lead to deadlock.
- Added auxiliary adjust_mdl_locks_upgradability() function.
sql/sql_partition.cc:
Adjusted code to the fact that reopen_tables() no longer has
"mark_share_as_old" argument. Got rid of comments which are no
longer true.
sql/sql_plist.h:
Added I_P_List template class for parametrized intrusive doubly
linked lists and I_P_List_iterator for corresponding iterator.
Unlike for I_List<> list elements of such list can participate in
several lists. Unlike List<> such lists are doubly-linked and
intrusive.
sql/sql_plugin.cc:
Allocate metadata lock requests objects (MDL_LOCK) on execution
memory root in cases when we use stack TABLE_LIST objects to open
tables.
sql/sql_prepare.cc:
Replaced boolean TABLE_LIST::create member with enum
open_table_type member. This allows easily handle situation in
which instead of opening the table we want only to take exclusive
metadata lock on it.
sql/sql_rename.cc:
Use new metadata locking subsystem in implementation of RENAME
TABLE.
sql/sql_servers.cc:
Allocate metadata lock requests objects (MDL_LOCK) on execution
memory root in cases when we use stack TABLE_LIST objects to open
tables. Got rid of redundant code by using unlock_locked_tables()
function.
sql/sql_show.cc:
Acquire shared metadata lock when we are getting information for
I_S table directly from TABLE_SHARE without doing full-blown table
open. We use high priority lock request in this situation in
order to avoid deadlocks.
Also allocate metadata lock requests objects (MDL_LOCK) on
execution memory root in cases when TABLE_LIST objects are also
allocated there
sql/sql_table.cc:
mysql_rm_table():
Removed comment which is no longer relevant.
mysql_rm_table_part2():
Now caller of mysql_ha_rm_tables() should not own LOCK_open.
Adjusted code to use new metadata locking subsystem instead of
name-locks.
lock_table_name_if_not_cached():
Moved this function from sql_base.cc to this file and
reimplemented it using metadata locking API.
mysql_create_table():
Adjusted code to use new MDL API.
wait_while_table_is_used():
Changed function to use new MDL subsystem. Made thread waiting
in it killable (this also led to introduction of return value so
caller can distinguish successful executions from situations
when waiting was aborted).
close_cached_tables():
Thread waiting in this function is killable now. As result it
has return value for distinguishing between succes and failure.
Got rid of redundant boradcast_refresh() call.
prepare_for_repair():
Use MDL subsystem instead of name-locks.
mysql_admin_table():
mysql_ha_rm_tables() now always assumes that caller doesn't own
LOCK_open.
mysql_repair_table():
We should mark all elements of table list as requiring
upgradable metadata locks.
mysql_create_table_like():
Use new MDL subsystem instead of name-locks.
create_temporary_tables():
We don't need to obtain metadata locks when creating temporary
table.
mysql_fast_or_online_alter_table():
Thread waiting in wait_while_table_is_used() is now killable.
mysql_alter_table():
Adjusted code to work with new MDL subsystem and to the fact
that threads waiting in what_while_table_is_used() and
close_cached_table() are now killable.
sql/sql_test.cc:
We no longer have separate table cache. TABLE instances are now
associated with/linked to TABLE_SHARE objects in table definition
cache.
sql/sql_trigger.cc:
Adjusted code to work with new metadata locking subsystem. Also
reopen_tables() no longer has mark_share_as_old argument (Instead
of relying on this parameter and related behavior FLUSH TABLES
WITH READ LOCK now takes global shared metadata lock).
sql/sql_udf.cc:
Allocate metadata lock requests objects (MDL_LOCK) on execution
memory root in cases when we use stack TABLE_LIST objects to open
tables.
sql/sql_update.cc:
Adjusted code to work with new meta-data locking subsystem.
sql/sql_view.cc:
Added proper meta-data locking to implementations of
CREATE/ALTER/DROP VIEW statements. Now we obtain exclusive
meta-data lock on a view before creating/ changing/dropping it.
This ensures that all concurrent statements that use this view
will finish before our statement will proceed and therefore we
will get correct order of statements in the binary log.
Also ensure that TABLE_LIST::mdl_upgradable attribute is properly
propagated for underlying tables of view.
sql/table.cc:
Added auxiliary alloc_mdl_locks() function for allocating metadata
lock request objects for all elements of table list.
sql/table.h:
TABLE_SHARE:
Got rid of unused members. Introduced members for storing lists
of used and unused TABLE objects for this share.
TABLE:
Added members for linking TABLE objects into per-share lists of
used and unused TABLE instances. Added member for holding
pointer to metadata lock for this table.
TABLE_LIST:
Replaced boolean TABLE_LIST::create member with enum
open_table_type member. This allows easily handle situation in
which instead of opening the table we want only to take
exclusive meta-data lock on it (we need this in order to handle
ALTER VIEW and CREATE VIEW statements).
Introduced new mdl_upgradable member for marking elements of
table list for which we need to take upgradable shared metadata
lock instead of plain shared metadata lock. Added pointer for
holding pointer to MDL_LOCK for the table.
Added auxiliary alloc_mdl_locks() function for allocating metadata
lock requests objects for all elements of table list. Added
auxiliary set_all_mdl_upgradable() function for marking all
elements in table list as requiring upgradable metadata locks.
storage/myisammrg/ha_myisammrg.cc:
Allocate MDL_LOCK objects for underlying tables of MERGE table.
To be reworked once Ingo pushes his patch for WL4144.
- Moved some code from innodb_plugin to xtradb, to ensure that all tests runs
- Did changes in pbxt and maria storage engines becasue of changes in thd->query
- Reverted wrong code in sql_table.cc for how ROW_FORMAT is used.
This is a re-commit of Monty's merge to eliminate an extra commit from
MySQL-5.1.42 that was accidentally included in the merge.
This is a merge of the MySQL 5.1.41 clone-off (clone-5.1.41-build). In
case there are any extra changes done before final MySQL 5.1.41
release, these will need to be merged later before MariaDB 5.1.41
release.
Conflicts
=========
Text conflict in .bzr-mysql/default.conf
Text conflict in libmysqld/CMakeLists.txt
Text conflict in libmysqld/Makefile.am
Text conflict in mysql-test/collections/default.experimental
Text conflict in mysql-test/extra/rpl_tests/rpl_row_sp006.test
Text conflict in mysql-test/suite/binlog/r/binlog_tmp_table.result
Text conflict in mysql-test/suite/rpl/r/rpl_loaddata.result
Text conflict in mysql-test/suite/rpl/r/rpl_loaddata_fatal.result
Text conflict in mysql-test/suite/rpl/r/rpl_row_create_table.result
Text conflict in mysql-test/suite/rpl/r/rpl_row_sp006_InnoDB.result
Text conflict in mysql-test/suite/rpl/r/rpl_stm_log.result
Text conflict in mysql-test/suite/rpl_ndb/r/rpl_ndb_circular_simplex.result
Text conflict in mysql-test/suite/rpl_ndb/r/rpl_ndb_sp006.result
Text conflict in mysql-test/t/mysqlbinlog.test
Text conflict in sql/CMakeLists.txt
Text conflict in sql/Makefile.am
Text conflict in sql/log_event_old.cc
Text conflict in sql/rpl_rli.cc
Text conflict in sql/slave.cc
Text conflict in sql/sql_binlog.cc
Text conflict in sql/sql_lex.h
21 conflicts encountered.
NOTE
====
mysql-5.1-rpl-merge has been made a mirror of mysql-next-mr:
- "mysql-5.1-rpl-merge$ bzr pull ../mysql-next-mr"
This is the first cset (merge/...) committed after pulling
from mysql-next-mr.
The BINLOG statement was sharing too much code with the slave SQL thread, introduced with
the patch for Bug#32407. This caused statements to be logged with the wrong server_id, the
id stored inside the events of the BINLOG statement rather than the id of the running
server.
Fix by rearranging code a bit so that only relevant parts of the code are executed by
the BINLOG statement, and the server_id of the server executing the statements will
not be overrided by the server_id stored in the 'format description BINLOG statement'.
mysql-test/extra/binlog_tests/binlog.test:
Added test to verify if the server_id stored in the 'format
description BINLOG statement' will override the server_id
of the server executing the statements.
mysql-test/suite/binlog/r/binlog_row_binlog.result:
Test result for bug#46640
mysql-test/suite/binlog/r/binlog_stm_binlog.result:
Test result for bug#46640
sql/log_event.cc:
Moved rows_event_stmt_clean() call from update_pos() to apply_event(). This in any case
makes more sense, and is needed as update_pos() is no longer called when executing
BINLOG statements.
Moved setting of rli->relay_log.description_event_for_exec from
Format_description_log_event::do_update_pos() to
Format_description_log_event::do_apply_event()
sql/log_event_old.cc:
Moved rows_event_stmt_clean() call from update_pos() to apply_event(). This in any case
makes more sense, and is needed as update_pos() is no longer called when executing
BINLOG statements.
sql/slave.cc:
The skip flag is no longer needed, as the code path for BINLOG statement has been
cleaned up.
sql/sql_binlog.cc:
Don't invoke the update_pos() code path for the BINLOG statement, as it contains code
that is redundant and/or harmful (especially setting thd->server_id).
NOTE: Backporting the patch to next-mr.
The fix proposed in BUG#35542 and BUG#31665 introduces a performance issue
when fsyncing the master.info, relay.info and relay-log.bin* after #th events.
Although such solution has been proposed to reduce the probability of corrupted
files due to a slave-crash, the performance penalty introduced by it has
made the approach impractical for highly intensive workloads.
In a nutshell, the option --syn-relay-log proposed in BUG#35542 and BUG#31665
simultaneously fsyncs master.info, relay-log.info and relay-log.bin* and
this is the main source of performance issues.
This patch introduces new options that give more control to the user on
what should be fsynced and how often:
1) (--sync-master-info, integer) which syncs the master.info after #th event;
2) (--sync-relay-log, integer) which syncs the relay-log.bin* after #th
events.
3) (--sync-relay-log-info, integer) which syncs the relay.info after #th
transactions.
To provide both performance and increased reliability, we recommend the following
setup:
1) --sync-master-info = 0 eventually the operating system will fsync it;
2) --sync-relay-log = 0 eventually the operating system will fsync it;
3) --sync-relay-log-info = 1 fsyncs it after every transaction;
Notice, that the previous setup does not reduce the probability of
corrupted master.info and relay-log.bin*. To overcome the issue, this patch also
introduces a recovery mechanism that right after restart throws away relay-log.bin*
retrieved from a master and updates the master.info based on the relay.info:
4) (--relay-log-recovery, boolean) which enables a recovery mechanism that
throws away relay-log.bin* after a crash.
However, it can only recover the incorrect binlog file and position in master.info,
if other informations (host, port password, etc) are corrupted or incorrect,
then this recovery mechanism will fail to work.
Adding comments to some of the high-level functions in replication.
sql/log_event.h:
Fixed some mistakes in comments.
sql/repl_failsafe.cc:
Added comment for show_slave_hosts()
sql/slave.cc:
Added comment for show_master_info(), handle_slave_[sql|io](), and next_event()
sql/sql_binlog.cc:
Added @param comment.
sql/sql_lex.h:
Added comment for st_lex_master_info.
sql/sql_repl.cc:
Added comments for functions executing a statement:
PURGE BINARY LOGS
START SLAVE
STOP SLAVE
RESET SLAVE
CHANGE MASTER
RESET MASTER
SHOW BINLOG EVENTS
SHOW MASTER STATUS
SHOW BINARY LOGS
The problem was because the event allocated in mysql_client_binlog_statement
was not freed when an error occured while applying the event.
sql/sql_binlog.cc:
Delete the event if apply it failed
tree with main.
sql/sql_binlog.cc:
Adding code to free memory after execution of BINLOG statement.
It caused a memory leak in the case that the execution failed
for any reason.
sql/sql_class.cc:
Since rli_fake is checked for NULL at various occations to mean
that no rli_fake is assigned, NULL is assigned to rli_fake after
deleting the instance.
sql/rpl_rli.cc:
Adding variable to mark an instance of Relay_log_info as fake.
sql/rpl_rli.h:
Adding variable to mark an instance of Relay_log_info as fake.
sql/slave.cc:
Not printing debug information if we are working with a fake
instance of Relay_log_info. This because the result of calling
update is nonsense, and trying to print it generates valgrind
warnings.
sql/sql_binlog.cc:
Marking newly created instance of Relay_log_info as a fake instance.
into dl145h.mysql.com:/data0/mkindahl/mysql-5.1-rpl
include/my_base.h:
Auto merged
mysql-test/include/show_binlog_events2.inc:
Auto merged
mysql-test/suite/binlog/t/binlog_old_versions.test:
Auto merged
mysql-test/suite/rpl/r/rpl_row_tabledefs_2myisam.result:
Auto merged
mysql-test/suite/rpl/r/rpl_row_tabledefs_3innodb.result:
Auto merged
mysql-test/suite/rpl/t/rpl_loaddata_map-master.opt:
Auto merged
mysql-test/suite/rpl/t/rpl_loaddata_map-slave.opt:
Auto merged
sql/log_event_old.cc:
Auto merged
sql/sql_binlog.cc:
Auto merged
mysql-test/suite/rpl/r/rpl_loaddata_map.result:
Manual merge.
mysql-test/suite/rpl/r/rpl_user.result:
Manual merge.
mysql-test/suite/rpl/t/rpl_loaddata_map.test:
Manual merge.
mysql-test/suite/rpl/t/rpl_user.test:
Manual merge.
sql/log_event.cc:
Manual merge.
sql/rpl_record.cc:
Manual merge.
Main problem: mysql 5.1 cannot read binlogs from 4.1.
Subproblem 1: There is a mistake in sql_ex_info::init. The read_str()
function updates its first argument to point to the next character to
read. However, it is applied only to a copy of the buffer pointer, so the
real buffer pointer is not updated.
Fix 1: do not take a copy of the buffer pointer. The copy was needed
because sql_ex_info::init does not use the const attribute on some of its
arguments. So we add the const attribute, too.
Subproblem 2: The first BINLOG statement is asserted to be a
FORMAT_DESCRIPTION_LOG_EVENT, but 4.1 binlogs begin with START_EVENT_V3.
Fix 2: allow START_EVENT_V3 too.
mysql-test/suite/binlog/std_data/binlog_old_version_4_1.000001:
New BitKeeper file ``mysql-test/suite/binlog/std_data/binlog_old_version_4_1.000001''
mysql-test/suite/binlog/r/binlog_old_versions.result:
Updated result file.
mysql-test/suite/binlog/t/binlog_old_versions.test:
Added a test reading an old 4.1 binlog.
sql/log_event.cc:
1. Added const keyword at the following places:
- input buffer for pretty_print_str
- input buffer for write_str
- input buffer, end pointer, and return value from sql_ex_info::init
2. Fixed the bug by not taking a copy of buf before calling read_str in
sql_ex_info::init().
sql/log_event.h:
Added const keyword to fields of the sql_ex_info struct.
Added const keyword to arguments and return value of sql_ex_info::init
sql/sql_binlog.cc:
The first BINLOG statement must describe the format for future BINLOG
statements. Otherwise, we do not know how to read the BINLOG statement.
Problem: only FORMAT_DESCRIPTION_EVENT is currently allowed as the first
event. Binlogs from 4.1 begin with a START_EVENT_V3, which serves the
same purpose.
Fix: We now allow the first BINLOG statement to be a START_EVENT_V3, as
well as a FORMAT_DESCRIPTION_EVENT.
into dl145h.mysql.com:/data0/mkindahl/mysql-5.1-rpl-merge
mysql-test/include/have_multi_ndb.inc:
Auto merged
mysql-test/lib/mtr_cases.pl:
Auto merged
mysql-test/lib/mtr_report.pl:
Auto merged
mysql-test/suite/binlog/r/binlog_stm_blackhole.result:
Auto merged
mysql-test/suite/rpl_ndb/r/rpl_row_basic_7ndb.result:
Auto merged
mysql-test/suite/rpl_ndb/t/disabled.def:
Auto merged
sql/ha_ndbcluster_binlog.cc:
Auto merged
sql/log.cc:
Auto merged
sql/log_event_old.cc:
Auto merged
sql/mysql_priv.h:
Auto merged
sql/mysqld.cc:
Auto merged
sql/rpl_rli.cc:
Auto merged
sql/set_var.cc:
Auto merged
sql/set_var.h:
Auto merged
sql/slave.cc:
Auto merged
sql/sql_binlog.cc:
Auto merged
sql/sql_class.h:
Auto merged
sql/sql_string.cc:
Auto merged
mysql-test/Makefile.am:
SCCS merged
mysql-test/mysql-test-run.pl:
Manual merge.
mysql-test/suite/binlog/t/disabled.def:
Manual merge.
mysql-test/suite/rpl/r/rpl_row_tabledefs_2myisam.result:
Manual merge.
mysql-test/suite/rpl/r/rpl_row_tabledefs_3innodb.result:
Manual merge.
sql/log_event.cc:
Manual merge.
,
Problem: it is unsafe to read base64-printed events without first
reading the Format_description_log_event (FD). Currently, mysqlbinlog
cannot print the FD.
As a side effect, another bug has also been fixed: When mysqlbinlog
--start-position=X was specified, no ROLLBACK was printed. I changed
this, so that ROLLBACK is always printed.
This patch does several things:
- Format_description_log_event (FD) now print themselves in base64
format.
- mysqlbinlog is now able to print FD events. It has three modes:
--base64-output=auto Print row events in base64 output, and print
FD event. The FD event is printed even if
it is outside the range specified with
--start-position, because it would not be
safe to read row events otherwise. This is
the default.
--base64-output=always Like --base64-output=auto, but also print
base64 output for query events. This is
like the old --base64-output flag, which
is also a shorthand for
--base64-output=always
--base64-output=never Never print base64 output, generate error if
row events occur in binlog. This is
useful to suppress the FD event in binlogs
known not to contain row events (e.g.,
because BINLOG statement is unsafe,
requires root privileges, is not SQL, etc)
- the BINLOG statement now handles FD events correctly, by setting
the thread's rli's relay log's description_event_for_exec to the
loaded event.
In fact, executing a BINLOG statement is almost the same as reading
an event from a relay log. Before my patch, the code for this was
separated (exec_relay_log_event in slave.cc executes events from
the relay log, mysql_client_binlog_statement in sql_binlog.cc
executes BINLOG statements). I needed to augment
mysql_client_binlog_statement to do parts of what
exec_relay_log_event does. Hence, I did a small refactoring and
moved parts of exec_relay_log_event to a new function, which I
named apply_event_and_update_pos. apply_event_and_update_pos is
called both from exec_relay_log_event and from
mysql_client_binlog_statement.
- When a non-FD event is executed in a BINLOG statement, without
previously executing a FD event in a BINLOG statement, it generates
an error, because that's unsafe. I took a new error code for that:
ER_NO_FORMAT_DESCRIPTION_EVENT_BEFORE_BINLOG_STATEMENTS.
In order to get a decent error message containing the name of the
event, I added the class method char*
Log_event::get_type_str(Log_event_type type), which returns a
string name for the given Log_event_type. This is just like the
existing char* Log_event::get_type_str(), except it is a class
method that takes the log event type as parameter.
I also added PRE_GA_*_ROWS_LOG_EVENT to Log_event::get_type_str(),
so that names of old rows event are properly printed.
- When reading an event, I added a check that the event type is known
by the current Format_description_log_event. Without this, it may
crash on bad input (and I was struck by this several times).
- I patched the following test cases, which all contain BINLOG
statements for row events which must be preceded by BINLOG
statements for FD events:
- rpl_bug31076
While I was here, I fixed some small things in log_event.cc:
- replaced hard-coded 4 by EVENT_TYPE_OFFSET in 3 places
- replaced return by DBUG_VOID_RETURN in one place
- The name of the logfile can be '-' to indicate stdin. Before my
patch, the code just checked if the first character is '-'; now it
does a full strcmp(). Probably, all arguments that begin with a -
are already handled somewhere else as flags, but I still think it
is better that the code reflects what it is supposed to do, with as
little dependencies as possible on other parts of the code. If we
one day implement that all command line arguments after -- are
files (as most unix tools do), then we need this.
I also fixed the following in slave.cc:
- next_event() was declared twice, and queue_event was not static but
should be static (not used outside the file).
client/client_priv.h:
Declared the new option for base64 output.
client/mysqlbinlog.cc:
- Change from using the two-state command line option
"default/--base64-output" to the three-state
"--base64-output=[never|auto|always]"
- Print the FD event even if it is outside the --start-position range.
- Stop if a row event is about to be printed without a preceding FD
event.
- Minor fixes:
* changed 4 to EVENT_TYPE_OFFSET in some places
* Added comments
* before, "mysqlbinlog -xyz" read from stdin; now it does not
(only "mysqlbinlog -" reads stdin).
mysql-test/r/mysqlbinlog2.result:
Updated result file: mysqlbinlog now prints ROLLBACK always.
mysql-test/suite/binlog/t/disabled.def:
The test must be disabled since it reveals another bug: see BUG#33247.
mysql-test/suite/rpl/r/rpl_bug31076.result:
Updated result file
mysql-test/suite/rpl/r/rpl_row_mysqlbinlog.result:
Updated result file
mysql-test/suite/rpl/t/rpl_bug31076.test:
Had to add explicit Format_description_log_event before other BINLOG
statements
mysql-test/t/mysqlbinlog2.test:
we must suppress base64 output in result file because it contains a
timestamp
sql/log_event.cc:
- Made FD events able to print themselves
- Added check that the current FD event knows about the event type, when
an event is about to be read. (Hint to reviewers: I had to re-indent
a big block because of this; use diff -b)
* To get a decent error message, I also added a class method
const char* Log_event::get_type_str(Log_event_type)
which converts number to event type string without having a
Log_event object.
* Made Log_event::get_type_str aware of PRE_GA_*_ROWS_LOG_EVENT.
- Minor fixes:
* Changed return to DBUG_VOID_RETURN
sql/log_event.h:
- Declared enum to describe the three base64_output modes
- Use the enum instead of a flag
- Declare the new class method get_type_str (see log_event.cc)
sql/share/errmsg.txt:
Added error msg.
sql/slave.cc:
- Factored out part of exec_relay_log_event to the new function
apply_event_and_update_pos, because that code is needed when executing
BINLOG statements. (this is be functionally equivalent to the
previous code, except: (1) skipping events is now optional, controlled
by a parameter to the new function (2) the return value of
exec_relay_log_event has changed; see next item).
- Changed returned error value to always be 1. Before, it would return
the error value from apply_log_event, which was unnecessary. This
change is safe because the exact return value of exec_relay_log_event
is never examined; it is only tested to be ==0 or !=0.
- Added comments describing exec_relay_log_event and
apply_event_and_update_pos.
- Minor fixes:
* Removed duplicate declaration of next_event, made queue_event
static.
* Added doxygen code to include this file.
sql/slave.h:
Declared the new apply_event_and_update_pos
sql/sql_binlog.cc:
- Made mysql_binlog_statement set the current FD event when the given
event is an FD event. This entails using the new function
apply_event_and_update_pos from slave.cc instead of just calling the
ev->apply method.
- Made mysql_binlog_statement fail if the first BINLOG statement is not
an FD event.
mysql-test/suite/binlog/r/binlog_base64_flag.result:
New test file needs new result file
mysql-test/suite/binlog/t/binlog_base64_flag.test:
Added test case to verify that:
- my patch fixes the bug
- the new --base64-output flag works as expected
- base64 events not preceded by an FD event give an error
- an event of a type not known by the current FD event fails cleanly.
mysql-test/suite/binlog/std_data/binlog-bug32407.000001:
BitKeeper file /home/sven/bk/b32407-5.1-new-rpl-mysqlbinlog_base64/mysql-test/suite/binlog/std_data/binlog-bug32407.000001
cause ROLLBACK of statement", part 1. Review fixes.
Do not send OK/EOF packets to the client until we reached the end of
the current statement.
This is a consolidation, to keep the functionality that is shared by all
SQL statements in one place in the server.
Currently this functionality includes:
- close_thread_tables()
- log_slow_statement().
After this patch and the subsequent patch for Bug#12713, it shall also include:
- ha_autocommit_or_rollback()
- net_end_statement()
- query_cache_end_of_result().
In future it may also include:
- mysql_reset_thd_for_next_command().
include/mysql_com.h:
Rename now unused members of NET: no_send_ok, no_send_error, report_error.
These were server-specific variables related to the client/server
protocol. They have been made obsolete by this patch.
Previously the same members of NET were used to store the error message
both on the client and on the server.
The error message was stored in net.last_error (client: mysql->net.last_error,
server: thd->net.last_error).
The error code was stored in net.last_errno (client: mysql->net.last_errno,
server: thd->net.last_errno).
The server error code and message are now stored elsewhere
(in the Diagnostics_area), thus NET members are no longer used by the
server.
Rename last_error to client_last_error, last_errno to client_last_errno
to avoid potential bugs introduced by merges.
include/mysql_h.ic:
Update the ABI file to reflect a rename.
Renames do not break the binary compatibility.
libmysql/libmysql.c:
Rename last_error to client_last_error, last_errno to client_last_errno.
This is necessary to ensure no unnoticed bugs introduced by merged
changesets.
Remove net.report_error, net.no_send_ok, net.no_send_error.
libmysql/manager.c:
Rename net.last_errno to net.client_last_errno.
libmysqld/lib_sql.cc:
Rename net.last_errno to net.client_last_errno.
Update the embedded implementation of the client-server protocol to
reflect the refactoring of protocol.cc.
libmysqld/libmysqld.c:
Rename net.last_errno to net.client_last_errno.
mysql-test/r/events.result:
Update to reflect the change in mysql_rm_db(). Now we drop stored
routines and events for a given database name only if there
is a directory for this database name. ha_drop_database() and
query_cache_invalidate() are called likewise.
Previously we would attempt to drop routines/events even if database
directory was not found (it worked, since routines and events are stored
in tables). This fixes Bug 29958 "Weird message on DROP DATABASE if mysql.proc
does not exist".
The change was done because the previous code used to call send_ok()
twice, which led to an assertion failure when asserts against it were
added by this patch.
mysql-test/r/grant.result:
Fix the patch for Bug 16470, now FLUSH PRIVILEGES produces an error
if mysql.procs_priv is missing.
This fixes the assert that send_ok() must not called after send_error()
(the original patch for Bug 16470 was prone to this).
mysql-test/suite/rpl/r/rpl_row_tabledefs_2myisam.result:
Produce a more detailed error message.
mysql-test/suite/rpl/r/rpl_row_tabledefs_3innodb.result:
Produce a more detailed error message.
mysql-test/t/grant.test:
Update the test, now FLUSH PRIVILEGES returns an error if mysql.procs_priv
is missing.
server-tools/instance-manager/mysql_connection.cc:
Rename net.last_errno to net.client_last_errno.
sql/ha_ndbcluster_binlog.cc:
Add asserts.
Use getters to access statement status information.
Add a comment why run_query() is broken. Reset the diagnostics area
in the end of run_query() to fulfill the invariant that the diagnostics_area
is never assigned twice per statement (see the comment in the code
when this can happen). We still do not clear thd->is_fatal_error and
thd->is_slave_error, which may lead to bugs, I consider the whole affair
as something to be dealt with separately.
sql/ha_partition.cc:
fatal_error() doesn't set an error by itself. Perhaps we should
remove this method altogether and instead add a flag to my_error
to set thd->is_fatal_error property.
Meanwhile, this change is a part of inspection made to the entire source
code with the goal to ensure that fatal_error()
is always accompanied by my_error().
sql/item_func.cc:
There is no net.last_error anymore. Remove the obsolete assignment.
sql/log_event.cc:
Use getters to access statement error status information.
sql/log_event_old.cc:
Use getters to access statement error status information.
sql/mysqld.cc:
Previously, if a continue handler for an error was found, my_message_sql()
would not set an error in THD. Since the current statement
must be aborted in any case, find_handler() had a hack to assign
thd->net.report_error to 1.
Remove this hack. Set an error in my_message_sql() even if the continue
handler is found. The error will be cleared anyway when the handler
is executed. This is one action among many in this patch to ensure the
invariant that whenever thd->is_error() is TRUE, we have a message in
thd->main_da.message().
sql/net_serv.cc:
Use a full-blown my_error() in net_serv.cc to report an error,
instead of just setting net->last_errno. This ensures the invariant that
whenever thd->is_error() returns TRUE, we have a message in
thd->main_da.message().
Remove initialization of removed NET members.
sql/opt_range.cc:
Use my_error() instead of just raising thd->net.report_error.
This ensures the invariant that whenever thd->is_error() returns TRUE,
there is a message in thd->main_da.message().
sql/opt_sum.cc:
Move invocation of fatal_error() right next to the place where
we set the error message. That makes it easier to track that whenever
fatal_error() is called, there is a message in THD.
sql/protocol.cc:
Rename send_ok() and send_eof() to net_send_ok() and net_send_eof()
respectively. These functions write directly to the network and are not
for use anywhere outside the client/server protocol code.
Remove the code that was responsible for cases when either there is
no error code, or no error message, or both.
Instead the calling code ensures that they are always present. Asserts
are added to enforce the invariant.
Instead of a direct access to thd->server_status and thd->total_warn_count
use function parameters, since these from now on don't always come directly
from THD.
Introduce net_end_statement(), the single-entry-point replacement API for
send_ok(), send_eof() and net_send_error().
Implement Protocol::end_partial_result_set to use in select_send::abort()
when there is a continue handler.
sql/protocol.h:
Update declarations.
sql/repl_failsafe.cc:
Use getters to access statement status information in THD.
Rename net.last_error to net.client_last_error.
sql/rpl_record.cc:
Set an error message in prepare_record() if there is no default
value for the field -- later we do print this message to the client.
sql/rpl_rli.cc:
Use getters to access statement status information in THD.
sql/slave.cc:
In create_table_from_dump() (a common function that is used in
LOAD MASTER TABLE SQL statement and COM_LOAD_MASTER_DATA), instead of hacks
with no_send_ok, clear the diagnostics area when mysql_rm_table() succeeded.
Update has_temporary_error() to work correctly when no error is set.
This is the case when Incident_log_event is executed: it always returns
an error but does not set an error message.
Use getters to access error status information.
sql/sp_head.cc:
Instead of hacks with no_send_error, work through the diagnostics area
interface to suppress sending of OK/ERROR packets to the client.
Move query_cache_end_of_result before log_slow_statement(), similarly
to how it's done in dispatch_command().
sql/sp_rcontext.cc:
Remove hacks with assignment of thd->net.report_error, they are not
necessary any more (see the changes in mysqld.cc).
sql/sql_acl.cc:
Use getters to access error status information in THD.
sql/sql_base.cc:
Access thd->main_da.sql_errno() only if there is an error. This fixes
a bug when auto-discovery, that was effectively disabled under pre-locking.
sql/sql_binlog.cc:
Remove hacks with no_send_ok/no_send_error, they are not necessary
anymore: the caller is responsible for network communication.
sql/sql_cache.cc:
Disable sending of OK/ERROR/EOF packet in the end of dispatch_command
if the response has been served from the query cache. This raises the
question whether we should store EOF packet in the query cache at all,
or generate it anew for each statement (we should generate it anew), but
this is to be addressed separately.
sql/sql_class.cc:
Implement class Diagnostics_area. Please see comments in sql_class.h
for details.
Fix a subtle coding mistake in select_send::send_data: when on slave,
an error in Item::send() was ignored.
The problem became visible due to asserts that the diagnostics area is
never double assigned.
Remove initialization of removed NET members.
In select_send::abort() do not call select_send::send_eof(). This is
not inheritance-safe. Even if a stored procedure continue handler is
found, the current statement is aborted, not succeeded.
Instead introduce a Protocol API to send the required response,
Protocol::end_partial_result_set().
This simplifies implementation of select_send::send_eof(). No need
to add more asserts that there is no error, there is an assert inside
Diagnostics_area::set_ok_status() already.
Leave no trace of no_send_* in the code.
sql/sql_class.h:
Declare class Diagnostics_area.
Remove the hack with no_send_ok from
Substatement_state.
Provide inline implementations of send_ok/send_eof.
Add commetns.
sql/sql_connect.cc:
Remove hacks with no_send_error.
Since now an error in THD is always set if net->error, it's not necessary
to check both net->error and thd->is_error() in the do_command loop.
Use thd->main_da.message() instead of net->last_errno.
Remove the hack with is_slave_error in sys_init_connect. Since now we do not
reset the diagnostics area in net_send_error (it's reset at the beginning
of the next statement), we can access it safely even after
execute_init_command.
sql/sql_db.cc:
Update the code to satisfy the invariant that the diagnostics area is never
assigned twice.
Incidentally, this fixes Bug 29958 "Weird message on DROP DATABASE if
mysql.proc does not exist".
sql/sql_delete.cc:
Change multi-delete to abort in abort(), as per select_send protocol.
Fixes the merge error with the test for Bug 29136
sql/sql_derived.cc:
Use getters to access error information.
sql/sql_insert.cc:
Use getters to access error information.
sql-common/client.c:
Rename last_error to client_last_error, last_errno to client_last_errno.
sql/sql_parse.cc:
Remove hacks with no_send_error. Deploy net_end_statement().
The story of COM_SHUTDOWN is interesting. Long story short, the server
would become on its death's door, and only no_send_ok/no_send_error assigned
by send_ok()/net_send_error() would hide its babbling from the client.
First of all, COM_QUIT does not require a response. So, the comment saying
"Let's send a response to possible COM_QUIT" is not only groundless
(even mysqladmin shutdown/mysql_shutdown() doesn't send COM_QUIT after
COM_SHUTDOWN), it's plainly incorrect.
Secondly, besides this additional 'OK' packet to respond to a hypothetical
COM_QUIT, there was the following code in dispatch_command():
if (thd->killed)
thd->send_kill_message();
if (thd->is_error()
net_send_error(thd);
This worked out really funny for the thread through which COM_SHUTDOWN
was delivered: we would get COM_SHUTDOWN, say okay, say okay again,
kill everybody, get the kill signal ourselves, and then attempt to say
"Server shutdown in progress" to the client that is very likely long gone.
This all became visible when asserts were added that the Diagnostics_area
is not assigned twice.
Move query_cache_end_of_result() to the end of dispatch_command(), since
net_send_eof() has been moved there. This is safe, query_cache_end_of_result()
is a no-op if there is no started query in the cache.
Consistently use select_send interface to call abort() or send_eof()
depending on the operation result.
Remove thd->fatal_error() from reset_master(), it was a no-op.
in hacks with no_send_error woudl save us
from complete breakage of the client/server protocol.
Consistently use select_send::abort() whenever there is an error,
and select_send::send_eof() in case of success.
The issue became visible due to added asserts.
sql/sql_partition.cc:
Always set an error in THD whenever there is a call to fatal_error().
sql/sql_prepare.cc:
Deploy class Diagnostics_area.
Remove the unnecessary juggling with the protocol in
Select_fetch_protocol_binary::send_eof(). EOF packet format is
protocol-independent.
sql/sql_select.cc:
Call fatal_error() directly in opt_sum_query.
Call my_error() whenever we call thd->fatal_error().
sql/sql_servers.cc:
Use getters to access error information in THD.
sql/sql_show.cc:
Use getters to access error information in THD.
Add comments.
Call my_error() whenever we call fatal_error().
sql/sql_table.cc:
Replace hacks with no_send_ok with the interface of the diagnostics area.
Clear the error if ENOENT error in ha_delete_table().
sql/sql_update.cc:
Introduce multi_update::abort(), which is the proper way to abort a
multi-update. This fixes the merge conflict between this patch and
the patch for Bug 29136.
sql/table.cc:
Use a getter to access error information in THD.
sql/tztime.cc:
Use a getter to access error information in THD.
