Backport from mysql-5.5 to mysql-5.1
Bug# 19699237: UNINITIALIZED VARIABLE IN
ITEM_FIELD::STR_RESULT LEADS TO INCORRECT
BEHAVIOR
ISSUE:
------
When the following conditions are satisfied in a query, a
server crash occurs:
a) Two rows are compared using a NULL-safe equal-to operator.
b) Each of these rows belong to different charsets.
SOLUTION:
---------
When one charset is converted to another for comparision,
the constructor of "Item_func_conv_charset" is called.
This will attempt to use the Item_cache if the string is a
constant. This check succeeds because the "used_table_map"
of the Item_cache class is never set to the correct value.
Since it is mistakenly assumed to be a constant, it tries
to fetch the relevant null value related fields which are
yet to be initialized. This results in valgrind issues
and wrong results.
The fix is to update the "used_table_map" of "Item_cache".
This will allow "Item_func_conv_charset" to realise that
this is not a constant.
Problem: UDF doesn't handle the arguments properly when they
are of string type due to a misplaced break.
The length of arguments is also not set properly
when the argument is NULL.
Solution: Fixed the code by putting the break at right place
and setting the argument length to zero when the
argument is NULL.
Backport from mysql-5.5 to mysql-5.1
Bug#19880368 : GROUP_CONCAT CRASHES AFTER DUMP_LEAF_KEY
Problem:
find_order_by_list does not update the address of order_item
correctly after resolving.
Solution:
Change the ref_by address for a order_by field if its
SUM_FUNC_ITEM to the address of the field present in
all_fields.
Backport from mysql-5.5 to mysql-5.1
Bug #19612819 : FILESORT: ASSERTION FAILED: POS->FIELD != 0 || POS->ITEM != 0
Problem:
While getting the temp table field for a REF_ITEM
make_sortorder is using the real_item. As a result
server fails later with an assert.
Solution:
Do not use real_item to get the temp table field.
Instead use the REF_ITEM itself as temp table fields
are created for REF_ITEM not the real_item.
Backport from mysql-5.5 to mysql-5.1 of:
Bug19770858: MYSQLD CAN BE DRIVEN TO OOM WITH TWO SIMPLE SESSION VARS
The problem was that the maximum value of the transaction_prealloc_size
session system variable was ULONG_MAX which meant that it was possible
to cause the server to allocate excessive amounts of memory.
This patch fixes the problem by reducing the maxmimum value of
transaction_prealloc_size and transaction_alloc_block_size down
to 128K.
Note that transactions will still be able to allocate more than
128K if needed, this patch just reduces the amount that can be
preallocated - as well as the maximum size of the incremental
allocation blocks.
(cherry picked from commit 540c9f7ebb428bbf9ec028feabe1f7f919fdefd9)
Conflicts:
mysql-test/suite/sys_vars/r/transaction_alloc_block_size_basic.result
mysql-test/suite/sys_vars/r/transaction_alloc_block_size_basic_64.result
mysql-test/suite/sys_vars/t/disabled.def
mysql-test/suite/sys_vars/t/transaction_alloc_block_size_basic.test
sql/sys_vars.cc
BINLOGGED INCORRECTLY - BREAKS A SLAVE
Submitted a incomplete patch with my previous push,
re submitting the extra changes the required to make
the patch complete.
Analysis:
In row based replication, Master does not send temp table information
to Slave. If there are any DDLs that involves in regular table that needs
to be sent to Slave and a temp tables (which will not be available at Slave),
the Master rewrites the query replacing temp table with it's defintion.
Eg: create table regular_table like temptable.
In rewrite logic, server is ignoring the database of regular table
which can cause problems mentioned in this bug.
Fix: dont ignore database information (if available) while
rewriting the query
3RD EXECUTION OF PS
Problem:
When order by is by a column number for a group concat function
which has an outer reference, server fails in case of prepared
statements on the third execution
Analysis:
When a group concat function has order by, the fields in order by
are not resolved until execution if the input is a column number.
During execution they get resolved after the temp table gets created.
As a result they will be pointing to temp table fields which are
runtime created objects. This results in dangling pointers leading
to server failure.
Solution:
Reset the pointers for the order by fields to point to the original
arguments after execution as they are invalid.
Done in Item_func_group_concat::cleanup.
ISSUE:
------
There can be up to MERGEBUFF2 number of sorted merge chunks,
We need enough buffer space for at least one record from
each merge chunks. If estimates are wrong(very low) and we
allocate buffer space for less than MERGEBUFF2, then we will
have issue in merge_buffers, if actual number of rows to be
sorted is bigger than estimate and external filesort is
chosen.
SOLUTION:
---------
Set number of rows to sort to be at least MERGEBUFF2.
Problem:
find_order_by_list does not update the address of order_item
correctly after resolving.
Solution:
Change the ref_by address for a order_by field if its
SUM_FUNC_ITEM to the address of the field present in
all_fields.
Problem:
While getting the temp table field for a REF_ITEM
make_sortorder is using the real_item. As a result
server fails later with an assert.
Solution:
Do not use real_item to get the temp table field.
Instead use the REF_ITEM itself as temp table fields
are created for REF_ITEM not the real_item.
In versions 5.5 and 5.6 the MySQL version is not logged until
server is started and ready to accept connections. Exiting
server before this point will not have server version information
in the log. But in 5.7 code, we log a server version information
just after we prepare server_version string and logging is initialized.
For 5.5 and 5.6 code also adding this code to print server version
information.
Test results:
================
5.5
-----
Server version will be logged as below on server startup:
141218 8:45:48 [Note] /home/praveen/WorkDir/mysql_local/bug20052694/mysql/sql/mysqld (mysqld 5.5.42-debug-log) starting as process 19697 ...
5.6
----
Server version will be logged as below on server startup:
2014-12-18 09:08:43 0 [Note] /home/praveen/WorkDir/mysql_local/bug20052694/mysql-5.6/sql/mysqld (mysqld 5.6.23-debug-log) starting as process 18474 ...
LEADS TO INCORRECT BEHAVIOR
ISSUE:
------
When the following conditions are satisfied in a query, a
server crash occurs:
a) Two rows are compared using a NULL-safe equal-to operator.
b) Each of these rows belong to different charsets.
SOLUTION:
---------
When one charset is converted to another for comparision,
the constructor of "Item_func_conv_charset" is called.
This will attempt to use the Item_cache if the string is a
constant. This check succeeds because the "used_table_map"
of the Item_cache class is never set to the correct value.
Since it is mistakenly assumed to be a constant, it tries
to fetch the relevant null value related fields which are
yet to be initialized. This results in valgrind issues
and wrong results.
The fix is to update the "used_table_map" of "Item_cache".
This will allow "Item_func_conv_charset" to realise that
this is not a constant.
ISSUE:
------
We pre-allocate the ref_pointer_array before we resolve outer
references. This means that in some cases the
ref_pointer_array may not be large enough to hold all
references created. One such case is aggregate functions in
having clause of a subquery which may add items to select list
of outer query. So it is necessary to consider
select_n_having_items for subqueries while allocating
ref_pointer_array else we will get buffer overflow.
SOLUTION:
---------
Allocate a larger ref_pointer_array by aggregating
select_n_having_items for subqueries.
The fix in sql_yacc.yy is a backport from bug fix 18782905.
CRASHES WITH AUTO_INCREMENT COLUMN
Description:- Creating a federated table with AUTO_INCREMENT
column using LIKE clause results in a server crash.
Analysis:- Creating a federated table with AUTO_INCREMENT
column using LIKE clause results in a federated server
crash due to the uninitialized connection structure(mysql).
Also due to unassigned connection string for the remote
server, at the time of preparation of "create_info"
structure, the creation of any federated table using LIKE
clause fails with an error, "ERROR 1 (HY000): server name:
'' doesn't exist!". This bug is not only with
AUTO_INCREMENT but in all creations of federated tables with
LIKE clause.
Fix :- In ha_federated::info(), "mysql->insert_id" assigned
to "stats.auto_increment_value" only when there is an
active connection. This fixes the crash issue. For creating
the federated table with LIKE clause, connection string is
assigned at the time of preparation of "create_info"
structure.
The problem was that the maximum value of the transaction_prealloc_size
session system variable was ULONG_MAX which meant that it was possible
to cause the server to allocate excessive amounts of memory.
This patch fixes the problem by reducing the maxmimum value of
transaction_prealloc_size and transaction_alloc_block_size down
to 128K.
Note that transactions will still be able to allocate more than
128K if needed, this patch just reduces the amount that can be
preallocated - as well as the maximum size of the incremental
allocation blocks.
Problem Description And Fix:
Inserting a fudged record in mysql.proc with the dbname
column value as test and the name column as empty, will
cause a crash in mysqld when we run the command DROP
DATABASE test.
During DROP DATABASE test, mysql_rm_db subsequently
calls lock_db_routines. In the routine we fetch the
field 'name' from mysql.proc by calling the underlying
storage engine API in lock_db_routines. This cause NULL
value as the field column of mysql.proc and subsequent
dereference MDL_request::init leads to crash.
Modifying mysql.proc using SQL command by user is not
supported, but in principle, there is a possibility
of mysql.proc getting corrupted which can also lead
to empty fields and arbitary values. The patch fixes
the crash by checking NULL and propagating the appopriate
error code to the user.
special character sets like utf16, utf32, ucs2.
Analysis: MySQL server does not support few special character sets like
utf16,utf32 and ucs2 as "client's character set"(eg: utf16,utf32, ucs2).
It is known limitation listed in the documentation
http://dev.mysql.com/doc/refman/5.5/en/charset-connection.html.
The default value for default-character-set parameter is 'auto'
which means that if the server's character set is not supported,
then server automatically changes client's character set to
predefined character-set which is 'latin1' in the current code.
Eg:
$ ./mysql -uroot -S$SOCKET_FILE --default-character-set=utf16
ERROR 1231 (42000): Variable 'character_set_client' can't be set to the value of 'utf16'
$ ./mysql -uroot -S$SOCKET_FILE will be successfully connected to
server with 'latin1' as default client side character set.
When IO thread is trying to connect to Master, it sets server's character
set as client's character set. When Slave server is started with these
special character sets, IO thread (which is like a connection to Master)
fails because of the above said limitation.
Fix: Now even IO thread also behaves the same as a regular client behaves.
i.e., If server's character set is not supported as client's character set,
then set default's client character set(latin1) as client's character set.
Attempt to truncate temporary table using Blackhole storage and
locked by LOCK TABLES caused assertion failure and crashes.
Blackhole is a transaction-aware engine. While creating the temporary
table in transaction-aware engine, temporary table of type
"TRANSACTIONAL_TMP_TABLE" is created. For such temporary tables
a THR_LOCK lock is acquired by the LOCK TABLE operation. References
to them are also added into MYSQL_LOCK::table[] array. Also for
Blackhole engine, flag HTON_CAN_RECREATE is set.
While truncating temporary tables, no locks are taken and
recreate_temporary_table() is called for engines having
"HTON_CAN_RECREATE" in flag.
Function closefrm() is called from the recreate_temporary_table(),
to close the current temporary table. In closefrm(), the lock on
table expected is "F_UNLCK". In debug builds, assert condition on
this fails when lock of type "F_WRLCK" is acquired by LOCK TABLE
operation on temporary tables using Blackhole engine.
In non-debug builds closefrm() simply freed TABLE object leaving
dangling pointer to this object in MYSQL_LOCK::table[] array which
might lead to crashes later.
Fix:
---------
To fix this issue, we now unlock and remove table from MYSQL_LOCK::table[]
array before calling close_temporary_table() in recreate_temporary_table().
This is achieved by calling mysql_lock_remove() function for this table.
Fix:
===
Backport Bug#11756194 to mysql-5.5. slave breaks if
'drop database' fails on master and mismatched tables on
slave.
'DROP TABLE <deleted tables>' was binlogged when
'DROP DATABASE' failed and at least one table was deleted
from the database. The log event would lead slave SQL thread
stop if some of the tables did not exist on slave.
After this patch, It is always binlogged with 'IF EXISTS'
option.
CODE
Problem: UDF doesn't handle the arguments properly when they
are of string type due to a misplaced break.
The length of arguments is also not set properly
when the argument is NULL.
Solution: Fixed the code by putting the break at right place
and setting the argument length to zero when the
argument is NULL.
Description: When querying a subset of columns from the information_schema.TABLES
Analysis: When information about tables is collected for statements like
"SELECT ENGINE FROM I_S.TABLES" we do not perform full-blown table opens
in SE, instead we only use information from table shares from the Table
Definition Cache or .FRMs. Still in order to simplify I_S implementation
mock TABLE objects are created from TABLE_SHARE during this process.
This is done by calling open_table_from_share() function with special
arguments. Since this function always increments "Opened_tables" counter,
calls to it can be mistakingly interpreted as full-blown table opens in SE.
Note that claim that "'SELECT ENGINE FROM I_S.TABLES' statement doesn't
use Table Cache" is nevertheless factually correct. But it misses the
point, since such statements a) don't use full-blown TABLE objects and
therefore don't do table opens b) still use Table Definition Cache.
Fix: We are now incrementing the counter when db_stat(i.e open flags for ha_open(
we have considered an optimization which would use TABLE objects from
Table Cache when available instead of constructing mock TABLE objects,
but found it too intrusive for stable releases.
Analysis:
--------
Certain queries using intrinsic temporary tables may fail due to
name clashes in the file name for the temporary table when the
'temp-pool' enabled.
'temp-pool' tries to reduce the number of different filenames used for
temp tables by allocating them from small pool in order to avoid
problems in the Linux kernel by using a three part filename:
<tmp_file_prefix>_<pid>_<temp_pool_slot_num>.
The bit corresponding to the temp_pool_slot_num is set in the bit
map maintained for the temp-pool when it used for the file name.
It is cleared after the temp table is deleted for re-use.
The 'create_tmp_table()' function call under error condition
tries to clear the same bit twice by calling 'free_tmp_table()'
and 'bitmap_lock_clear_bit()'. 'free_tmp_table()' does a delete
of the table/file and clears the bit by calling the same function
'bitmap_lock_clear_bit()'.
The issue reported can be triggered under the timing window mentioned
below for an error condition while creating the temp table:
a) THD1: Due to an error clears the temp pool slot number used by it
by calling 'free_tmp_table'.
b) THD2: In the process of creating the temp table by using an unused
slot number in the bit map.
c) THD1: Clears the slot number used THD2 by calling
'bitmap_lock_clear_bit()' after completing the call 'free_tmp_table'.
d) THD3: Uses the slot number used the THD2 since it is freed by THD1.
When it tries to create the temp file using that slot number,
an error is reported since it is currently in use by THD2.
[The error: Error 'Can't create/write to file
'/tmp/#sql_277e_0.MYD' (Errcode: 17)']
Another issue which may occur in 5.6 and trunk is that:
When the open temporary table fails after its creation(due to ulimit
or OOM error), the file is not deleted. Thus further attempts to use
the same slot number in the 'temp-pool' results in failure.
Fix:
---
a) Under the error condition calling the 'bitmap_lock_clear_bit()'
function to clear the bit is unnecessary since 'free_tmp_table()'
deletes the table/file and clears the bit. Hence removed the
redundant call 'bitmap_lock_clear_bit()' in 'create_tmp_table()'
This prevents the timing window under which the issue reported
can be seen.
b) If open of the temporary table fails, then the file is deleted
thus allowing the temp-pool slot number to be utilized for the
subsequent temporary table creation.
c) Also if the attempt to create temp table fails since it already
exists, the temp-pool slot for it is marked as used, to avoid
the problem from re-appearing.
Issue :
-------
This seems for some platform -(LONGLONG_MIN) is
not flagged as out of range.
Fix:
----
Fix is backported from mysql-5.6 bug 14314156.
Fixed by adding an explicit test for this value in
Item_func_neg::int_op().
Bug#17959689: MAKE GCC AND CLANG GIVE CONSISTENT COMPILATION WARNINGS
Bug#18313717: ENABLE -WERROR IN MAINTANER MODE WHEN COMPILING WITH CLANG
Bug#18510941: REMOVE CMAKE WORKAROUNDS FOR OLDER VERSIONS OF OS X/XCODE
Backport from mysql-5.6 to mysql-5.5
Problem: For every event read, mysqlbinlog calls localtime() which in turn
calls stat(/etc/localtime) which is causing kernel mutex contention.
Analysis and Fix:
localtime() calls stat(/etc/localtime) for every instance of the call
where as localtime_r() the reentrant version was optimized to store
the read only tz internal structure. Hence it will not call
stat(/etc/localtime). It will call only once at the beginning.
The mysql server is calling localtime_r() and mysqlbinlog tool is
one place where we are still using localtime().
Once the process (mysqlbinlog) is started if timezone is changed
it will be not picked up the the process and it will continue
with the same values as the beginning of the process. This
behavior is in-lined with mysql server.
Also adding localtime_r() and gmtime_r() support for windows.
Problem:
========
In a master slave replication if a slave receives a
Start_log_event_v3 the payload is expected to be of fixed
size. If a payload which is smaller than the fixed size is
received it causes a read out of bounds issue.
Analysis:
========
According to documentation the fixed data part of
Start_log_event_v3 looks as shown below.
2 bytes: The binary log format version
50 bytes: The MySQL server's version
4 bytes: Timestamp in seconds when this event was created
Since the payload is expected to be of fixed size, therefore
ST_SERVER_VER_LEN (50) bytes are memcpy'ed into
server_version. But if a malicious master sends a shorter
payload it causes a read out of bounds issue.
Fix:
===
In Start_log_event_v3 event's constructor a check has been
added which expects the minimum payload length to be of size
common_header_len + ST_COMMON_HEADER_LEN_OFFSET bytes. If a
malicious packet of lesser length is received it will be
considered as an invalid event.
- Restarting mysqld with --expire-log-days=1 triggers 'log_in_use()' to be called while current_thd is NULL.
- Check current_thd before calling DEBUG_SYNC() to avoid passing NULL pointer to DEBUG_SYNC()
- Wrap debug code construct inside #ifndef DBUG_OFF like in other parts of the file
Normally, SET SESSION SQL_LOG_BIN is used by DBAs to run a
non-conflicting command locally only, ensuring it does not
get replicated.
Setting GLOBAL SQL_LOG_BIN would not require all sessions to
disconnect. When SQL_LOG_BIN is changed globally, it does not
immediately take effect for any sessions. It takes effect by
becoming the session-level default inherited at the start of
each new session, and this setting is kept and cached for the
duration of that session. Setting it intentionally is unlikely
to have a useful effect under any circumstance; setting it
unintentionally, such as while intending to use SET [SESSION]
is potentially disastrous. Accidentally using SET GLOBAL
SQL_LOG_BIN will not show an immediate effect to the user,
instead not having the desired session-level effect, and thus
causing other potential problems with local-only maintenance
being binlogged and executed on slaves; And transactions from
new sessions (after SQL_LOG_BIN is changed globally) are not
binlogged and replicated, which would result in irrecoverable
or difficult data loss.
This is the regular GLOBAL variables way to work, but in
replication context it does not look right on a working server
(with connected sessions) 'set global sql_log_bin' and none of
that connections is affected. Unexperienced DBA after noticing
that the command did "nothing" will change the session var and
most probably won't unset the global var, causing new sessions
to not be binlog.
Setting GLOBAL SQL_LOG_BIN allows DBA to stop binlogging on all
new sessions, which can be used to make a server "replication
read-only" without restarting the server. But this has such big
requirements, stop all existing connections, that it is more
likely to make a mess, it is too risky to allow the GLOBAL variable.
The statement 'SET GLOBAL SQL_LOG_BIN=N' will produce an error
in 5.5, 5.6 and 5.7. Reading the GLOBAL SQL_LOG_BIN will produce
a deprecation warning in 5.7.
AND IS_USED_LOCK().
Analysis:
-----------
In functions Item_func_is_free_lock::val_int() and
Item_func_is_used_lock::val_int(), for the specified user lock
name, pointer to its "User_level_lock" object is obtained from hash
"hash_user_locks". Mutex "LOCK_user_locks" is acquired for this
and released immediately. And we are accessing members of
User_level_lock after releasing the mutex. If same user lock is
deleted(released) from concurrent thread then accessing members
results in invalid(freed) memory access issue.
Deleting of user lock is also protected from the mutex
"LOCK_user_locks". Since this mutex is released in "val_int"
functions mentioned above, delete operation proceeds while concurrent
thread tries to access its members.
With the test case, valgrind reports invalid read issues in val_int
functions.
Fix:
-----------
To fix this issue, in "val_int" function of classes
"Item_func_is_free_lock" and "Item_func_is_used_lock", now releasing
mutex "LOCK_user_locks" after accessing User_level_lock members.
CHECK.
Analysis:
----------
Issue here is, while creating or altering the InnoDB table,
if the foreign key defined on the table references a parent
table on which the user has no access privileges then the
table is created without reporting any error.
Currently the privilege level REFERENCES_ACL is unused
and is not used for access evaluation while creating the
table with a foreign key constraint or adding the foreign
key constraint to a table. But when no privileges are granted
to user then also access evaluation on parent table is ignored.
Fix:
---------
For DMLs, irrelevant of the fact, support does not want any
changes to avoid permission checks on every operation.
So, as a fix, added a function "check_fk_parent_table_access"
to check whether any of the SELECT_ACL, INSERT_ACL, UDPATE_ACL,
DELETE_ACL or REFERENCE_ACL privileges are granted for user
at table level. If none of them is granted then error is reported.
This function is called during the table creation and alter
operation.
Problem: A corrupted header length in FORMAT_DESCRIPTION_LOG_EVENT
can cause server to crash.
Analysis: FORMAT_DESCRIPTION_EVENT will be considered invalid if
header len is too small (i.e. below OLD_HEADER_LEN).
Format_description_log_event:: Format_description_log_event(...)
{
...
if ((common_header_len=buf[ST_COMMON_HEADER_LEN_OFFSET]) < OLD_HEADER_LEN)
DBUG_VOID_RETURN; /* sanity check */
...
post_header_len= my_memdup(...)
}
In that case Format_description_log_event constructor will return early,
without allocating any memory for post_header_len. Thence this variable is
left uninitialized and making server to crash when server is trying
to free the uninitialized value.
Fix: When Format_description_log_event constructor returns early, assign
NULL to post_header_len.
Several string functions have optimizations for constant
sub-expressions which lead to setting max_length == 0.
For subqueries, where we need a temporary table to holde the result,
we need to ensure that we use a VARCHAR(0) column rather than a
CHAR(0) column when such expressions take part in grouping.
With CHAR(0) end_update() may write garbage into the next field.
GOES AWAY, MYSQL QUITS WORKING.
Analysis:
-----------------
Issue in this bug and in bug 11907705 is, the socket file or
fifo file is set for general log at command line while starting
the server. But currently, only regular file can be set for the
general log. Instead of reporting any error, the provided files
are opened for writing and continued. Because of this issues
mentioned in the bug reports are seen.
As mentioned, only when any non-regular file is set for general
log at command line while starting the server, these issues are
seen. If general log file is set to non-regular file from CLI
using system variable general_log_file then error is reported.
These issues can also be faced with slow query log file, if it is
set to non-regular file.
Fix:
-----------------
Currently while starting the server if we fail to open log file
then we report an error, disable logging to file and continue.
To fix issue reported code is modified to check whether file
is regular file or not before opening it. If file is not a
regular file then error is logged to error log and logging to
file is disabled.
Description:
THREAD_CONCURRENCY is deprecated and there is no
deprecation warning message while setting this variable
while starting the server.
Analysis:
This variable is specific to Solaris 8 and earlier systems
and is ignored on all other platforms. But since many
customers, who uses other than Solaris, still has this
variable in their configuration file, it is important to
have a deprecation warning.
Fix:
THREAD_CONCURRENCY deprecation warning message is added.
BACKGROUND:
This bug is a followup on Bug#16368875.
The assertion failure happens because in SQL layer the key
does not get promoted to PRIMARY KEY but InnoDB takes it
as PRIMARY KEY.
ANALYSIS:
Here we are trying to create an index on POINT (GEOMETRY)
data type which is a type of BLOB (since GEOMETRY is a
subclass of BLOB).
In general, we can't create an index over GEOMETRY family
type field unless we specify the length of the
keypart (similar to BLOB fields).
Only exception is the POINT field type. The POINT column
max size is 25. The problem is that the field is not treated
as PRIMARY KEY when we create a index on POINT column using
its max column size as key part prefix. The fix would allow
index on POINT column to be treated as PRIMARY KEY.
FIX:
Patch for Bug#16368875 is extended to take into account
GEOMETRY datatype, POINT in particular to consider it
as PRIMARY KEY in SQL layer.
