MDEV-4489 "Replication of big5, cp932, gbk, sjis strings makes wrong values on slave"
has been fixed.
Problem:
String constants of some Asian charsets (big5,cp932,gbk,sjis)
can have backslash '\' (0x5C) in the second byte of multi-byte characters.
Replicating of such constants using the standard '\'-escaping is dangerous.
Therefore, constants of these charsets are replicated using hex notation:
INSERT INTO t1 (a) VALUES (0x815C);
However, 0xHHHH constants do not work well in some cases,
because they can behave as strings and as numbers, depending on context
(for example, depending on the data type of the column in an INSERT statement).
This SQL script was not replicated correctly with statement-based replication:
SET NAMES gbk;
PREPARE STMT FROM 'INSERT INTO t1 (a) VALUES (?)';
SET @a = '1';
EXECUTE STMT USING @a;
The INSERT statement was replicated as:
INSERT INTO t1 (a) VALUES (0x31);
'1' was correctly converted to the number 1 on master.
But the 0x31 constant was treated as number 49 on slave.
Fix:
1. Binary log now uses X'HHHH' instead of 0xHHHH constants.
2. The X'HHHH' constants now work always as strings, in all contexts.
This is the SQL standard compliant behaviour.
After the fix, the above statement is replicated as:
INSERT INTO t1 (a) VALUES (X'31');
X'31' is treated as string '1' on slave, and is correctly converted to 1.
modified:
@ mysql-test/r/ctype_cp932_binlog_stm.result
@ mysql-test/r/select.result
@ mysql-test/r/select_jcl6.result
@ mysql-test/r/select_pkeycache.result
@ mysql-test/r/user_var-binlog.result
@ mysql-test/r/varbinary.result
@ mysql-test/suite/binlog/r/binlog_stm_ctype_ucs.result
@ mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result
@ mysql-test/suite/rpl/r/rpl_charset_sjis.result
@ mysql-test/suite/rpl/r/rpl_mdev382.result
@ mysql-test/suite/rpl/t/rpl_charset_sjis.test
@ mysql-test/t/ctype_cp932_binlog_stm.test
@ mysql-test/t/select.test
@ mysql-test/t/varbinary.test
Adding and updating tests
@ sql/item.cc
@ sql/item.h
@ sql/sql_yacc.yy
@ sql/sql_lex.cc
Splitting the implementations of X'HH' and 0xHH constants into two
separate classes. Fixing the parser to distinguish the two syntaxes.
@ sql/log_event.cc
Using X'HH' instead of 0xHH for binary logging for string constants
of the "dangerous" charsets.
@ sql/sql_string.h
Adding a helped method String::append_hex().
get_datetime_value() should not double-cache its own Item_cache_temporal items,
but it *should* cache other Item_cache items, such as Item_cache_str.
sql/item.h:
shortcut, to avoid going through the switch in Item::cmp_type()
sql/item_cmpfunc.cc:
even if the item is Item_cache_str - it still needs to be converted and cached.
sql/item_timefunc.h:
all descendants of Item_temporal_func always have cmp_type==TIME_RESULT.
Even Item_date_add_interval, that might have field_type == MYSQL_TYPE_STRING.
Some queries with the "SELECT ... FROM DUAL" nested subqueries
failed with an assertion on debug builds.
Non-debug builds were not affected.
There were a few different issues with similar assertion
failures on different queries:
1. The first problem was related to the incomplete propagation
of the "non-constant" item status from underlying subquery
items to the outer item tree: in some cases non-constants were
interpreted as constants and evaluated at the preparation stage
(val_int() calls withing fix_fields() etc).
Thus, the default implementation of Item_ref::const_item() from
the Item parent class didn't take into account the "const_item"
status of the referenced item tree -- it used the insufficient
"used_tables() == 0" check instead. This worked in most cases
since our "non-constant" functions like RAND() and SLEEP() set
the RAND_TABLE_BIT in the used table map, so they aren't
non-constant from Item_ref's "point of view". However, the
"SELECT ... FROM DUAL" subquery may have an empty map of used
tables, but at the same time subqueries are never "constant" at
the context analysis stage (preparation, view creation etc).
So, the non-contantness of such subqueries was missed.
Fix: the Item_ref::const_item() function has been overloaded to
take into account both (*ref)->const_item() status and tricky
Item_ref::used_tables() return values, since the only
(*ref)->const_item() call is not enough there.
2. In some cases instead of the const_item() call we check a
value of the Item::with_subselect field to recognize items
with nested subqueries. However, the Item_ref class didn't
propagate this value from the referenced item tree.
Fix: Item::has_subquery() and Item_ref::has_subquery()
functions have been backported from 5.6. All direct
references to the with_subselect fields of nested items have
been with the has_subquery() function call.
3. The Item_func_regex class didn't propagate with_subselect
as well, since it overloads the Item_func::fix_fields()
function with insufficient fix_fields() implementation.
Fix: the Item_func_regex::fix_fields() function has been
modified to gather "constant" statuses from inner items.
4. The Item_func_isnull::update_used_tables() function has
a special branch for the underlying item where the maybe_null
value is false: in this case it marks the Item_func_isnull
as a "const_item" and sets the cached_value to false.
However, the Item_func_isnull::val_int() was not in sync with
update_used_tables(): it didn't take into account neither
const_item_cache nor cached_value for the case of
"args[0]->maybe_null == false optimization".
As far as such an Item_func_isnull has "const_item() == true",
it's ok to call Item_func_isnull::val_int() etc from outer
items on preparation stage. In this case the server tried to
call Item_func_isnull::args[0]->isnull(), and if the args[0]
item contained a nested not-nullable subquery, it failed
with an assertion.
Fix: take the value of Item_func_isnull::const_item_cache into
account in the val_int() function.
5. The auxiliary Item_is_not_null_test class has a similar
optimization in the update_used_tables() function as the
Item_func_isnull class has, and the same issue in the val_int()
function.
In addition to that the Item_is_not_null_test::update_used_tables()
doesn't update the const_item_cache value, so the "maybe_null"
optimization is useless there. Thus, we missed some optimizations
of cases like these (before and after the fix):
< <is_not_null_test>(a),
---
> <cache>(<is_not_null_test>(a)),
or
< having (<is_not_null_test>(a) and <is_not_null_test>(a))
---
> having 1
etc.
Fix: update Item_is_not_null_test::const_item_cache in
update_used_tables() and take in into account in val_int().
Some queries with the "SELECT ... FROM DUAL" nested subqueries
failed with an assertion on debug builds.
Non-debug builds were not affected.
There were a few different issues with similar assertion
failures on different queries:
1. The first problem was related to the incomplete propagation
of the "non-constant" item status from underlying subquery
items to the outer item tree: in some cases non-constants were
interpreted as constants and evaluated at the preparation stage
(val_int() calls withing fix_fields() etc).
Thus, the default implementation of Item_ref::const_item() from
the Item parent class didn't take into account the "const_item"
status of the referenced item tree -- it used the insufficient
"used_tables() == 0" check instead. This worked in most cases
since our "non-constant" functions like RAND() and SLEEP() set
the RAND_TABLE_BIT in the used table map, so they aren't
non-constant from Item_ref's "point of view". However, the
"SELECT ... FROM DUAL" subquery may have an empty map of used
tables, but at the same time subqueries are never "constant" at
the context analysis stage (preparation, view creation etc).
So, the non-contantness of such subqueries was missed.
Fix: the Item_ref::const_item() function has been overloaded to
take into account both (*ref)->const_item() status and tricky
Item_ref::used_tables() return values, since the only
(*ref)->const_item() call is not enough there.
2. In some cases instead of the const_item() call we check a
value of the Item::with_subselect field to recognize items
with nested subqueries. However, the Item_ref class didn't
propagate this value from the referenced item tree.
Fix: Item::has_subquery() and Item_ref::has_subquery()
functions have been backported from 5.6. All direct
references to the with_subselect fields of nested items have
been with the has_subquery() function call.
3. The Item_func_regex class didn't propagate with_subselect
as well, since it overloads the Item_func::fix_fields()
function with insufficient fix_fields() implementation.
Fix: the Item_func_regex::fix_fields() function has been
modified to gather "constant" statuses from inner items.
4. The Item_func_isnull::update_used_tables() function has
a special branch for the underlying item where the maybe_null
value is false: in this case it marks the Item_func_isnull
as a "const_item" and sets the cached_value to false.
However, the Item_func_isnull::val_int() was not in sync with
update_used_tables(): it didn't take into account neither
const_item_cache nor cached_value for the case of
"args[0]->maybe_null == false optimization".
As far as such an Item_func_isnull has "const_item() == true",
it's ok to call Item_func_isnull::val_int() etc from outer
items on preparation stage. In this case the server tried to
call Item_func_isnull::args[0]->isnull(), and if the args[0]
item contained a nested not-nullable subquery, it failed
with an assertion.
Fix: take the value of Item_func_isnull::const_item_cache into
account in the val_int() function.
5. The auxiliary Item_is_not_null_test class has a similar
optimization in the update_used_tables() function as the
Item_func_isnull class has, and the same issue in the val_int()
function.
In addition to that the Item_is_not_null_test::update_used_tables()
doesn't update the const_item_cache value, so the "maybe_null"
optimization is useless there. Thus, we missed some optimizations
of cases like these (before and after the fix):
< <is_not_null_test>(a),
---
> <cache>(<is_not_null_test>(a)),
or
< having (<is_not_null_test>(a) and <is_not_null_test>(a))
---
> having 1
etc.
Fix: update Item_is_not_null_test::const_item_cache in
update_used_tables() and take in into account in val_int().
Early evaluation of subqueries in the WHERE conditions on I_S.*_STATUS tables,
otherwise the subquery on this same table will try to acquire LOCK_status twice.
sql/item.h:
remove unused method
This fixed failing test in group_by.test
mysql-test/r/join_outer.result:
Updated test case
mysql-test/r/join_outer_jcl6.result:
Updated test case
sql/item.cc:
Don't reset maybe_null in update_used_tables(); This breaks ROLLUP
sql/item.h:
Don't reset maybe_null in update_used_tables(); This breaks ROLLUP
sql/item_cmpfunc.h:
Don't reset maybe_null in update_used_tables(); This breaks ROLLUP
INCORRECT RESULTS
This is a backport of fix for Bug#13068506.
mysql-test/r/join_outer.result:
Added test result for Bug#13068506
mysql-test/t/join_outer.test:
Added test case for Bug#13068506
sql/item.h:
Implement Item_outer_ref::not_null_tables()
from a MERGE view.
The problem was in the lost ability to be null for the table of a left join if it
is a view/derived table.
It hapenned because setup_table_map(), was called earlier then we merged
the view or derived.
Fixed by propagating new maybe_null flag during Item::update_used_tables().
Change in join_outer.test and join_outer_jcl6.test appeared because
IS NULL reported no used tables (i.e. constant) for argument which could not be
NULL and new maybe_null flag was propagated for IS NULL argument (Item_field)
because table the Item_field belonged to changed its maybe_null status.
Analysis:
The following call stack shows that it is possible to set Item_cache::value_cached, and the relevant value
without setting Item_cache::example.
#0 Item_cache_temporal::store_packed at item.cc:8395
#1 get_datetime_value at item_cmpfunc.cc:915
#2 resolve_const_item at item.cc:7987
#3 propagate_cond_constants at sql_select.cc:12264
#4 propagate_cond_constants at sql_select.cc:12227
#5 optimize_cond at sql_select.cc:13026
#6 JOIN::optimize at sql_select.cc:1016
#7 st_select_lex::optimize_unflattened_subqueries at sql_lex.cc:3161
#8 JOIN::optimize_unflattened_subqueries at opt_subselect.cc:4880
#9 JOIN::optimize at sql_select.cc:1554
The fix is to set Item_cache_temporal::example even when the value is
set directly by Item_cache_temporal::store_packed. This makes the
Item_cache_temporal object consistent.
mysql-test/suite/innodb/include/restart_and_reinit.inc:
drop and recreate mysql.innodb* tables when deleting innodb table spaces
mysql-test/t/ssl_8k_key-master.opt:
with loose- prefix ssl errors are ignored
sql-common/client.c:
compiler warnings
sql/field.cc:
use the new function
sql/item.cc:
don't convert time to double or decimal via longlong,
this loses sub-second part.
Use dedicated functions.
sql/item.h:
incorrect cast_to_int type for params
sql/item_strfunc.cc:
use the new function
sql/lex.h:
unused
sql/my_decimal.h:
helper macro
sql/sql_plugin.cc:
workaround for a compiler warning
sql/sql_yacc.yy:
unused
sql/transaction.cc:
fix the merge for SERVER_STATUS_IN_TRANS_READONLY protocol flag
storage/sphinx/CMakeLists.txt:
compiler warnings
Link view/derived table fields to a real table to check turning the table record to null row.
Item_direct_view_ref wrapper now checks if table is turned to null row.
Documentation for class Item_outer_ref was wrong:
(*ref) may point to Item_field as well
(see e.g. Item_outer_ref::fix_fields)
So this casting in get_store_key() was wrong:
(*(Item_ref**)((Item_ref*)keyuse->val)->ref)->ref_type()
Documentation for class Item_outer_ref was wrong:
(*ref) may point to Item_field as well
(see e.g. Item_outer_ref::fix_fields)
So this casting in get_store_key() was wrong:
(*(Item_ref**)((Item_ref*)keyuse->val)->ref)->ref_type()
