Merge branch '10.3' into trunk
Both field_visibility and VERS_HIDDEN_FLAG exist independently.
TODO:
VERS_HIDDEN_FLAG should be replaced with SYSTEM_INVISIBLE (or COMPLETELY_INVISIBLE?).
TODO:
- Make get_thd_memroot() inline
- To do this, we need to reduce dependence of include files, especially
so that sql_class.h is not depending in item.h
Make differentiation between pullout for merge and pulout of outer field during exists2in transformation.
In last case the field was outer and so we can safely start from name resolution context of the SELECT where it was pulled.
Old behavior lead to inconsistence between list of tables and outer name resolution context (which skips one SELECT for merge purposes) which creates problem vor name resolution.
As a result of this merge the code for the following tasks appears in 10.3:
- MDEV-12172 Implement tables specified by table value constructors
- MDEV-12176 Transform [NOT] IN predicate with long list of values INTO
[NOT] IN subquery.
Side effect: the second debug Note in cache_temporal_4265.result disappeared.
Before this change:
- During JOIN::cache_const_exprs(),
Item::get_cache() for Item_date_add_interval() was called.
The data type for date_add('2001-01-01',interval 5 day) is VARCHAR,
because the first argument is VARCHAR (not temporal).
Item_get_cache() created Item_cache_str('2001-01-06').
- During evaluate_join_record(), get_datetime_value() was called,
which called Item::get_date() for Item_cache_str('2001-01-06').
This gave the second Note. Then, get_datetime_value() created
a new cache, now Item_cache_temporal for '2001-01-06', so not
further str_to_datetime() happened.
After this change:
- During tem_bool_rowready_func2::fix_length_and_dec(),
Arg_comparator::set_cmp_func_datetime() is called,
which immediately creates an instance of Item_cache_date for
the result of date_add('2001-01-01',interval 5 day).
So later no str_to_datetime happens any more,
neither during JOIN::cache_const_exprs(),
nor during evaluate_join_record().
Fixing the data type for the "fuzzydate" parameter to
Item_func_hybrid_field_type::date_op() from uint to ulonglong,
for consistency with Item::get_date().
- Fix win64 pointer truncation warnings
(usually coming from misusing 0x%lx and long cast in DBUG)
- Also fix printf-format warnings
Make the above mentioned warnings fatal.
- fix pthread_join on Windows to set return value.
This is a preparatory step for MDEV-13864.
It does not change behavior in any ways. It simply splits methods into smaller peaces.
The intent of this separate patch is to make more readable the main patch for
MDEV-13864 (which will actually move the predicant to args[0]).
1. Splitting fix_length_and_dec() into smaller pieces, adding:
- bool aggregate_then_and_else_arguments(THD *thd);
- bool aggregate_switch_and_when_arguments(THD *thd);
2. Splitting find_item() into smaller pieces, adding:
- Item *find_item_searched();
- Item *find_item_simple();
3. Splitting print() into smaller pieces, adding:
- void print_when_then_arguments(String *str, enum_query_type query_type,
Item **items, uint count);
- void print_else_argument(String *str, enum_query_type query_type, Item *item)
4. Moving the maybe_null handling part related to ELSE from fix_length_and_dec()
to fix_fields(), as in all other Item_func's.
5. Removing the unused String* argument from find_item().
6. Moving find_item() from public to private, as it's not needed outside.
Item_cond did not correctly propagate with_window_func flag in the Item
tree. Without it, we would not call Item::split_sum_func correctly and the
window function's result_field would remain NULL.
- Added sql/mariadb.h file that should be included first by files in sql
directory, if sql_plugin.h is not used (sql_plugin.h adds SHOW variables
that must be done before my_global.h is included)
- Removed a lot of include my_global.h from include files
- Removed include's of some files that my_global.h automatically includes
- Removed duplicated include's of my_sys.h
- Replaced include my_config.h with my_global.h
JSON_EXTRACT behaves specifically in the comparison,
so we have to implement specific method for that in
Arg_comparator.
Conflicts:
sql/item_cmpfunc.cc
1. use Regexp_processor_pcre::set_recursion_limit() to set the
recursion limit depending on the current available stack size
2. make pcre stack frame to be estimated no less than 500 bytes.
sometimes pcre estimates it too low, even though the manual
says 500+16 bytes (it was estimated only 188 for me, actual
frame size was 512).
3. do it for embedded too
It was possible to construct a PCRE expression that exceeded the stack.
resulting in a crash:
With fix:
MariaDB [(none)]> SELECT 1
-> FROM dual
-> WHERE ('Alpha,Bravo,Charlie,Delta,Echo,Foxtrot,StrataCentral,Golf,Hotel,India,Juliet,Kilo,Lima,Mike,StrataL3,November,Oscar,StrataL2,Sand,P3,P4SwitchTest,Arsys,Poppa,ExtensionMgr,Arp,Quebec,Romeo,StrataApiV2,PtReyes,Sierra,SandAcl,Arrow,Artools,BridgeTest,Tango,SandT,PAlaska,Namespace,Agent,Qos,PatchPanel,ProjectReport,Ark,Gimp,Agent,SliceAgent,Arnet,Bgp,Ale,Tommy,Central,AsicPktTestLib,Hsc,SandL3,Abuild,Pca9555,Standby,ControllerDut,CalSys,SandLib,Sb820,PointV2,BfnLib,Evpn,BfnSdk,Sflow,ManagementActive,AutoTest,GatedTest,Bgp,Sand,xinetd,BfnAgentLib,bf-utils,Hello,BfnState,Eos,Artest,Qos,Scd,ThermoMgr,Uniform,EosUtils,Eb,FanController,Central,BfnL3,BfnL2,tcp_wrappers,Victor,Environment,Route,Failover,Whiskey,Xray,Gimp,BfnFixed,Strata,SoCal,XApi,Msrp,XpProfile,tcpdump,PatchPanel,ArosTest,FhTest,Arbus,XpAcl,MacConc,XpApi,telnet,QosTest,Alpha2,BfnVlan,Stp,VxlanControllerTest,MplsAgent,Bravo2,Lanz,BfnMbb,Intf,XCtrl,Unicast,SandTunnel,L3Unicast,Ipsec,MplsTest,Rsvp,EthIntf,StageMgr,Sol,MplsUtils,Nat,Ira,P4NamespaceDut,Counters,Charlie2,Aqlc,Mlag,Power,OpenFlow,Lag,RestApi,BfdTest,strongs,Sfa,CEosUtils,Adt746,MaintenanceMode,MlagDut,EosImage,IpEth,MultiProtocol,Launcher,Max3179,Snmp,Acl,IpEthTest,PhyEee,bf-syslibs,tacc,XpL2,p4-ar-switch,p4-bf-switch,LdpTest,BfnPhy,Mirroring,Phy6,Ptp'
->
-> REGEXP '^((?!\b(Strata|StrataApi|StrataApiV2)\b).)*$');
Empty set, 1 warning (0.00 sec)
MariaDB [(none)]> show warnings;
+---------+------+---------------------------------------------------------+
| Level | Code | Message |
+---------+------+---------------------------------------------------------+
| Warning | 1139 | Got error 'pcre_exec: Internal error (-21)' from regexp |
+---------+------+---------------------------------------------------------+
Other things
- Ensure that ut_d() is set to EXPR if ut_ad() is DEBUG_ASSERT()
If not, we will get a crash in purge_sys_t::~purge_sys_t() as
this ut_ad() code expect's that the ut_d() codes has been executed
- Adding Type_handler::make_table_field() and moving pieces of the code
from Item::tmp_table_field_from_field_type() to virtual implementations
for various type handlers.
- Adding a new Type_all_attributes, to access to Item's extended
attributes, such as decimal_precision() and geometry_type().
- Adding a new class Record_addr, to pass record related information
to Type_handler methods (ptr, null_ptr and null_bit) as a single structure.
Note, later it will possibly be extended for BIT-alike field purposes,
by adding new members (bit_ptr_arg, bit_ofs_arg).
- Moving the code from Field_new_decimal::create_from_item()
to Type_handler_newdecimal::make_table_field().
- Removing Field_new_decimal() and Field_geom() helper constructor
variants that were used for temporary field creation.
- Adding Item_field::type_handler(), Field::type_handler() and
Field_blob::type_handler() to return correct type handlers for
blob variants, according to Field_blob::packlength.
- Adding Type_handler_blob_common, as a common parent for
Type_handler_tiny_blob, Type_handler_blob, Type_handler_medium_blob
and Type_handler_long_blob.
- Implementing Type_handler_blob_common::Item_hybrid_func_fix_attributes().
It's needed for cases when TEXT variants of different character sets are mixed
in LEAST, GREATEST, CASE and its abreviations (IF, IFNULL, COALESCE), e.g.:
CREATE TABLE t1 (
a TINYTEXT CHARACTER SET latin1,
b TINYTEXT CHARACTER SET utf8
);
CREATE TABLE t2 AS SELECT COALESCE(a,b) FROM t1;
Type handler aggregation returns TINYTEXT as a common data type
for the two columns. But as conversion from latin1 to utf8
happens for "a", the maximum possible length of "a" grows from 255 to 255*3.
Type_handler_blob_common::Item_hybrid_func_fix_attributes() makes sure
to update the blob type handler according to max_length.
- Adding Type_handler::blob_type_handler(uint max_octet_length).
- Adding a few m_type_aggregator_for_result.add() pairs, because
now Item_xxx::type_handler() can return pointers to type_handler_tiny_blob,
type_handler_blob, type_handler_medium_blob, type_handler_long_blob.
Before the patch only type_handler_blob was possible result of type_handler().
- Making type_handler_tiny_blob, type_handler_blob, type_handler_medium_blob,
type_handler_long_blob public.
- Removing the condition in Item_sum_avg::create_tmp_field()
checking Item_sum_avg::result_type() against DECIMAL_RESULT.
Now both REAL_RESULT and DECIMAL_RESULT are symmetrically handled
by tmp_table_field_from_field_type().
- Removing Item_geometry_func::create_field_for_create_select(),
as the inherited version perfectly works.
- Fixing Item_func_as_wkb::field_type() to return MYSQL_TYPE_LONG_BLOB
rather than MYSQL_TYPE_BLOB. It's needed to make sure that
tmp_table_field_from_field_type() creates a LONGBLOB field for AsWKB().
- Fixing Item_func_as_wkt::fix_length_and_dec() to set max_length to
UINT32_MAX rather than MAX_BLOB_WIDTH, to make sure that
tmp_table_field_from_field_type() creates a LONGTEXT field for AsWKT().
- Removing Item_func_set_user_var::create_field_for_create_select(),
as the inherited version works fine.
- Adding Item_func_get_user_var::create_field_for_create_select() to
make sure that "CREATE TABLE t1 AS SELECT @string_user variable"
always creates a field of LONGTEXT/LONGBLOB type.
- Item_func_ifnull::create_field_for_create_select()
behavior has changed. Before the patch it passed set_blob_packflag=false,
which meant to create LONGBLOB for all blob variants.
Now it takes into account max_length, which gives better column
data types for:
CREATE TABLE t2 AS SELECT IFNULL(blob_column1, blob_column2) FROM t1;
- Fixing Item_func_nullif::fix_length_and_dec() to use
set_handler(args[2]->type_handler()) instead of
set_handler_by_field_type(args[2]->field_type()).
This is needed to distinguish between BLOB variants.
- Implementing Item_blob::type_handler(), to make sure to create
proper BLOB field variant, according to max_length, for queries like:
CREATE TABLE t1 AS
SELECT some_blob_field FROM INFORMATION_SCHEMA.SOME_TABLE;
- Fixing Item_field::real_type_handler() to make sure that
the code aggregating fields for UNION gets a proper BLOB
variant type handler from fields.
- Adding a special code into Item_type_holder::make_field_by_type(),
to make sure that after aggregating field types it also properly
takes into account max_length when mixing TEXT variants of different
character sets and chooses a proper TEXT variant:
CREATE TABLE t1 (
a TINYTEXT CHARACTER SET latin1,
b TINYTEXT CHARACTER SET utf8
);
CREATE TABLE t2 AS SELECT a FROM t1 UNION SELECT b FROM t1;
- Adding tests, for better coverage of IFNULL, NULLIF, UNION.
- The fact that tmp_table_field_from_field_type() now takes
into account BLOB variants (instead of always creating LONGBLOB),
tests results for WEIGHT_STRING() and NULLIF() and UNION
have become more precise.
Also, implement MDEV-11027 a little differently from 5.5 and 10.0:
recv_apply_hashed_log_recs(): Change the return type back to void
(DB_SUCCESS was always returned).
Report progress also via systemd using sd_notifyf().
Problem: integer literals may be converted to floats for
comparison with decimal data. If the integers are large,
we may lose precision, and give wrong results.
Fix: for
<non-const decimal expression> <cmp> <const string expression>
or
<const string expression> <cmp> <non-const decimal expression>
we override the compare_type chosen by item_cmp_type(), and
do comparison as decimal rather than float.
(cherry picked from commit 1cf3489ba4 and edited by Johannes Weißl <jargon@molb.org>)
it used current_thd->alloc() and allocated on the thd's execution arena,
not on table->expr_arena.
Remove THD::arena_for_cached_items that is temporarily set in
update_virtual_fields(), and replaces THD arena in get_datetime_value().
Instead set THD arena to table->expr_arena for the whole duration
of update_virtual_fields()
Item_func_le included Arg_comparator. Arg_comparator remembered
the current_thd during fix_fields and used that value during
execution to allocate Item_cache in get_datetime_value().
But for vcols fix_fields and val_int can happen in different threads.
Same bug for Item_func_in using in_datetime or cmp_item_datetime,
both also remembered current_thd at fix_fields() to use it later
for get_datetime_value().
As a fix, these objects no longer remember the current_thd,
and get_datetime_value() uses current_thd at run time. This
should not increase the number of current_thd calls much, as
Item_cache is created only once anyway.
This patch fixes a number of data type aggregation problems in IN and CASE:
- MDEV-11497 Wrong result for (int_expr IN (mixture of signed and unsigned expressions))
- MDEV-11514 IN with a mixture of TIME and DATETIME returns a wrong result
- MDEV-11554 Wrong result for CASE on a mixture of signed and unsigned expressions
- MDEV-11555 CASE with a mixture of TIME and DATETIME returns a wrong result
1. The problem reported in MDEV-11514 and MDEV-11555 was in the wrong assumption
that items having the same cmp_type() can reuse the same cmp_item instance.
So Item_func_case and Item_func_in used a static array of cmp_item*,
one element per one XXX_RESULT.
TIME and DATETIME cannot reuse the same cmp_item, because arguments of
these types are compared very differently. TIME and DATETIME must have
different instances in the cmp_item array. Reusing the same cmp_item
for TIME and DATETIME leads to unexpected result and unexpected warnings.
Note, after adding more data types soon (e.g. INET6), the problem would
become more serious, as INET6 will most likely have STRING_RESULT, but
it won't be able to reuse the same cmp_item with VARCHAR/TEXT.
This patch introduces a new class Predicant_to_list_comparator,
which maintains an array of cmp_items, one element per distinct
Type_handler rather than one element per XXX_RESULT.
2. The problem reported in MDEV-11497 and MDEV-11554 happened because
Item_func_in and Item_func_case did not take into account the fact
that UNSIGNED and SIGNED values must be compared as DECIMAL rather than INT,
because they used item_cmp_type() to aggregate the arguments.
The relevant code now resides in Predicant_to_list_comparator::add_value()
and uses Type_handler_hybrid_field_type::aggregate_for_comparison(),
like Item_func_between does.
As agreed with Sanja, debug messages printed in
Item_func_in::fix_length_and_dec() now have the "DBUG:" prefix,
to make the *.result files more readable.
Also changing level from WARN_LEVEL_WARN to WARN_LEVEL_NOTE.
This patch implements the task according to the description:
1. The old code from Item_func_in::fix_length_and_dec() was decomposed
into smaller methods:
- all_items_are_consts()
- compatible_types_scalar_bisection_possible()
- compatible_types_row_bisection_possible()
- fix_in_vector()
- fix_for_scalar_comparison_using_bisection()
- fix_for_scalar_comparison_using_cmp_items()
- fix_for_row_comparison_using_bisection()
- fix_for_row_comparison_using_cmp_items()
The data type dependend pieces where moved as methods to Type_handler.
2. Splits in_datetime into separate:
- in_datetime, for DATETIME and DATE,
- in_time, for TIME
to make the code more symmetric across data types.
Additionally:
- Adds a test func_debug.test to see which calculation strategy
(bisect or no bisect) is chosen to handle IN with various arguments.
- Adds a new helper method (to avoid duplicate code):
cmp_item_rows::prepare_comparators()
- Changes the propotype for cmp_item_row::alloc_comparators(),
to avoid duplicate code, and to use less current_thd.
- Changes "friend" sections in cmp_item_row and in_row from
an exact Item_func_in method to the entire class Item_func_in,
as their internals are now needed in multiple Item_func_in methods.
- Added more comments (e.g. on bisection, on the problem reported in MDEV-11511)
- Removes "Item_result Item_func_opt_neg::m_compare_type" and introduces
"Type_handler_hybrid_field_type Item_func_opt_neg::m_comparator" instead.
- Removes Item_func_between::compare_as_dates, because
the new member m_comparator now contains the precise information
about the data type that is used for comparison, which is important
for TIME vs DATETIME.
- Adds a new method:
Type_handler_hybrid_field_type::aggregate_for_comparison(const Type_handler*),
as a better replacement for item_cmp_type(), which additionally can handle
TIME vs DATE/DATETIME/TIMESTAMP correctly. Additionally, it correctly
handles TIMESTAMP which fixes the problem reported in MDEV-11482.
The old compare_as_dates/find_date_time_item() based code didn't handle
comparison between TIME and TIMESTAMP correctly and erroneously used TIME
comparison instead of DATETIME comparison.
- Adds a new method:
Type_handler_hybrid_field_type::aggregate_for_comparison(Item **, uint nitems),
as a better replacement for agg_cmp_type(), which can handle TIME.
- Splits Item_func_between::val_int() into pieces val_int_cmp_xxx(),
one new method per XXX_RESULT.
- Adds a new virtual method Type_handler::Item_func_between_val_int()
whose implementations use Item_func_between::val_int_cmp_xxx().
- Makes type_handler_longlong and type_handler_newdecimal public,
as they are now needed in item_cmpfunc.cc.
Note:
This patch does not change Item_func_in to use the new aggregation methods,
so it still uses collect_cmp_type()/item_cmp_type() based aggregation.
Item_func_in will be changed in a separate patch and item_cmp_type() will be
removed.
This patch:
- Adds a new virtual method Type_handler::Item_get_cache
- Splits moves Item_cache::get_cache() into the new method, every
"case XXX_RESULT" to the corresponding Type_handler_xxx::Item_get_cache.
- Adds Item::get_cache as a convenience wrapper, to make the caller code
shorter.
- Changes the last argument of Arg_comparator::cache_converted_constant()
from Item_result to "const Type_handler *".
- Removes subselect_engine::cmp_type, subselect_engine::res_type,
subselect_engine::res_field_type and derives subselect_engine
from Type_handler_hybrid_field_type instead.
- Makes Type_handler_varchar public, as it's now needed as the
default data type handler for subselect_engine.
This patch:
- Introduces a new virtuial method Type_handler::set_comparator_func
and moves pieces of the code from the switch in
Arg_comparator::set_compare_func into the corresponding
Type_handler_xxx::set_comparator_func.
- Adds Type_handler::get_handler_by_cmp_type()
- Moves Type_handler_hybrid_field_type::get_handler_by_result_type() to
a static method Type_handler::get_handler_by_result_type(),
for symmetry with similar methods:
* Type_handler::get_handler_by_field_type()
* Type_handler::get_handler_by_real_type()
* Type_handler::get_handler_by_cmp_type()
- Introduces Type_handler_row, to unify the code for the scalar
data types and the ROW data type (currently for comparison purposes only).
- Adds public type_handler_row, as it's now needed in item_row.h
- Makes type_handler_null public, as it's now needed in item_cmpfunc.h
Note, other type_handler_xxx will become public as well later.
- Removes the global variable Arg_comparator::comparator_matrix,
as it's not needed any more.
10.1 introduced a problem:
Execution time for various recursive stages
(walk, update_used_table, and propagate_equal_fields)
in NULLIF is O(recursion_level^2), because complexity is
doubled on every recursion level when we copy args[0] to args[2].
This change fixes to avoid unnecessary recursion in:
- Item_func_nullif::walk
- Item_func_nullif::update_used_tables
- Item_func_nullif::propagate_equal_fields
when possible.
Integer comparison of INT expressions with different signess in BETWEEN
is not safe. Switching to DECIMAL comparison in case if INT arguments
have different signess.
This is a backport of the patch for MDEV-9653 (fixed earlier in 10.1.13).
The code in Item_func_case::fix_length_and_dec() did not
calculate max_length and decimals properly.
In case of any numeric result (DECIMAL, REAL, INT) a generic method
Item_func_case::agg_num_lengths() was called, which could erroneously result
into a DECIMAL item with max_length==0 and decimals==0, so the constructor of
Field_new_decimals tried to create a field of DECIMAL(0,0) type,
which caused a crash.
Unlike Item_func_case, the code responsible for merging attributes in
Item_func_coalesce::fix_length_and_dec() works fine: it has specific execution
branches for all distinct numeric types and correctly creates a DECIMAL(1,0)
column instead of DECIMAL(0,0) for the same set of arguments.
The fix does the following:
- Moves the attribute merging code from Item_func_coalesce::fix_length_and_dec()
to a new method Item_func_hybrid_result_type::fix_attributes()
- Removes the wrong code from Item_func_case::fix_length_and_dec()
and reuses fix_attributes() in both Item_func_coalesce::fix_length_and_dec()
and Item_func_case::fix_length_and_dec()
- Fixes count_real_length() and count_decimal_length() to get an array
of Items as an argument, instead of using Item::args directly.
This is needed for Item_func_case::fix_length_and_dec().
- Moves methods Item_func::count_xxx_length() from "public" to "protected".
- Removes Item_func_case::agg_num_length(), as it's not used any more.
- Additionally removes Item_func_case::agg_str_length(),
as it also was not used (dead code).
Item_func_ifnull::date_op() and Item_func_coalesce::date_op() could
erroneously return 0000-00-00 instead of NULL when get_date()
was called with the TIME_FUZZY_DATES flag, e.g. from LEAST().
- Removing the "diff_if_only_endspace_difference" argument from
MY_COLLATION_HANDLER::strnncollsp(), my_strnncollsp_simple(),
as well as in the function template MY_FUNCTION_NAME(strnncollsp)
in strcoll.ic
- Removing the "diff_if_only_space_different" from ha_compare_text(),
hp_rec_key_cmp().
- Adding a new function my_strnncollsp_padspace_bin() and reusing
it instead of duplicate code pieces in my_strnncollsp_8bit_bin(),
my_strnncollsp_latin1_de(), my_strnncollsp_tis620(),
my_strnncollsp_utf8_cs().
- Adding more tests for better coverage of the trailing space handling.
- Removing the unused definition of HA_END_SPACE_ARE_EQUAL
* only copy args[0] to args[2] after fix_fields (when all item
substitutions have already happened)
* change QT_ITEM_FUNC_NULLIF_TO_CASE (that allows to print NULLIF
as CASE) to QT_ITEM_ORIGINAL_FUNC_NULLIF (that prohibits it).
So that NULLIF-to-CASE is allowed by default and only disabled
explicitly for SHOW VIEW|FUNCTION|PROCEDURE and mysql_make_view.
By default it is allowed (in particular in error messages and
debug output, that can happen anytime before or after optimizer).
don't do special SUM_FUNC_ITEM treatment in NULLIF for views
(as before), but do it for derived tables (when
context_analysis_only == CONTEXT_ANALYSIS_ONLY_DERIVED)
"Re-factor the code for post-join operations".
The patch mainly contains the code ported from mysql-5.6 and
created for two essential architectural changes:
1. WL#5558: Resolve ORDER BY execution method at the optimization stage
2. WL#6071: Inline tmp tables into the nested loops algorithm
The first task was implemented for mysql-5.6 by Ole John Aske.
It allows to make all decisions on ORDER BY operation at the optimization
stage.
The second task implemented for mysql-5.6 by Evgeny Potemkin adds JOIN_TAB
nodes for post-join operations that require temporary tables. It allows
to execute these operations within the nested loops algorithm that used to
be used before this task only for join queries. Besides these task moves
all planning on the execution of these operations from the execution phase
to the optimization phase.
Some other re-factoring changes of mysql-5.6 were pulled in, mainly because
it was easier to pull them in than roll them back. In particular all
changes concerning Ref_ptr_array were incorporated.
The port required some changes in the MariaDB code that concerned the
functionality of EXPLAIN and ANALYZE. This was done mainly by Sergey
Petrunia.
DBUG_ASSERT() added in the patch for MDEV-9181 did not take into account
special circumstances for the prepared statement EXECUTE. Fixig the assert.
Also, extending and fixing comments made during MDEV-9181.
MDEV-9408 CREATE TABLE SELECT MAX(int_column) creates different columns for table vs view
There were three almost identical pieces of the code:
- Field *Item_func::tmp_table_field();
- Field *Item_sum::create_tmp_field();
- Field *create_tmp_field_from_item();
with a difference in very small details (hence the bugs):
Only Item_func::tmp_table_field() was correct, the other two were not.
Removing the two incorrect pieces of the redundant code.
Joining these three functions/methods into a single virtual method
Item::create_tmp_field().
Additionally, moving Item::make_string_field() and
Item::tmp_table_field_from_field_type() from the public into the
protected section of the class declaration, as they are now not
needed outside of Item.
The following left in semi-improved state to keep patch size reasonable:
- Field operator new: left thd_alloc(current_thd)
- Sql_alloc operator new: left thd_alloc(thd_get_current_thd())
- Item_args constructors: left thd_alloc(thd)
- Item_func_interval::fix_length_and_dec(): no THD arg, have to call current_thd
- Item_func_dyncol_exists::val_int(): same
- Item_dyncol_get::val_str(): same
- Item_dyncol_get::val_int(): same
- Item_dyncol_get::val_real(): same
- Item_dyncol_get::val_decimal(): same
- Item_singlerow_subselect::fix_length_and_dec(): same
Problem:
At the end of first execution select_lex->prep_where is pointing to
a runtime created object (temporary table field). As a result
server exits trying to access a invalid pointer during second
execution.
Analysis:
While optimizing the join conditions for the query, after the
permanent transformation, optimizer makes a copy of the new
where conditions in select_lex->prep_where. "prep_where" is what
is used as the "where condition" for the query at the start of execution.
W.r.t the query in question, "where" condition is actually pointing
to a field in the temporary table. As a result, for the second
execution the pointer is no more valid resulting in server exit.
Fix:
At the end of the first execution, select_lex->where will have the
original item of the where condition.
Make prep_where the new place where the original item of select->where
has to be rolled back.
Fixed in 5.7 with the wl#7082 - Move permanent transformations from
JOIN::optimize to JOIN::prepare
Patch for 5.5 includes the following backports from 5.6:
Bugfix for Bug12603141 - This makes the first execute statement in the testcase
pass in 5.5
However it was noted later in in Bug16163596 that the above bugfix needed to
be modified. Although Bug16163596 is reproducible only with changes done for
Bug12582849, we have decided include the fix.
Considering that Bug12582849 is related to Bug12603141, the fix is
also included here. However this results in Bug16317817, Bug16317685,
Bug16739050. So fix for the above three bugs is also part of this patch.
Problem was in rewriting left expression which had 2 references on it. Solved with making subselect reference main.
Item_in_optimized can have not Item_in_subselect reference in left part so type casting with no check is dangerous.
Item::cols() should be checked after Item::fix_fields().
Item_func_hybrid_field_type did not return correct field_type(), cmp_type()
and result_type() in some cases, because cached_result_type and
cached_field_type were set in independent pieces of the code and
did not properly match to each other.
Fix:
- Removing Item_func_hybrid_result_type
- Deriving Item_func_hybrid_field_type directly from Item_func
- Introducing a new class Type_handler which guarantees that
field_type(), cmp_type() and result_type() are always properly synchronized
and using the new class in Item_func_hybrid_field_type.
1. Removing the legacy code that disabled equal field propagation in cases
when comparison is done as VARBINARY. This is now correctly handled by
the new propagation code in Item_xxx::propagate_equal_fields() and
Field_str::can_be_substituted_to_equal_item (the bug fix).
2. Also, removing legacy (pre-MySQL-4.1) Arg_comparator methods
compare_binary_string() and compare_e_binary_string(), as VARBINARY
comparison is correcty handled in compare_string() and compare_e_string() by
the corresponding VARBINARY collation handler implemented in my_charset_bin.
(not really a part of the bug fix)
WHERE COALESCE(time_column)=TIME('00:00:00')
AND COALESCE(time_column)=DATE('2015-09-11')
MDEV-8814 Wrong result for WHERE datetime_column > TIME('00:00:00')
__MEMMOVE_SSSE3_BACK FROM STRING::COPY
Issue:
-----
While using row comparators, the store_value functions call
val_xxx functions in the prepare phase. This can cause
valgrind issues.
SOLUTION:
---------
Setting up of the comparators should be done by
alloc_comparators in the prepare phase. Also, make sure
store_value will be called only during execute phase.
This is a backport of the fix for Bug#17755540.
MDEV-8754 Wrong result for SELECT..WHERE year_field=2020 AND NULLIF(year_field,2010)='2020'
Problems:
1. Item_func_nullif stored a copy of args[0] in a private member m_args0_copy,
which was invisible for the inherited Item_func menthods, like
update_used_tables(). As a result, after equal field propagation
things like Item_func_nullif::const_item() could return wrong result
and a non-constant NULLIF() was erroneously treated as a constant
at optimize_cond() time.
Solution: removing m_args0_copy and storing the return value item
in args[2] instead.
2. Equal field propagation did not work well for Item_fun_nullif.
Solution: using ANY_SUBST for args[0] and args[1], as they are in
comparison, and IDENTITY_SUBST for args[2], as it's not in comparison.
removing IMPOSSIBLE_RESULT from Item_result, as it's not
needed any more. The fact that an Item is not in a comparison
context is now always designated by IDENTITY_SUBST in Subst_constraint.
Previously IMPOSSIBLE_RESULT and IDENTITY_SUBST co-existed but
actually meant the same thing.
Item::cmp_context was inconsistently used in combination with cmp_type()
and result_type() in different places of the code. Fixed to use cmp_type()
in all places where cmp_context is involved, to avoid unexpected results
for temporal data types (which have result_type()==STRING_RESULT and
cmp_type==TIME_RESULT).
- Part 4: Removing calls to sql_alloc() and sql_calloc()
Other things:
- Added current_thd in some places to make it clear that it's called (easier to remove later)
- Move memory allocation from Item_func_case::fix_length_and_dec() to Item_func_case::fix_fields()
- Added mem_root to some new calls
- Fixed some wrong UNINIT_VAR() calls
- Fixed a bug in generate_partition_syntax() in case of errors
- Added mem_root to argument to new thread_info
- Simplified my_parse_error() call in sql_yacc.yy
- Part 3: Adding mem_root to push_back() and push_front()
Other things:
- Added THD as an argument to some partition functions.
- Added memory overflow checking for XML tag's in read_xml()
- Added mem_root to all calls to new Item
- Added private method operator new(size_t size) to Item to ensure that
we always use a mem_root when creating an item.
This saves use once call to current_thd per Item creation
Added mandatory thd parameter to Item (and all derivative classes) constructor.
Added thd parameter to all routines that may create items.
Also removed "current_thd" from Item::Item. This reduced number of
pthread_getspecific() calls from 290 to 177 per OLTP RO transaction.
- Changed ER(ER_...) to ER_THD(thd, ER_...) when thd was known or if there was many calls to current_thd in the same function.
- Changed ER(ER_..) to ER_THD_OR_DEFAULT(current_thd, ER...) in some places where current_thd is not necessary defined.
- Removing calls to current_thd when we have access to thd
Part of this is optimization (not calling current_thd when not needed),
but part is bug fixing for error condition when current_thd is not defined
(For example on startup and end of mysqld)
Notable renames done as otherwise a lot of functions would have to be changed:
- In JOIN structure renamed:
examined_rows -> join_examined_rows
record_count -> join_record_count
- In Field, renamed new_field() to make_new_field()
Other things:
- Added DBUG_ASSERT(thd == tmp_thd) in Item_singlerow_subselect() just to be safe.
- Removed old 'tab' prefix in JOIN_TAB::save_explain_data() and use members directly
- Added 'thd' as argument to a few functions to avoid calling current_thd.
Fixed several optimizer issues relatied to GROUP BY:
a) Refering to a SELECT column in HAVING sometimes calculated it twice, which caused problems with non determinstic functions
b) Removing duplicate fields and constants from GROUP BY was done too late for "using index for group by" optimization to work
c) EXPLAIN SELECT ... GROUP BY did wrongly show 'Using filesort' in some cases involving "Using index for group-by"
a) was fixed by:
- Changed last argument to Item::split_sum_func2() from bool to int to allow more flags
- Added flag argument to Item::split_sum_func() to allow on to specify if the item was in the SELECT part
- Mark all split_sum_func() calls from SELECT with SPLIT_SUM_SELECT
- Changed split_sum_func2() to do nothing if called with an argument that is not a sum function and doesn't include sum functions, if we are not an argument to SELECT.
This ensures that in a case like
select a*sum(b) as f1 from t1 where a=1 group by c having f1 <= 10;
That 'a' in the SELECT part is stored as a reference in the temporary table togeher with sum(b) while the 'a' in having isn't (not needed as 'a' is already a reference to a column in the result)
b) was fixed by:
- Added an extra remove_const() pass for GROUP BY arguments before make_join_statistics() in case of one table SELECT.
This allowes get_best_group_min_max() to optimize things better.
c) was fixed by:
- Added test for group by optimization in JOIN::exec_inner for
select->quick->get_type() == QUICK_SELECT_I::QS_TYPE_GROUP_MIN_MAX
item.cc:
- Simplifed Item::split_sum_func2()
- Split test to make them faster and easier to read
- Changed last argument to Item::split_sum_func2() from bool to int to allow more flags
- Added flag argument to Item::split_sum_func() to allow on to specify if the item was in the SELECT part
- Changed split_sum_func2() to do nothing if called with an argument that is not a sum function and doesn't include sum functions, if we are not an argument to SELECT.
opt_range.cc:
- Simplified get_best_group_min_max() by calcuating first how many group_by elements.
- Use join->group instead of join->group_list to test if group by, as join->group_list may be NULL if everything was optimized away.
sql_select.cc:
- Added an extra remove_const() pass for GROUP BY arguments before make_join_statistics() in case of one table SELECT.
- Use group instead of group_list to test if group by, as group_list may be NULL if everything was optimized away.
- Moved printing of "Error in remove_const" to remove_const() instead of having it in caller.
- Simplified some if tests by re-ordering code.
- update_depend_map_for_order() and remove_const() fixed to handle the case where make_join_statistics() has not yet been called (join->join_tab is 0 in this case)
count_sargable_conds() instead for Item_func_in, Item_func_null_predicate,
Item_bool_func2. There other Item_int_func descendants that used to set
"sargable" to true (Item_func_between, Item_equal) already have their
own implementation of count_sargable_conds(). There is no sense to
have two parallel coding models for the same thing.
Pass THD to find_all_keys() and Item_equal::Item_equal().
In MRR use table->in_use instead of current_thd.
This reduces number of pthread_getspecific() calls from 354 to 320.
- Changing Comp_creator::create() and create_swap() to return
Item_bool_rowready_func2 instead of Item_bool_func2, as they
can never return neither Item_func_like nor Item_func_xor
- Changing the first argument of Comp_create::create() and create_swap()
from THD to MEM_ROOT, so the method implementations can now reside in
item_cmpfunc.h instead of item_cmpfunc.cc and thus make the code slightly
easier to read.
sql_alloc() has additional costs compared to direct mem_root allocation:
- function call: it is defined in a separate translation unit and can't be
inlined
- it needs to call pthread_getspecific() to get THD::mem_root
It is called dozens of times implicitely at least by:
- List<>::push_back()
- List<>::push_front()
- new (for Sql_alloc derived classes)
- sql_memdup()
Replaced lots of implicit sql_alloc() calls with direct mem_root allocation,
passing through THD pointer whenever it is needed.
Number of sql_alloc() calls reduced 345 -> 41 per OLTP RO transaction.
pthread_getspecific() overhead dropped 0.76 -> 0.59
sql_alloc() overhed dropped 0.25 -> 0.06
- Adding a new class Item_args, represending regular function or
aggregate function arguments array.
- Adding a new class Item_func_or_sum,
a parent class for Item_func and Item_sum
- Moving Item_result_field::name() to Item_func_or_sum(),
as name() is not needed on Item_result_field level.
Fixing a wrong assymetric code in Arg_comparator::set_cmp_func().
It existed for a long time, but showed up in 10.0.14 after the fix
for "MDEV-6666 Malformed result for CONCAT(utf8_column, binary_string)".
The bug is not very important per se, but it was helpful to move
Item_func_strcmp out of Item_bool_func2 (to Item_int_func),
for the purposes of "MDEV-4912 Add a plugin to field types (column types)".
The reason for the failure was a bug in an include file on debian that causes 'struct stat'
to have different sized depending on the environment.
This patch fixes so that we always include my_global.h or my_config.h before we include any other files.
Other things:
- Removed #include <my_global.h> in some include files; Better to always do this at the top level to have as few
"always-include-this-file-first' files as possible.
- Removed usage of some include files that where already included by my_global.h or by other files.
client/mysql_plugin.c:
Use my_global.h first
client/mysqlslap.c:
Remove duplicated include files
extra/comp_err.c:
Remove duplicated include files
include/m_string.h:
Remove duplicated include files
include/maria.h:
Remove duplicated include files
libmysqld/emb_qcache.cc:
Use my_global.h first
plugin/semisync/semisync.h:
Use my_pthread.h first
sql/datadict.cc:
Use my_global.h first
sql/debug_sync.cc:
Use my_global.h first
sql/derror.cc:
Use my_global.h first
sql/des_key_file.cc:
Use my_global.h first
sql/discover.cc:
Use my_global.h first
sql/event_data_objects.cc:
Use my_global.h first
sql/event_db_repository.cc:
Use my_global.h first
sql/event_parse_data.cc:
Use my_global.h first
sql/event_queue.cc:
Use my_global.h first
sql/event_scheduler.cc:
Use my_global.h first
sql/events.cc:
Use my_global.h first
sql/field.cc:
Use my_global.h first
Remove duplicated include files
sql/field_conv.cc:
Use my_global.h first
sql/filesort.cc:
Use my_global.h first
Remove duplicated include files
sql/gstream.cc:
Use my_global.h first
sql/ha_ndbcluster.cc:
Use my_global.h first
sql/ha_ndbcluster_binlog.cc:
Use my_global.h first
sql/ha_ndbcluster_cond.cc:
Use my_global.h first
sql/ha_partition.cc:
Use my_global.h first
sql/handler.cc:
Use my_global.h first
sql/hash_filo.cc:
Use my_global.h first
sql/hostname.cc:
Use my_global.h first
sql/init.cc:
Use my_global.h first
sql/item.cc:
Use my_global.h first
sql/item_buff.cc:
Use my_global.h first
sql/item_cmpfunc.cc:
Use my_global.h first
sql/item_create.cc:
Use my_global.h first
sql/item_geofunc.cc:
Use my_global.h first
sql/item_inetfunc.cc:
Use my_global.h first
sql/item_row.cc:
Use my_global.h first
sql/item_strfunc.cc:
Use my_global.h first
sql/item_subselect.cc:
Use my_global.h first
sql/item_sum.cc:
Use my_global.h first
sql/item_timefunc.cc:
Use my_global.h first
sql/item_xmlfunc.cc:
Use my_global.h first
sql/key.cc:
Use my_global.h first
sql/lock.cc:
Use my_global.h first
sql/log.cc:
Use my_global.h first
sql/log_event.cc:
Use my_global.h first
sql/log_event_old.cc:
Use my_global.h first
sql/mf_iocache.cc:
Use my_global.h first
sql/mysql_install_db.cc:
Remove duplicated include files
sql/mysqld.cc:
Remove duplicated include files
sql/net_serv.cc:
Remove duplicated include files
sql/opt_range.cc:
Use my_global.h first
sql/opt_subselect.cc:
Use my_global.h first
sql/opt_sum.cc:
Use my_global.h first
sql/parse_file.cc:
Use my_global.h first
sql/partition_info.cc:
Use my_global.h first
sql/procedure.cc:
Use my_global.h first
sql/protocol.cc:
Use my_global.h first
sql/records.cc:
Use my_global.h first
sql/records.h:
Don't include my_global.h
Better to do this at the upper level
sql/repl_failsafe.cc:
Use my_global.h first
sql/rpl_filter.cc:
Use my_global.h first
sql/rpl_gtid.cc:
Use my_global.h first
sql/rpl_handler.cc:
Use my_global.h first
sql/rpl_injector.cc:
Use my_global.h first
sql/rpl_record.cc:
Use my_global.h first
sql/rpl_record_old.cc:
Use my_global.h first
sql/rpl_reporting.cc:
Use my_global.h first
sql/rpl_rli.cc:
Use my_global.h first
sql/rpl_tblmap.cc:
Use my_global.h first
sql/rpl_utility.cc:
Use my_global.h first
sql/set_var.cc:
Added comment
sql/slave.cc:
Use my_global.h first
sql/sp.cc:
Use my_global.h first
sql/sp_cache.cc:
Use my_global.h first
sql/sp_head.cc:
Use my_global.h first
sql/sp_pcontext.cc:
Use my_global.h first
sql/sp_rcontext.cc:
Use my_global.h first
sql/spatial.cc:
Use my_global.h first
sql/sql_acl.cc:
Use my_global.h first
sql/sql_admin.cc:
Use my_global.h first
sql/sql_analyse.cc:
Use my_global.h first
sql/sql_audit.cc:
Use my_global.h first
sql/sql_base.cc:
Use my_global.h first
sql/sql_binlog.cc:
Use my_global.h first
sql/sql_bootstrap.cc:
Use my_global.h first
Use my_global.h first
sql/sql_cache.cc:
Use my_global.h first
sql/sql_class.cc:
Use my_global.h first
sql/sql_client.cc:
Use my_global.h first
sql/sql_connect.cc:
Use my_global.h first
sql/sql_crypt.cc:
Use my_global.h first
sql/sql_cursor.cc:
Use my_global.h first
sql/sql_db.cc:
Use my_global.h first
sql/sql_delete.cc:
Use my_global.h first
sql/sql_derived.cc:
Use my_global.h first
sql/sql_do.cc:
Use my_global.h first
sql/sql_error.cc:
Use my_global.h first
sql/sql_explain.cc:
Use my_global.h first
sql/sql_expression_cache.cc:
Use my_global.h first
sql/sql_handler.cc:
Use my_global.h first
sql/sql_help.cc:
Use my_global.h first
sql/sql_insert.cc:
Use my_global.h first
sql/sql_lex.cc:
Use my_global.h first
sql/sql_load.cc:
Use my_global.h first
sql/sql_locale.cc:
Use my_global.h first
sql/sql_manager.cc:
Use my_global.h first
sql/sql_parse.cc:
Use my_global.h first
sql/sql_partition.cc:
Use my_global.h first
sql/sql_plugin.cc:
Added comment
sql/sql_prepare.cc:
Use my_global.h first
sql/sql_priv.h:
Added error if we use this before including my_global.h
This check is here becasue so many files includes sql_priv.h first.
sql/sql_profile.cc:
Use my_global.h first
sql/sql_reload.cc:
Use my_global.h first
sql/sql_rename.cc:
Use my_global.h first
sql/sql_repl.cc:
Use my_global.h first
sql/sql_select.cc:
Use my_global.h first
sql/sql_servers.cc:
Use my_global.h first
sql/sql_show.cc:
Added comment
sql/sql_signal.cc:
Use my_global.h first
sql/sql_statistics.cc:
Use my_global.h first
sql/sql_table.cc:
Use my_global.h first
sql/sql_tablespace.cc:
Use my_global.h first
sql/sql_test.cc:
Use my_global.h first
sql/sql_time.cc:
Use my_global.h first
sql/sql_trigger.cc:
Use my_global.h first
sql/sql_udf.cc:
Use my_global.h first
sql/sql_union.cc:
Use my_global.h first
sql/sql_update.cc:
Use my_global.h first
sql/sql_view.cc:
Use my_global.h first
sql/sys_vars.cc:
Added comment
sql/table.cc:
Use my_global.h first
sql/thr_malloc.cc:
Use my_global.h first
sql/transaction.cc:
Use my_global.h first
sql/uniques.cc:
Use my_global.h first
sql/unireg.cc:
Use my_global.h first
sql/unireg.h:
Removed inclusion of my_global.h
storage/archive/ha_archive.cc:
Added comment
storage/blackhole/ha_blackhole.cc:
Use my_global.h first
storage/csv/ha_tina.cc:
Use my_global.h first
storage/csv/transparent_file.cc:
Use my_global.h first
storage/federated/ha_federated.cc:
Use my_global.h first
storage/federatedx/federatedx_io.cc:
Use my_global.h first
storage/federatedx/federatedx_io_mysql.cc:
Use my_global.h first
storage/federatedx/federatedx_io_null.cc:
Use my_global.h first
storage/federatedx/federatedx_txn.cc:
Use my_global.h first
storage/heap/ha_heap.cc:
Use my_global.h first
storage/innobase/handler/handler0alter.cc:
Use my_global.h first
storage/maria/ha_maria.cc:
Use my_global.h first
storage/maria/unittest/ma_maria_log_cleanup.c:
Remove duplicated include files
storage/maria/unittest/test_file.c:
Added comment
storage/myisam/ha_myisam.cc:
Move sql_plugin.h first as this includes my_global.h
storage/myisammrg/ha_myisammrg.cc:
Use my_global.h first
storage/oqgraph/oqgraph_thunk.cc:
Use my_config.h and my_global.h first
One could not include my_global.h before oqgraph_thunk.h (don't know why)
storage/spider/ha_spider.cc:
Use my_global.h first
storage/spider/hs_client/config.cpp:
Use my_global.h first
storage/spider/hs_client/escape.cpp:
Use my_global.h first
storage/spider/hs_client/fatal.cpp:
Use my_global.h first
storage/spider/hs_client/hstcpcli.cpp:
Use my_global.h first
storage/spider/hs_client/socket.cpp:
Use my_global.h first
storage/spider/hs_client/string_util.cpp:
Use my_global.h first
storage/spider/spd_conn.cc:
Use my_global.h first
storage/spider/spd_copy_tables.cc:
Use my_global.h first
storage/spider/spd_db_conn.cc:
Use my_global.h first
storage/spider/spd_db_handlersocket.cc:
Use my_global.h first
storage/spider/spd_db_mysql.cc:
Use my_global.h first
storage/spider/spd_db_oracle.cc:
Use my_global.h first
storage/spider/spd_direct_sql.cc:
Use my_global.h first
storage/spider/spd_i_s.cc:
Use my_global.h first
storage/spider/spd_malloc.cc:
Use my_global.h first
storage/spider/spd_param.cc:
Use my_global.h first
storage/spider/spd_ping_table.cc:
Use my_global.h first
storage/spider/spd_sys_table.cc:
Use my_global.h first
storage/spider/spd_table.cc:
Use my_global.h first
storage/spider/spd_trx.cc:
Use my_global.h first
storage/xtradb/handler/handler0alter.cc:
Use my_global.h first
storage/xtradb/handler/i_s.cc:
Use my_global.h first
This is port of fix for MySQL BUG#17647863.
revno: 5572
revision-id: jon.hauglid@oracle.com-20131030232243-b0pw98oy72uka2sj
committer: Jon Olav Hauglid <jon.hauglid@oracle.com>
timestamp: Thu 2013-10-31 00:22:43 +0100
message:
Bug#17647863: MYSQL DOES NOT COMPILE ON OSX 10.9 GM
Rename test() macro to MY_TEST() to avoid conflict with libc++.
The earlier pushed fix for the bug was incomplete. It did not remove
the main cause of the problem: the function remove_eq_conds()
removed always true multiple equalities from any conjunct, but did not
adjust the list of them stored in Item_cond_and::cond_equal.current_level.
Simplified the test case for the bug and moved it to another test file.
The fix triggered changes in EXPLAIN EXTENDED for some queries.
1. Transformation of row IN subquery made the same as single value.
2. replace_where_subcondition() made working on several layers of OR/AND because it called on expression before fix_fields().
Singular Item_equal support added.
The problem was that during constant table substitution Item_equal become containing only one constant which was not supported internally.
Objects of the classes Item_func_isnull and Item_func_isnotnull
must have the flag sargable set to TRUE.
Set the value of the flag sargable only in constructors of the
classes inherited from Item_int_func.
The bug caused a memory overwrite in the function update_ref_and_keys()
It happened due to a wrong value of SELECT_LEX::cond_count. This value
historically was calculated by the fix_fields method. Now the logic of
calling this method became too complicated and, as a result, this value
is calculated not always correctly.
The patch changes the way how and when the values of SELECT_LEX::cond_count
and of SELECT_LEX::between_count are calculated. The new code does it just at
the beginning of update_ref_and_keys().
Other fix of maybe_null problem and revert of revno: 3608 "MDEV-3873 & MDEV-3876 & MDEV-3912 : Wrong result (extra rows) with ALL subquery from a MERGE view."
