- Adding new virtual methods in Type_handler:
* Column_definition_prepare_stage1()
* Column_definition_prepare_stage2()
* calc_pack_length()
- Using new methods to remove type specific code in:
* Global function calc_pack_length()
* Column_definition::prepare_create_field()
* The loop body mysql_prepare_create_table()
* Column_definition::sp_prepare_create_field()
1. Implementing the task according to the description:
a. Adding Type_handler::type_handler_for_tmp_table().
b. Adding Type_handler::type_handler_for_union_table.
c. Adding helper methods Type_handler::varstring_type_handler(const Item*),
Type_handler::blob_type_handler(const Item*)
d. Removing Item::make_string_field() and
Item_func_group_concat::make_string_field().
They are not needed any more.
e. Simplifying Item::tmp_table_field_from_field_type() to just two lines.
f. Renaming Item_type_holder::make_field_by_type() and implementing
virtual Item_type_holder::create_tmp_field() instead.
The new implementation is also as simple as two lines.
g. Adding a new virtual method Type_all_attributes::get_typelib(),
to access to TYPELIB definitions for ENUM and SET columns.
h. Simplifying the code branch for TIME_RESULT, DECIMAL_RESULT, STRING_RESULT
in Item::create_tmp_field(). It's now just one line.
i. Implementing Type_handler_enum::make_table_field() and
Type_handler_set::make_table_field().
2. Code simplification in Field_str constructor calls.
a. Changing the "CHARSET_INFO *cs" argument in constuctors for Field_str
and its descendants to "const DTCollation &collation". This is to
avoid two step initialization:
- setting Field_str::derivation and Field_str::repertoire to the
default values first
- then resetting them using:
set_derivation(item->derivation, item->repertoire).
b. Removing Field::set_derivation()
c. Adding a new constructor DTCollation(CHARSET_INFO *cs),
for the old code compatibility.
3. Changes in test results
As a side effect some test results have changed, because
in the old version Item::make_string_field() converted
TINYBLOB to VARCHAR(255). Now TINYBLOB is preserved.
a. sp-row.result
This query:
CREATE TABLE t1 AS SELECT tinyblob_sp_variable;
Now preserves TINYBLOB as the data type.
Before the patch a VARCHAR(255) was created.
b. gis-debug.result
This is a debug test, to make sure that + and - operators
are commutative and non-commutative correspondingly.
The exact data type is not really important.
(But anyway, it now chooses a better data type that fits the result)
This is a joint patch for:
- MDEV-12426 Add Field::type_handler()
- MDEV-12432 Range optimizer for ENUM and SET does not return "Impossible WHERE" in some case
With the new type handler approach being added to Field, it was easier to fix
MDEV-12432 rather than to reproduce the old ENUM/SET behavior.
The patch does the following:
1. Adds Field::type_handler(), according to the task description.
2. Fixes the asymmetry between Fields and Items of ENUM and SET field types.
Field_enum::cmp_type() returned INT_RESULT
Item*::cmp_type() returned STRING_RESULT for ENUM and SET expressions
This asymmetry was originally done for easier coding in the optimizer sources.
However, in 10.1 we moved a lot of code to methods of the class Field:
- test_if_equality_guarantees_uniqueness()
- can_be_substituted_to_equal_item()
- get_equal_const_item()
- can_optimize_keypart_ref()
- can_optimize_hash_join()
- can_optimize_group_min_max()
- can_optimize_range()
- can_optimize_outer_join_table_elimination()
As of 10.2 only a few lines of the code in opt_range.cc, field.cc and field.h
still relayed on the fact that Field_enum::cmp_type() returns INT_RESULT:
- Some asserts in field.cc
- Field_year::get_copy_func()
- Item_func_like::get_mm_leaf()
- Item_bool_func::get_mm_leaf()
These lines have been fixed.
3. Item_bool_func::get_mm_leaf() did not work well for ENUM/SET,
see MDEV-12432. So the ENUM/SET code was rewritten, and the relevant
code in Field_enum::store() and Field_set::store() was fixed to
properly return errors to the caller.
4. The result of Field_decimal::result_type() was changed from REAL_RESULT
to DECIMAL_RESULT. Data type aggregation (e.g. in COALESCE()) is now more
precise for old DECIMAL, because Item::decimal_precision() now goes through
the DECIMAL_RESULT branch. Earlier it went through the REAL_RESULT branch.
- Adding Type_handler::make_table_field() and moving pieces of the code
from Item::tmp_table_field_from_field_type() to virtual implementations
for various type handlers.
- Adding a new Type_all_attributes, to access to Item's extended
attributes, such as decimal_precision() and geometry_type().
- Adding a new class Record_addr, to pass record related information
to Type_handler methods (ptr, null_ptr and null_bit) as a single structure.
Note, later it will possibly be extended for BIT-alike field purposes,
by adding new members (bit_ptr_arg, bit_ofs_arg).
- Moving the code from Field_new_decimal::create_from_item()
to Type_handler_newdecimal::make_table_field().
- Removing Field_new_decimal() and Field_geom() helper constructor
variants that were used for temporary field creation.
- Adding Item_field::type_handler(), Field::type_handler() and
Field_blob::type_handler() to return correct type handlers for
blob variants, according to Field_blob::packlength.
- Adding Type_handler_blob_common, as a common parent for
Type_handler_tiny_blob, Type_handler_blob, Type_handler_medium_blob
and Type_handler_long_blob.
- Implementing Type_handler_blob_common::Item_hybrid_func_fix_attributes().
It's needed for cases when TEXT variants of different character sets are mixed
in LEAST, GREATEST, CASE and its abreviations (IF, IFNULL, COALESCE), e.g.:
CREATE TABLE t1 (
a TINYTEXT CHARACTER SET latin1,
b TINYTEXT CHARACTER SET utf8
);
CREATE TABLE t2 AS SELECT COALESCE(a,b) FROM t1;
Type handler aggregation returns TINYTEXT as a common data type
for the two columns. But as conversion from latin1 to utf8
happens for "a", the maximum possible length of "a" grows from 255 to 255*3.
Type_handler_blob_common::Item_hybrid_func_fix_attributes() makes sure
to update the blob type handler according to max_length.
- Adding Type_handler::blob_type_handler(uint max_octet_length).
- Adding a few m_type_aggregator_for_result.add() pairs, because
now Item_xxx::type_handler() can return pointers to type_handler_tiny_blob,
type_handler_blob, type_handler_medium_blob, type_handler_long_blob.
Before the patch only type_handler_blob was possible result of type_handler().
- Making type_handler_tiny_blob, type_handler_blob, type_handler_medium_blob,
type_handler_long_blob public.
- Removing the condition in Item_sum_avg::create_tmp_field()
checking Item_sum_avg::result_type() against DECIMAL_RESULT.
Now both REAL_RESULT and DECIMAL_RESULT are symmetrically handled
by tmp_table_field_from_field_type().
- Removing Item_geometry_func::create_field_for_create_select(),
as the inherited version perfectly works.
- Fixing Item_func_as_wkb::field_type() to return MYSQL_TYPE_LONG_BLOB
rather than MYSQL_TYPE_BLOB. It's needed to make sure that
tmp_table_field_from_field_type() creates a LONGBLOB field for AsWKB().
- Fixing Item_func_as_wkt::fix_length_and_dec() to set max_length to
UINT32_MAX rather than MAX_BLOB_WIDTH, to make sure that
tmp_table_field_from_field_type() creates a LONGTEXT field for AsWKT().
- Removing Item_func_set_user_var::create_field_for_create_select(),
as the inherited version works fine.
- Adding Item_func_get_user_var::create_field_for_create_select() to
make sure that "CREATE TABLE t1 AS SELECT @string_user variable"
always creates a field of LONGTEXT/LONGBLOB type.
- Item_func_ifnull::create_field_for_create_select()
behavior has changed. Before the patch it passed set_blob_packflag=false,
which meant to create LONGBLOB for all blob variants.
Now it takes into account max_length, which gives better column
data types for:
CREATE TABLE t2 AS SELECT IFNULL(blob_column1, blob_column2) FROM t1;
- Fixing Item_func_nullif::fix_length_and_dec() to use
set_handler(args[2]->type_handler()) instead of
set_handler_by_field_type(args[2]->field_type()).
This is needed to distinguish between BLOB variants.
- Implementing Item_blob::type_handler(), to make sure to create
proper BLOB field variant, according to max_length, for queries like:
CREATE TABLE t1 AS
SELECT some_blob_field FROM INFORMATION_SCHEMA.SOME_TABLE;
- Fixing Item_field::real_type_handler() to make sure that
the code aggregating fields for UNION gets a proper BLOB
variant type handler from fields.
- Adding a special code into Item_type_holder::make_field_by_type(),
to make sure that after aggregating field types it also properly
takes into account max_length when mixing TEXT variants of different
character sets and chooses a proper TEXT variant:
CREATE TABLE t1 (
a TINYTEXT CHARACTER SET latin1,
b TINYTEXT CHARACTER SET utf8
);
CREATE TABLE t2 AS SELECT a FROM t1 UNION SELECT b FROM t1;
- Adding tests, for better coverage of IFNULL, NULLIF, UNION.
- The fact that tmp_table_field_from_field_type() now takes
into account BLOB variants (instead of always creating LONGBLOB),
tests results for WEIGHT_STRING() and NULLIF() and UNION
have become more precise.
Benefits of this patch:
- Removed a lot of calls to strlen(), especially for field_string
- Strings generated by parser are now const strings, less chance of
accidently changing a string
- Removed a lot of calls with LEX_STRING as parameter (changed to pointer)
- More uniform code
- Item::name_length was not kept up to date. Now fixed
- Several bugs found and fixed (Access to null pointers,
access of freed memory, wrong arguments to printf like functions)
- Removed a lot of casts from (const char*) to (char*)
Changes:
- This caused some ABI changes
- lex_string_set now uses LEX_CSTRING
- Some fucntions are now taking const char* instead of char*
- Create_field::change and after changed to LEX_CSTRING
- handler::connect_string, comment and engine_name() changed to LEX_CSTRING
- Checked printf() related calls to find bugs. Found and fixed several
errors in old code.
- A lot of changes from LEX_STRING to LEX_CSTRING, especially related to
parsing and events.
- Some changes from LEX_STRING and LEX_STRING & to LEX_CSTRING*
- Some changes for char* to const char*
- Added printf argument checking for my_snprintf()
- Introduced null_clex_str, star_clex_string, temp_lex_str to simplify
code
- Added item_empty_name and item_used_name to be able to distingush between
items that was given an empty name and items that was not given a name
This is used in sql_yacc.yy to know when to give an item a name.
- select table_name."*' is not anymore same as table_name.*
- removed not used function Item::rename()
- Added comparision of item->name_length before some calls to
my_strcasecmp() to speed up comparison
- Moved Item_sp_variable::make_field() from item.h to item.cc
- Some minimal code changes to avoid copying to const char *
- Fixed wrong error message in wsrep_mysql_parse()
- Fixed wrong code in find_field_in_natural_join() where real_item() was
set when it shouldn't
- ER_ERROR_ON_RENAME was used with extra arguments.
- Removed some (wrong) ER_OUTOFMEMORY, as alloc_root will already
give the error.
TODO:
- Check possible unsafe casts in plugin/auth_examples/qa_auth_interface.c
- Change code to not modify LEX_CSTRING for database name
(as part of lower_case_table_names)
Fixed the bug by failing the statement with an error message that explains
that an auto-increment column may not be used in an expression for a
check constraint.
Added a test case in check_constraint.test.
Updated existing tests and results.
Fixed that the Column_definition::pack_flag member corresponding to
ROW-type SP variables and their fields was not properly initialized.
This lead to sporadic test failures. Valgrind complained about jumps
depending on uninitialized value in VALGRIND builds.
This patch makes sure that sp_head::fill_spvar_definition() is always
called for ROW variables and their fields.
Additionally, fixed that a function with a scalar parameter
erroneously acceptes ROWs with one fields. Now an error is returned.
Implementing cursor%ROWTYPE variables, according to the task description.
This patch includes a refactoring in how sp_instr_cpush and sp_instr_copen
work. This is needed to implement MDEV-10598 later easier, to allow variable
declarations go after cursor declarations (which is currently not allowed).
Before this patch, sp_instr_cpush worked as a Query_arena associated with
the cursor. sp_instr_copen::execute() switched to the sp_instr_cpush's
Query_arena when executing the cursor SELECT statement.
Now the Query_arena associated with the cursor is stored inside an instance
of a new class sp_lex_cursor (a LEX descendand) that contains the cursor SELECT
statement.
This simplifies the implementation, because:
- It's easier to follow the code when everything related to execution
of the cursor SELECT statement is stored inside the same sp_lex_cursor
object (rather than distributed between LEX and sp_instr_cpush).
- It's easier to link an sp_instr_cursor_copy_struct to
sp_lex_cursor rather than to sp_instr_cpush.
- Also, it allows to perform sp_instr_cursor_copy_struct::exec_core()
without having a pointer to sp_instr_cpush, using a pointer to sp_lex_cursor
instead. This will be important for MDEV-10598, because sp_instr_cpush will
happen *after* sp_instr_cursor_copy_struct.
After MDEV-10598 is done, this declaration:
DECLARE
CURSOR cur IS SELECT * FROM t1;
rec cur%ROWTYPE;
BEGIN
OPEN cur;
FETCH cur INTO rec;
CLOSE cur;
END;
will generate about this code:
+-----+--------------------------+
| Pos | Instruction |
+-----+--------------------------+
| 0 | cursor_copy_struct rec@0 | Points to sp_cursor_lex through m_lex_keeper
| 1 | set rec@0 NULL |
| 2 | cpush cur@0 | Points to sp_cursor_lex through m_lex_keeper
| 3 | copen cur@0 | Points to sp_cursor_lex through m_cursor
| 4 | cfetch cur@0 rec@0 |
| 5 | cclose cur@0 |
| 6 | cpop 1 |
+-----+--------------------------+
Notice, "cursor_copy_struct" and "set" will go before "cpush".
Instructions at positions 0, 2, 3 point to the same sp_cursor_lex instance.
Define my_thread_id as an unsigned type, to avoid mismatch with
ulonglong. Change some parameters to this type.
Use size_t in a few more places.
Declare many flag constants as unsigned to avoid sign mismatch
when shifting bits or applying the unary ~ operator.
When applying the unary ~ operator to enum constants, explictly
cast the result to an unsigned type, because enum constants can
be treated as signed.
In InnoDB, change the source code line number parameters from
ulint to unsigned type. Also, make some InnoDB functions return
a narrower type (unsigned or uint32_t instead of ulint;
bool instead of ibool).
Optionally do table->update_default_fields() even for INSERT
that supposedly provides values for all column. Because these
"values" might be DEFAULT, which would need table->update_default_fields()
at the end.
Also set Item_default_value::used_tables() from the default expression.
Non-zero used_field() means that mysql_insert() will initialize all
fields to their default values (with restore_record()) even if
all columns are later provided with values. Because default expressions
may refer to other columns and they must be initialized.
Found and fixed 2 problems:
- Filesort addon fields didn't mark virtual columns properly
- multi-range-read calculated vcol bitmap but was not using it.
This caused wrong vcol field to be calculated on read, which caused the assert.
When updating a table with virtual BLOB columns, the following might
happen:
- an old record is read from the table, it has no virtual blob values
- update_virtual_fields() is run, vcol blob gets its value into the
record. But only a pointer to the value is in the table->record[0],
the value is in Field_blob::value String (but it doesn't have to be!
it can be in the record, if the column is just a copy of another
columns: ... b VARCHAR, c BLOB AS (b) ...)
- store_record(table,record[1]), old record now is in record[1]
- fill_record() prepares new values in record[0], vcol blob is updated,
new value replaces the old one in the Field_blob::value
- now both record[1] and record[0] have a pointer that points to the
*new* vcol blob value. Or record[1] has a pointer to nowhere if
Field_blob::value had to realloc.
To fix this I have introduced a new String object 'read_value' in
Field_blob. When updating virtual columns when a row has been read,
the allocated value is stored in 'read_value' instead of 'value'. The
allocated blobs for the new row is stored in 'value' as before.
I also made, as a safety precaution, the insert delayed handling of
blobs more general by using value to store strings instead of the
record. This ensures that virtual functions on delayed insert should
work in as in the case of normal insert.
Triggers are now properly updating the read, write and vcol maps for used
fields. This means that we don't need VCOL_UPDATE_FOR_READ_WRITE anymore
and there is no need for any other special handling of triggers in
update_virtual_fields().
To be able to test how many times virtual fields are invoked, I also
relaxed rules that one can use local (@) variables in DEFAULT and non
persistent virtual field expressions.
The problem happened because Item_ident_for_show::field_type() always
returned MYSQL_TYPE_DOUBLE and ignored the actual data type of the
referenced Field. As a result, the execution always used
Item_ident_for_show::val_real() to send the default value of the field,
so most default values for non-numeric types were displayed as '0'.
This patch:
1. Cleanup:
a. Removes Send_field::charsetnr, as it's been unused since
introduction of Item::charset_for_protocol() in MySQL-5.5.
b. Adds the "const" qualifier to Field::char_length().
This is needed for (5.a), see below.
2. Introduces a new virtual method Type_handler::charset_for_protocol(),
returning item->collation.collation for string data types, or
&my_charset_bin for non-string data types.
3. Changes Item::charset_for_protocol() from virtual to non-virtual.
It now calls type_handler()->charset_for_protocol().
As a good side effect, duplicate code in Item::charset_for_protocol() and
Item_temporal_hybrid_func::charset_for_protocol() is now gone.
4. Fixes Item_ident_for_show::field_type() to correctly return
its data type according to the data type of the referenced field.
This actually fixes the problem reported in MDEV-11672.
Now the default value is sent using a correct method, e.g.
val_str() for VARCHAR/TEXT, or val_int() for INT/BIGINT.
This required additional changes:
a. in DBUG_ASSERT in Protocol::store(const char *,size_t,CHARSET_INFO),
This method is now used by mysqld_list_fields(), which
(unlike normal SELECT queries) does not set
field_types/field_pos/field_count.
b. Item_ident_for_show::Item_ident_for_show() now set standard attributes
(collation, decimals, max_length, unsigned_flag) according to the
referenced field, to make charset_for_protocol() return the correct
value and to make mysqld_list_fields() correctly send default
values.
5. In order to share the code between Item_field::set_field() and
Item_ident_for_show::Item_ident_for_show():
a. Introduces a new method Type_std_attributes::set(const Field*)
b. To make (a) possible, moves Item::fix_char_length() from Item
to Type_std_attributes, also moves char_to_byte_length_safe()
from item.h to sql_type.h
c. Additionally, moves Item::fix_length_and_charset() and
Item::max_char_length() from Item to Type_std_attributes.
This is not directly needed for the fix and is done just for symmetry
with fix_char_length(), as these three methods are directly related
to each other.
This is an extraction from the patch for MDEV-10577, which is not
directly related to %TYPE implementation. This patch does the following:
- Moves LEX::set_last_field_type() to Column_definition::set_attributes()
- Adds initialization of Column_definition members length, decimals,
charset, on_update into the constructor.
- Column_definition::set_attributes() now does not set length and decimal
to 0 any more, as they are not initialized in the constructor.
- Move Column_definition::prepare_interval_field() from field.h to field.cc,
as it's too huge.
Some upcoming tasks, e.g.:
- MDEV-10577 sql_mode=ORACLE: %TYPE in variable declarations
- MDEV-10914 ROW data type for stored routine variables
will need to reuse the code implemented in prepare_create_field(),
sp_prepare_create_field(), prepare_blob_field().
Before reusing this code, it's a good idea to move these global functions
as methods to Column_definition.
This patch:
- actually moves prepare_create_field(), sp_prepare_create_field(),
prepare_blob_field() as methods to Column_definition
- makes sp_prepare_create_field() call prepare_create_field() at the end,
to avoid duplicate code in MDEV-10577 and MDEV-10914.
- changes the return data type for prepare_create_field() from int to bool,
to make it consistent with all other functions returning "ok" or "error".
- moves the implementation sp_head::fill_field_definition() from sp_head.cc
to sp_head.h, as it now uses globally visible Column_definition methods,
and is very simple, so inlining is now possible.
- removes the unused "LEX*" argument from sp_head::fill_field_definition()
There was a duplicate code to create TYPELIB from List<String>:
- In typelib() and mysql_prepare_create_table(), which was used to initialize
table fields.
- create_typelib() and sp_prepare_create_field(), which was used to initialize
SP variables.
create_typelib() was incomplete and didn't check for wrong SET values.
Fix:
- Moving the code from create_typelib() and mysql_prepare_create_field()
to news methods Column_definition::create_interval_from_interval_list()
and Column_definition::prepare_interval_field().
- Moving the code from calculate_interval_lengths() in sql_table.cc
to a new method Column_definition::calculate_interval_lengths(), as it's now
needed only in Column_definition::create_interval_from_interval_list()
- Reusing the new method Column_definition::prepare_interval_field() in both
mysql_prepare_create_table() and sp_prepare_create_field(), instead of the
old duplicate code pieces
- Removing global functions typelib() and create_typelib()
This patch also fixes:
MDEV-11155 Bad error message when creating a SET column with comma and non-ASCII characters
The problem was that ErrCongString() was called with a wrong "charset" parameter.
otherwise we'd need to store sql_mode *per vcol*
(consider CREATE INDEX...) and how SHOW CREATE TABLE would
support that?
Additionally, get rid of vcol::expr_str, just to make sure
the string is always generated and never leaked in the
original form.
When updating a table with virtual BLOB columns, the following might happen:
- an old record is read from the table, it has no virtual blob values
- update_virtual_fields() is run, vcol blob gets its value into the
record. But only a pointer to the value is in the table->record[0],
the value is in Field_blob::value String (but it doesn't have to be!
it can be in the record, if the column is just a copy of another
columns: ... b VARCHAR, c BLOB AS (b) ...)
- store_record(table,record[1]), old record now is in record[1]
- fill_record() prepares new values in record[0], vcol blob is updated,
new value replaces the old one in the Field_blob::value
- now both record[1] and record[0] have a pointer that points to the
*new* vcol blob value. Or record[1] has a pointer to nowhere if
Field_blob::value had to realloc.
To resolve this we unlink vcol blobs from the pointer to the
data (in the record[1]). Because the value is not *always* in
the Field_blob::value String, we need to remember what blobs
were unlinked. The orphan memory must be freed manually.
To complicate the matter, ha_update_row() is also used in
multi-update, in REPLACE, in INSERT ... ON DUPLICATE KEY UPDATE,
also on REPLACE ... SELECT, REPLACE DELAYED, and LOAD DATA REPLACE, etc
* don't issue an error for ER_KEY_BASED_ON_GENERATED_VIRTUAL_COLUMN
* support keyread on vcols
* callback into the server to compute vcol values from mi_check/mi_repair
* DMLs just work. Automatically.
