Commit graph

2806 commits

Author SHA1 Message Date
Marko Mäkelä
b4911f5a34 Merge 10.6 into 10.7 2021-10-13 16:37:12 +03:00
Marko Mäkelä
a8379e53e8 Merge 10.5 into 10.6
The changes to galera.galear_var_replicate_myisam_on
in commit d9b933bec6
are omitted due to conflicts
with commit 27d66d644c.
2021-10-13 13:28:12 +03:00
Marko Mäkelä
99bb3fb656 Merge 10.4 into 10.5 2021-10-13 12:33:56 +03:00
Marko Mäkelä
a736a3174a Merge 10.3 into 10.4 2021-10-13 12:03:32 +03:00
Marko Mäkelä
4a7dfda373 Merge 10.2 into 10.3 2021-10-13 11:38:21 +03:00
Aleksey Midenkov
911c803db1 MDEV-22660 System versioning cleanups
- Cleaned up Vers_parse_info::check_sys_fields();
- Renamed VERS_SYS_START_FLAG, VERS_SYS_END_FLAG to VERS_ROW_START,
  VERS_ROW_END.
2021-10-11 13:36:06 +03:00
Aleksey Midenkov
89936f11e9 MDEV-18278 Misleading error message in error log upon failed table creation
If error_reported is not set upper caller open_table_from_share()
throws error ER_NOT_FORM_FILE itself via open_table_error().
2021-10-11 12:26:43 +03:00
Sergei Golubchik
0299ec29d4 cleanup: MY_BITMAP mutex
in about a hundred of users of MY_BITMAP, only two were using its
built-in mutex, and only one of those two was actually needing it.

Remove the mutex from MY_BITMAP, remove all associated conditions
and checks in bitmap functions. Use an external LOCK_temp_pool
mutex and temp_pool_set_next/temp_pool_clear_bit acccessors.

Remove bitmap_init/bitmap_free, always use my_* versions.
2021-08-26 23:39:52 +02:00
Marko Mäkelä
2b66cd2493 Merge 10.3 into 10.4 2021-08-23 10:44:06 +03:00
Marko Mäkelä
8a33d36dac Fix GCC 11.2.0 -Wmaybe-uninitialized
TABLE_LIST::calc_md5(): Remove an untruthful const qualifier.

thd_get_query_start_data(): Pass empty_clex_str instead of
an uninitialized LEX_CSTRING.
2021-08-23 09:00:37 +03:00
Marko Mäkelä
f3fcf5f45c Merge 10.5 to 10.6 2021-08-19 12:25:00 +03:00
Marko Mäkelä
4a25957274 Merge 10.4 into 10.5 2021-08-18 18:22:35 +03:00
Marko Mäkelä
f84e28c119 Merge 10.3 into 10.4 2021-08-18 16:51:52 +03:00
Oleksandr Byelkin
7ae6ef5236 Merge branch '10.5' into 10.6 2021-08-03 11:21:22 +02:00
Oleksandr Byelkin
850b2ba15d Merge branch '10.4' into 10.5 2021-08-02 16:53:37 +02:00
Oleksandr Byelkin
4902b0fdc9 Merge branch '10.3' into 10.4 2021-08-02 16:50:28 +02:00
Oleksandr Byelkin
7f264997dd Merge branch '10.2' into 10.3 2021-08-02 11:41:00 +02:00
Nikita Malyavin
b549af6913 MDEV-26220 Server crashes with indexed by prefix virtual column
Server crashes in Field::register_field_in_read_map upon select from
partitioned table with indexed by prefix virtual column.

After several read-mark fixes a problem has surfaced:
Since KEY (c(10),a) uses only a prefix of c, a new field is created,
duplicated from table->field[3], with a new length. However,
vcol_inco->expr is not copied.

Therefore, (*key_info)->key_part[i].field->vcol_info->expr was left NULL
in ha_partition::index_init().

Solution: copy vcol_info from table field when it's set up.
2021-08-02 10:31:22 +02:00
Oleksandr Byelkin
8b6c8a6ce9 Revert "MDEV-26220 Server crashes with indexed by prefix virtual column"
This reverts commit 9b8e207ce0.
2021-08-02 10:30:18 +02:00
Oleksandr Byelkin
6efb5e9f5e Merge branch '10.5' into 10.6 2021-08-02 10:11:41 +02:00
Oleksandr Byelkin
ae6bdc6769 Merge branch '10.4' into 10.5 2021-07-31 23:19:51 +02:00
Oleksandr Byelkin
7841a7eb09 Merge branch '10.3' into 10.4 2021-07-31 22:59:58 +02:00
Nikita Malyavin
9b8e207ce0 MDEV-26220 Server crashes with indexed by prefix virtual column
Server crashes in Field::register_field_in_read_map upon select from
partitioned table with indexed by prefix virtual column.

After several read-mark fixes a problem has surfaced:
Since KEY (c(10),a) uses only a prefix of c, a new field is created,
duplicated from table->field[3], with a new length. However,
vcol_inco->expr is not copied.

Therefore, (*key_info)->key_part[i].field->vcol_info->expr was left NULL
in ha_partition::index_init().

Solution: initialize vcols before key initialization

Also key initialization is moved to a function.
2021-07-28 11:13:24 +02:00
Sergei Golubchik
6190a02f35 Merge branch '10.2' into 10.3 2021-07-21 20:11:07 +02:00
Nikita Malyavin
f64a4f672a follow-up MDEV-18166: rename marking functions
Reformulate mark_columns_used_by_index* function family in a more laconic
way:

mark_columns_used_by_index -> mark_index_columns
mark_columns_used_by_index_for_read_no_reset -> mark_index_columns_for_read
mark_columns_used_by_index_no_reset -> mark_index_columns_no_reset
static mark_index_columns -> do_mark_index_columns
2021-07-12 22:00:40 +03:00
Nikita Malyavin
0f6a5b4390 [2/2] MDEV-18166 ASSERT_COLUMN_MARKED_FOR_READ failed on tables with vcols
Several different test cases were failing under the same reason: the
fields in a vcol expression were not marked during marking columns of a key
contatining virtual column for read.

Fix: make marking columns of a key for read a special case where
register_field_in_read_map() is done instead of plain bitmap_set_bit().

Some test cases are only reproducible in 10.4+, but the fix is applicable
to 10.2+
2021-07-12 22:00:39 +03:00
Nikita Malyavin
7d9ba57da4 [1/2] MDEV-18166 ASSERT_COLUMN_MARKED_FOR_READ failed on tables with vcols
This is a 10.2+ part of a jira task

The two bugs regarding virtual column marking have been fixed:

1. UPDATE of a partitioned table, where the optimizer has chosen a
 secondary index to make a filesort;
2. INSERT into a table with a nonblob field generated from a blob, with
 binlog enabled and binlog_row_image=noblob.

3. DELETE from a view on a table with virtual column.

Generally the assertion happens from update_virtual_fields() call

These bugs are root-caused by missing field marking for dependant fields
of a virtual column.

Therefore a fix is: mark all the fields involved in the vcol expression by
calling field->register_field_in_read_map() instead just setting a single
bit.

3 was reproducible only on 10.4+, however the problem might has just been
invisible in the earlier versions. The fix is applicable to 10.2-10.3 as
well.
2021-07-12 22:00:39 +03:00
Sergei Golubchik
b5f50e2de8 errors after altering a table has finished aren't fatal
We cannot revert the ALTER, so anything happening after
the point of no return should not be treated as an error. A
very unfortunate condition that a user needs to be warned about - yes,
but we cannot say "ALTER TABLE has failed" if the table was successfully
altered.
2021-07-02 16:44:00 +02:00
Sergei Golubchik
add782a13e fix JSON_ARRAYAGG not to over-quote json in joins
This replaces 8711adb786

if a temptable field is created for some json expression (is_json_type()
returns true), make this temptable field a proper json field.

A field is a json field (see Item_field::is_json_type()) if it
has a CHECK constraint of JSON_VALID(field).

Note that it will never be actually checked for temptable fields,
so it won't cause a run-time slowdown.
2021-06-30 22:09:19 +02:00
Monty
be093c81a7 MDEV-24089 support oracle syntax: rownum
The ROWNUM() function is for SELECT mapped to JOIN->accepted_rows, which is
incremented for each accepted rows.
For Filesort, update, insert, delete and load data, we map ROWNUM() to
internal variables incremented when the table is changed.
The connection between the row counter and Item_func_rownum is done
in sql_select.cc::fix_items_after_optimize() and
sql_insert.cc::fix_rownum_pointers()

When ROWNUM() is used anywhere in query, the optimization to ignore ORDER
BY in sub queries are disabled. This was done to get the following common
Oracle query to work:
select * from (select * from t1 order by a desc) as t where rownum() <= 2;
MDEV-3926 "Wrong result with GROUP BY ... WITH ROLLUP" contains a discussion
about this topic.

LIMIT optimization is enabled when in a top level WHERE clause comparing
ROWNUM() with a numerical constant using any of the following expressions:
- ROWNUM() < #
- ROWNUM() <= #
- ROWNUM() = 1
ROWNUM() can be also be the right argument to the comparison function.

LIMIT optimization is done in two cases:
- For the current sub query when the ROWNUM comparison is done on the top
  level:
  SELECT * from t1 WHERE rownum() <= 2 AND t1.a > 0
- For an inner sub query, when the upper level has only a ROWNUM comparison
  in the WHERE clause:
  SELECT * from (select * from t1) as t WHERE rownum() <= 2

In Oracle mode, one can also use ROWNUM without parentheses.

Other things:
- Fixed bug where the optimizer tries to optimize away sub queries
  with RAND_TABLE_BIT set (non-deterministic queries). Now these
  sub queries will not be converted to joins.  This bug fix was also
  needed to get rownum() working inside subqueries.
- In remove_const() remove setting simple_order to FALSE if ROLLUP is
  USED. This code was disable a long time ago because of wrong assignment
  in the following code.  Instead we set simple_order to false if
  RAND_TABLE_BIT was used in the SELECT list.  This ensures that
  we don't delete ORDER BY if the result set is not deterministic, like
  in 'SELECT RAND() AS 'r' FROM t1 ORDER BY r';
- Updated parameters for Sort_param::init_for_filesort() to be able
  to provide filesort with information where the number of accepted
  rows should be stored
- Reordered fields in class Filesort to optimize storage layout
- Added new error messsage to tell that a function can't be used in HAVING
- Added field 'with_rownum' to THD to mark that ROWNUM() is used in the
  query.

Co-author: Oleksandr Byelkin <sanja@mariadb.com>
           LIMIT optimization for sub query
2021-05-19 22:54:11 +02:00
Monty
7b134ffa3d Make LEX::can_not_use_merged more general 2021-05-19 22:54:11 +02:00
Monty
949d10bea2 Don't reset StringBuffers in loops when not needed
- Moved out creating StringBuffers in loops and instead create them
  outside and just reset the buffer if it was not allocated (to avoid
  a possible malloc/free for every entry)

Other things related to set_buffer_if_not_allocated()
- Changed Valuebuffer to not call set_buffer_if_not_allocated() when
  it is created.
- Fixed geometry functions to reset string length before calling
  String::reserve().  This is because one should not access length()
  of an undefined.
- Added Item_func_conv_charset::save_in_field() as the item is using
  str_value to store cached values, which conflicts with
  Item::save_str_in_field().
- Changed Item_proc_string to not store the string value in sql_string
  as this clashes with Item::save_str_in_field().
- Locally store value of full_name_cstring() in analyse::end_of_records()
  as Item::save_str_in_field() may overwrite it.
- Marked some strings as set_thread_specific()
- Added String::free_buffer() to be used internally in String functions
  to just free the buffer but not reset other String values.
- Fixed uses_buffer_owned_by() to check for allocated length instead of
  strlength, which could be marked MEM_UNDEFINED().
2021-05-19 22:54:11 +02:00
Monty
a206658b98 Change CHARSET_INFO character set and collaction names to LEX_CSTRING
This change removed 68 explict strlen() calls from the code.

The following renames was done to ensure we don't use the old names
when merging code from earlier releases, as using the new variables
for print function could result in crashes:
- charset->csname renamed to charset->cs_name
- charset->name renamed to charset->coll_name

Almost everything where mechanical changes except:
- Changed to use the new Protocol::store(LEX_CSTRING..) when possible
- Changed to use field->store(LEX_CSTRING*, CHARSET_INFO*) when possible
- Changed to use String->append(LEX_CSTRING&) when possible

Other things:
- There where compiler issues with ensuring that all character set names
  points to the same string: gcc doesn't allow one to use integer constants
  when defining global structures (constant char * pointers works fine).
  To get around this, I declared defines for each character set name
  length.
2021-05-19 22:54:07 +02:00
Monty
b6ff139aa3 Reduce usage of strlen()
Changes:
- To detect automatic strlen() I removed the methods in String that
  uses 'const char *' without a length:
  - String::append(const char*)
  - Binary_string(const char *str)
  - String(const char *str, CHARSET_INFO *cs)
  - append_for_single_quote(const char *)
  All usage of append(const char*) is changed to either use
  String::append(char), String::append(const char*, size_t length) or
  String::append(LEX_CSTRING)
- Added STRING_WITH_LEN() around constant string arguments to
  String::append()
- Added overflow argument to escape_string_for_mysql() and
  escape_quotes_for_mysql() instead of returning (size_t) -1 on overflow.
  This was needed as most usage of the above functions never tested the
  result for -1 and would have given wrong results or crashes in case
  of overflows.
- Added Item_func_or_sum::func_name_cstring(), which returns LEX_CSTRING.
  Changed all Item_func::func_name()'s to func_name_cstring()'s.
  The old Item_func_or_sum::func_name() is now an inline function that
  returns func_name_cstring().str.
- Changed Item::mode_name() and Item::func_name_ext() to return
  LEX_CSTRING.
- Changed for some functions the name argument from const char * to
  to const LEX_CSTRING &:
  - Item::Item_func_fix_attributes()
  - Item::check_type_...()
  - Type_std_attributes::agg_item_collations()
  - Type_std_attributes::agg_item_set_converter()
  - Type_std_attributes::agg_arg_charsets...()
  - Type_handler_hybrid_field_type::aggregate_for_result()
  - Type_handler_geometry::check_type_geom_or_binary()
  - Type_handler::Item_func_or_sum_illegal_param()
  - Predicant_to_list_comparator::add_value_skip_null()
  - Predicant_to_list_comparator::add_value()
  - cmp_item_row::prepare_comparators()
  - cmp_item_row::aggregate_row_elements_for_comparison()
  - Cursor_ref::print_func()
- Removes String_space() as it was only used in one cases and that
  could be simplified to not use String_space(), thanks to the fixed
  my_vsnprintf().
- Added some const LEX_CSTRING's for common strings:
  - NULL_clex_str, DATA_clex_str, INDEX_clex_str.
- Changed primary_key_name to a LEX_CSTRING
- Renamed String::set_quick() to String::set_buffer_if_not_allocated() to
  clarify what the function really does.
- Rename of protocol function:
  bool store(const char *from, CHARSET_INFO *cs) to
  bool store_string_or_null(const char *from, CHARSET_INFO *cs).
  This was done to both clarify the difference between this 'store' function
  and also to make it easier to find unoptimal usage of store() calls.
- Added Protocol::store(const LEX_CSTRING*, CHARSET_INFO*)
- Changed some 'const char*' arrays to instead be of type LEX_CSTRING.
- class Item_func_units now used LEX_CSTRING for name.

Other things:
- Fixed a bug in mysql.cc:construct_prompt() where a wrong escape character
  in the prompt would cause some part of the prompt to be duplicated.
- Fixed a lot of instances where the length of the argument to
  append is known or easily obtain but was not used.
- Removed some not needed 'virtual' definition for functions that was
  inherited from the parent. I added override to these.
- Fixed Ordered_key::print() to preallocate needed buffer. Old code could
  case memory overruns.
- Simplified some loops when adding char * to a String with delimiters.
2021-05-19 22:27:48 +02:00
Monty
6079b46d8d Split item->flags into base_flags and with_flags
This was done to simplify copying of with_* flags

Other things:
- Changed Flags to C++ enums, which enables gdb to print
  out bit values for the flags. This also enables compiler
  errors if one tries to manipulate a non existing bit in
  a variable.
- Added set_maybe_null() as a shortcut as setting the
  MAYBE_NULL flags was used in a LOT of places.
- Renamed PARAM flag to SP_VAR to ensure it's not confused with persistent
  statement parameters.
2021-05-19 22:27:28 +02:00
Monty
7ca4e381f7 Removed Item::is_fixed() and Item::has_subquery()
One should instead use Item::fixed() and Item::with_subquery()

Removed Item::is_fixed() and has_subquery() and did the following replace:
replace is_fixed() fixed() -- *.*
replace 'has_subquery()' 'with_subquery()' -- *.*
2021-05-19 22:27:28 +02:00
Michael Widenius
3105c9e7a5 Change bitfields in Item to an uint16
The reason for the change is that neither clang or gcc can do efficient
code when several bit fields are change at the same time or when copying
one or more bits between identical bit fields.
Updated bits explicitely with & and | is MUCH more efficient than what
current compilers can do.
2021-05-19 22:27:28 +02:00
Monty
963e5e406d Changed field_index to use field_index_t instead of uint16 2021-05-19 22:27:28 +02:00
Monty
36cdd5c3cd Optimize usage of c_ptr(), c_ptr_quick() and String::alloc()
The problem was that when one used String::alloc() to allocate a string,
the String ensures that there is space for an extra NULL byte in the
buffer and if not, reallocates the string. This is a problem with the
String::set_int() that calls alloc(21), which forces extra
malloc/free calls to happen.

- We do not anymore re-allocate String if alloc() is called with the
  Allocated_length. This reduces number of malloc() allocations,
  especially one big re-allocation in Protocol::send_result_Set_metadata()
  for almost every query that produced a result to the connnected client.
- Avoid extra mallocs when using LONGLONG_BUFFER_SIZE
  This can now be done as alloc() doesn't increase buffers if new length is
  not bigger than old one.
- c_ptr() is redesigned to be safer (but a bit longer) than before.
- Remove wrong usage of c_ptr_quick()
  c_ptr_quick() was used in many cases to get the pointer to the used
  buffer, even when it didn't need to be \0 terminated. In this case
  ptr() is a better substitute.
  Another problem with c_ptr_quick() is that it did not guarantee that
  the string would be \0 terminated.
- item_val_str(), an API function not used currently by the server,
  now always returns a null terminated string (before it didn't always
  do that).
- Ensure that all String allocations uses STRING_PSI_MEMORY_KEY. The old
  mixed usage of performance keys caused assert's when String buffers
  where shrunk.
- Binary_string::shrink() is simplifed
- Fixed bug in String(const char *str, size_t len, CHARSET_INFO *cs) that
  used Binary_string((char *) str, len) instead of Binary_string(str,len).
- Changed argument to String() creations and String.set() functions to use
  'const char*' instead of 'char*'. This ensures that Alloced_length is
  not set, which gives safety against someone trying to change the
  original string. This also would allow us to use !Alloced_length in
  c_ptr() if needed.
- Changed string_ptr_cmp() to use memcmp() instead of c_ptr() to avoid
  a possible malloc during string comparision.
2021-05-19 22:27:27 +02:00
Rucha Deodhar
2fdb556e04 MDEV-8334: Rename utf8 to utf8mb3
This patch changes the main name of 3 byte character set from utf8 to
utf8mb3. New old_mode UTF8_IS_UTF8MB3 is added and set TRUE by default,
so that utf8 would mean utf8mb3. If not set, utf8 would mean utf8mb4.
2021-05-19 06:48:36 +02:00
Marko Mäkelä
b2e0a45d7a Merge 10.5 into 10.6 2021-05-05 13:15:06 +03:00
Jan Lindström
f673277491 MDEV-25586 : SIGSEGV in my_strcasecmp_utf8mb3
Fixed NULL pointer reference to db.str
2021-05-05 07:29:10 +03:00
Alexey Botchkov
1db1f85996 MDEV-17399 JSON_TABLE.
test crashing after any_db assinged nonzero lenght fixed.
2021-04-21 10:21:46 +04:00
Alexey Botchkov
1b81e23737 MDEV-17339 JSON_TABLE.
add_table_to_list - remove the table_function argument.
2021-04-21 10:21:46 +04:00
Alexey Botchkov
e9fd327ee3 MDEV-17399 Add support for JSON_TABLE.
The specific table handler for the table functions was introduced,
and used to implement JSON_TABLE.
2021-04-21 10:21:43 +04:00
Marko Mäkelä
1900c2ede5 Merge 10.5 into 10.6 2021-04-08 10:11:36 +03:00
Daniel Black
553ef1a78b MDEV-13115: Implement SELECT SKIP LOCKED
Adds an implementation for SELECT ... FOR UPDATE SKIP LOCKED /
SELECT ... LOCK IN SHARED MODE SKIP LOCKED

This is implemented only InnoDB at the moment, not in RockDB yet.

This adds a new hander flag HA_CAN_SKIP_LOCKED than
will be used when the storage engine advertises the flag.

When a storage engine indicates this flag it will get
TL_WRITE_SKIP_LOCKED and TL_READ_SKIP_LOCKED transaction types.

The Lex structure has been updated to store both the FOR UPDATE/LOCK IN
SHARE as well as the SKIP LOCKED so the SHOW CREATE VIEW
implementation is simplier.

"SELECT FOR UPDATE ... SKIP LOCKED" combined with CREATE TABLE AS or
INSERT.. SELECT on the result set is not safe for STATEMENT based
replication. MIXED replication will replicate this as row based events."

Thanks to guidance from Facebook commit
193896c466
This helped verify basic test case, and components that need implementing
(even though every part was implemented differently).

Thanks Marko for guidance on simplier InnoDB implementation.

Reviewers: Marko, Monty
2021-04-08 16:51:36 +10:00
Daniel Black
058484687a Add TL_FIRST_WRITE in SQL layer for determining R/W
Use < TL_FIRST_WRITE for determining a READ transaction.

Use TL_FIRST_WRITE as the relative operator replacing TL_WRITE_ALLOW_WRITE
as the minimium WRITE lock type.
2021-04-08 16:51:36 +10:00
Marko Mäkelä
2a7810759d MDEV-22775: Merge 10.4 into 10.5 2021-04-08 08:08:53 +03:00
Alexander Barkov
58780b5afb MDEV-22775 [HY000][1553] Changing name of primary key column with foreign key constraint fails.
Problem:

The problem happened because of a conceptual flaw in the server code:

a. The table level CHARSET/COLLATE clause affected all data types,
  including numeric and temporal ones:

   CREATE TABLE t1 (a INT) CHARACTER SET utf8 [COLLATE utf8_general_ci];

  In the above example, the Column_definition_attributes
  (and then the FRM record) for the column "a" erroneously inherited
  "utf8" as its character set.

b. The "ALTER TABLE t1 CONVERT TO CHARACTER SET csname" statement
   also erroneously affected Column_definition_attributes::charset
   for numeric and temporal data types and wrote "csname" as their
   character set into FRM files.

So now we have arbitrary non-relevant charset ID values for numeric
and temporal data types in all FRM files in the world :)

The code in the server and the other engines did not seem to be affected
by this flaw. Only InnoDB inplace ALTER was affected.

Solution:

Fixing the code in the way that only character string data types
(CHAR,VARCHAR,TEXT,ENUM,SET):
- inherit the table level CHARSET/COLLATE clause
- get the charset value according to "CONVERT TO CHARACTER SET csname".

Numeric and temporal data types now always get &my_charset_numeric
in Column_definition_attributes::charset and always write its ID into FRM files:
- no matter what the table level CHARSET/COLLATE clause is, and
- no matter what "CONVERT TO CHARACTER SET" says.

Details:

1. Adding helper classes to pass small parts of HA_CREATE_INFO
   into Type_handler methods:

   - Column_derived_attributes - to pass table level CHARSET/COLLATE,
     so columns that do not have explicit CHARSET/COLLATE clauses
     can derive them from the table level, e.g.

       CREATE TABLE t1 (a VARCHAR(1), b CHAR(1)) CHARACTER SET utf8;

   - Column_bulk_alter_attributes - to pass bulk attribute changes
     generated by the ALTER related code. These bulk changes affect
     multiple columns at the same time:

       ALTER TABLE ... CONVERT TO CHARACTER SET csname;

   Note, passing the whole HA_CREATE_INFO directly to Type_handler
   would not be good: HA_CREATE_INFO is huge and would need not desired
   dependencies in sql_type.h and sql_type.cc. The Type_handler API should
   use smallest possible data types!

2. Type_handler::Column_definition_prepare_stage1() is now responsible
   to set Column_definition::charset properly, according to the data type,
   for example:

   - For string data types, Column_definition_attributes::charset is set from
     the table level CHARSET/COLLATE clause (if not specified explicitly in
     the column definition).

   - For numeric and temporal fields, Column_definition_attributes::charset is
     set to &my_charset_numeric, no matter what the table level
     CHARSET/COLLATE says.

   - For GEOMETRY, Column_definition_attributes::charset is set to
     &my_charset_bin, no matter what the table level CHARSET/COLLATE says.

   Previously this code (setting `charset`) was outside of of
   Column_definition_prepare_stage1(), namely in
   mysql_prepare_create_table(), and was erroneously called for
   all data types.

3. Adding Type_handler::Column_definition_bulk_alter(), to handle
   "ALTER TABLE .. CONVERT TO". Previously this code was inside
   get_sql_field_charset() and was erroneously called for all data types.

4. Removing the Schema_specification_st parameter from
   Type_handler::Column_definition_redefine_stage1().
   Column_definition_attributes::charset is now fully properly initialized by
   Column_definition_prepare_stage1(). So we don't need access to the
   table level CHARSET/COLLATE clause in Column_definition_redefine_stage1()
   any more.

5. Other changes:
   - Removing global function get_sql_field_charset()

   - Moving the part of the former get_sql_field_charset(), which was
     responsible to inherit the table level CHARSET/COLLATE clause to
     new methods:
      -- Column_definition_attributes::explicit_or_derived_charset() and
      -- Column_definition::prepare_charset_for_string().
     This code is only needed for string data types.
     Previously it was erroneously called for all data types.

   - Moving another part, which was responsible to apply the
     "CONVERT TO" clause, to
     Type_handler_general_purpose_string::Column_definition_bulk_alter().

   - Replacing the call for get_sql_field_charset() in sql_partition.cc
     to sql_field->explicit_or_derived_charset() - it is perfectly enough.
     The old code was redundant: get_sql_field_charset() was called from
     sql_partition.cc only when there were no a "CONVERT TO CHARACTER SET"
     clause involved, so its purpose was only to inherit the table
     level CHARSET/COLLATE clause.

   - Moving the code handling the BINCMP_FLAG flag from
     mysql_prepare_create_table() to
     Column_definition::prepare_charset_for_string():
     This code is responsible to resolve the BINARY comparison style
     into the corresponding _bin collation, to do the following transparent
     rewrite:
        CREATE TABLE t1 (a VARCHAR(10) BINARY) CHARSET utf8;  ->
        CREATE TABLE t1 (a VARCHAR(10) CHARACTER SET utf8 COLLATE utf8_bin);
     This code is only needed for string data types.
     Previously it was erroneously called for all data types.

6. Renaming Table_scope_and_contents_source_pod_st::table_charset
   to alter_table_convert_to_charset, because the only purpose it's used for
   is handlering "ALTER .. CONVERT". The new name is much more self-descriptive.
2021-04-07 12:09:53 +04:00
Marko Mäkelä
356c149603 Merge 10.5 into 10.6 2021-03-26 11:50:32 +02:00
Otto Kekäläinen
cebf9ee204 Fix various spelling errors still found in code
Reseting -> Resetting
Unknow -> Unknown
capabilites -> capabilities
choosen -> chosen
direcory -> directory
informations -> information
openned -> opened
refered -> referred
to access -> one to access
missmatch -> mismatch
succesfully -> successfully
dont -> don't
2021-03-22 18:10:39 +11:00
Varun Gupta
f691d9865b MDEV-7317: Make an index ignorable to the optimizer
This feature adds the functionality of ignorability for indexes.
Indexes are not ignored be default.

To control index ignorability explicitly for a new index,
use IGNORE or NOT IGNORE as part of the index definition for
CREATE TABLE, CREATE INDEX, or ALTER TABLE.

Primary keys (explicit or implicit) cannot be made ignorable.

The table INFORMATION_SCHEMA.STATISTICS get a new column named IGNORED that
would store whether an index needs to be ignored or not.
2021-03-04 22:50:00 +05:30
Sergei Golubchik
f33e57a9e6 Merge branch '10.4' into 10.5 2021-02-23 13:06:22 +01:00
Sergei Golubchik
e841957416 Merge branch '10.3' into 10.4 2021-02-23 09:25:57 +01:00
Sergei Golubchik
0ab1e3914c Merge branch '10.2' into 10.3 2021-02-22 22:42:27 +01:00
Varun Gupta
b87c342da5 MDEV-11172: EXPLAIN shows non-sensical value for key_len with type=index
The issue happens when the secondary keys are extended with primary
key parts. Inside the function TABLE_SHARE::init_from_binary_frm_image()
adds the length bytes for the primary key key parts to the length of the
secondary key. This is not needed because when the extended keys are
used we recalculate the length for the used key parts.

Also removed TABLE_SHARE::total_key_length as it is not used in the code

Apporved-by: Monty <monty@mariadb.org>
2021-01-30 14:41:43 +05:30
Marko Mäkelä
961c7938bb Merge 10.4 into 10.5 2021-01-25 12:44:24 +02:00
Marko Mäkelä
3467f63764 Merge 10.3 into 10.4 2021-01-25 11:02:07 +02:00
Sergei Golubchik
3ffd5f28f0 MDEV-17227 Server crash in TABLE_SHARE::init_from_sql_statement_string upon table discovery with non-existent database
* failed init_from_binary_frm_image can clear share->db_plugin,
  don't use it on the error path
* cleanup the test a bit
2021-01-12 10:25:04 +01:00
Sergei Golubchik
f144ce2cfa MDEV-20763 Table corruption or Assertion `btr_validate_index(index, 0, false)' failed in row_upd_sec_index_entry with virtual column and EMPTY_STRING_IS_NULL SQL mode
unset empty_string_is_null mode when parsing generated columns in a table,
this mode affects pasring.
2021-01-12 10:25:04 +01:00
Marko Mäkelä
6a1e655cb0 Merge 10.4 into 10.5 2020-12-02 18:29:49 +02:00
Marko Mäkelä
589cf8dbf3 Merge 10.3 into 10.4 2020-12-01 19:51:14 +02:00
Vicențiu Ciorbaru
f6549e9544 MDEV-18323 Convert MySQL JSON type to MariaDB TEXT in mysql_upgrade
This patch solves two key problems.
1. There is a type number clash between MySQL and MariaDB. The number
   245, used for MariaDB Virtual Fields is the same as MySQL's JSON.
   This leads to corrupt FRM errors if unhandled. The code properly
   checks frm table version number and if it matches 5.7+ (until 10.0+)
   it will assume it is dealing with a MySQL table with the JSON
   datatype.
2. MySQL JSON datatype uses a proprietary format to pack JSON data. The
   patch introduces a datatype plugin which parses the format and convers
   it to its string representation.

The intended conversion path is to only use the JSON datatype within
ALTER TABLE <table> FORCE, to force a table recreate. This happens
during mysql_upgrade or via a direct ALTER TABLE <table> FORCE.
2020-10-28 11:38:14 +02:00
Marko Mäkelä
1657b7a583 Merge 10.4 to 10.5 2020-10-22 17:08:49 +03:00
Nikita Malyavin
5896a49820 MDEV-19130 Assertion failed in handler::update_auto_increment
add store/restore_auto_increment in period portion insert/update functions
2020-10-14 21:57:58 +10:00
Sujatha
25ede13611 Merge branch '10.4' into 10.5 2020-09-29 16:59:36 +05:30
Monty
0b73ef0688 MDEV-21470 ASAN heap-use-after-free in my_hash_sort_bin
The problem was that the server was calling virtual functions on a record
that was not initialized with new data.
This happened when fill_record() was aborted in the middle because an
error in save_val() or save_in_field()
2020-09-25 13:07:04 +03:00
Marko Mäkelä
97a4a3872e Merge 10.4 into 10.5 2020-08-26 12:02:07 +03:00
Alexander Barkov
04ce29354b MDEV-23551 Performance degratation in temporal literals in 10.4
Problem:

Queries like this showed performance degratation in 10.4 over 10.3:

  SELECT temporal_literal FROM t1;
  SELECT temporal_literal + 1 FROM t1;
  SELECT COUNT(*) FROM t1 WHERE temporal_column = temporal_literal;
  SELECT COUNT(*) FROM t1 WHERE temporal_column = string_literal;

Fix:

Replacing the universal member "MYSQL_TIME cached_time" in
Item_temporal_literal to data type specific containers:
- Date in Item_date_literal
- Time in Item_time_literal
- Datetime in Item_datetime_literal

This restores the performance, and make it even better in some cases.
See benchmark results in MDEV.

Also, this change makes futher separations of Date, Time, Datetime
from each other, which will make it possible not to derive them from
a too heavy (40 bytes) MYSQL_TIME, and replace them to smaller data
type specific containers.
2020-08-24 09:17:47 +04:00
Oleksandr Byelkin
48b5777ebd Merge branch '10.4' into 10.5 2020-08-04 17:24:15 +02:00
Marko Mäkelä
50a11f396a Merge 10.4 into 10.5 2020-08-01 14:42:51 +03:00
Marko Mäkelä
9216114ce7 Merge 10.3 into 10.4 2020-07-31 18:09:08 +03:00
Marko Mäkelä
66ec3a770f Merge 10.2 into 10.3 2020-07-31 13:51:28 +03:00
Marko Mäkelä
4ec032b492 Merge 10.4 into 10.5 2020-07-21 17:33:16 +03:00
Marko Mäkelä
b1538f4d60 Merge 10.3 into 10.4 2020-07-21 16:36:47 +03:00
Nikita Malyavin
ebca70ead3 fix c++98 build 2020-07-21 23:12:32 +10:00
Nikita Malyavin
5acd391e8b MDEV-16039 Crash when selecting virtual columns generated using functions with DAYNAME()
* Allocate items on thd->mem_root while refixing vcol exprs
* Make vcol tree changes register and roll them back after the statement is executed.

Explanation:
Due to collation implementation specifics an Item tree could change while fixing.
The tricky thing here is to make it on a proper arena.
It's usually not a problem when a field is deterministic, however, makes a pain vice-versa, during allocation allocating.
A non-deterministic field should be refixed on each statement, since it depends on the environment state.
Changing the tree will be temporary and therefore it should be reverted after the statement execution.
2020-07-21 16:18:00 +10:00
Aleksey Midenkov
af83ed9f0e MDEV-20661 Virtual fields are not recalculated on system fields value assignment
Fix stale virtual field value in 4 cases: when virtual field depends
on row_start/row_end in timestamp/trx_id versioned table. row_start
dep is recalculated in vers_update_fields() (SQL and InnoDB
layer). row_end dep is recalculated on history row insert.
2020-07-20 18:28:08 +03:00
Marko Mäkelä
a85f81af03 MDEV-22535 fixup: Define a single-caller function inline
Let us avoid any overhead in release builds, for an empty function.
2020-07-04 14:28:11 +03:00
Monty
0fd89a1a89 Merge remote-tracking branch 'origin/10.4' into 10.5 2020-07-03 23:31:12 +03:00
Sergei Golubchik
7a4afad969 compilation fix
include/my_valgrind.h:88:112: error: ‘void* memset(void*, int, size_t)’ writing to an object of non-trivial type ‘key_map’ {aka ‘class Bitmap<64>’}; use assignment instead [-Werror=class-memaccess]

in this case it's safe, Bitmap<> is trivial enough
2020-07-03 15:01:21 +02:00
Monty
5211af1c16 Merge remote-tracking branch 'origin/10.3' into 10.4 2020-07-03 00:35:28 +03:00
Marko Mäkelä
b6ec1e8bbf MDEV-20377 post-fix: Introduce MEM_MAKE_ADDRESSABLE
In AddressSanitizer, we only want memory poisoning to happen
in connection with custom memory allocation or freeing.

The primary use of MEM_UNDEFINED is for declaring memory uninitialized
in Valgrind or MemorySanitizer. We do not want MEM_UNDEFINED to
have the unwanted side effect that AddressSanitizer would no longer
be able to complain about accessing unallocated memory.

MEM_UNDEFINED(): Define as no-op for AddressSanitizer.

MEM_MAKE_ADDRESSABLE(): Define as MEM_UNDEFINED() or
ASAN_UNPOISON_MEMORY_REGION().

MEM_CHECK_ADDRESSABLE(): Wrap also __asan_region_is_poisoned().
2020-07-02 17:59:28 +03:00
Monty
65f831d17c Fixed bugs found by valgrind
- Some of the bug fixes are backports from 10.5!
- The fix in innobase/fil/fil0fil.cc is just a backport to get less
  error messages in mysqld.1.err when running with valgrind.
- Renamed HAVE_valgrind_or_MSAN to HAVE_valgrind
2020-07-02 17:57:34 +03:00
Monty
6cee9b1953 MDEV-22535 TABLE::initialize_quick_structures() takes 0.5% in oltp_read_only
Fixed by:
- Make all quick_* variable allocated according to real number keys instead
  of MAX_KEY
- Store all the quick* items in separated allocated structure (OPT_RANGE)
- Ensure we don't access any quick* variable without first checking
  opt_range_keys.is_set().  Thanks to this, we don't need any
  pre-initialization of quick* variables anymore.

Some renames was done to use the new structure:
table->quick_keys                -> table->opt_range_keys
table->quick_rows[X]             -> table->opt_range[X].rows
table->quick_key_parts[X]        -> table->opt_range[X].key_parts
table->quick_costs[X]            -> table->opt_range[X].cost
table->quick_index_only_costs[X] -> table->opt_range[X].index_only_cost
table->quick_n_ranges[X]         -> table->opt_range[X].ranges
table->quick_condition_rows      -> table->opt_range_condition_rows

This patch should both decrease memory needed for TABLE objects
(3528 -> 984 + keyinfo) and increase performance, thanks to less
initializations per query, and more localized memory, thanks to the
opt_range structure.
2020-07-02 16:59:14 +03:00
Monty
3f2044ae99 MDEV-22535 TABLE::initialize_quick_structures() takes 0.5% in oltp_read_only
- Removed not needed bzero in void TABLE::initialize_quick_structures().
- Replaced bzero with TRASH_ALLOC() to have this change verfied with
  memory checkers
- Added missing table->quick_keys.is_set in table_cond_selectivity()
2020-07-02 14:25:41 +03:00
Marko Mäkelä
1813d92d0c Merge 10.4 into 10.5 2020-07-02 09:41:44 +03:00
Varun Gupta
cc0dca3663 MDEV-22910: SIGSEGV in Opt_trace_context::is_started & SIGSEGV in Json_writer::add_table_name (on optimized builds)
Make sure to initialize members of TABLE::reginfo when TABLE::init is called. In this case the problem
was that table->reginfo.join_tab was set for the SELECT query and then was reused by the UPDATE query.
This case occurred only when the SELECT query had a degenerate join.
2020-06-30 18:29:02 +05:30
Monty
6a3b581b90 MDEV-19745 BACKUP STAGE BLOCK_DDL hangs on flush sequence table
Problem was that FLUSH TABLES where trying to read latest sequence state
which conflicted with a running ALTER SEQUENCE. Removed the reading
of the state, when opening a table for FLUSH, as it's not needed in this
case.

Other thing:
- Fixed a potential issue with concurrently running ALTER SEQUENCE where
  the later ALTER could potentially read old data
2020-06-14 19:39:43 +03:00
Aleksey Midenkov
762bf7a03b MDEV-22602 Disable UPDATE CASCADE for SQL constraints
CHECK constraint is checked by check_expression() which walks its
items and gets into Item_field::check_vcol_func_processor() to check
for conformity with foreign key list.

WITHOUT OVERLAPS is checked for same conformity in
mysql_prepare_create_table().

Long uniques are already impossible with InnoDB foreign keys. See
ER_CANT_CREATE_TABLE in test case.

2 accompanying bugs fixed (test main.constraints failed):

1. check->name.str lived on SP execute mem_root while "check" obj
itself lives on SP main mem_root. On second SP execute check->name.str
had garbage data. Fixed by allocating from thd->stmt_arena->mem_root
which is SP main mem_root.

2. CHECK_CONSTRAINT_IF_NOT_EXISTS value was mixed with
VCOL_FIELD_REF. VCOL_FIELD_REF is assigned in check_expression() and
then detected as CHECK_CONSTRAINT_IF_NOT_EXISTS in
handle_if_exists_options().

Existing cases for MDEV-16932 in main.constraints cover both fixes.
2020-06-12 11:12:40 +03:00
Sergei Golubchik
89a33303c4 remove dead code
reduce the amount of engine-specific code in the server,
particularly as it does not serve any purpose now.

may be needed for VP engine,
to be reconsidered in MDEV-7795
2020-06-09 14:32:43 +02:00
Marko Mäkelä
0e69f601aa Merge 10.4 into 10.5 2020-06-07 12:22:06 +03:00
Sachin
e208f91ba8 MDEV-21804 Assertion `marked_for_read()' failed upon INSERT into table with long unique blob under binlog_row_image=NOBLOB
Problem:- Calling mark_columns_per_binlog_row_image() earlier may change the
result of mark_virtual_columns_for_write() , Since it can set the bitmap on
for virtual column, and henceforth  mark_virtual_column_deps(field) will
never be called in mark_virtual_column_with_deps.

This bug is not specific for long unique, It also fails for this case
   create table t2(id int primary key, a blob, b varchar(20) as (LEFT(a,2)));
2020-06-07 12:07:36 +05:30
Kentoku SHIBA
23c8adda74 MDEV-6268 SPIDER table with no COMMENT clause causes queries to wait forever
Add looping check

Conflicts:
	sql/table.h
2020-06-05 17:29:59 +09:00
Nikita Malyavin
35d327fddb MDEV-22599 WITHOUT OVERLAPS does not work with prefix indexes
cmp_max is used instead of cmp to compare key_parts
2020-06-05 20:04:37 +10:00
Marko Mäkelä
701efbb25b Merge 10.4 into 10.5 2020-06-03 09:45:39 +03:00
Marko Mäkelä
8059148154 Merge 10.3 into 10.4 2020-06-03 07:32:09 +03:00
Marko Mäkelä
8300f639a1 Merge 10.2 into 10.3 2020-06-02 10:25:11 +03:00
Marko Mäkelä
d72eebaa3d Merge 10.1 into 10.2 2020-06-01 09:33:03 +03:00