Commit graph

73823 commits

Author SHA1 Message Date
Monty
8edef482a7 Changed some malloc() calls to my_malloc()
- hostnames in hostname_cache added
- Some Galera (WSREP) allocations
- Table caches
2023-10-03 08:25:30 +03:00
Monty
e3b36b8f1b MDEV-31957 Concurrent ALTER and ANALYZE collecting statistics can result in stale statistical data
Example of what causes the problem:
T1: ANALYZE TABLE starts to collect statistics
T2: ALTER TABLE starts by deleting statistics for all changed fields,
    then creates a temp table and copies data to it.
T1: ANALYZE ends and writes to the statistics tables.
T2: ALTER TABLE renames temp table in place of the old table.

Now the statistics from analyze matches the old deleted tables.

Fixed by waiting to delete old statistics until ALTER TABLE is
the only one using the old table and ensure that rename of columns
can handle swapping of column names.

rename_columns_in_stat_table() (former rename_column_in_stat_tables())
now takes a list of columns to rename. It uses the following algorithm
to update column_stats to be able to handle circular renames

- While there are columns to be renamed and it is the first loop or
  last rename loop did change something.
  - Loop over all columns to be renamed
    - Change column name in column_stat
      - If fail because of duplicate key
      - If this is first change attempt for this column
         - Change column name to a temporary column name
         - If there was a conflicting row, replace it with the current row.
    else
     - Remove entry from column list

- Loop over all remaining columns in the list
 - Remove the conflicting row
 - Change column from temporary name to final name in column_stat

Other things:
- Don't flush tables for every operation. Only flush when all updates
  are done.
- Rename of columns was not handled in case of ALGORITHM=copy (old bug).
  - Fixed that we do not collect statistics for hidden hash columns
    used by UNIQUE constraint on long values.
  - Fixed that we do not collect statistics for blob columns referred by
    generated virtual columns. This was achieved by storing the fields for
    which we want to have statistics in table->has_value_set instead of
    in table->read_set.
- Rename of indexes was not handled for persistent statistics.
  - This is now handled similar as rename of columns. Renamed columns
    are now stored in 'rename_stat_indexes' and handled in
    Alter_info::delete_statistics() together with drooped indexes.
- ALTER TABLE .. ADD INDEX may instead of creating a new index rename
  an existing generated foreign key index. This was not reflected in
  the index_stats table because this was handled in
  mysql_prepare_create_table instead instead of in the mysql_alter() code.
  Fixed by adding a call in mysql_prepare_create_table() to drop the
  changed index.
  I also had to change the code that 'marked the index' to be ignored
  with code that would not destroy the original index name.

Reviewer: Sergei Petrunia <sergey@mariadb.com>
2023-10-03 08:25:30 +03:00
Andrei
388296a1e6 MDEV-32257 dangling XA-rollback in binlog from emtpy XA in pseudo_slave_mode
In `pseudo_slave_mode=1` aka "psedo-slave" mode any prepared XA
transaction disconnects from the user session, as if the user
connection drops. The xid of such transaction remains in the server,
and should the prepared transaction be read-only, it is marked.
The marking makes sure that the following termination of the
read-only transaction  ends up with ER_XA_RBROLLBACK.
This did not take place actually for  `pseudo_slave_mode=1` read-only.

Fixed with checking the read-only status of a prepared transaction
at time it disconnects from the `pseudo_slave_mode=1` session, to mark
its xid when that's the case.
2023-09-29 15:54:09 +03:00
Vladislav Vaintroub
a2a08eda8a MDEV-27943 - reduce calls to mysql_socket_set_thread_owner() in threadpool
It is enough to just once, during connection phase
2023-09-27 15:02:54 +02:00
Dmitry Shulga
ddffae0a88 MDEV-31871: maria-install-db fails on MacOS
Follow-up to fix issue with access to probably not-initialized mutex/cond_var

Constructor of the class st_debug_sync_globals was changed to initialize
the data members dsp_hits, dsp_executed, dsp_max_active with zero.
Formerly, these data members were filled with zeroes by C-runtime since
the variable debug_sync_global was declared as static and according with C rules
the static variable initialized with zero bytes.

By the same reason, the data members
  debug_sync_global->ds_mutex
  debug_sync_global->ds_cond
were initialized by zeros before the patch for MDEV-31871. After this patch
the memory for the synch primitives debug_sync_global->ds_mutex
and debug_sync_global->ds_cond are initialized explicitly by calling
the functions mysql_mutex_init/mysql_cond_init so access to these synch
primitives should be done only after such initialization be completed.
Guarded access to these synch primitives has been added to the function
debug_sync_end_thread() that is called on clean up since that was single
problem place detected by MSAN. Theoretically problem places located in the
function debug_sync_execute were not protected with similar check since
it is not obvious that the variables debug_sync_global->ds_mutex
and debug_sync_global->ds_cond could be not initilialized for use cases where
the function debug_sync_execute() is called. It is required additional study
to conclude whether it does need or not.
2023-09-22 12:02:47 +07:00
Dmitry Shulga
541433508f MDEV-31871: maria-install-db fails on MacOS
mariadb-install crashes on start when the static variable debug_sync_global
of the class st_debug_sync_globals is initialized by constructor.

Definition of the class st_debug_sync_globals has a data member of the type
Hash_set whose implementation depends on thread-specific data associated
with the key THD_KEY_mysys. This dependency results from constructor of
the class Hash_set that runs my_hash_init2() which in turn invokes my_malloc.
The thread-specific data value associated with the key THD_KEY_mysys is used
by the function sf_malloc to get id of the current thread. The key
THD_KEY_mysys is defined as static variable at my_thr_init.c initialized
with the value -1. Proper initialization of the key THD_KEY_mysys is done
with the library call pthread_key_create but it happens at the  my_init()
that called much later after the first time the THD_KEY_mysys() has been
invoked. In according with Single Unix Specification, the effect of calling
pthread_setspecific() or pthread_getspecific() with a key value not obtained
from pthread_key_create() is undefined. That is the reason why mariadb-install
crashes on MacOS.

To fix the issue, the static variable debug_sync_global is converted to
a pointer to the class st_debug_sync_globals and its instantiation is done
explicitly at the function debug_sync_init() that is called at right time.

This is the follow-up patch to the commits
  8885225de6
  f6ecadfee8
where was introduced a statically instantiated object debug_sync_global of
the structure st_debug_sync_globals and the key THR_KEY_mysys for this
thread-specific data was initialized with the value -1.
2023-09-22 12:01:52 +07:00
Marko Mäkelä
8096139b3a Merge 10.5 into 10.6 2023-09-19 10:47:26 +03:00
Marko Mäkelä
6c05edfdcd Merge 10.4 into 10.5 2023-09-19 10:20:09 +03:00
Dmitry Shulga
68353dc92a MDEV-23902: MariaDB crash on calling function
On creation of a VIEW that depends on a stored routine an instance of
the class Item_func_sp is allocated on a memory root of SP statement.
It happens since mysql_make_view() calls the method
  THD::activate_stmt_arena_if_needed()
before parsing definition of the view.
On the other hand, when sp_head's rcontext is created an instance of
the class Field referenced by the data member
  Item_func_sp::result_field
is allocated on the Item_func_sp's Query_arena (call arena) that set up
inside the method
  Item_sp::execute_impl
just before calling the method
  sp_head::execute_function()

On return from the method sp_head::execute_function() all items allocated
on the Item_func_sp's Query_arena are released and its memory root is freed
(see implementation of the method Item_sp::execute_impl). As a consequence,
the pointer
  Item_func_sp::result_field
references to the deallocated memory. Later, when the method
  sp_head::execute
cleans up items allocated for just executed SP instruction the method
  Item_func_sp::cleanup is invoked and tries to delete an object referenced
by data member Item_func_sp::result_field that points to already deallocated
memory, that results in a server abnormal termination.

To fix the issue the current active arena shouldn't be switched to
a statement arena inside the function mysql_make_view() that invoked indirectly
by the method sp_head::rcontext_create. It is implemented by introducing
the new Query_arena's state STMT_SP_QUERY_ARGUMENTS that is set when explicit
Query_arena is created for placing SP arguments and other caller's side items
used during SP execution. Then the method THD::activate_stmt_arena_if_needed()
checks Query_arena's state and returns immediately without switching to
statement's arena.
2023-09-19 08:57:36 +07:00
Marko Mäkelä
6a470db552 Merge 10.5 into 10.6 2023-09-14 15:25:53 +03:00
Yuchen Pei
cb1965bd9d
Merge branch '10.4' into 10.5 2023-09-14 16:30:11 +10:00
Marko Mäkelä
0f9acce3f2 Merge 10.5 into 10.6 2023-09-14 09:01:15 +03:00
Brandon Nesterenko
1407f99963 MDEV-31177: SHOW SLAVE STATUS Last_SQL_Errno Race Condition on Errored Slave Restart
The SQL thread and a user connection executing SHOW SLAVE STATUS
have a race condition on Last_SQL_Errno, such that a slave which
previously errored and stopped, on its next start, SHOW SLAVE STATUS
can show that the SQL Thread is running while the previous error is
also showing.

The fix is to move when the last error is cleared when the SQL
thread starts to occur before setting the status of
Slave_SQL_Running.

Thanks to Kristian Nielson for his work diagnosing the problem!

Reviewed By:
============
Andrei Elkin <andrei.elkin@mariadb.com>
Kristian Nielson <knielsen@knielsen-hq.org>
2023-09-13 12:01:47 -06:00
Daniel Black
1831f8e4d7 MDEV-31369 Disable TLS v1.0 and 1.1 for MariaDB
Remove TLSv1.1 from the default tls_version system variable.

Output a warning if TLSv1.0 or TLSv1.1 are selected.

Thanks Tingyao Nian for the feature request.
2023-09-13 20:17:29 +10:00
sjaakola
a3cbc44b24 MDEV-31833 replication breaks when using optimistic replication and replica is a galera node
MariaDB async replication SQL thread was stopped for any failure
in applying of replication events and error message logged for the failure
was: "Node has dropped from cluster". The assumption was that event applying
failure is always due to node dropping out.
With optimistic parallel replication, event applying can fail for natural
reasons and applying should be retried to handle the failure. This retry
logic was never exercised because the slave SQL thread was stopped with first
applying failure.

To support optimistic parallel replication retrying logic this commit will
now skip replication slave abort, if node remains in cluster (wsrep_ready==ON)
and replication is configured for optimistic or aggressive retry logic.

During the development of this fix, galera.galera_as_slave_nonprim test showed
some problems. The test was analyzed, and it appears to need some attention.
One excessive sleep command was removed in this commit, but it will need more
fixes still to be fully deterministic. After this commit galera_as_slave_nonprim
is successful, though.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
2023-09-12 02:37:30 +02:00
Jan Lindström
632a503ce7 MDEV-29861 : Galera "notify" test cases hang
Problem was that if wsrep_notify_cmd was set it was called
with a new status "joined" it tries to connect to the server
to update some table, but the server isn't initialized yet,
it's not listening for connections. So the server waits for the
script to finish, script waits for mariadb client to connect,
and the client cannot connect, because the server isn't listening.

Fix is to call script only when Galera has already formed a
view or when it is synched or donor.

This fix also enables following test cases:
* galera.MW-284
* galera.galera_binlog_checksum
* galera_var_notify_ssl_ipv6

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
2023-09-11 23:06:15 +02:00
Marko Mäkelä
0dd25f28f7 Merge 10.5 into 10.6 2023-09-11 14:46:39 +03:00
Marko Mäkelä
f8f7d9de2c Merge 10.4 into 10.5 2023-09-11 11:29:31 +03:00
Kristian Nielsen
e937a64d46 MDEV-10356: rpl.rpl_parallel_temptable failure due to incorrect commit optimization of temptables
The problem was that parallel replication of temporary tables using
statement-based binlogging could overlap the COMMIT in one thread with a DML
or DROP TEMPORARY TABLE in another thread using the same temporary table.
Temporary tables are not safe for concurrent access, so this caused
reference to freed memory and possibly other nastiness.

The fix is to disable the optimisation with overlapping commits of one
transaction with the start of a later transaction, when temporary tables are
in use. Then the following event groups will be blocked from starting until
the one using temporary tables is completed.

This also fixes occasional test failures of rpl.rpl_parallel_temptable seen
in Buildbot.

Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
2023-09-07 14:40:05 +02:00
Sergei Golubchik
d762e9d943 MDEV-32093 long uniques break old->new replication
recalculate long unique hash in Write_rows_log_event
and Update_rows_log_event.

normally generated columns (stored and indexed virtual)
are deterministic and their values don't need to be recalculated
on the slave as they're already present in the row image.
but the long unique hash function was changed in MDEV-27653,
so a row event from the old master will have the old hash,
but a table created on the new slave will need a new hash.
2023-09-06 22:38:41 +02:00
Sergei Golubchik
65b3c89430 MDEV-32015 insert into an empty table fails with hash unique
don't enable bulk insert when table->s->long_unique_table
2023-09-06 22:38:41 +02:00
Sergei Golubchik
382c543f53 MDEV-32012 hash unique corrupts index on virtual blobs
as always when copying record[0] aside one needs to detach
Field_blob::value's from it, and restore them when record[0]
is restored from a backup.
2023-09-06 22:38:41 +02:00
Sergei Golubchik
e78ce63291 MDEV-17711 Assertion `arena_for_set_stmt== 0' failed in LEX::set_arena_for_set_stmt upon SET STATEMENT
restore SET STATEMENT variables between statements in a multi-statement
2023-09-06 22:38:41 +02:00
Monty
f009c4da91 Small corrections to MDEV-29693 ANALYZE TABLE
32 bit MariaDB crashed in innodb.innodb-16k and a few other tests.
Fixed by using correct sizeof() calls.

Histograms where not read if first read was without histograms.
2023-09-05 19:37:07 +03:00
Dmitry Shulga
de5dba9ebe Merge branch '10.5' into 10.6 2023-09-05 14:44:52 +07:00
Dmitry Shulga
68a925b325 Merge branch '10.4' into 10.5 2023-09-05 12:41:49 +07:00
Daniel Black
91ab819451 MDEV-25177 Better indication of refusing to start because of ProtectHome
Create test for for case insensitive gives a basic warning on creating
a test file and the next thing a user might see is an abort.

ProtectHome and other systemd setting protect system services from
accessing user data. Unfortunately some of our users do put things
on /home due space or other reasons.

Rather than enumberate the systemd options in a very clunkly fragile
way we put an error associated with the "Can't create test file" and
hope the user can work it out from there.

%M tip thanks Sergei.
2023-09-04 21:26:05 +10:00
Marko Mäkelä
b0a43818b4 Merge 10.5 into 10.6 2023-09-04 10:15:02 +03:00
Marko Mäkelä
59952b2625 Merge 10.4 into 10.5 2023-09-04 09:40:26 +03:00
Dmitry Shulga
d0a872c20e MDEV-14959: Fixed memory leak relating with view and IS
Fixed memory leak taken place on executing a prepared statement or
a stored routine that querying a view and this view constructed
on an information schema table. For example,

Lets consider the following definition of the view 'v1'
CREATE VIEW v1 AS SELECT table_name FROM information_schema.views
ORDER BY table_name;

Querying this view in PS mode result in hit of assert.
PREPARE stmt FROM "SELECT * FROM v1";
EXECUTE stmt;
EXECUTE stmt; (*)

Running the statement marked with (*) leads to a crash in case
server build with mode to control allocation of a memory from SP/PS
memory root on the second and following executions of PS/SP.

The reason of leaking the memory is that a memory allocated on
processing of FRM file for the view requested from a PS/PS memory
root meaning that this memory be released only when a stored routine
be evicted from SP-cache or a prepared statement be deallocated
that typically happens on termination of a user session.

To fix the issue switch to a memory root specially created for
allocation of short-lived objects that requested on parsing FRM.
2023-09-02 13:00:00 +07:00
Dmitry Shulga
be02356206 MDEV-14959: Fixed memory leak happened on re-parsing a view that substitutes a table
In case a table accessed by a PS/SP is dropped after the first execution of
PS/SP and a view created with the same name as a table just dropped then
the second execution of PS/SP leads to allocation of a memory on SP/PS
memory root already marked as read only on first execution.

For example, the following test case:
CREATE TABLE t1 (a INT);
PREPARE stmt FROM "INSERT INTO t1 VALUES (1)";
EXECUTE stmt;
DROP TABLE t1;
CREATE VIEW t1 S SELECT 1;
--error ER_NON_INSERTABLE_TABLE
EXECUTE stmt; # (*)
DROP VIEW t1;

will hit assert on running the statement 'EXECUTE stmt' marked with (*)
when allocation of a memory be performed on parsing the view.

Memory allocation is requested inside the function mysql_make_view
when a view definition being parsed. In order to avoid an assertion
failure, call of the function mysql_make_view() must be moved after
invocation of the function check_and_update_table_version().
It will result in re-preparing the whole PS statement or current
SP instruction that will free currently allocated items and reset
read_only flag for the memory root.
2023-09-02 13:00:00 +07:00
Dmitry Shulga
1d502a29e5 MDEV-14959: Fixed possible memory leaks that could happen on running PS/SP depending on a trigger
Moved call of the function check_and_update_table_version() just
before the place where the function extend_table_list() is invoked
in order to avoid allocation of memory on a PS/SP memory root
marked as read only. It happens by the reason that the function
extend_table_list() invokes sp_add_used_routine() to add a trigger
created for the table in time frame between execution the statement
EXECUTE `stmt_id` .

For example, the following test case
create table t1 (a int);

prepare stmt from "insert into t1 (a) value (1)";
execute stmt;

create trigger t1_bi before insert on t1 for each row
  set @message= new.a;

execute stmt; # (*)

adds the trigger t1_bi to a list of used routines that involves
allocation of a memory on PS memory root that has been already marked
as read only on first run of the statement 'execute stmt'.
In result, when the statement marked with (*) is executed it results in
assert hit.

To fix the issue call the function check_and_update_table_version()
before invocation of extend_table_list() to force re-compilation of
PS/SP that resets read-only flag of its memory root.
2023-09-02 13:00:00 +07:00
Dmitry Shulga
d8574dbba3 MDEV-14959: Moved calculation the number of items reserved for exists to in transformation
It is done now before call of select_lex->setup_ref_array()
in order to avoid allocation of SP/PS's memory on its second invocation.
2023-09-02 13:00:00 +07:00
Dmitry Shulga
0d4be10a8a MDEV-14959: Control over memory allocated for SP/PS
This patch adds support for controlling of memory allocation
done by SP/PS that could happen on second and following executions.
As soon as SP or PS has been executed the first time its memory root
is marked as read only since no further memory allocation should
be performed on it. In case such allocation takes place it leads to
the assert hit for invariant that force no new memory allocations
takes place as soon as the SP/PS has been marked as read only.

The feature for control of memory allocation made on behalf SP/PS
is turned on when both debug build is on and the cmake option
-DWITH_PROTECT_STATEMENT_MEMROOT is set.

The reason for introduction of the new cmake option
  -DWITH_PROTECT_STATEMENT_MEMROOT
to control memory allocation of second and following executions of
SP/PS is that for the current server implementation there are too many
places where such memory allocation takes place. As soon as all such
incorrect allocations be fixed the cmake option
 -DWITH_PROTECT_STATEMENT_MEMROOT
can be removed and control of memory allocation made on second and
following executions can be turned on only for debug build. Before
every incorrect memory allocation be fixed it makes sense to guard
the checking of memory allocation on read only memory by extra cmake
option else we would get a lot of failing test on buildbot.

Moreover, fixing of all incorrect memory allocations could take pretty
long period of time, so for introducing the feature without necessary
to wait until all places throughout the source code be fixed it makes
sense to add the new cmake option.
2023-09-02 13:00:00 +07:00
Marko Mäkelä
2325f8f339 Merge 10.5 into 10.6 2023-08-31 13:01:42 +03:00
Daniel Black
9b1b4a6f69 MDEV-31545 Revert "Fix gcc warning for wsrep_plug"
This reverts commit 38fe266ea9.

The correct fix was pushed to the 10.4 branch
(fbc157ab33)
2023-08-30 16:24:38 +10:00
Alexander Barkov
53499cd1ea MDEV-31303 Key not used when IN clause has both signed and usigned values
Summary:

This patch enables possible index optimization when
the WHERE clause has an IN condition of the form:

signed_or_unsigned_column IN (signed_or_unsigned_constant,
                              signed_or_unsigned_constant
                              [,signed_or_unsigned_constant]*)

when the IN list constants are of different signess, e.g.:
  WHERE signed_column   IN (signed_constant, unsigned_constant ...)
  WHERE unsigned_column IN (signed_constant, unsigned_constant ...)

Details:

In a condition like:
   WHERE unsigned_predicant IN (1, LONGLONG_MAX + 1)

comparison handlers for individual (predicant,value) pairs are
calculated as follows:

* unsigned_predicant and 1 produce &type_handler_newdecimal
* unsigned_predicant and (LONGLONG_MAX + 1) produce &type_handler_slonglong

The old code decided that it could not use bisection because
the two pairs had different comparison handlers.
As a result, bisection was not allowed, and, in case of
an indexed integer column predicant the index on the column was not used.

The new code catches special cases like:
    signed_predicant   IN (signed_constant, unsigned_constant)
    unsigned_predicant IN (signed_constant, unsigned_constant)

It enables bisection using in_longlong, which supports a mixture
of predicant and values of different signess.
In case when the predicant is an indexed column this change
automatically enables index range optimization.

Thanks to Vicențiu Ciorbaru for proposing the idea and for preparing MTR tests.
2023-08-29 13:20:10 +03:00
Dmitry Shulga
1fde785315 MDEV-31890: Compilation failing on MacOS (unknown warning option -Wno-unused-but-set-variable)
For clang compiler the compiler's flag -Wno-unused-but-set-variable
was set based on compiler version. This approach could result in
false positive detection for presence of compiler option since
only first three groups of digits in compiler version taken into account
and it could lead to inaccuracy in determining of supported compiler's
features.

Correct way to detect options supported by a compiler is to use
the macros  MY_CHECK_CXX_COMPILER_FLAG and to check the result of
variable with prefix have_CXX__
So, to check whether compiler does support the option
 -Wno-unused-but-set-variable
the macros
 MY_CHECK_CXX_COMPILER_FLAG(-Wno-unused-but-set-variable)
should be called and the result variable
 have_CXX__Wno_unused_but_set_variable
be tested for assigned value.
2023-08-28 16:47:00 +07:00
Thirunarayanan Balathandayuthapani
c438284863 MDEV-31835 Remove unnecesary extra HA_EXTRA_IGNORE_INSERT call
- HA_EXTRA_IGNORE_INSERT call is being called for every inserted row,
and on partitioned tables on every row * every partition.
This leads to slowness during load..data operation

- Under bulk operation, multiple insert statement error handling
will end up emptying the table. This behaviour introduced by the
commit 8ea923f55b (MDEV-24818).
This makes the HA_EXTRA_IGNORE_INSERT call redundant. We can
use the same behavior for insert..ignore statement as well.

- Removed the extra call HA_EXTRA_IGNORE_INSERT as the solution
to improve the performance of load command.
2023-08-25 17:22:17 +05:30
Marko Mäkelä
448c2077fb Merge 10.5 into 10.6 2023-08-21 15:50:31 +03:00
Monty
a6bf4b5807 MDEV-29693 ANALYZE TABLE still flushes table definition cache when engine-independent statistics is used
This commits enables reloading of engine-independent statistics
without flushing the table from table definition cache.

This is achieved by allowing multiple version of the
TABLE_STATISTICS_CB object and having independent pointers to it in
TABLE and TABLE_SHARE.  The TABLE_STATISTICS_CB object have reference
pointers and are freed when no one is pointing to it anymore.

TABLE's TABLE_STATISTICS_CB pointer is updated to use the
TABLE_SHARE's pointer when read_statistics_for_tables() is called at
the beginning of a query.

Main changes:
- read_statistics_for_table() will allocate an new TABLE_STATISTICS_CB
  object.
- All get_stat_values() functions has a new parameter that tells
  where collected data should be stored. get_stat_values() are not
  using the table_field object anymore to store data.
- All get_stat_values() functions returns 1 if they found any
  data in the statistics tables.

Other things:
- Fixed INSERT DELAYED to not read statistics tables.
- Removed Statistics_state from TABLE_STATISTICS_CB as this is not
  needed anymore as wer are not changing TABLE_SHARE->stats_cb while
  calculating or loading statistics.
- Store values used with store_from_statistical_minmax_field() in
  TABLE_STATISTICS_CB::mem_root. This allowed me to remove the function
  delete_stat_values_for_table_share().
  - Field_blob::store_from_statistical_minmax_field() is implemented
    but is not normally used as we do not yet support EIS statistics
    for blobs. For example Field_blob::update_min() and
    Field_blob::update_max() are not implemented.
    Note that the function can be called if there is an concurrent
    "ALTER TABLE MODIFY field BLOB" running because of a bug in
    ALTER TABLE where it deletes entries from column_stats
    before it has an exclusive lock on the table.
- Use result of field->val_str(&val) as a pointer to the result
  instead of val (safetly fix).
- Allocate memory for collected statistics in THD::mem_root, not in
  in TABLE::mem_root. This could cause the TABLE object to grow if a
  ANALYZE TABLE was run many times on the same table.
  This was done in allocate_statistics_for_table(),
  create_min_max_statistical_fields_for_table() and
  create_min_max_statistical_fields_for_table_share().
- Store in TABLE_STATISTICS_CB::stats_available which statistics was
  found in the statistics tables.
- Removed index_table from class Index_prefix_calc as it was not used.
- Added TABLE_SHARE::LOCK_statistics to ensure we don't load EITS
  in parallel. First thread will load it, others will reuse the
  loaded data.
- Eliminate read_histograms_for_table(). The loading happens within
  read_statistics_for_tables() if histograms are needed.
  One downside is that if we have read statistics without histograms
  before and someone requires histograms, we have to read all statistics
  again (once) from the statistics tables.
  A smaller downside is the need to call alloc_root() for each
  individual histogram. Before we could allocate all the space for
  histograms with a single alloc_root.
- Fixed bug in MyISAM and Aria where they did not properly notice
  that table had changed after analyze table. This was not a problem
  before this patch as then the MyISAM and Aria tables where flushed
  as part of ANALYZE table which did hide this issue.
- Fixed a bug in ANALYZE table where table->records could be seen as 0
  in collect_statistics_for_table(). The effect of this unlikely bug
  was that a full table scan could be done even if
  analyze_sample_percentage was not set to 1.
- Changed multiple mallocs in a row to use multi_alloc_root().
- Added a mutex protection in update_statistics_for_table() to ensure
  that several tables are not updating the statistics at the same time.

Some of the changes in sql_statistics.cc are based on a patch from
Oleg Smirnov <olernov@gmail.com>

Co-authored-by: Oleg Smirnov <olernov@gmail.com>
Co-authored-by: Vicentiu Ciorbaru <cvicentiu@gmail.com>
Reviewer: Sergei Petrunia <sergey@mariadb.com>
2023-08-18 13:28:39 +03:00
Marko Mäkelä
5895a3622b Merge 10.4 into 10.5 2023-08-17 10:33:36 +03:00
Kristian Nielsen
34e8585437 MDEV-29974: Missed kill waiting for worker queues to drain
When the SQL driver thread goes to wait for room in the parallel slave
worker queue, there was a race where a kill at the right moment could
be ignored and the wait proceed uninterrupted by the kill.

Fix by moving the THD::check_killed() to occur _after_ doing ENTER_COND().

This bug was seen as sporadic failure of the testcase rpl.rpl_parallel
(rpl.rpl_parallel_gco_wait_kill since 10.5), with "Slave stopped with
wrong error code".

Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
2023-08-16 14:07:06 +02:00
Alexander Barkov
88dd50b80a After-merge cleanup for MDEV-27207 + MDEV-31719
Something went wrong during a merge (from 10.5 to 10.6)
of 68403eeda3
(fixing bugs MDEV-27207 and MDEV-31719).

Originally (in 10.5) the fix was done in_inet6::set() in
plugin/type_inet/sql_type_inet.cc.
In 10.6 this code resides in a different place:
in the method in_fbt::set() of a template class
in sql/sql_type_fixedbin.h.

During the merge:
- the fix did not properly migrate to in_fbt::set()
- the related MTR tests disappeared

This patch fixes in_fbt::set() properly and restores MTR tests.
2023-08-16 10:02:18 +04:00
Monty
ca5c122adc MDEV-9938 Prepared statement return wrong result (missing row)
The problem is that the first execution of the prepared statement makes
a permanent optimization of converting the LEFT JOIN to an INNER JOIN.

This is based on the assumption that all the user parameters (?) are
always constants and that parameters to Item_cond() will not change value
from true and false between different executions.

(The example was using IS NULL, which will change value if parameter
depending on if the parameter is NULL or not).

The fix is to change Item_cond::fix_fields() and
Item_cond::eval_not_null_tables() to not threat user parameters as
constants. This will ensure that we don't do the LEFT_JOIN -> INNER
JOIN conversion that causes problems.

There is also some things that needs to be improved regarding
calculations of not_null_tables_cache as we get a different value for
WHERE 1 or t1.a=1
compared to
WHERE t1.a= or 1

Changes done:
- Mark Item_param with the PARAM flag to be able to quickly check
  in Item_cond::eval_not_null_tables() if an item contains a
  prepared statement parameter (just like we check for stored procedure
  parameters).
- Fixed that Item_cond::not_null_tables_cache is not depending on
  order of arguments.
- Don't call item->eval_const_cond() for items that are NOT on the top
  level of the WHERE clause. This removed a lot of unnecessary
  warnings in the test suite!
- Do not reset not_null_tables_cache for not top level items.
- Simplified Item_cond::fix_fields by calling eval_not_null_tables()
  instead of having duplication of all the code in
  eval_not_null_tables().
- Return an error if Item_cond::fix_field() generates an error
  The old code did generate an error in some cases, but not in all
   cases.
  - Fixed all handling of the above error in make_cond_for_tables().
    The error handling by the callers did not exists before which
    could lead to asserts in many different places in the old code).
  - All changes in sql_select.cc are just checking the return value of
    fix_fields() and make_cond_for_tables() and returning an error
    value if fix_fields() returns true or make_cond_for_tables()
    returns NULL and is_error() is set.
- Mark Item_cond as const_item if all arguments returns true for
  can_eval_in_optimize().

Reviewer: Sergei Petrunia <sergey@mariadb.com>
2023-08-15 21:41:01 +03:00
Kristian Nielsen
7c9837ce74 Merge 10.4 into 10.5
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
2023-08-15 18:02:18 +02:00
Kristian Nielsen
805e0668c9 MDEV-31482: Lock wait timeout with INSERT-SELECT, autoinc, and statement-based replication
Remove the exception that InnoDB does not report auto-increment locks waits
to the parallel replication.

There was an assumption that these waits could not cause conflicts with
in-order parallel replication and thus need not be reported. However, this
assumption is wrong and it is possible to get conflicts that lead to hangs
for the duration of --innodb-lock-wait-timeout. This can be seen with three
transactions:

1. T1 is waiting for T3 on an autoinc lock
2. T2 is waiting for T1 to commit
3. T3 is waiting on a normal row lock held by T2

Here, T3 needs to be deadlock killed on the wait by T1.

Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
2023-08-15 16:40:02 +02:00
Kristian Nielsen
18acbaf416 MDEV-31655: Parallel replication deadlock victim preference code errorneously removed
Restore code to make InnoDB choose the second transaction as a deadlock
victim if two transactions deadlock that need to commit in-order for
parallel replication. This code was erroneously removed when VATS was
implemented in InnoDB.

Also add a test case for InnoDB choosing the right deadlock victim.
Also fixes this bug, with testcase that reliably reproduces:

MDEV-28776: rpl.rpl_mark_optimize_tbl_ddl fails with timeout on sync_with_master

Reviewed-by: Marko Mäkelä <marko.makela@mariadb.com>
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
2023-08-15 16:39:49 +02:00
Kristian Nielsen
900c4d6920 MDEV-31655: Parallel replication deadlock victim preference code errorneously removed
Restore code to make InnoDB choose the second transaction as a deadlock
victim if two transactions deadlock that need to commit in-order for
parallel replication. This code was erroneously removed when VATS was
implemented in InnoDB.

Also add a test case for InnoDB choosing the right deadlock victim.
Also fixes this bug, with testcase that reliably reproduces:

MDEV-28776: rpl.rpl_mark_optimize_tbl_ddl fails with timeout on sync_with_master

Note: This should be null-merged to 10.6, as a different fix is needed
there due to InnoDB locking code changes.

Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
2023-08-15 16:35:30 +02:00
Kristian Nielsen
920789e9d4 MDEV-31482: Lock wait timeout with INSERT-SELECT, autoinc, and statement-based replication
Remove the exception that InnoDB does not report auto-increment locks waits
to the parallel replication.

There was an assumption that these waits could not cause conflicts with
in-order parallel replication and thus need not be reported. However, this
assumption is wrong and it is possible to get conflicts that lead to hangs
for the duration of --innodb-lock-wait-timeout. This can be seen with three
transactions:

1. T1 is waiting for T3 on an autoinc lock
2. T2 is waiting for T1 to commit
3. T3 is waiting on a normal row lock held by T2

Here, T3 needs to be deadlock killed on the wait by T1.

Note: This should be null-merged to 10.6, as a different fix is needed
there due to InnoDB lock code changes.

Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
2023-08-15 16:34:09 +02:00